OK... I know what's going on, but I wish I don't. I tried to reproduce it a couple of times, and it seems random. For example, adding a tcpdump in parallel makes the issue disappear. Finally I added some debugging in the fetch call in Landscape, and here's what I saw returned by the EC2 API server:
Basically, it adds a stupid warning *in the middle of the body response*: "2010-04-21 07:21:14: (server.c.1357) [note] sockets disabled, connection limit reached \n"
I guess Amazon changed his backend to limit the number of parallel connections. Or maybe they're having high load. Of course, it's completely retarded that it appears in the middle of response.
The solution is to run those HTTP calls one by one instead of in parallel.
OK... I know what's going on, but I wish I don't. I tried to reproduce it a couple of times, and it seems random. For example, adding a tcpdump in parallel makes the issue disappear. Finally I added some debugging in the fetch call in Landscape, and here's what I saw returned by the EC2 API server:
2010-04-21 07:21:14: (server.c.1357) [note] sockets disabled, connection limit reached \nHTTP/1.0 200 OK\r\nContent-Type: application/ octet-stream\ r\nAccept- Ranges: bytes\r\nETag: "1976824587" \r\nLast- Modified: Wed, 21 Apr 2010 07:10:35 GMT\r\nContent- Length: 195\r\nConnection: close\r\nDate: Wed, 21 Apr 2010 07:21:14 GMT\r\nServer: EC2ws\r\ n\r\nds12: exchange- urls46:https:/ /landscape. canonical. com/message- systems4: otpsls64: hv1dsip9GPOCp6Q gs0kdCEQuT248rT M6IgYaX1VEDzz6o vu1I70QfqJL0eAi XFXs;s8: ping-urls35: http:// landscape. canonical. com/ping;
Basically, it adds a stupid warning *in the middle of the body response*: "2010-04-21 07:21:14: (server.c.1357) [note] sockets disabled, connection limit reached \n"
I guess Amazon changed his backend to limit the number of parallel connections. Or maybe they're having high load. Of course, it's completely retarded that it appears in the middle of response.
The solution is to run those HTTP calls one by one instead of in parallel.