Comment 11 for bug 402623

There is a patch up for review.
In testing this, I went ahead and used bzr.dev to download http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel
I set -Dhttp and then used some silly grep hacks to check Content-Length headers:
grep -A 17 "http.*cix" .bzr.log | grep Content-Length
< Content-Length: 4096
< Content-Length: 94208
< Content-Length: 15846927
< Content-Length: 433
< Content-Length: 17119
< Content-Length: 583
< Content-Length: 298
< Content-Length: 610
< Content-Length: 12349440
< Content-Length: 12004988
< Content-Length: 11701226
< Content-Length: 1092692

Which in total is 50.7MiB

Versus just 'grep Content-Length' which was:
 180.5 MiB

So out of 180.5 MiB downloaded, 28% of that was the .cix headers being downloaded 4+ times. (The last one is probably reading the parent_id,basename=>file_id index pages, which are significantly less than the full content.)

In comparison, doing the same action with my patch resulted in:
< Content-Length: 4096
< Content-Length: 94208
< Content-Length: 15846927
< Content-Length: 433
< Content-Length: 17119
< Content-Length: 583
< Content-Length: 298
< Content-Length: 610

    15.2 MiB / 143.1 MiB

Peak memory did increase to about 650MiB during the transfer. Though I'm trying
to address that sort of issue with different patches.