[RFC 0/3] dumb HTTP transport speedups

[Eric Wong <e@80x24.org>]

TL;DR: dumb HTTP clone from a certain badly-packed repo goes from
~2 hours to ~30 min memory usage drops from 2G to 360M


I hadn't packed the public repo at https://public-inbox.org/git
for a few weeks.  As an admin of a small server limited memory
and CPU resources but fairly good bandwidth, I prefer clients
use dumb HTTP for initial clones.

Unfortunately, I noticed my dinky netbook runs out-of-memory
when using GIT_SMART_HTTP=0 to clone this giant repo; and a
machine with more memory still takes over two hours depending
on network conditions (and uses around 2GB RSS!).

Anyways, https://public-inbox.org/git is better packed, now;
but I've kept https://80x24.org/git-i-forgot-to-pack available
with over 7K loose objects to illustrate the problem:

	(this is dumb HTTP-only)
	git clone --mirror https://80x24.org/git-i-forgot-to-pack

The primary problem is fixed by PATCH 3/3 in this series, and I
can now clone the above in around 30 minutes and "only" seems to
use around 360M memory.

I'll leave git-i-forgot-to-pack up for a few months/year
so others can test and hammer away at it.

The following changes since commit 5c589a73de4394ad125a4effac227b3aec856fa1:

  Third batch of topics for 2.10 (2016-07-06 13:42:58 -0700)

are available in the git repository at:

  git://bogomips.org/git-svn.git dumb-speedups

for you to fetch changes up to b9d5aca4b8e6c9f7fb5ee4e0ce33bb42c4ea2992:

  http-walker: reduce O(n) ops with doubly-linked list (2016-07-11 20:25:51 +0000)

----------------------------------------------------------------
Eric Wong (3):
      http-walker: remove unused parameter from fetch_object
      http: avoid disconnecting on 404s for loose objects
      http-walker: reduce O(n) ops with doubly-linked list

 http-walker.c |  55 ++++++++++----------
 http.c        |  16 +++++-
 list.h        | 164 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 204 insertions(+), 31 deletions(-)
 create mode 100644 list.h