Partitioned packs

["Chris Lee" <clee@kde.org>]

I've been running some experiments, as hinted earlier by the
discussion about just how much git-index-pack sucks (which, really,
isn't much since the gaping memleak is gone now).

These experiments include trying to see if there's a noticeable
performance improvement by splitting out objects of different types
into different packs. So far, it definitely seems to make a
difference, though not the one I was initially expecting. For all of
these tests, I did 'sysctl -w vm.drop_caches=3' before running, to
effectively simulate a cold-cache run.

Single 3.1GB pack file containing all commits, blobs, and trees
First run (cold cache):
git-rev-list --all > /dev/null  5.52s user 0.32s system 45% cpu 12.872 total
git-blame -- kdelibs/kdeui/kmenubar.cpp  0.00s user 0.01s system 0%
cpu 40.218s total
git-archive --format=tar HEAD -- kdelibs >> /dev/null  0.48s user
0.10s system 5% cpu 10.143 total

Subsequent runs (warm cache):
git-rev-list --all > /dev/null  5.19s user 0.48s system 98% cpu 5.750 total
git-blame -- kdelibs/kdeui/kmenubar.cpp  0.00s user 0.00s system 0%
cpu 11.960 total
git-archive --format=tar HEAD -- kdelibs >> /dev/null  0.43s user
0.04s system 100% cpu 0.472 total


Single pack for commit objects and another pack for the rest
First run (cold cache):
git-rev-list --all > /dev/null  5.84s user 0.34s system 31% cpu 19.427 total
git-blame -- kdelibs/kdeui/kmenubar.cpp  0.00s user 0.00s system 0%
cpu 9:42.74 total
git-archive --format=tar HEAD -- kdelibs >> /dev/null  0.50s user
0.26s system 0% cpu 1:35.44 total

Subsequent runs (warm cache):
git-rev-list --all > /dev/null  5.94s user 0.26s system 99% cpu 6.204 total
git-blame -- kdelibs/kdeui/kmenubar.cpp  0.00s user 0.00s system 0%
cpu 12.394 total
git-archive --format=tar HEAD -- kdelibs >> /dev/null  0.41s user
0.07s system 98% cpu 0.486 total

Fully-partitioned separate packs for commit, tree, and blob objects
First run (cold cache):
git-rev-list --all > /dev/null  6.24s user 0.32s system 25% cpu 25.689 total
git-blame -- kdelibs/kdeui/kmenubar.cpp  0.00s user 0.00s system 0%
cpu 1:08.76 total
git-archive --format=tar HEAD -- kdelibs >> /dev/null  0.38s user
0.30s system 0% cpu 1:35.89 total

Subsequent runs (warm cache):
git-rev-list --all > /dev/null  6.28s user 0.24s system 99% cpu 6.527 total
git-blame -- kdelibs/kdeui/kmenubar.cpp  0.00s user 0.00s system 0%
cpu 13.895 total
git-archive --format=tar HEAD -- kdelibs >> /dev/null  0.42s user
0.06s system 99% cpu 0.476 total

I packed all of these using --delta-base-offset, with a window of 100
and a depth of 10.

-clee