Re: 'git gc --aggressive' effectively unusable

[Michael Poole <mdpoole@troilus.org>]

Miles Bader writes:

> Frans Pop <elendil@planet.nl> writes:
>>> > I haven't had the patience to let it finish
>>>
>>> There's your problem.
>>
>> Yes, I had seen that. But there's a difference between taking much more 
>> time and slowing down to such an extend that it never finishes.
>>
>> I've tried it today on my linux-2.6 repo as well and the same thing 
>> happened. At first the progress is not fast but reasonable. When it gets 
>> to about 45% percent it starts slowing down a lot: from ~1500 objects per 
>> update of the counters to ~300 objects per update. And who knows what the 
>> progress is going to be when it reaches 70% or 90%: 10 per update?
>
> Are you sure it doesn't subsequently speed up again?

I have seen asymptotic slowdown as "git gc --aggressive" progresses on
certain repositories.  It is particularly bad with
git://git.infradead.org/gcc.git (on an x86-64 system with 4 GB RAM).
git seemed to be thrashing swap badly as time went on.  I don't know
that git gc --aggressive would *never* finish on my gcc-git repository.
I just know that it got to about 80% done in less than an hour, to 90%
after twelve hours, and about 94% after another twelve hours.  (The same
operation on linux-2.6.git takes about 40 minutes with all the default
settings.)

I may have been dreaming, but I thought with some 1.6.x version of git,
reducing core.packedGitLimit and pack.windowLimit (now windowMemory?)
mostly made the thrashing go away.  When I try again with v1.7.0.2,
though, it doesn't seem to help very much -- there is still a lot of
swapping, and the git process got to about 7 GB virtual size before I
killed it after about 10 hours of operation.

Michael Poole