Re: git gc --aggressive led to about 40 times slower "git log --raw"

[Duy Nguyen <pclouds@gmail.com>]

On Tue, Feb 18, 2014 at 3:55 PM, David Kastrup <dak@gnu.org> wrote:
> Christian Jaeger <chrjae@gmail.com> writes:
>
>> I've got a repository where "git log --raw > _somefile" took a few
>> seconds in the past, but after an attempt at merging some commits that
>> were collected in a clone of the same repo that was created about a
>> year ago, I noticed that this command was now taking 3 minutes 7
>> seconds. "git gc", "git fsck", "git clone file:///the/repo/.git" also
>> now each took between ~4-10 minutes, also "git log --raw somefile" got
>> equally unusably slow. With the help of the people on the IRC, I
>> tracked it down to my recent use of "git gc --aggressive" in this
>> repo. Running "git repack -a -d -f" solved it, now it's again taking
>> 4-5 seconds. After running "git gc --aggressive" again for
>> confirmation, "git log --raw > _somefile" was again slowed down,
>> although now 'only' to 1 minute 34 seconds;
>
> [...]
>
>> I've now learned to avoid "git gc --aggressive". Perhaps there are
>> some other conclusions to be drawn, I don't know.
>
> I've seen the same with my ongoing work on git-blame with the current
> Emacs Git mirror.  Aggressive packing reduces the repository size to
> about a quarter, but it blows up the system time (mainly I/O)
> significantly, quite reducing the total benefits of my algorithmic
> improvements there.

Likely because --aggressive passes --depth=250 to pack-objects. Long
delta chains could reduce pack size and increase I/O as well as zlib
processing signficantly. Christian can try "git repack -adf" which is
really close to --aggressive (except it uses default --depth=50) and
see if it makes any difference.

> There is also some quite visible additional time spent in zlib, so a
> wild guess would be that zlib is not really suited to the massive amount
> of directory entries of a Git object store.  Since the system time still
> dominates, this guess would only make sense if Git over zlib kept
> rereading the directory section of whatever compressed file we are
> talking about.  But that's really a rather handwavy wild guess without
> anything better than a hunch to back it up.  I don't even know what kind
> of compression and/or packs are used: I've only ever messed myself with
> the delta coding of the normal "unpacked" operation (there are a few
> older commits from me on that).
>
> --
> David Kastrup
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Duy