Re: Performance issue: initial git clone causes massive repack

[Jon Smirl <jonsmirl@gmail.com>]

On Mon, Apr 6, 2009 at 10:19 AM, Nicolas Pitre <nico@cam.org> wrote:
> On Mon, 6 Apr 2009, Jon Smirl wrote:
>
>> On Mon, Apr 6, 2009 at 1:15 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> > Nicolas Pitre <nico@cam.org> writes:
>> >
>> >> What git-pack-objects does in this case is not a full repack.  It
>> >> instead _reuse_ as much of the existing packs as possible, and only does
>> >> the heavy packing processing for loose objects and/or inter pack
>> >> boundaryes when gluing everything together for streaming over the net.
>> >> If for example you have a single pack because your repo is already fully
>> >> packed, then the "packing operation" involved during a clone should
>> >> merely copy the existing pack over with no further attempt at delta
>> >> compression.
>> >
>> > One possibile scenario that you still need to spend memory and cycle is if
>> > the cloned repository was packed to an excessive depth to cause many of
>> > its objects to be in deltified form on insanely deep chains, while cloning
>> > send-pack uses a depth that is more reasonable.  Then pack-objects invoked
>> > by send-pack is not allowed to reuse most of the objects and would end up
>> > redoing the delta on them.
>>
>> That seems broken. You went through all of the trouble to make the
>> pack file smaller to reduce transmission time, and then clone undoes
>> the work.
>
> And as I already explained, this is indeed not what happens.
>
>> What about making a very simple special case for an initial clone?
>
> There should not be any need for initial clone hacks.
>
>> First thing an initial clone does is copy all of the pack files from
>> the server to the client without even looking at them.
>
> This is a no go for reasons already stated many times.  There are
> security implications (those packs might contain stuff that you didn't
> intend to be publically accessible) and there might be efficiency
> reasons as well (you might have a shared object store with lots of stuff
> unrelated to the particular clone).

How do you deal with dense history packs? These packs take many hours
to make (on a server class machine) and can be half the size of a
regular pack. Shouldn't there be a way to copy these packs intact on
an initial clone? It's ok if these packs are specially marked as being
ok to copy.

>
> The biggest cost right now when cloning a big packed repo is object
> enumeration.  Any other issues related to memory costs in the GB range
> simply has no reason for it, and is mostly due to misconfigurations or
> bugs that have to be fixed.  Trying to work around the issue by all
> sorts of hacks is simply counter productive.
>
> In the case that started this very thread, I suspect that a small
> misfeature of some delta caching might be the culprit.  I asked Robin H.
> Johnson to perform a really simple config addition to his repo and
> retest, for which we still haven't seen any results yet.
>
>
> Nicolas
>



-- 
Jon Smirl
jonsmirl@gmail.com