Re: [RFC] Add --create-cache to repack

[A Large Angry SCM <gitzilla@gmail.com>]

On 01/30/2011 12:14 PM, Nicolas Pitre wrote:
> On Sat, 29 Jan 2011, Junio C Hamano wrote:
>
>> Shawn Pearce<spearce@spearce.org>  writes:
>>
>>> I fully implemented the reuse of a cached pack behind a thin pack idea
>>> I was trying to describe in this thread.  It saved 1m7s off the JGit
>>> running time, but increased the data transfer by 25 MiB.  I didn't
>>> expect this much of an increase, I honestly expected the thin pack
>>> portion to be well, thinner.  The issue is the thin pack cannot delta
>>> against all of the history, its only delta compressing against the tip
>>> of the cached pack.  So long-lived side branches that forked off an
>>> older part of the history aren't delta compressing well, or at all,
>>> and that is significantly bloating the thin pack.  (Its also why that
>>> "newer" pack is 57M, but should be 14M if correctly combined with the
>>> cached pack.)  If I were to consider all of the objects in the cached
>>> pack as potential delta base candidates for the thin pack, the entire
>>> benefit of the cached pack disappears.
>>
>> What if you instead use the cached pack this way?
>>
>>   0. You perform the proposed pre-traversal until you hit the tip of cached
>>      pack(s), and realize that you will end up sending everything.
>>
>>   1. Instead of sending the new part of the history first and then sending
>>      the cached pack(s), you send the contents of cached pack(s), but also
>>      note what objects you sent;
>>
>>   2. Then you send the new part of the history, taking full advantage of
>>      what you have already sent, perhaps doing only half of the reuse-delta
>>      logic (i.e. you reuse what you can reuse, but you do _not_ punt on an
>>      object that is not a delta in an existing pack).
>
> The problem is to determine the best base object to delta against.  If
> you end up listing all the already sent objects and perform delta
> attempts against them for the remaining non delta objects to find the
> best match then you might end up taking more CPU time than the current
> enumeration phase.

Why worry about best here? Just add the object (or one of the objects) 
with the same path from the commit you found in step 0, above, to the 
delta base search for each object to pack.