Re: git pull is slow

["Shawn O. Pearce" <spearce@spearce.org>]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> On Sun, 13 Jul 2008, Shawn O. Pearce wrote:
> > Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > ...
> > > I expect this to touch the resolve_delta() function of index-pack.c in a 
> > > major way, though.
> > 
> > Yea, that's going to be ugly.  The "cache" you speak of above is held on 
> > the call stack as resolv_delta() recurses through the delta chain to 
> > reconstruct objects and generate their SHA-1s.  There isn't a way to 
> > release these objects when memory gets low so your worst case scenario 
> > is a 100M+ blob with a delta chain of 50 or more - that will take you 5G 
> > of memory to pass through index-pack.
> 
> Actually, there is...
> 
> You would only need to tap into the "release_pack_memory()" mechanism, 
> adding a sort of a "smart pointer" that knows how to reconstruct its 
> contents.

I guess you don't really know how index-pack is organized than.
It is not quite that simple.

I think we want to use the core.deltabasecachelimit parameter inside
of index-pack to limit the size of the cache, but evict based on
the callstack depth.  Objects deeper back in the callstack should
evict before objects closer to the top of the stack.

Reconstruction is a bit complex as we don't have the machinary in
sha1_file.c available to us, as the index is not available, as we
are still in the process of creating the damn thing.

I'm working up a patch series to resolve this.  I'm going to shutup
now and just try to post working code sometime this evening.

-- 
Shawn.