Re: pack operation is thrashing my server

[Nicolas Pitre <nico@cam.org>]

On Thu, 14 Aug 2008, Linus Torvalds wrote:

> 
> 
> On Wed, 13 Aug 2008, Nicolas Pitre wrote:
> > 
> > A much better solution would consist of finding just _why_ object 
> > enumeration is so slow.  This is indeed my biggest grip with git 
> > performance at the moment.
> > 
> > |nico@xanadu:linux-2.6> time git rev-list --objects --all > /dev/null
> > |
> > |real    0m21.742s
> > |user    0m21.379s
> > |sys     0m0.360s
> > 
> > That's way too long for 1030198 objects (roughly 48k objects/sec).
> 
> Why do you think that's horribly slow?

Call it gut feeling.  Or 60% CPU wasted in zlib.

> Doing a rev-list of all objects is a fairly rare operation, but even if 
> you want to clone/repack all of your archives the whole time, please 
> realize that listing objects is _not_ a simple operation. It opens up and 
> parses every single tree in the whole history. That's a _lot_ of data to 
> unpack.

I disagree.  Well, right _now_ it is not a simple operation.  But if you 
remember, I'm one of the co-investigator of the pack v4 format which 
goal is to make history and tree walking much much cheaper, while making 
their packed representation denser too.  Even with early prototypes of 
the format with the overhead of converting objects back into the current 
format on the fly in unpack_entry() the object enumeration was _faster_ 
than current git.

So this might just be what was needed to bring back some incentive 
behind the pack v4 effort.


Nicolas