Re: pack operation is thrashing my server

[Linus Torvalds <torvalds@linux-foundation.org>]

On Thu, 14 Aug 2008, Andi Kleen wrote:
> 
> I would have thought that zlib has a sequential access pattern that the
> CPU prefetchers have a easy time with hiding latency.

No, the lookup tables for the patterns are quite non-sequential. It does 
do a lot of indirect accesses, ie it loads data from the input stream and 
then looks things up through that. 

But it's quite possible that we should use different compression factors 
for different object types. Right now we have different (configurable) 
compression levels for loose objects and packs, but it might be 
interesting to see what happens for just "packed tree objects".

The trees really end up having rather different access patterns in 
pack-files. They also tend to be rather less compressible than other 
blobs, since the SHA1's in there are just random binary data. They also 
delta very well - obviously regular blobs do that _too_, but regular blobs 
are seldom as performance-critical in git (ie once you actually unpack a 
blob, there are other things going on like actually generating a diff - 
but trees get unpacked over and over for "internal git reasons")

		Linus