Re: pack operation is thrashing my server

["Jon Smirl" <jonsmirl@gmail.com>]

On 9/6/08, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>  I'm starting to hate zlib. I actually spent almost a week trying to clean
>  up the zlib source code and make it something that gcc can compile into
>  clean code, but the fact is, zlib isn't amenable to that. The whole "shift
>  <n> bits in from the buffer" approach means that there is no way to make
>  zlib generate good code unless you are an insanely competent assembly
>  hacker or have tons of registers to keep all the temporaries live in.
>
>  Now, I still do think that all my reasons for choosing zlib were pretty
>  solid (it's a well-tested piece of code and it is _everywhere_ and easy to
>  use), but boy do I wish there had been alternatives.

Some alternative algorithms are here...
http://cs.fit.edu/~mmahoney/compression
It is possible to beat zlib by 2x at the cost of CPU time and memory.

Of course switching to these algorithms would involve a lot of testing
and benchmarking. I'm also not sure how PAQ would fare on lots of
small git objects instead of large files.

Turning a 500MB packfile into a 250MB has lots of advantages in IO
reduction so it is worth some CPU/memory to create it.

You can even win 50'000€ for a better algorithm.
http://prize.hutter1.net/

-- 
Jon Smirl
jonsmirl@gmail.com