Re: pack operation is thrashing my server

["Shawn O. Pearce" <spearce@spearce.org>]

Andi Kleen <andi@firstfloor.org> wrote:
> > If you have only 256 MB of RAM, I'm afraid the machine dives into swap 
> > the moment it attempts to process that single 150-MB object during 
> > repacking.  Objects are always allocated entirely, including the 
> > deflated and inflated copy at some point.  Making git handle partial 
> > objects in memory would add complexity all over the map so I don't think 
> > it'll ever be implemented nor be desirable.
> 
> If the access pattern is sequential and not much reuse it might be possible
> to madvise() strategically to do prefetch and early unmap of not used
> anymore data. I used that successfully in a few programs in the past that did
> aggressive mmap on very large files.

We actually do something better where we can.  However parts of
Git assume that it can get back a contiguous block of memory which
contains the entire file content, decompressed.  The data is stored
on disk compressed, so we cannot just mmap the data from disk.

-- 
Shawn.