Re: pack operation is thrashing my server

["Shawn O. Pearce" <spearce@spearce.org>]

Ken Pratt <ken@kenpratt.net> wrote:
> I just went as low as:
> 
> [core]
>         packedGitWindowSize = 1m
>         packedGitLimit = 4m
> [pack]
>         threads = 1
>         windowMemory = 4m
>         deltaCacheSize = 128k
> 
> And it didn't make a dent in memory usage. Server is still swapping
> within ~10 seconds of starting object compression.
> 
> I'm starting to think repacking is just not feasible on a 64-bit
> server with 256MB of RAM (which is a very popular configuration in the
> VPS market).

What is the largest object in that repository? Do you have a
rough guess?  You said earlier:

> The remote repository is bare, and is 180MB in size (says du), with
> 1824 objects.

That implies there is at least one really large object in that
repository.  The average of 101KB per object is not going to be
a correct figure here as most commits and trees are _very_ tiny.
It must be a large object.  Those big objects are going to consume
a lot of memory if they get inflated in memory.

You may very well be right that this particular repository of
yours is simply not packable on a 64 bit system with only 256M.
Packing takes a good chunk of memory as we maintain data about
every single object, plus we need working space to unpack several
objects at once so we can perform diffs to find deltas.

I'm not sure there are any more tunables you can try to tweak to
reduce the memory usage further.  The configuration above is pushed
down about as low as it will go.  For the most part the code is
pretty good about not exploding memory usage.

You said earlier this was Git 1.5.6.4.  I recently fixed a bug in
the code that reads data from packs to prevent it from blowing out
memory usage, but that bug fix was included in 1.5.6.4.


On the up side, packing should only be consuming huge memory like
this when it needs to move loose objects into a pack file.  I think
Martin Langhoff suggested packing this on your laptop then using
rsync over SSH to copy the pack file and .idx file to the server, so
the server didn't have to spend time figuring out the deltas itself.

Even though the clone command will fire off git-pack-objects the
pack-objects command will have a lot less work to do if the data
it needs is already stored in existing pack files.

-- 
Shawn.