Re: "malloc failed"

["Shawn O. Pearce" <spearce@spearce.org>]

David Abrahams <dave@boostpro.com> wrote:
> I've been abusing Git for a purpose it wasn't intended to serve:
> archiving a large number of files with many duplicates and
> near-duplicates.  Every once in a while, when trying to do something
> really big, it tells me "malloc failed" and bails out (I think it's
> during "git add" but because of the way I issued the commands I can't
> tell: it could have been a commit or a gc).  This is on a 64-bit linux
> machine with 8G of ram and plenty of swap space, so I'm surprised.
> 
> Git is doing an amazing job at archiving and compressing all this stuff
> I'm putting in it, but I have to do it a wee bit at a time or it craps
> out.  Bug?

No, not really.  Above you said you are "abusing git for a purpose
it wasn't intended to serve"...

Git was never designed to handle many large binary blobs of data.
It was mostly designed for source code, where the majority of the
data stored in it is some form of text file written by a human.

By their very nature these files need to be relatively short (e.g.
under 1 MB each) as no human can sanely maintain a text file that
large without breaking it apart into different smaller files (like
the source code for an operating system kernel).

As a result of this approach, the git code assumes it can malloc()
at least two blocks large enough for each file: one of the fully
decompressed content, and another for the fully compressed content.
Try doing git add on a large file and its very likely malloc
will fail due to ulimit issues, or you just don't have enough
memory/address space to go around.

git gc likewise needs a good chunk of memory, but it shouldn't
usually report "malloc failed".  Usually in git gc if a malloc fails
it prints a warning and degrades the quality of its data compression.
But there are critical bookkeeping data structures where we must be
able to malloc the memory, and if those fail because we've already
exhausted the heap early on, then yea, it can fail too.

-- 
Shawn.