Re: error: git-fast-import died of signal 11

[Michael Haggerty <mhagger@alum.mit.edu>]

On 10/15/2012 05:53 PM, Uri Moszkowicz wrote:
> I'm trying to convert a CVS repository to Git using cvs2git. I was able to
> generate the dump file without problem but am unable to get Git to
> fast-import it. The dump file is 328GB and I ran git fast-import on a
> machine with 512GB of RAM.
> 
> fatal: Out of memory? mmap failed: Cannot allocate memory
> fast-import: dumping crash report to fast_import_crash_18192
> error: git-fast-import died of signal 11
> 
> How can I import the repository?

What versions of git and of cvs2git are you using?  If not the current
versions, please try with the current versions.

What is the nature of your repository (i.e., why is it so big)?  Does it
consist of extremely large files?  A very deep history?  Extremely many
branches/tags?  Extremely many files?

Did you check whether the RAM usage of git-fast-import process was
growing gradually to fill RAM while it was running vs. whether the usage
seemed reasonable until it suddenly crashed?

There are a few obvious possibilities:

0. There is some reason that too little of your computer's RAM is
available to git-fast-import (e.g., ulimit, other processes running at
the same time, much RAM being used as a ramdisk, etc).

1. Your import is simply too big for git-fast-import to hold in memory
the accumulated things that it has to remember.  I'm not familiar with
the internals of git-fast-import, but I believe that the main thing that
it has to keep in RAM is the list of "marks" (references to git objects
that can be referred to later in the import).  From your crash file, it
looks like there were about 350k marks loaded at the time of the crash.
 Supposing each mark is about 100 bytes, this would only amount to 35
Mb, which should not be a problem (*if* my assumptions are correct).

2. Your import contains a gigantic object which individually is so big
that it overflows some component of the import.  (I don't know whether
large objects are handled streamily; they might be read into memory at
some point.)  But since your computer had so much RAM this is hardly
imaginable.

3. git-fast-import has a memory leak and the accumulated memory leakage
is exhausting your RAM.

4. git-fast-import has some other kind of a bug.

5. The contents of the dumpfile are corrupt in a way that is triggering
the problem.  This could either be invalid input (e.g., an object that
is reported to be quaggabytes large), or some invalid input that
triggers a bug in git-fast-import.

If (1), then you either need a bigger machine or git-fast-import needs
architectural changes.

If (2), then you either need a bigger machine or git-fast-import and/or
git needs architectural changes.

If (3), then it would be good to get more information about the problem
so that the leak can be fixed.  If this is the case, it might be
possible to work around the problem by splitting the dumpfile into
several parts and loading them one after the other (outputting the marks
from one run and loading them into the next).

If (4) or (5), then it would be helpful to narrow down the problem.  It
might be possible to do so by following the instructions in the cvs2svn
FAQ [1] for systematically shrinking a test case to smaller size using
destroy_repository.py and shrink_test_case.py.  If you can create a
small repository that triggers the same problem, then there is a good
chance that it is easy to fix.

Michael
(the cvs2git maintainer)

[1] http://cvs2svn.tigris.org/faq.html#testcase

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/