Re: error: git-fast-import died of signal 11

[Uri Moszkowicz <uri@4refs.com>]

Hi Michael,
Looks like the changes to limit solved the problem. I didn't verify if
it was the stacksize or descriptors but one of those. Final repository
size was 14GB from a 328GB dump file.

Thanks,
Uri

On Tue, Oct 16, 2012 at 2:18 AM, Michael Haggerty <mhagger@alum.mit.edu> wrote:
> On 10/15/2012 05:53 PM, Uri Moszkowicz wrote:
>> I'm trying to convert a CVS repository to Git using cvs2git. I was able to
>> generate the dump file without problem but am unable to get Git to
>> fast-import it. The dump file is 328GB and I ran git fast-import on a
>> machine with 512GB of RAM.
>>
>> fatal: Out of memory? mmap failed: Cannot allocate memory
>> fast-import: dumping crash report to fast_import_crash_18192
>> error: git-fast-import died of signal 11
>>
>> How can I import the repository?
>
> What versions of git and of cvs2git are you using?  If not the current
> versions, please try with the current versions.
>
> What is the nature of your repository (i.e., why is it so big)?  Does it
> consist of extremely large files?  A very deep history?  Extremely many
> branches/tags?  Extremely many files?
>
> Did you check whether the RAM usage of git-fast-import process was
> growing gradually to fill RAM while it was running vs. whether the usage
> seemed reasonable until it suddenly crashed?
>
> There are a few obvious possibilities:
>
> 0. There is some reason that too little of your computer's RAM is
> available to git-fast-import (e.g., ulimit, other processes running at
> the same time, much RAM being used as a ramdisk, etc).
>
> 1. Your import is simply too big for git-fast-import to hold in memory
> the accumulated things that it has to remember.  I'm not familiar with
> the internals of git-fast-import, but I believe that the main thing that
> it has to keep in RAM is the list of "marks" (references to git objects
> that can be referred to later in the import).  From your crash file, it
> looks like there were about 350k marks loaded at the time of the crash.
>  Supposing each mark is about 100 bytes, this would only amount to 35
> Mb, which should not be a problem (*if* my assumptions are correct).
>
> 2. Your import contains a gigantic object which individually is so big
> that it overflows some component of the import.  (I don't know whether
> large objects are handled streamily; they might be read into memory at
> some point.)  But since your computer had so much RAM this is hardly
> imaginable.
>
> 3. git-fast-import has a memory leak and the accumulated memory leakage
> is exhausting your RAM.
>
> 4. git-fast-import has some other kind of a bug.
>
> 5. The contents of the dumpfile are corrupt in a way that is triggering
> the problem.  This could either be invalid input (e.g., an object that
> is reported to be quaggabytes large), or some invalid input that
> triggers a bug in git-fast-import.
>
> If (1), then you either need a bigger machine or git-fast-import needs
> architectural changes.
>
> If (2), then you either need a bigger machine or git-fast-import and/or
> git needs architectural changes.
>
> If (3), then it would be good to get more information about the problem
> so that the leak can be fixed.  If this is the case, it might be
> possible to work around the problem by splitting the dumpfile into
> several parts and loading them one after the other (outputting the marks
> from one run and loading them into the next).
>
> If (4) or (5), then it would be helpful to narrow down the problem.  It
> might be possible to do so by following the instructions in the cvs2svn
> FAQ [1] for systematically shrinking a test case to smaller size using
> destroy_repository.py and shrink_test_case.py.  If you can create a
> small repository that triggers the same problem, then there is a good
> chance that it is easy to fix.
>
> Michael
> (the cvs2git maintainer)
>
> [1] http://cvs2svn.tigris.org/faq.html#testcase
>
> --
> Michael Haggerty
> mhagger@alum.mit.edu
> http://softwareswirl.blogspot.com/