git-svn clone -- runs out of memory

git-svn clone -- runs out of memory

[Jack Moore]

I'm trying to transition from subversion to git.  I have a very poorly structured subversion repository.  It has multiple (dozens) copies of many (thousands) of binary files that don't change from version to version.  When I "git-svn clone" the repository it takes about 30 hour (yes hours)  to (appears to) retrieve the files.  Then git goes into either a packing or garbage collecting phase and eventually runs out of memory.  I get an error relating to "mmap ... out of memory".  When it runs out of memory the "cloned" repository is ~2.4 GBi.

I don't know what files it's failing on.  I've tried excluding the directories that contain the binary files.  Does anyone have suggestions on isolating the problem or working around the memory limitation?

Thanks
Jack

Re: git-svn clone -- runs out of memory

[Ævar Arnfjörð Bjarmason]

On Wed, May 19, 2010 at 05:51, Jack Moore <jhmoore719@verizon.net> wrote:

> When I "git-svn clone" the repository it takes about 30 hour (yes
> hours)  to (appears to) retrieve the files.

That's relatively short actually with git-svn. I've taken 3 weeks to
clone one large (~70k revisions + lots of branches) repository, and
that was via local file:// access.

> Then git goes into either a packing or garbage collecting phase and
> eventually runs out of memory.  I get an error relating to "mmap
> ... out of memory".  When it runs out of memory the "cloned"
> repository is ~2.4 GBi.

This could be one of a few things:

  * git-svn leaks *lots* of memory.

    The solution is to just kill it and restart it every few
    hours. After it's killed run `git svn fetch`. You should be doing
    this anyway on any clone that takes >1 hour, aside from leaking
    memory it gets a lot slower.

  * The known problem with some git utilities doing things in-memory.

    If that's the case the solution is to just add lots of swap (or
    patch the core).

  * Something I'm forgetting

In any case, run it with GIT_TRACE=1 so you can see what command is
the one that failed.