git-svn -- out of memory

git-svn -- out of memory

[Alfred M. Szmidt]

Hi,

when doing a `git svn clone' on a subversion repository that contains
a large file (>1GiB), `git svn' dies with the following:

ams@trillian:~$ git svn clone REPO -T trunk -b branches -t tags
[...]
fatal: Out of memory? mmap failed: Cannot allocate memory
hash-object -w --stdin-paths: command returned error: 128

error closing pipe: Bad file descriptor at /usr/local/bin/libexec/git-core//git-svn line 0
error closing pipe: Bad file descriptor at /usr/local/bin/libexec/git-core//git-svn line 0

Doing a `git repack' doesn't help either.  Anyone got a clue how to
either solve, or work around this?

git version 1.6.1.3

Cheers.

Re: git-svn -- out of memory

[Eric Wong]

"Alfred M. Szmidt" <ams@gnu.org> wrote:
> Hi,
> 
> when doing a `git svn clone' on a subversion repository that contains
> a large file (>1GiB), `git svn' dies with the following:
> 
> ams@trillian:~$ git svn clone REPO -T trunk -b branches -t tags
> [...]
> fatal: Out of memory? mmap failed: Cannot allocate memory
> hash-object -w --stdin-paths: command returned error: 128
> 
> error closing pipe: Bad file descriptor at /usr/local/bin/libexec/git-core//git-svn line 0
> error closing pipe: Bad file descriptor at /usr/local/bin/libexec/git-core//git-svn line 0
> 
> Doing a `git repack' doesn't help either.  Anyone got a clue how to
> either solve, or work around this?
> 
> git version 1.6.1.3

Hi Alfred,

Core git requires objects to be able to fit into (virtual) memory.
Adding more swap space should solve the problem for you, but this is not
only a git-svn problem.

Fwiw, git-svn itself has always had a pretty strict "no slurping"
policy.

-- 
Eric Wong

Re: git-svn -- out of memory

[Alfred M. Szmidt]

   > when doing a `git svn clone' on a subversion repository that contains
   > a large file (>1GiB), `git svn' dies with the following:
   > 
   > ams@trillian:~$ git svn clone REPO -T trunk -b branches -t tags
   > [...]
   > fatal: Out of memory? mmap failed: Cannot allocate memory
   > hash-object -w --stdin-paths: command returned error: 128
   > 
   > error closing pipe: Bad file descriptor at /usr/local/bin/libexec/git-core//git-svn line 0
   > error closing pipe: Bad file descriptor at /usr/local/bin/libexec/git-core//git-svn line 0
   > 
   > Doing a `git repack' doesn't help either.  Anyone got a clue how to
   > either solve, or work around this?
   > 
   > git version 1.6.1.3

   Hi Alfred,

   Core git requires objects to be able to fit into (virtual) memory.
   Adding more swap space should solve the problem for you, but this is not
   only a git-svn problem.

In this case, the machine has 32G RAM, including ~20G swap.  So I do
not think that this is the case, it could be I suppose...

Cheers.

Re: git-svn -- out of memory

[Junio C Hamano]

"Alfred M\. Szmidt" <ams@gnu.org> writes:

>    > when doing a `git svn clone' on a subversion repository that contains
>    > a large file (>1GiB), `git svn' dies with the following:
>    > 
>    > ams@trillian:~$ git svn clone REPO -T trunk -b branches -t tags
>    > [...]
>    > fatal: Out of memory? mmap failed: Cannot allocate memory
>    > hash-object -w --stdin-paths: command returned error: 128

Ding...

>    > git version 1.6.1.3
>
>    Core git requires objects to be able to fit into (virtual) memory.
>    Adding more swap space should solve the problem for you, but this is not
>    only a git-svn problem.
>
> In this case, the machine has 32G RAM, including ~20G swap.  So I do
> not think that this is the case, it could be I suppose...

The hash-object error message sounds like the issue that was fixed with
915308b (avoid 31-bit truncation in write_loose_object, 2009-01-29).  Does
v1.6.2-rc1 help?

Re: git-svn -- out of memory

[Jeff King]

On Mon, Feb 16, 2009 at 02:05:54AM -0800, Junio C Hamano wrote:

> "Alfred M\. Szmidt" <ams@gnu.org> writes:
> 
> >    > when doing a `git svn clone' on a subversion repository that contains
> >    > a large file (>1GiB), `git svn' dies with the following:
> >    > 
> >    > ams@trillian:~$ git svn clone REPO -T trunk -b branches -t tags
> >    > [...]
> >    > fatal: Out of memory? mmap failed: Cannot allocate memory
> >    > hash-object -w --stdin-paths: command returned error: 128
> 
> Ding...
> [...]
> The hash-object error message sounds like the issue that was fixed with
> 915308b (avoid 31-bit truncation in write_loose_object, 2009-01-29).  Does
> v1.6.2-rc1 help?

Hmm. The original problem I had with the 32-bit wrapping actually caused
malloc to fail, whereas this is an mmap failure. So it might not be the
same thing (but certainly testing with v1.6.2-rc1 will be a good data
point).

-Peff

Re: git-svn -- out of memory

[Alfred M. Szmidt]

   >    > when doing a `git svn clone' on a subversion repository that contains
   >    > a large file (>1GiB), `git svn' dies with the following:
   >    > 
   >    > ams@trillian:~$ git svn clone REPO -T trunk -b branches -t tags
   >    > [...]
   >    > fatal: Out of memory? mmap failed: Cannot allocate memory
   >    > hash-object -w --stdin-paths: command returned error: 128

   Ding...

   >    > git version 1.6.1.3
   >
   >    Core git requires objects to be able to fit into (virtual) memory.
   >    Adding more swap space should solve the problem for you, but this is not
   >    only a git-svn problem.
   >
   > In this case, the machine has 32G RAM, including ~20G swap.  So I do
   > not think that this is the case, it could be I suppose...

   The hash-object error message sounds like the issue that was fixed with
   915308b (avoid 31-bit truncation in write_loose_object, 2009-01-29).  Does
   v1.6.2-rc1 help?

ams@trillian:~/# git --version
git version 1.6.2.rc1
ams@trillian:~/# git svn clone REPO -T trunk -b branches -t tags
[...]
fatal: Out of memory? mmap failed: Cannot allocate memory
hash-object -w --stdin-paths: command returned error: 128

Nope, behaves the same way.

I'll try adding more swap and see what happens...

Thanks for the help.

Re: git-svn -- out of memory

[Alfred M. Szmidt]

Added an extra 20G of swap, still the same error.

Re: git-svn -- out of memory

[Sverre Rabbelier]

On Mon, Feb 16, 2009 at 15:52, Alfred M. Szmidt <ams@gnu.org> wrote:
> Added an extra 20G of swap, still the same error.

Silly question, did you have a look at how much memory the git-svn
process is using? If so, does it have anything interesting to tell?

-- 
Cheers,

Sverre Rabbelier

Re: git-svn -- out of memory

[Jeff King]

On Mon, Feb 16, 2009 at 03:23:40PM +0100, Alfred M. Szmidt wrote:

> ams@trillian:~/# git --version
> git version 1.6.2.rc1
> ams@trillian:~/# git svn clone REPO -T trunk -b branches -t tags
> [...]
> fatal: Out of memory? mmap failed: Cannot allocate memory
> hash-object -w --stdin-paths: command returned error: 128
> 
> Nope, behaves the same way.

Hmm. It would help to know what object it is failing on. I assume you
repo is too large to be easily shared? If not, can you try instrumenting
hash-object with the patch below and seeing which file it is barfing on?

I'd be curious to see the size of the file (in case it is >2G and we
have another signed integer problem) and its position in the list (in
case we are leaking memory and you are hitting a system limit).

---
diff --git a/hash-object.c b/hash-object.c
index 37e6677..20371d0 100644
--- a/hash-object.c
+++ b/hash-object.c
@@ -44,6 +44,7 @@ static void hash_stdin_paths(const char *type, int write_objects)
 				die("line is badly quoted");
 			strbuf_swap(&buf, &nbuf);
 		}
+		fprintf(stderr, "debug: hashing %s\n", buf.buf);
 		hash_object(buf.buf, type, write_objects, buf.buf);
 	}
 	strbuf_release(&buf);

Re: git-svn -- out of memory

[Jeff King]

On Mon, Feb 16, 2009 at 03:52:17PM +0100, Alfred M. Szmidt wrote:

> Added an extra 20G of swap, still the same error.

You might have to also tweak your ulimit to be able to use that much
memory in a single process. Is this a 64-bit machine?

-Peff