how to keep git-fetch from running out of memory?

how to keep git-fetch from running out of memory?

[Kartik Agaram]

I have a git repo with some large files that I'm no longer able to
update. git fetch keeps running out of memory:

  fatal: Out of memory, malloc failed
  fatal: unpack-objects died with error code 128

Anybody know how to keep it from compressing the refs into packfiles?
I've experimented with core.compression, pack.compression,
pack.windowMemory, pack.packSizeLimit, all without luck :(

Re: how to keep git-fetch from running out of memory?

[Shawn Pearce]

On Sat, May 14, 2011 at 22:24, Kartik Agaram <ak@akkartik.com> wrote:
> I have a git repo with some large files that I'm no longer able to
> update. git fetch keeps running out of memory:
>
>  fatal: Out of memory, malloc failed
>  fatal: unpack-objects died with error code 128
>
> Anybody know how to keep it from compressing the refs into packfiles?
> I've experimented with core.compression, pack.compression,
> pack.windowMemory, pack.packSizeLimit, all without luck :(

Instead of playing with these settings, try transfer.unpackLimit 1. It
will force the code to use index-pack rather than unpack-objects,
which has a different memory profile.

However, that may still be insufficient. A big object must still be
allocated in memory in order to compute its SHA-1. If you don't have
sufficient memory, you need to increase your ulimits, or reconfigure
your system to have more virtual memory available to the process.

-- 
Shawn.

Re: how to keep git-fetch from running out of memory?

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> However, that may still be insufficient. A big object must still be
> allocated in memory in order to compute its SHA-1.

I wonder if we can stream this?

Re: how to keep git-fetch from running out of memory?

[Shawn Pearce]

On Sun, May 15, 2011 at 13:25, Junio C Hamano <gitster@pobox.com> wrote:
> Shawn Pearce <spearce@spearce.org> writes:
>
>> However, that may still be insufficient. A big object must still be
>> allocated in memory in order to compute its SHA-1.
>
> I wonder if we can stream this?

If its a non-delta, yes. We already do this in JGit for blobs over
50M. (Trees, commits and tags are not streamed because we might need
to do fsck on them and the fsck code for these object types requires
the entire object in a single buffer.)

Its probably not a huge change to index-pack.c. But I haven't dug
around in there in a while. I wish I could say I will look at this on
Monday, but I don't have the time.

-- 
Shawn.