git-repack keeps running out of memory

git-repack keeps running out of memory

[Phil Susi]

I keep having git-repack run out of virtual memory ( 32 bit system ) 
when trying to repack my linux kernel repo.  It keeps making it right up 
to 99% then barfing saying mmap failed: Cannot allocate memory.

I thought I could help this by limiting the pack size, and using 
--window-memory to limit the memory usage, but it still happens with 
this full command line:

git repack -a -d --max-pack-size=500m -f -F --depth=20 --window=250 
--window-memory=500m

The key factor seems to be the --window... with 50 it works fine, but 
with 100 or more, even with the very low --window-memory limit, it crashes.

Re: git-repack keeps running out of memory

[Junio C Hamano]

Phil Susi <psusi@ubuntu.com> writes:

> I keep having git-repack run out of virtual memory ( 32 bit system )
> when trying to repack my linux kernel repo.  It keeps making it right
> up to 99% then barfing saying mmap failed: Cannot allocate memory.
>
> I thought I could help this by limiting the pack size, and using
> --window-memory to limit the memory usage, but it still happens with
> this full command line:
>
> git repack -a -d --max-pack-size=500m -f -F --depth=20 --window=250
> --window-memory=500m
>
> The key factor seems to be the --window... with 50 it works fine, but
> with 100 or more, even with the very low --window-memory limit, it
> crashes.

Unfortunately, that is totally expected.  Window tells us to keep
enough information to compare with that many objects in-core.  I do
not think max-pack-size would affect much.

Re: git-repack keeps running out of memory

[Phil Susi]

On 6/1/2015 2:43 PM, Junio C Hamano wrote:
> Phil Susi <psusi@ubuntu.com> writes:
>
>> I keep having git-repack run out of virtual memory ( 32 bit system )
>> when trying to repack my linux kernel repo.  It keeps making it right
>> up to 99% then barfing saying mmap failed: Cannot allocate memory.
>>
>> I thought I could help this by limiting the pack size, and using
>> --window-memory to limit the memory usage, but it still happens with
>> this full command line:
>>
>> git repack -a -d --max-pack-size=500m -f -F --depth=20 --window=250
>> --window-memory=500m
>>
>> The key factor seems to be the --window... with 50 it works fine, but
>> with 100 or more, even with the very low --window-memory limit, it
>> crashes.
>
> Unfortunately, that is totally expected.  Window tells us to keep
> enough information to compare with that many objects in-core.  I do
> not think max-pack-size would affect much.

It's more the --window-memory argument that is important here: it is 
supposed to prevent exactly this problem.  I guess I tried adding the 
--max-pack-size as well on the off chance that it would help.

Re: git-repack keeps running out of memory

[Jeff King]

On Mon, Jun 01, 2015 at 03:07:16PM -0400, Phil Susi wrote:

> It's more the --window-memory argument that is important here: it is
> supposed to prevent exactly this problem.  I guess I tried adding the
> --max-pack-size as well on the off chance that it would help.

How many processors do you have? The window-memory is per-thread. Try
with --threads=1.

-Peff

Re: git-repack keeps running out of memory

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 06/01/2015 04:58 PM, Jeff King wrote:
> How many processors do you have? The window-memory is per-thread.
> Try with --threads=1.

Ahh, right... 8... that explains it ;)


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCgAGBQJVbPMBAAoJENRVrw2cjl5R2rkH/0qa9HerYM7Bw5qfcV3zahFz
5g/NGoW3M9IutqqLiyDzAQboj62+TzrEuM/fo7BgZrKYBsA7/4SNdJMmabaUb5Kr
rtW1qfpErIHUQUY0wAxBwXX7gpihB3HTYRsZAX+7Itfae7bw8r3krj/L1Ty2nj3Y
QLIm9uwijiBNpHoF9Jnl6gbklaVFbBBfdGH2uPLzoso5YdlJ0L6o4VDmeD55njjx
30cOLDwInQ4zxdGKBH1SaIXGq+2Vi1txLrzYaaEOA+NW9+Rp7J9Anqcv/IH5yyX/
y9H0yDXXP/3gPwTHvYiRTVWMjAvjxSdc4n4ZgySLBPTruW1utxjPdaTSffm2xsg=
=wza7
-----END PGP SIGNATURE-----