git gc exit with out of memory, malloc failed error

git gc exit with out of memory, malloc failed error

[Alexander Kostikov]

I'm new to git and I'm getting the following out of memory error on git gc:

$ git gc
Counting objects: 80818, done.
Delta compression using up to 8 threads.
fatal: Out of memory, malloc failed (tried to allocate 24359675 bytes)
error: failed to run repack

The only advice I found in the internet suggested to run repack with
--window-memory parameter specified. But this call also fails:

$ git repack -adf --window-memory=0
Counting objects: 80818, done.
Delta compression using up to 8 threads.
warning: suboptimal pack - out of memory
fatal: Out of memory, malloc failed (tried to allocate 24356363 bytes)

How do I cleanup my repository?

$ git version
git version 1.7.6.msysgit.0

OS: Windows Server 2008 R2 SP1 (x64)
Physical memory: 24 GB
The commands listed were executed under x64 console process.

-- 
Thanks,
Alexander Kostikov

Re: git gc exit with out of memory, malloc failed error

[David Michael Barr]

On Thu, Sep 15, 2011 at 11:33 AM, Alexander Kostikov
<alex.kostikov@gmail.com> wrote:
> I'm new to git and I'm getting the following out of memory error on git gc:
>
> $ git gc
> Counting objects: 80818, done.
> Delta compression using up to 8 threads.
> fatal: Out of memory, malloc failed (tried to allocate 24359675 bytes)
> error: failed to run repack
>
> The only advice I found in the internet suggested to run repack with
> --window-memory parameter specified. But this call also fails:
>
> $ git repack -adf --window-memory=0
> Counting objects: 80818, done.
> Delta compression using up to 8 threads.
> warning: suboptimal pack - out of memory
> fatal: Out of memory, malloc failed (tried to allocate 24356363 bytes)
>
> How do I cleanup my repository?
>
> $ git version
> git version 1.7.6.msysgit.0
>
> OS: Windows Server 2008 R2 SP1 (x64)
> Physical memory: 24 GB
> The commands listed were executed under x64 console process.
>
> --
> Thanks,
> Alexander Kostikov
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Hi,

My understanding is that msysgit is a 32-bit build.
So if your existing pack is ~2GB, repack will fail.
Also, I think that setting window-memory to 0
means no limit, which is not what you want.
One value I have seen suggested is 256m.
In my experience, peak memory consumption of
repack is proportional to the value of --window,
so you might want to try tweaking that.
There are quite a few config parameters that
affect the memory consumption of repack.

-- 
David Barr

Re: git gc exit with out of memory, malloc failed error

[Alexander Kostikov]

Thanks!
--window-memory=50m worked for my project. 256m was still throwing
error. Probably real memory consumption is close to threads number *
window memory.


On Wed, Sep 14, 2011 at 7:20 PM, David Michael Barr
<davidbarr@google.com> wrote:
> On Thu, Sep 15, 2011 at 11:33 AM, Alexander Kostikov
> <alex.kostikov@gmail.com> wrote:
>> I'm new to git and I'm getting the following out of memory error on git gc:
>>
>> $ git gc
>> Counting objects: 80818, done.
>> Delta compression using up to 8 threads.
>> fatal: Out of memory, malloc failed (tried to allocate 24359675 bytes)
>> error: failed to run repack
>>
>> The only advice I found in the internet suggested to run repack with
>> --window-memory parameter specified. But this call also fails:
>>
>> $ git repack -adf --window-memory=0
>> Counting objects: 80818, done.
>> Delta compression using up to 8 threads.
>> warning: suboptimal pack - out of memory
>> fatal: Out of memory, malloc failed (tried to allocate 24356363 bytes)
>>
>> How do I cleanup my repository?
>>
>> $ git version
>> git version 1.7.6.msysgit.0
>>
>> OS: Windows Server 2008 R2 SP1 (x64)
>> Physical memory: 24 GB
>> The commands listed were executed under x64 console process.
>>
>> --
>> Thanks,
>> Alexander Kostikov
>> --
>> To unsubscribe from this list: send the line "unsubscribe git" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> Hi,
>
> My understanding is that msysgit is a 32-bit build.
> So if your existing pack is ~2GB, repack will fail.
> Also, I think that setting window-memory to 0
> means no limit, which is not what you want.
> One value I have seen suggested is 256m.
> In my experience, peak memory consumption of
> repack is proportional to the value of --window,
> so you might want to try tweaking that.
> There are quite a few config parameters that
> affect the memory consumption of repack.
>
> --
> David Barr
>



--
Your sincerely,
Alexander Kostikov

Re: git gc exit with out of memory, malloc failed error

[Brandon Casey]

[resend since gmail's Rich formatting was enabled]

On Wed, Sep 14, 2011 at 8:33 PM, Alexander Kostikov
<alex.kostikov@gmail.com> wrote:
>
> I'm new to git and I'm getting the following out of memory error on git gc:
>
> $ git gc
> Counting objects: 80818, done.
> Delta compression using up to 8 threads.
> fatal: Out of memory, malloc failed (tried to allocate 24359675 bytes)
> error: failed to run repack

Try reducing the number of threads that are used.  You must have some
pretty large objects if you have 24GB and ran out of memory.  The
following will configure git to use only one thread.
   git config pack.threads=1
-Brandon

Re: git gc exit with out of memory, malloc failed error

[Carlos Martín Nieto]

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]

On Wed, 2011-09-14 at 21:29 -0500, Brandon Casey wrote:
> [resend since gmail's Rich formatting was enabled]
> 
> On Wed, Sep 14, 2011 at 8:33 PM, Alexander Kostikov
> <alex.kostikov@gmail.com> wrote:
> >
> > I'm new to git and I'm getting the following out of memory error on git gc:
> >
> > $ git gc
> > Counting objects: 80818, done.
> > Delta compression using up to 8 threads.
> > fatal: Out of memory, malloc failed (tried to allocate 24359675 bytes)
> > error: failed to run repack
> 
> Try reducing the number of threads that are used.  You must have some
> pretty large objects if you have 24GB and ran out of memory.  The

IIRC the 32-bit Windows builds limit each user process to 2GB. If
msysgit is built in 32-bit mode, it's possible (probable even, in order
to remain backwards-compatible) that it's hitting this limit, rather
than the physical limit of the machine. If it's using 8 threads, it
means that there are 256MB available for each thread.

   cmn

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]