git doesn't like big files when pushing

git doesn't like big files when pushing

[Greg KH]

I have a mail archive stored with git, in mbox form, and I made some
changes to a few of the files and checked them back in.

That worked fine, but when I went to push the stuff to my server, I got
the following errors:

$ git push origin
updating 'refs/heads/master'
  from 490badd9bec9ada3a21be275c97fb2a3a390f49e
  to   16be8985abc8a9c89ad2cc8f46a0d8e9786e832f
Generating pack...
Done counting 8 objects.
Deltifying 8 objects.
fatal: Out of memory, malloc failed
fatal: early EOF
unpack unpacker exited with error code
ng refs/heads/master n/a (unpacker error)


And here are the file sizes of the files that were changed:
$ ls -lh linux-usb-devel.save.200*
-rw-r--r-- 1 greg users 41M Jan  6 14:30 linux-usb-devel.save.2001
-rw-r--r-- 1 greg users 80M Jan  6 14:30 linux-usb-devel.save.2002
-rw-r--r-- 1 greg users 74M Jan  6 14:30 linux-usb-devel.save.2003
-rw-r--r-- 1 greg users 99M Jan  6 14:30 linux-usb-devel.save.2004
-rw-r--r-- 1 greg users 89M Jan  6 14:30 linux-usb-devel.save.2005

So, am I just foolish for trying to use git for this?  Should I just go
back to using rsync to back stuff like this up with?

thanks,

greg k-h

Re: git doesn't like big files when pushing

[Greg KH]

On Wed, Mar 01, 2006 at 02:08:02PM -0800, Greg KH wrote:
> I have a mail archive stored with git, in mbox form, and I made some
> changes to a few of the files and checked them back in.
> 
> That worked fine, but when I went to push the stuff to my server, I got
> the following errors:
> 
> $ git push origin
> updating 'refs/heads/master'
>   from 490badd9bec9ada3a21be275c97fb2a3a390f49e
>   to   16be8985abc8a9c89ad2cc8f46a0d8e9786e832f
> Generating pack...
> Done counting 8 objects.
> Deltifying 8 objects.
> fatal: Out of memory, malloc failed
> fatal: early EOF
> unpack unpacker exited with error code
> ng refs/heads/master n/a (unpacker error)

Oh, and I'm using:
	$ git --version
	git version 1.2.3.g8c2f

if that helps or not.

thanks,

greg k-h

Re: git doesn't like big files when pushing

[Greg KH]

On Wed, Mar 01, 2006 at 02:08:40PM -0800, Greg KH wrote:
> On Wed, Mar 01, 2006 at 02:08:02PM -0800, Greg KH wrote:
> > I have a mail archive stored with git, in mbox form, and I made some
> > changes to a few of the files and checked them back in.
> > 
> > That worked fine, but when I went to push the stuff to my server, I got
> > the following errors:
> > 
> > $ git push origin
> > updating 'refs/heads/master'
> >   from 490badd9bec9ada3a21be275c97fb2a3a390f49e
> >   to   16be8985abc8a9c89ad2cc8f46a0d8e9786e832f
> > Generating pack...
> > Done counting 8 objects.
> > Deltifying 8 objects.
> > fatal: Out of memory, malloc failed
> > fatal: early EOF
> > unpack unpacker exited with error code
> > ng refs/heads/master n/a (unpacker error)
> 
> Oh, and I'm using:
> 	$ git --version
> 	git version 1.2.3.g8c2f

Hm, 1.2.4.g6177 seems better, it's still trying to pack things, after
about 10 minutes, but at least it isn't dying yet.  I'll let you know if
it finishes properly or not...

thanks,

greg k-h

Re: git doesn't like big files when pushing

[Junio C Hamano]

Greg KH <greg@kroah.com> writes:

> On Wed, Mar 01, 2006 at 02:08:02PM -0800, Greg KH wrote:
>> I have a mail archive stored with git, in mbox form, and I made some
>> changes to a few of the files and checked them back in.
>...

Ouch.  Running out of memory while deltifying sounds really bad.

> Oh, and I'm using:
> 	$ git --version
> 	git version 1.2.3.g8c2f
>
> if that helps or not.

It doen't, since I do not have 8c2fXXXX commit ;-).

I suspect "git push --thin origin" might help, if you are on my
"master" branch:

        diff-tree a79a276... (from 2245be3...)
        Author: Junio C Hamano <junkio@cox.net>
        Date:   Mon Feb 20 00:09:41 2006 -0800

            Add git-push --thin.

            Maybe we would want to make this default before it graduates to
            the master branch, but in the meantime to help testing things,
            this allows you to say "git push --thin destination".

            Signed-off-by: Junio C Hamano <junkio@cox.net>

        :100755 100755 706db99... 73dcf06... M	git-push.sh

Re: git doesn't like big files when pushing

[Greg KH]

On Wed, Mar 01, 2006 at 03:03:17PM -0800, Junio C Hamano wrote:
> Greg KH <greg@kroah.com> writes:
> 
> > On Wed, Mar 01, 2006 at 02:08:02PM -0800, Greg KH wrote:
> >> I have a mail archive stored with git, in mbox form, and I made some
> >> changes to a few of the files and checked them back in.
> >...
> 
> Ouch.  Running out of memory while deltifying sounds really bad.
> 
> > Oh, and I'm using:
> > 	$ git --version
> > 	git version 1.2.3.g8c2f
> >
> > if that helps or not.
> 
> It doen't, since I do not have 8c2fXXXX commit ;-).

Heh, that's a merge on my tree, sorry.  My tree only has one change to
the git-format-patch.sh script.

858cbfbabe4ede5f5eba32041eb7448319e53e2a is the most recent commit from
your tree.

> I suspect "git push --thin origin" might help, if you are on my
> "master" branch:
> 
>         diff-tree a79a276... (from 2245be3...)
>         Author: Junio C Hamano <junkio@cox.net>
>         Date:   Mon Feb 20 00:09:41 2006 -0800
> 
>             Add git-push --thin.
> 
>             Maybe we would want to make this default before it graduates to
>             the master branch, but in the meantime to help testing things,
>             this allows you to say "git push --thin destination".
> 
>             Signed-off-by: Junio C Hamano <junkio@cox.net>

Will try that.  I eventually gave up on the last push when it ran for 45
minutes at full cpu usage, and X got killed by the OOM killer in the
kernel for some reason...

thanks,

greg k-h

Re: git doesn't like big files when pushing

[Greg KH]

On Wed, Mar 01, 2006 at 03:27:19PM -0800, Greg KH wrote:
> On Wed, Mar 01, 2006 at 03:03:17PM -0800, Junio C Hamano wrote:
> > I suspect "git push --thin origin" might help, if you are on my
> > "master" branch:
> > 
> >         diff-tree a79a276... (from 2245be3...)
> >         Author: Junio C Hamano <junkio@cox.net>
> >         Date:   Mon Feb 20 00:09:41 2006 -0800
> > 
> >             Add git-push --thin.
> > 
> >             Maybe we would want to make this default before it graduates to
> >             the master branch, but in the meantime to help testing things,
> >             this allows you to say "git push --thin destination".
> > 
> >             Signed-off-by: Junio C Hamano <junkio@cox.net>
> 
> Will try that.  I eventually gave up on the last push when it ran for 45
> minutes at full cpu usage, and X got killed by the OOM killer in the
> kernel for some reason...

Nice, this worked!

Now what's the odds that when I pull from the server to another box
these same objects, the server will have the same problem as git-push
did?

will go try that now...

thanks,

greg k-h

Re: git doesn't like big files when pushing

[Greg KH]

On Wed, Mar 01, 2006 at 03:35:06PM -0800, Greg KH wrote:
> On Wed, Mar 01, 2006 at 03:27:19PM -0800, Greg KH wrote:
> > On Wed, Mar 01, 2006 at 03:03:17PM -0800, Junio C Hamano wrote:
> > > I suspect "git push --thin origin" might help, if you are on my
> > > "master" branch:
> > > 
> > >         diff-tree a79a276... (from 2245be3...)
> > >         Author: Junio C Hamano <junkio@cox.net>
> > >         Date:   Mon Feb 20 00:09:41 2006 -0800
> > > 
> > >             Add git-push --thin.
> > > 
> > >             Maybe we would want to make this default before it graduates to
> > >             the master branch, but in the meantime to help testing things,
> > >             this allows you to say "git push --thin destination".
> > > 
> > >             Signed-off-by: Junio C Hamano <junkio@cox.net>
> > 
> > Will try that.  I eventually gave up on the last push when it ran for 45
> > minutes at full cpu usage, and X got killed by the OOM killer in the
> > kernel for some reason...
> 
> Nice, this worked!
> 
> Now what's the odds that when I pull from the server to another box
> these same objects, the server will have the same problem as git-push
> did?

Ok, no problem there either.

thanks for the pointer to that option.

greg k-h

Re: git doesn't like big files when pushing

[Martin Langhoff]

On 3/2/06, Greg KH <greg@kroah.com> wrote:
> Ok, no problem there either.
>
> thanks for the pointer to that option.

Now that it has been tested twice, it's about time it becomes the default ;-)

cheers,


martin

Re: git doesn't like big files when pushing

[Linus Torvalds]

On Wed, 1 Mar 2006, Greg KH wrote:
> 
> That worked fine, but when I went to push the stuff to my server, I got
> the following errors:
> 
> $ git push origin
> updating 'refs/heads/master'
>   from 490badd9bec9ada3a21be275c97fb2a3a390f49e
>   to   16be8985abc8a9c89ad2cc8f46a0d8e9786e832f
> Generating pack...
> Done counting 8 objects.
> Deltifying 8 objects.
> fatal: Out of memory, malloc failed
> fatal: early EOF

Gaah. We probably have a memory leak somewhere, and it just normally 
doesn't much matter.

Git does want to keep the "window" of the objects it packs in memory while 
packing (it would be really costly to read them in one at a time, over and 
over again), but it should hopefully not really not need tons more memory 
than that. Since the window is normally 10, and you only have 8 objects, 
it really wants to have all eight in memory, but it shouldn't need a whole 
lot more.

But maybe it's really the case that you can't fit those 8 objects in 
memory. One option (which might also solve some of the performance issues) 
is to make the window be based on object _size_ rather than just be a 
fixed number (ie with an 80MB object, you'd only try a couple of objects 
around it, not the full window).

		Linus

Re: git doesn't like big files when pushing

[Simon Richter]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

Hi,

Linus Torvalds wrote:

> But maybe it's really the case that you can't fit those 8 objects in 
> memory. One option (which might also solve some of the performance issues) 
> is to make the window be based on object _size_ rather than just be a 
> fixed number (ie with an 80MB object, you'd only try a couple of objects 
> around it, not the full window).

Well, doesn't the pack process involve keeping objects of similar size 
in memory next to each other? My suspicion is that the system is running 
out of virtual address space because almost every malloc() will attempt 
to get an entirely new block because the one that was just freed is a 
few bytes too small.

    Simon

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 307 bytes --]