malloc fails when dealing with huge files

malloc fails when dealing with huge files

[Jonathan Blanton]

I'm using Git for a project that contains huge (multi-gigabyte) files.
 I need to track these files, but with some of the really big ones,
git-add aborts with the message "fatal: Out of memory, malloc failed".
 Also, git-gc sometimes fails because it can't allocate enough memory.
 I've been using the "--window-memory" option to git- repack to work
around the git-gc problem, but I don't know of a similar trick for
git-add.  Is there any way (aside from adding more memory, of course)
that I can deal with these huge files?  I'm using git 1.5.6.2 on
Debian 4.0.

Thanks in advance.

Jonathan Blanton

Re: malloc fails when dealing with huge files

[Linus Torvalds]

On Wed, 10 Dec 2008, Jonathan Blanton wrote:
>
> I'm using Git for a project that contains huge (multi-gigabyte) files.
>  I need to track these files, but with some of the really big ones,
> git-add aborts with the message "fatal: Out of memory, malloc failed".

git is _really_ not designed for huge files.

By design - good or bad - git does pretty much all single file operations 
with the whole file in memory as one single allocation. 

Now, some of that is hard to fix - or at least would generate much more 
complex code. The _particular_ case of "git add" could be fixed without 
undue pain, but it's not entirely trivial either.

The main offender is probably "index_fd()" that just mmap's the whole file 
in one go and then calls write_sha1_file() which really expects it to be 
one single memory area both for the initial SHA1 create and for the 
compression and writing out of the result.

Changing that to do big files in pieces would not be _too_ painful, but 
it's not just a couple of lines either.

However, git performance with big files would never be wonderful, and 
things like "git diff" would still end up reading not just the whole file, 
but _both_versions_ at the same time. Marking the big files as being 
no-diff might help, though.


			Linus

Re: malloc fails when dealing with huge files

[Jeff Whiteside]

i tried to do something like that over a year ago, having gotten the
insane idea that i wanted to version my whole harddrive.  binaries
were a huge problem.

checkouts were also a problem over slow connections because there is
no git-clone --resume, so if your connection is interrupted, you're
back at square one.  perhaps git-torrent will fix that.

git wasn't supposed to be file based, as much as line/code based.  let
me know if you find a better alternative to git for filesystems.

it's too bad there's not a better way to keep resources tagged to a
version by a sha1, but keep source separate.

On Wed, Dec 10, 2008 at 11:32 AM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
>
> On Wed, 10 Dec 2008, Jonathan Blanton wrote:
>>
>> I'm using Git for a project that contains huge (multi-gigabyte) files.
>>  I need to track these files, but with some of the really big ones,
>> git-add aborts with the message "fatal: Out of memory, malloc failed".
>
> git is _really_ not designed for huge files.
>
> By design - good or bad - git does pretty much all single file operations
> with the whole file in memory as one single allocation.
>
> Now, some of that is hard to fix - or at least would generate much more
> complex code. The _particular_ case of "git add" could be fixed without
> undue pain, but it's not entirely trivial either.
>
> The main offender is probably "index_fd()" that just mmap's the whole file
> in one go and then calls write_sha1_file() which really expects it to be
> one single memory area both for the initial SHA1 create and for the
> compression and writing out of the result.
>
> Changing that to do big files in pieces would not be _too_ painful, but
> it's not just a couple of lines either.
>
> However, git performance with big files would never be wonderful, and
> things like "git diff" would still end up reading not just the whole file,
> but _both_versions_ at the same time. Marking the big files as being
> no-diff might help, though.
>
>
>                        Linus
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: malloc fails when dealing with huge files

[Johannes Schindelin]

Hi,

On Wed, 10 Dec 2008, Linus Torvalds wrote:

> However, git performance with big files would never be wonderful, and 
> things like "git diff" would still end up reading not just the whole 
> file, but _both_versions_ at the same time. Marking the big files as 
> being no-diff might help, though.

Makes me wonder if we should not have a default cut-off, say, 10MB, at 
which files are automatically tagged with the no-diff attribute (unless 
overridden explicitely in .gitattributes)?

Ciao,
Dscho