Re: Errors cloning large repo

[Linus Torvalds <torvalds@linux-foundation.org>]

On Fri, 9 Mar 2007, Anton Tropashko wrote:
> 
> My problem is git-clone though since for commit it's no big deal
> to git commit [a-c]* , or use xargs as a workaround

Sure, but there were two problems.

The "git commit" problem is trivial, and in no way fundamental. The thing 
that uses tons of memory is literally just eyecandy, to show you *what* 
you're committing.

In fact, by the time it starts using tons of memory, the commit has 
literally already happened. It's just doing statistics afterwards that 
bloats it up.

> For git clone I got this

The "git clone" problem is different, in that it's due to the 2GB 
pack-file limit. It's not "fundmentally hard" either, but it's at least 
not just a small tiny silly detail.

In fact, you can just do

	git add .
	git commit -q

and the "-q" flag (or "--quiet") will mean that the diffstat is never 
done, and the commit should be almost instantaneous (all the real work is 
done by the "git add .")

So "git commit" issue really is just a small beauty wart.

> Deltifying 144511 objects.
>  100% (144511/144511) done
> 1625.375MB  (1713 kB/s)       
> 1729.057MB  (499 kB/s)       
> /usr/bin/git-clone: line 321: 24360 File size limit exceededgit-fetch-pack --all -k $quiet "$repo"
> 
> again after git repack and don't see how to work around that aside from artifically
> splitting the tree at the top or resorting to a tarball on an ftp site.

So the "git repack" actually worked for you? It really shouldn't have 
worked.

Is the server side perhaps 64-bit? If so, the limit ends up being 4GB 
instead of 2GB, and your 8.5GB project may actually fit.

If so, we can trivially fix it with the current index file even for a 
32-bit machine. The reason we limit pack-files to 2GB on 32-bit machines 
is purely that we don't use O_LARGEFILE. If we enable O_LARGEFILE, that 
moves the limit up from 31 bits to 32 bits, and it might be enough for 
you. No new data structures for the index necessary at all.

		Linus