Re: Problem with large files on different OSes

[Linus Torvalds <torvalds@linux-foundation.org>]

On Wed, 27 May 2009, Jakub Narebski wrote:
> 
> Do seting `pack.packSizeLimit`, or adjusting values of
> `core.packedGitWindowSize` and/or `core.packedGitLimit`
> (see git-config(1)) help in your situation?

No, that will help just the packfile mmap (and even there, it won't help 
with things like index file size - we'll always mmap the whole index 
file). It's definitely worth doing, though - but I think we already 
default to 32MB pack-file windows on 32-bit architectures.

Individual files we always handle in one go. It's what git was designed 
for, after all - fairly small files. And so git is limited to files 
smaller than the virtual address space.

On a 32-bit setup, that often limits you to roughly a gigabyte. You have 
4GB of virtual address space, of which one or two is used for the OS 
kernel. So say you have 2GB for user mode - you then have the executable 
mapping and libraries and stack, all spread out in that 2GB virtual 
address space.

In fact, even if it's 3GB for user (I don't know what OS X does), getting 
one contiguous area may well be limited to ~1GB depending on layout of 
shared library mappings etc VM fragmentation. Older Linux systems tended 
to map things in ways that made it hard to get more than 1GB of contiguous 
data mapping if you compiled with dynamic libraries.

64-bit mostly makes this a non-issue.

In fact, if you do "diff", you're going to be even _more_ limited, since 
for simplicity, our version of xdiff really wants both sources in memory 
at a time. So you can't really diff >500MB files. Not that you generally 
want to, of course.

I'll see if I can make us handle the "big file without diff" case better 
by chunking.

		Linus