Re: git status OOM on mmap of large file

[Jeff King <peff@peff.net>]

On Thu, Jan 24, 2019 at 02:18:36PM -0500, Jeff King wrote:

> > I did some benchmarking, using cat as the clean filter:
> > [...]
> > From this, it looks like the file has to be quite large before the
> > preallocation makes a sizable improvement to runtime, and the
> > smudge/clean filters have to be used for actual content filtering
> > (not for hash generation purposes as git-annex and git-lfs use it).
> > An unusual edge case I think. So hint == 0 seems fine.
> 
> Thanks for these timings! I agree that "hint == 0" is probably
> reasonable, then.

One other minor point to consider: on some systems over-allocating
actually isn't that expensive, because pages are actually allocated
until we write to them, and malloc() is perfectly happy to overcommit
memory.  Your case would only run into problems on Linux when malloc()
actually refuses the allocation (so limiting ourselves to "too large but
still reasonable" is a valid strategy there).

But I doubt that's something we should be relying on in general. There
are many systems that don't overcommit.

-Peff