Re: [PATCH v2] xmmap: inform Linux users of tuning knobs on ENOMEM

[Eric Wong <e@80x24.org>]

Jeff King <peff@peff.net> wrote:
> On Wed, Jun 30, 2021 at 12:01:32AM +0000, Eric Wong wrote:
> 
> > This series is now down to a single patch.
> > 
> > I wanted to make things more transparent to users without
> > privileges to raise sys.vm.max_map_count and/or RLIMIT_DATA;
> > but it doesn't seem possible to account for libc/zlib/etc. doing
> > mmap() without our knowledge (usually via malloc).
> 
> Oh, I should have read this one before reviewing the inner parts of v1. :)
> 
> In general I agree that trying to manage our map count can never be
> foolproof. As you note, other parts of the system may contribute to that
> count. But even within Git, we don't have any mechanism for unmapping
> many non-packfiles. So if you have 30,000 packs, you may hit the limit
> purely via the .idx files (and ditto for the new .rev files, and
> probably commit-graph files, etc).

Yeah, the most annoying thing with my original series was when
I hit "inflate: out of memory" once I stopped xmmap from dying.
I suspect that would be a worse far error message for users who
aren't familiar with how malloc works.

> That said, I'm not opposed to handling xmmap() failures more gracefully,
> as your series did. It's not foolproof, but it might help in some cases.

I've also been wondering if we can maintain a watermark based
on reading the contents of /proc/sys/vm/max_map_count and the
mappings we make.   Then we could start dropping mappings
when we hit half (or some other threshold) of that sysctl.
Similar for RLIMIT_DATA (though that defaults to unlimited
on my Debian system).

OTOH, I also wonder if we're overusing mmap when we could be
just as well served with pread.

I'm not up-to-date on modern mmap performance and maybe CPU
vulnerability mitigations nowadays make mmap more compelling.
However, once upon a time in 2006, pread could be a hair quicker:

https://lore.kernel.org/git/Pine.LNX.4.64.0612182234260.3479@woody.osdl.org/T/#u
(But that info could be out-of-date...)

I'm also somewhat paranoid when it comes to mmap since rogue
processes could be truncating mmap-ed files to cause bus errors.

> > So I think giving users some information to feed their sysadmins
> > is the best we can do in this situation:
> 
> This seems OK to me, too. Translators might complain a bit about the
> message-lego. I don't have a strong opinion.

*shrug*  I saw my original patches already ended up in `seen'
(commit 7b79212a93c375365c06cab5c0018ab97a4185cf)