From: Eric Wong <e@80x24.org>
To: Jeff King <peff@peff.net>, Junio C Hamano <gitster@pobox.com>
Cc: git@vger.kernel.org, "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Subject: Re: [PATCH v2] xmmap: inform Linux users of tuning knobs on ENOMEM
Date: Wed, 30 Jun 2021 10:07:22 +0000 [thread overview]
Message-ID: <20210630100722.GB2653@dcvr> (raw)
In-Reply-To: <YNva8PWFJ24QX2yF@coredump.intra.peff.net>
Jeff King <peff@peff.net> wrote:
> On Wed, Jun 30, 2021 at 12:01:32AM +0000, Eric Wong wrote:
>
> > This series is now down to a single patch.
> >
> > I wanted to make things more transparent to users without
> > privileges to raise sys.vm.max_map_count and/or RLIMIT_DATA;
> > but it doesn't seem possible to account for libc/zlib/etc. doing
> > mmap() without our knowledge (usually via malloc).
>
> Oh, I should have read this one before reviewing the inner parts of v1. :)
>
> In general I agree that trying to manage our map count can never be
> foolproof. As you note, other parts of the system may contribute to that
> count. But even within Git, we don't have any mechanism for unmapping
> many non-packfiles. So if you have 30,000 packs, you may hit the limit
> purely via the .idx files (and ditto for the new .rev files, and
> probably commit-graph files, etc).
Yeah, the most annoying thing with my original series was when
I hit "inflate: out of memory" once I stopped xmmap from dying.
I suspect that would be a worse far error message for users who
aren't familiar with how malloc works.
> That said, I'm not opposed to handling xmmap() failures more gracefully,
> as your series did. It's not foolproof, but it might help in some cases.
I've also been wondering if we can maintain a watermark based
on reading the contents of /proc/sys/vm/max_map_count and the
mappings we make. Then we could start dropping mappings
when we hit half (or some other threshold) of that sysctl.
Similar for RLIMIT_DATA (though that defaults to unlimited
on my Debian system).
OTOH, I also wonder if we're overusing mmap when we could be
just as well served with pread.
I'm not up-to-date on modern mmap performance and maybe CPU
vulnerability mitigations nowadays make mmap more compelling.
However, once upon a time in 2006, pread could be a hair quicker:
https://lore.kernel.org/git/Pine.LNX.4.64.0612182234260.3479@woody.osdl.org/T/#u
(But that info could be out-of-date...)
I'm also somewhat paranoid when it comes to mmap since rogue
processes could be truncating mmap-ed files to cause bus errors.
> > So I think giving users some information to feed their sysadmins
> > is the best we can do in this situation:
>
> This seems OK to me, too. Translators might complain a bit about the
> message-lego. I don't have a strong opinion.
*shrug* I saw my original patches already ended up in `seen'
(commit 7b79212a93c375365c06cab5c0018ab97a4185cf)
next prev parent reply other threads:[~2021-06-30 10:07 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-29 8:11 [PATCH 0/4] gracefully handling mmap failures Eric Wong
2021-06-29 8:11 ` [PATCH 1/4] use_pack: attempt to handle ENOMEM from mmap Eric Wong
2021-06-30 2:30 ` Jeff King
2021-06-29 8:11 ` [PATCH 2/4] map_loose_object_1: " Eric Wong
2021-06-30 2:41 ` Jeff King
2021-06-30 6:06 ` Ævar Arnfjörð Bjarmason
2021-06-29 8:11 ` [PATCH 3/4] check_packed_git_idx: " Eric Wong
2021-06-29 20:21 ` [HOLD " Eric Wong
2021-06-29 21:33 ` Junio C Hamano
2021-06-29 22:31 ` Eric Wong
2021-06-29 8:11 ` [PATCH 4/4] xmmap: inform Linux users of tuning knobs on ENOMEM Eric Wong
2021-06-30 0:01 ` [PATCH v2] " Eric Wong
2021-06-30 2:46 ` Jeff King
2021-06-30 10:07 ` Eric Wong [this message]
2021-06-30 17:18 ` Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210630100722.GB2653@dcvr \
--to=e@80x24.org \
--cc=avarab@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=peff@peff.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).