From: Tejun Heo <tj@kernel.org>
To: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Christian Brauner <christian@brauner.io>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
Meta kernel team <kernel-team@meta.com>,
linux-fsdevel@vger.kernel.org, driver-core@lists.linux.dev,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH] kernfs: don't hold kernfs_rwsem across notification delivery
Date: Thu, 10 Sep 2026 11:12:22 -1000 [thread overview]
Message-ID: <aqMdNoXA0LIYFpQW@slm.duckdns.org> (raw)
In-Reply-To: <20260910045406.485295-1-shakeel.butt@linux.dev>
On Wed, Sep 09, 2026 at 09:54:06PM -0700, Shakeel Butt wrote:
> Recently at Meta we noticed that on multi-tenant systems, a workload
> under extreme memory pressure negatively impacts unrelated workloads and
> system daemons like below [1] and fb-oomd [2]. Mainly we observed that
> system daemons were stuck on kernfs_rwsem in cgroup-related interfaces
> for long periods of time.
>
> On further inspection, we found that the workload under pressure had
> registered inotify watches on its memory.events and was continuously
> receiving limit-hitting notifications. The cgroup notification is done in
> kernfs_notify_workfn(), which holds kernfs_supers_rwsem and kernfs_rwsem
> and calls fsnotify(), which allocates the event with
> kmalloc(GFP_KERNEL_ACCOUNT|__GFP_RETRY_MAYFAIL) charged to the watching
> cgroup, which is already under memory pressure. So the notify worker
> ended up in memory reclaim of a cgroup already under pressure, and itself
> kept triggering the limit notifications.
>
> This notification path takes the locks in read mode, but a waiting writer
> blocks all future readers. That is exactly what is happening in the Meta
> fleet, and it leaves daemons that are critical to the reliability of the
> system stuck for long periods of time.
>
> Commit 400188ae361a ("kernfs: Acquire kernfs_rwsem in
> kernfs_notify_workfn().") added kernfs_rwsem just to safely read
> kernfs_node::name. Let's sample the name once before the loop and drop
> the lock. kernfs_supers_rwsem still covers the list, and the removal
> paths take that one for reading too, so a stalled worker can still hold
> a removal up once a mount or unmount is queued behind it.
>
> Two things change for a watcher. A name longer than NAME_MAX now gives
> an event with no name, where before it gave one with the full name;
> fsnotify() takes the name as optional, so the event still arrives. And
> the name and the parent are no longer sampled under one lock, so a rename
> between the two would name a file against the directory it moved to.
> That cannot happen today: kernfs_notify() rejects anything that is not a
> file, and no caller of kernfs_rename_ns() renames one.
>
> [1] https://github.com/facebookincubator/below
> [2] https://github.com/facebookincubator/oomd
>
> Fixes: 400188ae361a ("kernfs: Acquire kernfs_rwsem in kernfs_notify_workfn().")
> Assisted-by: LLM
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Tejun Heo <tj@kernel.org>
Thanks.
--
tejun
prev parent reply other threads:[~2026-09-10 21:12 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-09-10 4:54 [PATCH] kernfs: don't hold kernfs_rwsem across notification delivery Shakeel Butt
2026-09-10 21:12 ` Tejun Heo [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=aqMdNoXA0LIYFpQW@slm.duckdns.org \
--to=tj@kernel.org \
--cc=bigeasy@linutronix.de \
--cc=christian@brauner.io \
--cc=driver-core@lists.linux.dev \
--cc=gregkh@linuxfoundation.org \
--cc=kernel-team@meta.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=shakeel.butt@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.