The Linux Kernel Mailing List
 help / color / mirror / Atom feed
* [PATCH] cgroup: allocate the inode for cgroup_may_write() before cgroup_mutex
@ 2026-11-18 23:03 Shakeel Butt
  2026-11-20 11:24 ` Tejun Heo
  0 siblings, 1 reply; 2+ messages in thread
From: Shakeel Butt @ 2026-11-18 23:03 UTC (permalink / raw)
  To: Tejun Heo, Christian Brauner
  Cc: Johannes Weiner, Michal Koutný, Greg Kroah-Hartman,
	Jori Koolstra, Meta kernel team, cgroups, linux-fsdevel,
	driver-core, linux-kernel

In the Meta fleet, on 6.16 kernels, the memcg OOM killer ran from
cgroup_may_write() with cgroup_mutex and cgroup_threadgroup_rwsem held.
The callers were systemd in containers and build tools, short of memory
in their own cgroup.  Our lock monitor also saw cgroup_mutex held
across memcg reclaim there, for up to 152 ms.

cgroup_may_write() checks permission on the inode of a cgroup.procs
file.  kernfs frees that inode with its last reference, so it is often
not cached, and kernfs_get_inode() then allocates one, charged to the
caller's memory cgroup, with cgroup_mutex held.  A cgroup.procs write
usually holds cgroup_threadgroup_rwsem too.  While the allocation
reclaims, every cgroup operation, fork and exit on the host waits.
clone3(CLONE_INTO_CGROUP) allocates the same way.

Allocate the inode before taking cgroup_mutex, and add it under
kernfs_rwsem with kernfs_get_inode_locked(), as kernfs lookup does.
The cgroup.procs write allocates after dropping the file's active
reference, because rmdir waits for active references with cgroup_mutex
held.  clone3() checks that the fd is a cgroup2 directory before
allocating on its superblock.

Worst latency in a VM while a task in a full 256 MB memory cgroup moves
a process between two cgroups (60 s, worst of four runs):

                          before    after
  fork and exit            4.9 s     14 ms
  mkdir and rmdir          4.9 s    8.4 ms
  cgroup_mutex held        4.9 s    8.3 ms

Before, each run spent 2 to 16 s in memcg reclaim with cgroup_mutex
held.  After, none.

Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
This is based on my series "kernfs: stop taking kernfs_iattr_rwsem for
writing in mkdir/rmdir" [1], on top of vfs-7.5.kernfs, as both change
the comment of kernfs_get_inode().

[1] https://lore.kernel.org/all/20261118220154.2965621-1-shakeel.butt@linux.dev/

 fs/kernfs/inode.c      |  28 ++++-----
 include/linux/kernfs.h |   4 +-
 kernel/cgroup/cgroup.c | 128 +++++++++++++++++++++++++++--------------
 3 files changed, 98 insertions(+), 62 deletions(-)

diff --git a/fs/kernfs/inode.c b/fs/kernfs/inode.c
index 580095e0fad9..71e19d32e392 100644
--- a/fs/kernfs/inode.c
+++ b/fs/kernfs/inode.c
@@ -318,30 +318,24 @@ static void kernfs_init_inode(struct kernfs_node *kn, struct inode *inode)
 }
 
 /**
- *	kernfs_get_inode - get inode for kernfs_node
- *	@sb: super block
- *	@kn: kernfs_node to allocate inode for
+ *	kernfs_get_inode - get the inode of a kernfs_node
+ *	@inode: inode allocated with new_inode() on the target superblock
+ *	@kn: kernfs_node to get the inode for
  *
- *	Get inode for @kn.  If such inode doesn't exist, a new inode is
- *	allocated and basics are initialized.  New inode is returned
- *	locked.
+ *	kernfs_get_inode_locked() for callers outside kernfs.  Allocate
+ *	@inode before taking your own locks, as the allocation can wait for
+ *	reclaim.  @inode is consumed.
  *
  *	Locking:
- *	Kernel thread context (may sleep).  Not for directories: kernfs
- *	sets up their inodes itself, with kernfs_rwsem held.
+ *	Kernel thread context (may sleep).  Takes kernfs_rwsem for reading.
  *
  *	Return:
- *	Pointer to allocated inode on success, %NULL on failure.
+ *	The inode of @kn, or %NULL if @kn is no longer active.
  */
-struct inode *kernfs_get_inode(struct super_block *sb, struct kernfs_node *kn)
+struct inode *kernfs_get_inode(struct inode *inode, struct kernfs_node *kn)
 {
-	struct inode *inode;
-
-	inode = iget_locked(sb, kernfs_ino(kn));
-	if (inode && (inode_state_read_once(inode) & I_NEW))
-		kernfs_init_inode(kn, inode);
-
-	return inode;
+	guard(rwsem_read)(&kernfs_root(kn)->kernfs_rwsem);
+	return kernfs_get_inode_locked(inode, kn);
 }
 
 static int kernfs_inode_test(struct inode *inode, void *data)
diff --git a/include/linux/kernfs.h b/include/linux/kernfs.h
index 803294a67e47..cda9467693da 100644
--- a/include/linux/kernfs.h
+++ b/include/linux/kernfs.h
@@ -429,7 +429,7 @@ void kernfs_put(struct kernfs_node *kn);
 
 struct kernfs_node *kernfs_node_from_dentry(struct dentry *dentry);
 struct kernfs_root *kernfs_root_from_sb(struct super_block *sb);
-struct inode *kernfs_get_inode(struct super_block *sb, struct kernfs_node *kn);
+struct inode *kernfs_get_inode(struct inode *inode, struct kernfs_node *kn);
 
 struct dentry *kernfs_node_dentry(struct kernfs_node *kn,
 				  struct super_block *sb);
@@ -528,7 +528,7 @@ static inline struct kernfs_root *kernfs_root_from_sb(struct super_block *sb)
 { return NULL; }
 
 static inline struct inode *
-kernfs_get_inode(struct super_block *sb, struct kernfs_node *kn)
+kernfs_get_inode(struct inode *inode, struct kernfs_node *kn)
 { return NULL; }
 
 static inline struct kernfs_root *
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index a284225f8dc9..15e03faedbbc 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -1664,6 +1664,32 @@ void cgroup_kn_unlock(struct kernfs_node *kn)
 	cgroup_put(cgrp);
 }
 
+/*
+ * Pin the cgroup of @kn and drop @kn's active reference: the part of
+ * cgroup_kn_lock_live() before cgroup_mutex is taken.  cgroup_kn_unlock()
+ * undoes it.
+ */
+static struct cgroup *cgroup_kn_break_active(struct kernfs_node *kn)
+{
+	struct cgroup *cgrp;
+
+	if (kernfs_type(kn) == KERNFS_DIR)
+		cgrp = kn->priv;
+	else
+		cgrp = kn_priv(kn);
+
+	/*
+	 * We're gonna grab cgroup_mutex which nests outside kernfs
+	 * active_ref.  cgroup liveliness check alone provides enough
+	 * protection against removal.  Ensure @cgrp stays accessible and
+	 * break the active_ref protection.
+	 */
+	if (!cgroup_tryget(cgrp))
+		return ERR_PTR(-ENODEV);
+	kernfs_break_active_protection(kn);
+	return cgrp;
+}
+
 /**
  * cgroup_kn_lock_live - locking helper for cgroup kernfs methods
  * @kn: the kernfs_node being serviced
@@ -1687,20 +1713,9 @@ struct cgroup *cgroup_kn_lock_live(struct kernfs_node *kn, bool drain_offline)
 	struct cgroup *cgrp;
 	int ret;
 
-	if (kernfs_type(kn) == KERNFS_DIR)
-		cgrp = kn->priv;
-	else
-		cgrp = kn_priv(kn);
-
-	/*
-	 * We're gonna grab cgroup_mutex which nests outside kernfs
-	 * active_ref.  cgroup liveliness check alone provides enough
-	 * protection against removal.  Ensure @cgrp stays accessible and
-	 * break the active_ref protection.
-	 */
-	if (!cgroup_tryget(cgrp))
-		return ERR_PTR(-ENODEV);
-	kernfs_break_active_protection(kn);
+	cgrp = cgroup_kn_break_active(kn);
+	if (IS_ERR(cgrp))
+		return cgrp;
 
 	if (drain_offline) {
 		ret = cgroup_lock_and_drain_offline(cgrp);
@@ -5424,16 +5439,23 @@ static int cgroup_procs_show(struct seq_file *s, void *v)
 	return 0;
 }
 
-static int cgroup_may_write(const struct cgroup *cgrp, struct super_block *sb)
+/*
+ * *@new is an inode from new_inode(), allocated before cgroup_mutex as the
+ * allocation can wait for reclaim.  This uses it up and sets *@new to NULL.
+ */
+static int cgroup_may_write(const struct cgroup *cgrp, struct inode **new)
 {
+	struct inode *inode = *new;
 	int ret;
-	struct inode *inode;
 
 	lockdep_assert_held(&cgroup_mutex);
 
-	inode = kernfs_get_inode(sb, cgrp->procs_file.kn);
+	*new = NULL;
 	if (!inode)
 		return -ENOMEM;
+	inode = kernfs_get_inode(inode, cgrp->procs_file.kn);
+	if (!inode)
+		return -ENODEV;
 
 	ret = inode_permission(&nop_mnt_idmap, inode, MAY_WRITE);
 	iput(inode);
@@ -5442,7 +5464,7 @@ static int cgroup_may_write(const struct cgroup *cgrp, struct super_block *sb)
 
 static int cgroup_procs_write_permission(struct cgroup *src_cgrp,
 					 struct cgroup *dst_cgrp,
-					 struct super_block *sb,
+					 struct inode **new,
 					 struct cgroup_namespace *ns)
 {
 	struct cgroup *com_cgrp = src_cgrp;
@@ -5455,7 +5477,7 @@ static int cgroup_procs_write_permission(struct cgroup *src_cgrp,
 		com_cgrp = cgroup_parent(com_cgrp);
 
 	/* %current should be authorized to migrate to the common ancestor */
-	ret = cgroup_may_write(com_cgrp, sb);
+	ret = cgroup_may_write(com_cgrp, new);
 	if (ret)
 		return ret;
 
@@ -5473,12 +5495,12 @@ static int cgroup_procs_write_permission(struct cgroup *src_cgrp,
 
 static int cgroup_attach_permissions(struct cgroup *src_cgrp,
 				     struct cgroup *dst_cgrp,
-				     struct super_block *sb, bool threadgroup,
+				     struct inode **new, bool threadgroup,
 				     struct cgroup_namespace *ns)
 {
 	int ret = 0;
 
-	ret = cgroup_procs_write_permission(src_cgrp, dst_cgrp, sb, ns);
+	ret = cgroup_procs_write_permission(src_cgrp, dst_cgrp, new, ns);
 	if (ret)
 		return ret;
 
@@ -5498,12 +5520,25 @@ static ssize_t __cgroup_procs_write(struct kernfs_open_file *of, char *buf,
 	struct cgroup_file_ctx *ctx = of->priv;
 	struct cgroup *src_cgrp, *dst_cgrp;
 	struct task_struct *task;
+	struct inode *new;
 	ssize_t ret;
 	enum cgroup_attach_lock_mode lock_mode;
 
-	dst_cgrp = cgroup_kn_lock_live(of->kn, false);
+	/*
+	 * Allocate the inode cgroup_may_write() may need before taking
+	 * cgroup_mutex and with the file's active reference dropped: the
+	 * allocation can wait for reclaim, and rmdir waits for active
+	 * references with cgroup_mutex held.
+	 */
+	dst_cgrp = cgroup_kn_break_active(of->kn);
 	if (IS_ERR(dst_cgrp))
 		return PTR_ERR(dst_cgrp);
+	new = new_inode(of->file->f_path.dentry->d_sb);
+	cgroup_lock();
+	if (cgroup_is_dead(dst_cgrp)) {
+		ret = -ENODEV;
+		goto out_unlock;
+	}
 
 	task = cgroup_procs_write_start(buf, threadgroup, &lock_mode);
 	ret = PTR_ERR_OR_ZERO(task);
@@ -5521,8 +5556,7 @@ static ssize_t __cgroup_procs_write(struct kernfs_open_file *of, char *buf,
 	 * inherited fd attacks.
 	 */
 	scoped_with_creds(of->file->f_cred)
-		ret = cgroup_attach_permissions(src_cgrp, dst_cgrp,
-						of->file->f_path.dentry->d_sb,
+		ret = cgroup_attach_permissions(src_cgrp, dst_cgrp, &new,
 						threadgroup, ctx->ns);
 	if (ret)
 		goto out_finish;
@@ -5533,7 +5567,7 @@ static ssize_t __cgroup_procs_write(struct kernfs_open_file *of, char *buf,
 	cgroup_procs_write_finish(task, lock_mode);
 out_unlock:
 	cgroup_kn_unlock(of->kn);
-
+	iput(new);
 	return ret;
 }
 
@@ -6924,13 +6958,31 @@ static void cgroup_fork_child_unlock(struct task_struct *child)
 static int cgroup_css_set_fork(struct kernel_clone_args *kargs)
 	__acquires(&cgroup_mutex) __acquires(&cgroup_threadgroup_rwsem)
 {
+	struct inode *new_dst = NULL, *new_com = NULL;
+	struct file *file = NULL;
 	int ret;
 	struct cgroup *dst_cgrp = NULL;
 	struct css_set *cset;
-	struct super_block *sb;
 
-	if (kargs->flags & CLONE_INTO_CGROUP)
+	if (kargs->flags & CLONE_INTO_CGROUP) {
+		/*
+		 * Allocate the inodes cgroup_may_write() may need before
+		 * taking cgroup_mutex, as the allocation can wait for reclaim.
+		 * Check first that the fd is a cgroup2 directory, so they are
+		 * allocated on a cgroup2 superblock.
+		 */
+		file = fget_raw(kargs->cgroup);
+		if (!file)
+			return -EBADF;
+		dst_cgrp = cgroup_get_from_file(file);
+		if (IS_ERR(dst_cgrp)) {
+			fput(file);
+			return PTR_ERR(dst_cgrp);
+		}
+		new_dst = new_inode(file->f_path.dentry->d_sb);
+		new_com = new_inode(file->f_path.dentry->d_sb);
 		cgroup_lock();
+	}
 
 	cgroup_threadgroup_change_begin(current);
 
@@ -6945,20 +6997,6 @@ static int cgroup_css_set_fork(struct kernel_clone_args *kargs)
 		return 0;
 	}
 
-	CLASS(fd_raw, f)(kargs->cgroup);
-	if (fd_empty(f)) {
-		ret = -EBADF;
-		goto err;
-	}
-	sb = fd_file(f)->f_path.dentry->d_sb;
-
-	dst_cgrp = cgroup_get_from_file(fd_file(f));
-	if (IS_ERR(dst_cgrp)) {
-		ret = PTR_ERR(dst_cgrp);
-		dst_cgrp = NULL;
-		goto err;
-	}
-
 	if (cgroup_is_dead(dst_cgrp)) {
 		ret = -ENODEV;
 		goto err;
@@ -6969,7 +7007,7 @@ static int cgroup_css_set_fork(struct kernel_clone_args *kargs)
 	 * usually done by the vfs layer but since we're not going through
 	 * the vfs layer here we need to do it "manually".
 	 */
-	ret = cgroup_may_write(dst_cgrp, sb);
+	ret = cgroup_may_write(dst_cgrp, &new_dst);
 	if (ret)
 		goto err;
 
@@ -6987,7 +7025,7 @@ static int cgroup_css_set_fork(struct kernel_clone_args *kargs)
 	 * cgroup.procs of the cgroup indicated by @dfd_cgroup. This allows us
 	 * to always use the caller's credentials.
 	 */
-	ret = cgroup_attach_permissions(cset->dfl_cgrp, dst_cgrp, sb,
+	ret = cgroup_attach_permissions(cset->dfl_cgrp, dst_cgrp, &new_com,
 					!(kargs->flags & CLONE_THREAD),
 					current->nsproxy->cgroup_ns);
 	if (ret)
@@ -7002,11 +7040,15 @@ static int cgroup_css_set_fork(struct kernel_clone_args *kargs)
 	put_css_set(cset);
 	kargs->cgrp = dst_cgrp;
 	kargs->kill_seq = dst_cgrp->kill_seq;
+	fput(file);
 	return ret;
 
 err:
 	cgroup_threadgroup_change_end(current);
 	cgroup_unlock();
+	iput(new_dst);
+	iput(new_com);
+	fput(file);
 	if (dst_cgrp)
 		cgroup_put(dst_cgrp);
 	put_css_set(cset);

base-commit: ccf2a4692fed0fa77751d32d2690585127a152c5
-- 
2.53.1-Meta


^ permalink raw reply related	[flat|nested] 2+ messages in thread

* Re: [PATCH] cgroup: allocate the inode for cgroup_may_write() before cgroup_mutex
  2026-11-18 23:03 [PATCH] cgroup: allocate the inode for cgroup_may_write() before cgroup_mutex Shakeel Butt
@ 2026-11-20 11:24 ` Tejun Heo
  0 siblings, 0 replies; 2+ messages in thread
From: Tejun Heo @ 2026-11-20 11:24 UTC (permalink / raw)
  To: Shakeel Butt
  Cc: Christian Brauner, Johannes Weiner, Michal Koutný,
	Greg Kroah-Hartman, Jori Koolstra, Meta kernel team, cgroups,
	linux-fsdevel, driver-core, linux-kernel

Hello, Shakeel.

On Wed, Nov 18, 2026 at 03:03:45PM -0800, Shakeel Butt wrote:
> +	dst_cgrp = cgroup_kn_break_active(of->kn);
>  	if (IS_ERR(dst_cgrp))
>  		return PTR_ERR(dst_cgrp);
> +	new = new_inode(of->file->f_path.dentry->d_sb);
> +	cgroup_lock();

I don't really like new_inode() being called on every write, and twice on
every clone3(CLONE_INTO_CGROUP). The inode is often not cached but not
mostly. It's not a lot of overhead, but doing it the right way isn't that
complicated either. How about peeking at the inode cache before taking the
locks, preallocating only when the peek misses, and then looking up again
under cgroup_mutex? If the peek turns out wrong, it can drop the locks,
allocate and retry, so that nothing is allocated under the lock. Can we
apply the same shape in other places too?

> 	scoped_with_creds(of->file->f_cred)
> -		ret = cgroup_attach_permissions(src_cgrp, dst_cgrp,
> -						of->file->f_path.dentry->d_sb,
> +		ret = cgroup_attach_permissions(src_cgrp, dst_cgrp, &new,
> 						threadgroup, ctx->ns);

AFAICS, this changes behavior under Smack. Smack labels a new inode from the
allocating task's creds, and the allocation used to be inside
scoped_with_creds(). Now the inode gets the writer's label and the
permission check runs with the opener's creds, which can fail when the two
differ. Maybe the allocation can use the opener's creds too?

Thanks.

--
tejun

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2026-11-20 11:24 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 23:03 [PATCH] cgroup: allocate the inode for cgroup_may_write() before cgroup_mutex Shakeel Butt
2026-11-20 11:24 ` Tejun Heo

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox