* [PATCH 1/6] kernfs: report times and link count from the node in getattr
2026-11-18 22:01 [PATCH 0/6] kernfs: stop taking kernfs_iattr_rwsem for writing in mkdir/rmdir Shakeel Butt
@ 2026-11-18 22:01 ` Shakeel Butt
2026-11-18 22:01 ` [PATCH 2/6] kernfs: don't copy times and link count into the inode on refresh Shakeel Butt
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Shakeel Butt @ 2026-11-18 22:01 UTC (permalink / raw)
To: Christian Brauner
Cc: Greg Kroah-Hartman, Tejun Heo, Jori Koolstra, Ian Kent,
T . J . Mercier, Meta kernel team, linux-fsdevel, driver-core,
linux-kernel
getattr and permission copy a node's attributes into its inode while
holding kernfs_iattr_rwsem for reading, so several of them can write
the same inode at once. That is safe only because every update of
these attributes takes the lock for writing, so they all write the same
values. Commit aa1d058d48f2 ("kernfs: dont take i_lock on inode attr
read") relies on this. Setting up a node's first iattrs from setxattr
does not take the lock, but a refresh that misses them writes no times.
The next patches let a directory's times and link count change without
that lock. A slow refresh could then write an old value after a newer
one, and stat() could return a stale or torn time or link count.
Make getattr report the times and link count from the node instead of
the inode. Permission checks use neither. Nothing changes yet, except
that a stat() racing a node's first setxattr can report the new times.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
fs/kernfs/inode.c | 32 ++++++++++++++++++++++++++++++--
1 file changed, 30 insertions(+), 2 deletions(-)
diff --git a/fs/kernfs/inode.c b/fs/kernfs/inode.c
index 53b1a2796e0b..3616161a56cb 100644
--- a/fs/kernfs/inode.c
+++ b/fs/kernfs/inode.c
@@ -162,9 +162,22 @@ static inline void set_inode_attr(struct inode *inode,
inode_set_ctime_to_ts(inode, attrs->ia_ctime);
}
+/*
+ * Get the link count of a directory: its subdirectories plus two.
+ * Return false for other nodes and for a directory being removed.
+ */
+static bool kernfs_dir_nlink(struct kernfs_node *kn, unsigned int *nlink)
+{
+ if (kernfs_type(kn) != KERNFS_DIR || (kn->flags & KERNFS_REMOVING))
+ return false;
+ *nlink = kn->dir.subdirs + 2;
+ return true;
+}
+
static void kernfs_refresh_inode(struct kernfs_node *kn, struct inode *inode)
{
struct kernfs_iattrs *attrs;
+ unsigned int nlink;
inode->i_mode = kn->mode;
attrs = kernfs_iattrs_noalloc(kn);
@@ -175,8 +188,8 @@ static void kernfs_refresh_inode(struct kernfs_node *kn, struct inode *inode)
*/
set_inode_attr(inode, attrs);
- if (kernfs_type(kn) == KERNFS_DIR && !(kn->flags & KERNFS_REMOVING))
- set_nlink(inode, kn->dir.subdirs + 2);
+ if (kernfs_dir_nlink(kn, &nlink))
+ set_nlink(inode, nlink);
}
int kernfs_iop_getattr(const struct mnt_idmap *idmap,
@@ -186,10 +199,25 @@ int kernfs_iop_getattr(const struct mnt_idmap *idmap,
struct inode *inode = d_inode(path->dentry);
struct kernfs_node *kn = inode->i_private;
struct kernfs_root *root = kernfs_root(kn);
+ struct kernfs_iattrs *attrs;
+ unsigned int nlink;
down_read(&root->kernfs_iattr_rwsem);
kernfs_refresh_inode(kn, inode);
generic_fillattr(&nop_mnt_idmap, request_mask, inode, stat);
+
+ /*
+ * Other refreshes may store older times or link counts in the
+ * inode at the same time. Report the values from the node.
+ */
+ attrs = kernfs_iattrs_noalloc(kn);
+ if (attrs) {
+ stat->atime = attrs->ia_atime;
+ stat->mtime = attrs->ia_mtime;
+ stat->ctime = attrs->ia_ctime;
+ }
+ if (kernfs_dir_nlink(kn, &nlink))
+ stat->nlink = nlink;
up_read(&root->kernfs_iattr_rwsem);
return 0;
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 2/6] kernfs: don't copy times and link count into the inode on refresh
2026-11-18 22:01 [PATCH 0/6] kernfs: stop taking kernfs_iattr_rwsem for writing in mkdir/rmdir Shakeel Butt
2026-11-18 22:01 ` [PATCH 1/6] kernfs: report times and link count from the node in getattr Shakeel Butt
@ 2026-11-18 22:01 ` Shakeel Butt
2026-11-18 22:01 ` [PATCH 3/6] kernfs: don't take kernfs_iattr_rwsem for the directory counters Shakeel Butt
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Shakeel Butt @ 2026-11-18 22:01 UTC (permalink / raw)
To: Christian Brauner
Cc: Greg Kroah-Hartman, Tejun Heo, Jori Koolstra, Ian Kent,
T . J . Mercier, Meta kernel team, linux-fsdevel, driver-core,
linux-kernel
Every permission check and getattr copies a node's times, if it has
iattrs, and a directory's link count into the shared inode. Since the
previous patch, getattr reports both from the node, and permission
checks use only the mode, owner and group. The only other users of a
kernfs inode's times are the mmap write fault and the atime update,
which just update the inode's copy. The inode's link count matters
only to the IN_DELETE_SELF check, which needs 0 once __kernfs_remove()
clears it.
Copy only the mode, owner and group, which also takes work off every
permission check. Set a directory's link count when its inode is set
up. Every caller that sets up a directory inode holds kernfs_rwsem,
and __kernfs_remove() holds it for writing while it clears the count,
so the two cannot race. Assert that, and note in the kernfs_get_inode()
comment that it is not for directories; its one user outside kernfs,
cgroup_may_write(), gets a file.
getattr fell back to the inode's link count for a directory being
removed. That count is no longer kept up to date, so report 0 instead:
KERNFS_REMOVING is never cleared, and lookups already fail.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
fs/kernfs/inode.c | 32 ++++++++++++++++++--------------
1 file changed, 18 insertions(+), 14 deletions(-)
diff --git a/fs/kernfs/inode.c b/fs/kernfs/inode.c
index 3616161a56cb..2d8da8b4e7da 100644
--- a/fs/kernfs/inode.c
+++ b/fs/kernfs/inode.c
@@ -157,27 +157,23 @@ static inline void set_inode_attr(struct inode *inode,
{
inode->i_uid = attrs->ia_uid;
inode->i_gid = attrs->ia_gid;
- inode_set_atime_to_ts(inode, attrs->ia_atime);
- inode_set_mtime_to_ts(inode, attrs->ia_mtime);
- inode_set_ctime_to_ts(inode, attrs->ia_ctime);
}
/*
- * Get the link count of a directory: its subdirectories plus two.
- * Return false for other nodes and for a directory being removed.
+ * Get a directory's link count: its subdirectories plus two, or 0 once
+ * its removal has started. Return false for other nodes.
*/
static bool kernfs_dir_nlink(struct kernfs_node *kn, unsigned int *nlink)
{
- if (kernfs_type(kn) != KERNFS_DIR || (kn->flags & KERNFS_REMOVING))
+ if (kernfs_type(kn) != KERNFS_DIR)
return false;
- *nlink = kn->dir.subdirs + 2;
+ *nlink = (kn->flags & KERNFS_REMOVING) ? 0 : kn->dir.subdirs + 2;
return true;
}
static void kernfs_refresh_inode(struct kernfs_node *kn, struct inode *inode)
{
struct kernfs_iattrs *attrs;
- unsigned int nlink;
inode->i_mode = kn->mode;
attrs = kernfs_iattrs_noalloc(kn);
@@ -187,9 +183,6 @@ static void kernfs_refresh_inode(struct kernfs_node *kn, struct inode *inode)
* persistent copy in kernfs_node.
*/
set_inode_attr(inode, attrs);
-
- if (kernfs_dir_nlink(kn, &nlink))
- set_nlink(inode, nlink);
}
int kernfs_iop_getattr(const struct mnt_idmap *idmap,
@@ -207,8 +200,8 @@ int kernfs_iop_getattr(const struct mnt_idmap *idmap,
generic_fillattr(&nop_mnt_idmap, request_mask, inode, stat);
/*
- * Other refreshes may store older times or link counts in the
- * inode at the same time. Report the values from the node.
+ * The inode's times and link count are not kept up to date.
+ * Report them from the node.
*/
attrs = kernfs_iattrs_noalloc(kn);
if (attrs) {
@@ -225,6 +218,8 @@ int kernfs_iop_getattr(const struct mnt_idmap *idmap,
static void kernfs_init_inode(struct kernfs_node *kn, struct inode *inode)
{
+ unsigned int nlink;
+
kernfs_get(kn);
inode->i_private = kn;
inode->i_mapping->a_ops = &ram_aops;
@@ -238,6 +233,14 @@ static void kernfs_init_inode(struct kernfs_node *kn, struct inode *inode)
/* initialize inode according to type */
switch (kernfs_type(kn)) {
case KERNFS_DIR:
+ /*
+ * The link count is set only here; getattr reports it from
+ * the node. Holding kernfs_rwsem keeps this from racing
+ * with __kernfs_remove() clearing it.
+ */
+ lockdep_assert_held(&kernfs_root(kn)->kernfs_rwsem);
+ if (kernfs_dir_nlink(kn, &nlink))
+ set_nlink(inode, nlink);
inode->i_op = &kernfs_dir_iops;
inode->i_fop = &kernfs_dir_fops;
if (kn->flags & KERNFS_EMPTY_DIR)
@@ -267,7 +270,8 @@ static void kernfs_init_inode(struct kernfs_node *kn, struct inode *inode)
* locked.
*
* Locking:
- * Kernel thread context (may sleep).
+ * Kernel thread context (may sleep). Not for directories: kernfs
+ * sets up their inodes itself, with kernfs_rwsem held.
*
* Return:
* Pointer to allocated inode on success, %NULL on failure.
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 3/6] kernfs: don't take kernfs_iattr_rwsem for the directory counters
2026-11-18 22:01 [PATCH 0/6] kernfs: stop taking kernfs_iattr_rwsem for writing in mkdir/rmdir Shakeel Butt
2026-11-18 22:01 ` [PATCH 1/6] kernfs: report times and link count from the node in getattr Shakeel Butt
2026-11-18 22:01 ` [PATCH 2/6] kernfs: don't copy times and link count into the inode on refresh Shakeel Butt
@ 2026-11-18 22:01 ` Shakeel Butt
2026-11-18 22:01 ` [PATCH 4/6] kernfs: don't take kernfs_iattr_rwsem to publish KERNFS_REMOVING Shakeel Butt
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Shakeel Butt @ 2026-11-18 22:01 UTC (permalink / raw)
To: Christian Brauner
Cc: Greg Kroah-Hartman, Tejun Heo, Jori Koolstra, Ian Kent,
T . J . Mercier, Meta kernel team, linux-fsdevel, driver-core,
linux-kernel
kernfs_link_sibling() and kernfs_unlink_sibling() take
kernfs_iattr_rwsem for writing to update the parent's dir.subdirs and
dir.rev. Every permission check and getattr takes the lock for
reading, so under load the writer waits, while it holds kernfs_rwsem
and, for cgroups, cgroup_mutex.
Both writers hold kernfs_rwsem for writing, which is enough. Commit
0559f63057f9 ("kernfs: fix missing kernfs_iattr_rwsem locking") added
the lock to keep the link count stable for getattr, which now reads it
from the node; the refresh no longer copies it. The other readers,
kernfs_dir_changed(), kernfs_set_rev(), kobject_has_children() and
resctrl, never took the lock. A racing stat() sees the old or the new
link count. While a subdirectory is renamed within the directory, it
can also see one less, as it could before.
Drop the lock, mark the accesses, and assert that kernfs_rwsem is held.
kernfs_iattr_rwsem write acquisitions (lock_stat), before -> after:
5 dummy netdevs created and removed 1850 -> 1110
100 cgroup mkdirs and 100 rmdirs 7500 -> 4500
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
fs/kernfs/dir.c | 12 ++++++------
fs/kernfs/inode.c | 3 ++-
2 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/fs/kernfs/dir.c b/fs/kernfs/dir.c
index fb1dcf35d900..ef23eac684c6 100644
--- a/fs/kernfs/dir.c
+++ b/fs/kernfs/dir.c
@@ -387,6 +387,8 @@ static int kernfs_link_sibling(struct kernfs_node *kn)
struct kernfs_node *kn_parent;
struct rb_node **node;
+ lockdep_assert_held_write(&kernfs_root(kn)->kernfs_rwsem);
+
kn_parent = kernfs_parent(kn);
node = &kn_parent->dir.children.rb_node;
@@ -410,11 +412,9 @@ static int kernfs_link_sibling(struct kernfs_node *kn)
rb_insert_color(&kn->rb, &kn_parent->dir.children);
/* successfully added, account subdir number */
- down_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
if (kernfs_type(kn) == KERNFS_DIR)
- kn_parent->dir.subdirs++;
+ WRITE_ONCE(kn_parent->dir.subdirs, kn_parent->dir.subdirs + 1);
kernfs_inc_rev(kn_parent);
- up_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
return 0;
}
@@ -436,15 +436,15 @@ static bool kernfs_unlink_sibling(struct kernfs_node *kn)
{
struct kernfs_node *kn_parent;
+ lockdep_assert_held_write(&kernfs_root(kn)->kernfs_rwsem);
+
if (RB_EMPTY_NODE(&kn->rb))
return false;
kn_parent = kernfs_parent(kn);
- down_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
if (kernfs_type(kn) == KERNFS_DIR)
- kn_parent->dir.subdirs--;
+ WRITE_ONCE(kn_parent->dir.subdirs, kn_parent->dir.subdirs - 1);
kernfs_inc_rev(kn_parent);
- up_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
rb_erase(&kn->rb, &kn_parent->dir.children);
RB_CLEAR_NODE(&kn->rb);
diff --git a/fs/kernfs/inode.c b/fs/kernfs/inode.c
index 2d8da8b4e7da..2f463c0354a5 100644
--- a/fs/kernfs/inode.c
+++ b/fs/kernfs/inode.c
@@ -167,7 +167,8 @@ static bool kernfs_dir_nlink(struct kernfs_node *kn, unsigned int *nlink)
{
if (kernfs_type(kn) != KERNFS_DIR)
return false;
- *nlink = (kn->flags & KERNFS_REMOVING) ? 0 : kn->dir.subdirs + 2;
+ *nlink = (kn->flags & KERNFS_REMOVING) ? 0 :
+ READ_ONCE(kn->dir.subdirs) + 2;
return true;
}
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 4/6] kernfs: don't take kernfs_iattr_rwsem to publish KERNFS_REMOVING
2026-11-18 22:01 [PATCH 0/6] kernfs: stop taking kernfs_iattr_rwsem for writing in mkdir/rmdir Shakeel Butt
` (2 preceding siblings ...)
2026-11-18 22:01 ` [PATCH 3/6] kernfs: don't take kernfs_iattr_rwsem for the directory counters Shakeel Butt
@ 2026-11-18 22:01 ` Shakeel Butt
2026-11-18 22:01 ` [PATCH 5/6] kernfs: update a directory's times without kernfs_iattr_rwsem Shakeel Butt
2026-11-18 22:01 ` [PATCH 6/6] kernfs: don't take kernfs_iattr_rwsem in __kernfs_remove() Shakeel Butt
5 siblings, 0 replies; 7+ messages in thread
From: Shakeel Butt @ 2026-11-18 22:01 UTC (permalink / raw)
To: Christian Brauner
Cc: Greg Kroah-Hartman, Tejun Heo, Jori Koolstra, Ian Kent,
T . J . Mercier, Meta kernel team, linux-fsdevel, driver-core,
linux-kernel
__kernfs_remove() takes kernfs_iattr_rwsem for writing while it marks
a subtree KERNFS_REMOVING, inside kernfs_rwsem. Commit 507d8ce13f5b
("kernfs: Don't set_nlink for directories being removed") added it so
that a refresh would not set the i_nlink of a directory whose removal
has started. The refresh no longer sets i_nlink, so the lock is not
needed.
The flag is only set here, under kernfs_rwsem. Inode setup reads it
under kernfs_rwsem. getattr reads it without and reports the old link
count or 0, either of which is right for a stat() racing the removal.
Drop the lock, mark the store and the load, and read the flags once in
kernfs_dir_nlink(), as the type bits share the word.
kernfs_iattr_rwsem write acquisitions (lock_stat), before -> after:
5 dummy netdevs created and removed 1110 -> 740
100 cgroup mkdirs and 100 rmdirs 4500 -> 3000
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
fs/kernfs/dir.c | 10 ++++++----
fs/kernfs/inode.c | 7 +++++--
2 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/fs/kernfs/dir.c b/fs/kernfs/dir.c
index ef23eac684c6..890aca5d8d70 100644
--- a/fs/kernfs/dir.c
+++ b/fs/kernfs/dir.c
@@ -1604,15 +1604,17 @@ static void __kernfs_remove(struct kernfs_node *kn)
pr_debug("kernfs %s: removing\n", kernfs_rcu_name(kn));
- /* prevent new usage by marking all nodes removing and deactivating */
- down_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
+ /*
+ * Prevent new usage by marking all nodes removing and deactivating.
+ * KERNFS_REMOVING is only set here, under kernfs_rwsem.
+ * kernfs_dir_nlink() reads it without that lock.
+ */
pos = NULL;
while ((pos = kernfs_next_descendant_post(pos, kn))) {
- pos->flags |= KERNFS_REMOVING;
+ WRITE_ONCE(pos->flags, pos->flags | KERNFS_REMOVING);
if (kernfs_active(pos))
atomic_add(KN_DEACTIVATED_BIAS, &pos->active);
}
- up_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
/* deactivate and unlink the subtree node-by-node */
do {
diff --git a/fs/kernfs/inode.c b/fs/kernfs/inode.c
index 2f463c0354a5..ca234f69882b 100644
--- a/fs/kernfs/inode.c
+++ b/fs/kernfs/inode.c
@@ -165,9 +165,12 @@ static inline void set_inode_attr(struct inode *inode,
*/
static bool kernfs_dir_nlink(struct kernfs_node *kn, unsigned int *nlink)
{
- if (kernfs_type(kn) != KERNFS_DIR)
+ /* KERNFS_REMOVING can be set concurrently; read the flags once. */
+ unsigned short flags = READ_ONCE(kn->flags);
+
+ if ((flags & KERNFS_TYPE_MASK) != KERNFS_DIR)
return false;
- *nlink = (kn->flags & KERNFS_REMOVING) ? 0 :
+ *nlink = (flags & KERNFS_REMOVING) ? 0 :
READ_ONCE(kn->dir.subdirs) + 2;
return true;
}
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 5/6] kernfs: update a directory's times without kernfs_iattr_rwsem
2026-11-18 22:01 [PATCH 0/6] kernfs: stop taking kernfs_iattr_rwsem for writing in mkdir/rmdir Shakeel Butt
` (3 preceding siblings ...)
2026-11-18 22:01 ` [PATCH 4/6] kernfs: don't take kernfs_iattr_rwsem to publish KERNFS_REMOVING Shakeel Butt
@ 2026-11-18 22:01 ` Shakeel Butt
2026-11-18 22:01 ` [PATCH 6/6] kernfs: don't take kernfs_iattr_rwsem in __kernfs_remove() Shakeel Butt
5 siblings, 0 replies; 7+ messages in thread
From: Shakeel Butt @ 2026-11-18 22:01 UTC (permalink / raw)
To: Christian Brauner
Cc: Greg Kroah-Hartman, Tejun Heo, Jori Koolstra, Ian Kent,
T . J . Mercier, Meta kernel team, linux-fsdevel, driver-core,
linux-kernel
kernfs_add_one() and __kernfs_remove() update the parent's ctime and
mtime under kernfs_iattr_rwsem for writing, inside kernfs_rwsem. This
is where cgroup mkdirs block on Meta fleet: the writer waits for every
path walk's permission check while it holds kernfs_rwsem and
cgroup_mutex.
The lock keeps the times stable for getattr, now their only reader, and
a timespec64 is too wide for a single store. Give the three times a
seqlock instead. setattr takes it in addition to kernfs_iattr_rwsem,
adding or removing an entry takes only the seqlock, and getattr retries
if a write was in progress. Adding or removing an entry reads the clock
with the seqlock held, so updates store their times in the order they
happen. A racing stat() can see the new link count with the old times,
or the reverse.
A directory without iattrs still skips the update. So that an update
racing with setting up the iattrs is not lost, take the seqlock of new
iattrs before publishing them and set their times after. An update
that found no iattrs ran before the times were set, and a writer that
finds the new iattrs waits for them. mkdir no longer takes
kernfs_iattr_rwsem for writing.
kernfs_iattr_rwsem write acquisitions (lock_stat), before -> after:
5 dummy netdevs created and removed 740 -> 370
100 cgroup mkdirs and 100 rmdirs 3000 -> 1500
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
fs/kernfs/dir.c | 26 ++----------
fs/kernfs/inode.c | 85 ++++++++++++++++++++++++++++++-------
fs/kernfs/kernfs-internal.h | 2 +
3 files changed, 75 insertions(+), 38 deletions(-)
diff --git a/fs/kernfs/dir.c b/fs/kernfs/dir.c
index 890aca5d8d70..8686219b826f 100644
--- a/fs/kernfs/dir.c
+++ b/fs/kernfs/dir.c
@@ -825,7 +825,6 @@ struct kernfs_node *kernfs_find_and_get_node_by_id(struct kernfs_root *root,
int kernfs_add_one(struct kernfs_node *kn)
{
struct kernfs_root *root = kernfs_root(kn);
- struct kernfs_iattrs *ps_iattr;
struct kernfs_node *parent;
bool has_ns;
int ret;
@@ -853,16 +852,7 @@ int kernfs_add_one(struct kernfs_node *kn)
if (ret)
goto out_unlock;
- /* Update timestamps on the parent */
- down_write(&root->kernfs_iattr_rwsem);
-
- ps_iattr = parent->iattr;
- if (ps_iattr) {
- ktime_get_real_ts64(&ps_iattr->ia_ctime);
- ps_iattr->ia_mtime = ps_iattr->ia_ctime;
- }
-
- up_write(&root->kernfs_iattr_rwsem);
+ kernfs_dir_update_times(parent);
/*
* Activate the new node unless CREATE_DEACTIVATED is requested.
@@ -1635,20 +1625,12 @@ static void __kernfs_remove(struct kernfs_node *kn)
* to decide who's responsible for cleanups.
*/
if (!parent || kernfs_unlink_sibling(pos)) {
- struct kernfs_iattrs *ps_iattr =
- parent ? parent->iattr : NULL;
-
down_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
-
kernfs_clear_inode_nlink(pos);
-
- /* update timestamps on the parent */
- if (ps_iattr) {
- ktime_get_real_ts64(&ps_iattr->ia_ctime);
- ps_iattr->ia_mtime = ps_iattr->ia_ctime;
- }
-
up_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
+
+ if (parent)
+ kernfs_dir_update_times(parent);
kernfs_put(pos);
}
diff --git a/fs/kernfs/inode.c b/fs/kernfs/inode.c
index ca234f69882b..580095e0fad9 100644
--- a/fs/kernfs/inode.c
+++ b/fs/kernfs/inode.c
@@ -24,6 +24,22 @@ static const struct inode_operations kernfs_iops = {
.listxattr = kernfs_iop_listxattr,
};
+/* Copy the times of @attrs. Writers hold ia_time_lock. */
+static void kernfs_iattrs_get_times(struct kernfs_iattrs *attrs,
+ struct timespec64 *atime,
+ struct timespec64 *mtime,
+ struct timespec64 *ctime)
+{
+ unsigned int seq;
+
+ do {
+ seq = read_seqbegin(&attrs->ia_time_lock);
+ *atime = attrs->ia_atime;
+ *mtime = attrs->ia_mtime;
+ *ctime = attrs->ia_ctime;
+ } while (read_seqretry(&attrs->ia_time_lock, seq));
+}
+
static struct kernfs_iattrs *__kernfs_iattrs(struct kernfs_node *kn, bool alloc)
{
struct kernfs_iattrs *ret __free(kfree) = NULL;
@@ -41,16 +57,28 @@ static struct kernfs_iattrs *__kernfs_iattrs(struct kernfs_node *kn, bool alloc)
/* assign default attributes */
ret->ia_uid = GLOBAL_ROOT_UID;
ret->ia_gid = GLOBAL_ROOT_GID;
-
- ktime_get_real_ts64(&ret->ia_atime);
- ret->ia_mtime = ret->ia_atime;
- ret->ia_ctime = ret->ia_atime;
+ seqlock_init(&ret->ia_time_lock);
simple_xattr_limits_init(&ret->xattr_limits);
+ /*
+ * Publish with ia_time_lock held and set the times after. An entry
+ * added or removed while @kn had no iattrs is then covered, and a
+ * writer that finds the new iattrs waits instead of being
+ * overwritten.
+ */
+ write_seqlock(&ret->ia_time_lock);
+
/* If someone raced us, recognize it. */
- if (!try_cmpxchg(&kn->iattr, &attr, ret))
+ if (!try_cmpxchg(&kn->iattr, &attr, ret)) {
+ write_sequnlock(&ret->ia_time_lock);
return READ_ONCE(kn->iattr);
+ }
+
+ ktime_get_real_ts64(&ret->ia_atime);
+ ret->ia_mtime = ret->ia_atime;
+ ret->ia_ctime = ret->ia_atime;
+ write_sequnlock(&ret->ia_time_lock);
return no_free_ptr(ret);
}
@@ -78,12 +106,16 @@ int __kernfs_setattr(struct kernfs_node *kn, const struct iattr *iattr)
attrs->ia_uid = iattr->ia_uid;
if (ia_valid & ATTR_GID)
attrs->ia_gid = iattr->ia_gid;
- if (ia_valid & ATTR_ATIME)
- attrs->ia_atime = iattr->ia_atime;
- if (ia_valid & ATTR_MTIME)
- attrs->ia_mtime = iattr->ia_mtime;
- if (ia_valid & ATTR_CTIME)
- attrs->ia_ctime = iattr->ia_ctime;
+ if (ia_valid & (ATTR_ATIME | ATTR_MTIME | ATTR_CTIME)) {
+ write_seqlock(&attrs->ia_time_lock);
+ if (ia_valid & ATTR_ATIME)
+ attrs->ia_atime = iattr->ia_atime;
+ if (ia_valid & ATTR_MTIME)
+ attrs->ia_mtime = iattr->ia_mtime;
+ if (ia_valid & ATTR_CTIME)
+ attrs->ia_ctime = iattr->ia_ctime;
+ write_sequnlock(&attrs->ia_time_lock);
+ }
if (ia_valid & ATTR_MODE)
kn->mode = iattr->ia_mode;
return 0;
@@ -159,6 +191,29 @@ static inline void set_inode_attr(struct inode *inode,
inode->i_gid = attrs->ia_gid;
}
+/**
+ * kernfs_dir_update_times - set a directory's ctime and mtime to now
+ * @kn: the directory, kept alive by the caller
+ *
+ * Call after adding or removing an entry, with kernfs_rwsem held for
+ * writing. Does nothing if @kn has no iattrs yet.
+ */
+void kernfs_dir_update_times(struct kernfs_node *kn)
+{
+ struct kernfs_iattrs *attrs = kernfs_iattrs_noalloc(kn);
+
+ lockdep_assert_held_write(&kernfs_root(kn)->kernfs_rwsem);
+
+ if (!attrs)
+ return;
+
+ /* Read the clock under the lock, so updates store times in order. */
+ write_seqlock(&attrs->ia_time_lock);
+ ktime_get_real_ts64(&attrs->ia_ctime);
+ attrs->ia_mtime = attrs->ia_ctime;
+ write_sequnlock(&attrs->ia_time_lock);
+}
+
/*
* Get a directory's link count: its subdirectories plus two, or 0 once
* its removal has started. Return false for other nodes.
@@ -208,11 +263,9 @@ int kernfs_iop_getattr(const struct mnt_idmap *idmap,
* Report them from the node.
*/
attrs = kernfs_iattrs_noalloc(kn);
- if (attrs) {
- stat->atime = attrs->ia_atime;
- stat->mtime = attrs->ia_mtime;
- stat->ctime = attrs->ia_ctime;
- }
+ if (attrs)
+ kernfs_iattrs_get_times(attrs, &stat->atime, &stat->mtime,
+ &stat->ctime);
if (kernfs_dir_nlink(kn, &nlink))
stat->nlink = nlink;
up_read(&root->kernfs_iattr_rwsem);
diff --git a/fs/kernfs/kernfs-internal.h b/fs/kernfs/kernfs-internal.h
index 2e364b626c1e..9bba1fc8acfd 100644
--- a/fs/kernfs/kernfs-internal.h
+++ b/fs/kernfs/kernfs-internal.h
@@ -25,6 +25,7 @@ struct kernfs_iattrs {
struct timespec64 ia_atime;
struct timespec64 ia_mtime;
struct timespec64 ia_ctime;
+ seqlock_t ia_time_lock; /* protects the times above */
struct list_head xattrs;
struct simple_xattr_limits xattr_limits;
@@ -188,6 +189,7 @@ int kernfs_iop_getattr(const struct mnt_idmap *idmap,
u32 request_mask, unsigned int query_flags);
ssize_t kernfs_iop_listxattr(struct dentry *dentry, char *buf, size_t size);
int __kernfs_setattr(struct kernfs_node *kn, const struct iattr *iattr);
+void kernfs_dir_update_times(struct kernfs_node *kn);
/*
* dir.c
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 6/6] kernfs: don't take kernfs_iattr_rwsem in __kernfs_remove()
2026-11-18 22:01 [PATCH 0/6] kernfs: stop taking kernfs_iattr_rwsem for writing in mkdir/rmdir Shakeel Butt
` (4 preceding siblings ...)
2026-11-18 22:01 ` [PATCH 5/6] kernfs: update a directory's times without kernfs_iattr_rwsem Shakeel Butt
@ 2026-11-18 22:01 ` Shakeel Butt
5 siblings, 0 replies; 7+ messages in thread
From: Shakeel Butt @ 2026-11-18 22:01 UTC (permalink / raw)
To: Christian Brauner
Cc: Greg Kroah-Hartman, Tejun Heo, Jori Koolstra, Ian Kent,
T . J . Mercier, Meta kernel team, linux-fsdevel, driver-core,
linux-kernel
__kernfs_remove() takes kernfs_iattr_rwsem for writing around
kernfs_clear_inode_nlink() for every removed node. A cgroup rmdir
removes the directory and all its files, so it takes the lock once per
file, each time behind the readers. Our lock monitor saw rmdirs wait
in __kernfs_remove() for several seconds.
Commit eea5d2bb34ba ("kernfs: Send IN_DELETE_SELF and IN_IGNORED")
clears i_nlink inside this section, so a refresh in progress could not
set a removed directory's i_nlink again, the race described in commit
507d8ce13f5b ("kernfs: Don't set_nlink for directories being removed").
The refresh no longer sets i_nlink. Only inode setup does, under
kernfs_rwsem, which __kernfs_remove() holds for writing.
A racing fstat() of a file can read its i_nlink while it is cleared,
and sees 1 or 0, as on other filesystems. A directory reports its link
count from the node.
Drop the lock. rmdir no longer takes kernfs_iattr_rwsem for writing.
kernfs_iattr_rwsem write acquisitions (lock_stat), before -> after:
5 dummy netdevs created and removed 370 -> 0
100 cgroup mkdirs and 100 rmdirs 1500 -> 0
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
---
fs/kernfs/dir.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/fs/kernfs/dir.c b/fs/kernfs/dir.c
index 8686219b826f..c1402a461405 100644
--- a/fs/kernfs/dir.c
+++ b/fs/kernfs/dir.c
@@ -1625,9 +1625,7 @@ static void __kernfs_remove(struct kernfs_node *kn)
* to decide who's responsible for cleanups.
*/
if (!parent || kernfs_unlink_sibling(pos)) {
- down_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
kernfs_clear_inode_nlink(pos);
- up_write(&kernfs_root(kn)->kernfs_iattr_rwsem);
if (parent)
kernfs_dir_update_times(parent);
--
2.53.1-Meta
^ permalink raw reply related [flat|nested] 7+ messages in thread