[PATCH 0/3] kernfs, driver core: Cut per-node lock traffic in bulk registration

[PATCH 0/3] kernfs, driver core: Cut per-node lock traffic in bulk registration

[Pavol Sakac]

Registering VFs in parallel, introduced in [1], runs into three lock
costs in kernfs and the driver core. Two scale with the number of
sysfs nodes created: the kernfs root rwsem is write-taken twice per
node, and every inode ID is allocated under kernfs_idr_lock. The third
scales with the number of devices: get_device_parent()'s glue
directory lookup walks a flat list under the global gdp_mutex.

The glue directory is how the driver core groups class devices under a
parent that is not itself a class device: when vfio-pci binds, each VF
gets a "vfio-dev" glue directory holding its vfio-dev class device.
get_device_parent() finds or creates it by walking the class's flat
glue_dirs list under gdp_mutex, so with thousands of VFs each
registration walks thousands of entries - quadratic work, serialized
on one global mutex.

Patch 1 (kernfs) activates a new node inside the same kernfs_rwsem
write hold that links it: one write acquisition per node instead of
two. Patch 2 (driver core) indexes glue directories by parent kobject
in an rbtree, replacing the linear walk; gdp_mutex still serializes as
before. Patch 3 (kernfs) pre-allocates inode IDs in batches of 16,
caches them per CPU, and installs the node pointer with an RCU store
via idr_replace(), cutting kernfs_idr_lock acquisitions 16x.

Lock statistics and SR-IOV init time for 4x PF (NVMe, 255 VFs each), on
the reproducer from the parallel VF initialization cover letter [1]:

  lock_stat:
  Lock                     wait: Before     After   contentions: Before   After
  iommu_probe_device_lock      10471 ms   9154 ms                   841     783
  &vfio.group_lock              3614 ms   3823 ms                   736     730
  &root->kernfs_rwsem           1834 ms   1285 ms                116316   55459
  &root->kernfs_idr_lock           5 ms      0 ms                  4580      77

  iommu_probe_device_lock and vfio.group_lock are shown for scale;
  neither is touched by this series - they are addressed by the
  IOMMU [2] and VFIO [3] series posted separately.

  kernfs_rwsem write acquisitions  168302 (2/node) -> 84661 (1/node)
  kernfs_idr_lock acquisitions     83641 -> 5229 (batch 16)
  gdp_mutex avg hold               18 -> 7 us (2040 acq. both arms)

  Stage                 SR-IOV init time:
  S0 (baseline)         3027 ms
  S1                     999 ms
  S2                     995 ms
  S3 (this series)       991 ms

Reproducer disclaimer:
I lean primarily on lock_stat numbers to defend the improvements. In
the reproducer, the residual iommu_probe_device_lock dominates the
window and masks the later series' wall-time gains; reducing that lock
further is out of scope for this set. On real hardware the five series
together cut SR-IOV initialization by 65%, more in [1].

The staged sysfs RFC [4] builds on top of these changes (mainly the
IDR batching).

This series adds a KUnit suite. The lock_stat and timing figures come
from the public reproducer. The full series has also been tested on
current datacenter server hardware with thousands of VFs.


[1] https://lore.kernel.org/r/20260911-vfopt-s1-v1-0-693271dc0226@amazon.de
[2] https://lore.kernel.org/r/20260911-vfopt-s2-v1-0-fff3db7e01c2@amazon.de
[3] https://lore.kernel.org/r/20260911-vfopt-s4-v1-0-98ba1d2ef7ab@amazon.de
[4] https://lore.kernel.org/r/20260911-vfopt-s5-v1-0-fa4cacdb6ca8@amazon.de

Pavol Sakac (3):
  kernfs: activate nodes while linking them
  driver core: Index class glue directories by parent kobject
  kernfs: batch inode ID allocation per CPU

 drivers/base/base.h                  |   4 +
 drivers/base/core.c                  | 112 +++++--
 drivers/base/test/.kunitconfig       |   1 +
 drivers/base/test/Kconfig            |  12 +
 drivers/base/test/Makefile           |   2 +
 drivers/base/test/glue-dir-test.c    | 466 +++++++++++++++++++++++++++
 drivers/base/test/root-device-test.c | 215 ++++++++++++
 fs/kernfs/dir.c                      | 139 +++++++-
 fs/kernfs/kernfs-internal.h          |   3 +
 fs/sysfs/mount.c                     |   3 +-
 include/linux/kernfs.h               |  13 +
 11 files changed, 930 insertions(+), 40 deletions(-)
 create mode 100644 drivers/base/test/glue-dir-test.c


base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
-- 
2.47.3

[PATCH 1/3] kernfs: activate nodes while linking them

[Pavol Sakac]

kernfs_add_one() links a new node, releases kernfs_rwsem, and takes it
again through kernfs_activate(). A new node has no descendants, so the
second hold only activates that node.

Activate ordinary nodes before releasing the linking hold, removing a
second write-side acquisition and the linked-but-inactive interval.
KERNFS_ROOT_CREATE_DEACTIVATED roots retain explicit subtree activation.

Removing the unlock-to-lock pair also removes its publication ordering.
The in-tree lockless ID lookups use CREATE_DEACTIVATED roots and retain
kernfs_activate(); unsynchronized callers cannot rely on observing a
node.

Assisted-by: LLM
Signed-off-by: Pavol Sakac <sakacpav@amazon.de>
---
 fs/kernfs/dir.c | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/fs/kernfs/dir.c b/fs/kernfs/dir.c
index 82bbaeb326aa..d68bce0b0b41 100644
--- a/fs/kernfs/dir.c
+++ b/fs/kernfs/dir.c
@@ -41,6 +41,8 @@ static bool kernfs_active(struct kernfs_node *kn)
 	return __kernfs_active(kn);
 }
 
+static void kernfs_activate_one(struct kernfs_node *kn);
+
 static bool kernfs_lockdep(struct kernfs_node *kn)
 {
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
@@ -855,17 +857,16 @@ int kernfs_add_one(struct kernfs_node *kn)
 	}
 
 	up_write(&root->kernfs_iattr_rwsem);
-	up_write(&root->kernfs_rwsem);
 
 	/*
-	 * Activate the new node unless CREATE_DEACTIVATED is requested.
-	 * If not activated here, the kernfs user is responsible for
-	 * activating the node with kernfs_activate().  A node which hasn't
-	 * been activated is not visible to userland and its removal won't
-	 * trigger deactivation.
+	 * A freshly linked node has no descendants, so activating just @kn
+	 * under the linking hold is equivalent to the kernfs_activate()
+	 * walk; CREATE_DEACTIVATED roots keep deferred activation.
 	 */
-	if (!(kernfs_root(kn)->flags & KERNFS_ROOT_CREATE_DEACTIVATED))
-		kernfs_activate(kn);
+	if (!(root->flags & KERNFS_ROOT_CREATE_DEACTIVATED))
+		kernfs_activate_one(kn);
+
+	up_write(&root->kernfs_rwsem);
 	return 0;
 
 out_unlock:
-- 
2.47.3

[PATCH 2/3] driver core: Index class glue directories by parent kobject

[Pavol Sakac]

get_device_parent() finds a parent's glue directory by walking the
class's glue_dirs kset list under gdp_mutex, so one per parent, as
vfio-dev needs per SR-IOV VF, is quadratic.

Index them by parent kobject in an rbtree embedded in the class's
subsys_private, which dies with the kset list it indexes, so no
per-entry class check is needed: two classes below one parent are told
apart by tree selection. The key is a kobject because a parentless
class device hangs off the shared "virtual" kobject, referenced by the
glue dir while indexed. The rb_node lives in struct class_dir, adding
no allocation and no failure mode.

gdp_mutex serializes the index, as it has glue dir lookup/create/remove
since commit 77d3d7c1d561f
("driver-core: fix race condition in get_device_parent()") and
commit e4a60d1390609
("sysfs: driver core: Fix glue dir race condition by gdp_mutex").

A kernfs name lookup in the parent's directory needs no new state, but
takes the kernfs root rwsem under gdp_mutex, behind the writes
concurrent sysfs directory creation generates.

A KUnit suite covers the index: reuse of one parent's glue directory,
two classes below one parent, the parentless "virtual" cases, reap and
recreate, name collision, many parents, device_move(), and class
teardown.

Assisted-by: LLM
Signed-off-by: Pavol Sakac <sakacpav@amazon.de>
---
 drivers/base/base.h               |   4 +
 drivers/base/core.c               | 112 +++++--
 drivers/base/test/.kunitconfig    |   1 +
 drivers/base/test/Kconfig         |  12 +
 drivers/base/test/Makefile        |   2 +
 drivers/base/test/glue-dir-test.c | 466 ++++++++++++++++++++++++++++++
 6 files changed, 566 insertions(+), 31 deletions(-)
 create mode 100644 drivers/base/test/glue-dir-test.c

diff --git a/drivers/base/base.h b/drivers/base/base.h
index a5b7abc10ff0..f5d608f4aaa5 100644
--- a/drivers/base/base.h
+++ b/drivers/base/base.h
@@ -11,6 +11,7 @@
  *
  */
 #include <linux/notifier.h>
+#include <linux/rbtree_types.h>
 
 /**
  * struct subsys_private - structure to hold the private to the driver core
@@ -32,6 +33,8 @@
  * @dev_root: Default device to use as the parent.
  * @glue_dirs: "glue" directory to put in-between the parent device to
  *	       avoid namespace conflicts
+ * @glue_dirs_index: the class's glue dirs by parent kobject, under gdp_mutex;
+ *		     zeroed is an empty rb_root
  * @class: pointer back to the struct class that this structure is associated
  *	   with.
  * @lock_key: Lock class key for use by the lock validator
@@ -55,6 +58,7 @@ struct subsys_private {
 	struct device *dev_root;
 
 	struct kset glue_dirs;
+	struct rb_root glue_dirs_index;
 	const struct class *class;
 
 	struct lock_class_key lock_key;
diff --git a/drivers/base/core.c b/drivers/base/core.c
index 4c0c373998a1..5dea641cbdb6 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -26,6 +26,7 @@
 #include <linux/of.h>
 #include <linux/of_device.h>
 #include <linux/pm_runtime.h>
+#include <linux/rbtree.h>
 #include <linux/sched/mm.h>
 #include <linux/sched/signal.h>
 #include <linux/slab.h>
@@ -3263,6 +3264,8 @@ struct kobject *virtual_device_parent(void)
 struct class_dir {
 	struct kobject kobj;
 	const struct class *class;
+	/* in the class's glue_dirs_index, keyed by kobj.parent (gdp_mutex) */
+	struct rb_node index_node;
 };
 
 #define to_class_dir(obj) container_of(obj, struct class_dir, kobj)
@@ -3298,6 +3301,7 @@ static struct kobject *class_dir_create_and_add(struct subsys_private *sp,
 
 	dir->class = sp->class;
 	kobject_init(&dir->kobj, &class_dir_ktype);
+	RB_CLEAR_NODE(&dir->index_node);
 
 	dir->kobj.kset = &sp->glue_dirs;
 
@@ -3311,6 +3315,66 @@ static struct kobject *class_dir_create_and_add(struct subsys_private *sp,
 
 static DEFINE_MUTEX(gdp_mutex);
 
+/*
+ * Glue-dir lookup index: each class's glue dirs by parent kobject, in an
+ * rbtree embedded in its subsys_private.  The glue_dirs kset remains the
+ * membership and identity authority (kobj->kset); the index replaces only
+ * the list walk in get_device_parent() and dies with the subsys_private
+ * generation that owns both.
+ */
+static int glue_dir_cmp_key(const void *key, const struct rb_node *node)
+{
+	const struct class_dir *cd = rb_entry(node, struct class_dir,
+					      index_node);
+
+	if ((unsigned long)key < (unsigned long)cd->kobj.parent)
+		return -1;
+	return (unsigned long)key > (unsigned long)cd->kobj.parent;
+}
+
+static bool glue_dir_less(struct rb_node *lhs, const struct rb_node *rhs)
+{
+	const struct class_dir *cd = rb_entry(lhs, struct class_dir,
+					      index_node);
+
+	return glue_dir_cmp_key(cd->kobj.parent, rhs) < 0;
+}
+
+static struct kobject *glue_dir_lookup(struct subsys_private *sp,
+				       struct kobject *parent_kobj)
+{
+	struct rb_node *node;
+
+	lockdep_assert_held(&gdp_mutex);
+
+	node = rb_find(parent_kobj, &sp->glue_dirs_index, glue_dir_cmp_key);
+	if (!node)
+		return NULL;
+
+	return kobject_get(&rb_entry(node, struct class_dir, index_node)->kobj);
+}
+
+static void glue_dir_index(struct subsys_private *sp, struct kobject *glue_dir)
+{
+	lockdep_assert_held(&gdp_mutex);
+
+	/* Lookup-before-create under gdp_mutex keeps keys unique. */
+	rb_add(&to_class_dir(glue_dir)->index_node, &sp->glue_dirs_index,
+	       glue_dir_less);
+}
+
+/*
+ * rb_erase() uses only recorded tree links, but glue_dir_less() keys on
+ * kobj.parent: the dir, indexed at create, leaves the index before
+ * kobject_del() clears its parent -- an indexed node's key must never mutate.
+ */
+static void glue_dir_unindex(struct subsys_private *sp,
+			     struct kobject *glue_dir)
+{
+	lockdep_assert_held(&gdp_mutex);
+	rb_erase(&to_class_dir(glue_dir)->index_node, &sp->glue_dirs_index);
+}
+
 static struct kobject *get_device_parent(struct device *dev,
 					 struct device *parent)
 {
@@ -3338,13 +3402,7 @@ static struct kobject *get_device_parent(struct device *dev,
 		mutex_lock(&gdp_mutex);
 
 		/* find our class-directory at the parent and reference it */
-		spin_lock(&sp->glue_dirs.list_lock);
-		list_for_each_entry(k, &sp->glue_dirs.list, entry)
-			if (k->parent == parent_kobj) {
-				kobj = kobject_get(k);
-				break;
-			}
-		spin_unlock(&sp->glue_dirs.list_lock);
+		kobj = glue_dir_lookup(sp, parent_kobj);
 		if (kobj) {
 			mutex_unlock(&gdp_mutex);
 			subsys_put(sp);
@@ -3354,6 +3412,8 @@ static struct kobject *get_device_parent(struct device *dev,
 		/* or create a new class-directory at the parent device */
 		k = class_dir_create_and_add(sp, parent_kobj);
 		/* do not emit an uevent for this simple "glue" directory */
+		if (!IS_ERR(k))
+			glue_dir_index(sp, k);
 		mutex_unlock(&gdp_mutex);
 		subsys_put(sp);
 		return k;
@@ -3375,28 +3435,6 @@ static struct kobject *get_device_parent(struct device *dev,
 	return NULL;
 }
 
-static inline bool live_in_glue_dir(struct kobject *kobj,
-				    struct device *dev)
-{
-	struct subsys_private *sp;
-	bool retval;
-
-	if (!kobj || !dev->class)
-		return false;
-
-	sp = class_to_subsys(dev->class);
-	if (!sp)
-		return false;
-
-	if (kobj->kset == &sp->glue_dirs)
-		retval = true;
-	else
-		retval = false;
-
-	subsys_put(sp);
-	return retval;
-}
-
 static inline struct kobject *get_glue_dir(struct device *dev)
 {
 	return dev->kobj.parent;
@@ -3426,11 +3464,19 @@ static inline bool kobject_has_children(struct kobject *kobj)
  */
 static void cleanup_glue_dir(struct device *dev, struct kobject *glue_dir)
 {
+	struct subsys_private *sp;
 	unsigned int ref;
 
 	/* see if we live in a "glue" directory */
-	if (!live_in_glue_dir(glue_dir, dev))
+	if (!glue_dir || !dev->class)
+		return;
+	sp = class_to_subsys(dev->class);
+	if (!sp)
+		return;
+	if (glue_dir->kset != &sp->glue_dirs) {
+		subsys_put(sp);
 		return;
+	}
 
 	mutex_lock(&gdp_mutex);
 	/**
@@ -3482,10 +3528,14 @@ static void cleanup_glue_dir(struct device *dev, struct kobject *glue_dir)
 	 * for glue_dir kobj is 1.
 	 */
 	ref = kref_read(&glue_dir->kref);
-	if (!kobject_has_children(glue_dir) && !--ref)
+	if (!kobject_has_children(glue_dir) && !--ref) {
+		glue_dir_unindex(sp, glue_dir);
 		kobject_del(glue_dir);
+	}
 	kobject_put(glue_dir);
 	mutex_unlock(&gdp_mutex);
+	/* outside gdp_mutex: the last put runs the class's release callback */
+	subsys_put(sp);
 }
 
 static int device_add_class_symlinks(struct device *dev)
diff --git a/drivers/base/test/.kunitconfig b/drivers/base/test/.kunitconfig
index 473923f0998b..28322bad39a7 100644
--- a/drivers/base/test/.kunitconfig
+++ b/drivers/base/test/.kunitconfig
@@ -1,2 +1,3 @@
 CONFIG_KUNIT=y
 CONFIG_DM_KUNIT_TEST=y
+CONFIG_GLUE_DIR_KUNIT_TEST=y
diff --git a/drivers/base/test/Kconfig b/drivers/base/test/Kconfig
index 542ce07530a1..253b5bd96aff 100644
--- a/drivers/base/test/Kconfig
+++ b/drivers/base/test/Kconfig
@@ -24,3 +24,15 @@ config DRIVER_SWNODE_KUNIT_TEST
 	tristate "KUnit Tests for software node fw_devlink links" if !KUNIT_ALL_TESTS
 	depends on KUNIT
 	default KUNIT_ALL_TESTS
+
+config GLUE_DIR_KUNIT_TEST
+	tristate "KUnit Tests for class glue directories" if !KUNIT_ALL_TESTS
+	depends on KUNIT && SYSFS
+	default KUNIT_ALL_TESTS
+	help
+	  Enable this option to test the class glue directories the driver
+	  core places class devices under: that siblings share one, that a
+	  reaped one is created again, and that a same-named child which is
+	  not a glue directory is never mistaken for one.
+
+	  If unsure say N.
diff --git a/drivers/base/test/Makefile b/drivers/base/test/Makefile
index 9ced7bbd569f..f13f0c399bea 100644
--- a/drivers/base/test/Makefile
+++ b/drivers/base/test/Makefile
@@ -8,3 +8,5 @@ obj-$(CONFIG_DRIVER_PE_KUNIT_TEST) += property-entry-test.o
 CFLAGS_property-entry-test.o += $(DISABLE_STRUCTLEAK_PLUGIN)
 
 obj-$(CONFIG_DRIVER_SWNODE_KUNIT_TEST) += swnode-devlink-test.o
+
+obj-$(CONFIG_GLUE_DIR_KUNIT_TEST) += glue-dir-test.o
diff --git a/drivers/base/test/glue-dir-test.c b/drivers/base/test/glue-dir-test.c
new file mode 100644
index 000000000000..fcdfdd3acd6f
--- /dev/null
+++ b/drivers/base/test/glue-dir-test.c
@@ -0,0 +1,466 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * KUnit tests for the class glue-directory index: a class device whose
+ * parent is not itself a class device is placed under a per-parent "glue"
+ * directory named after the class, which get_device_parent() finds
+ * through a per-class index keyed by the parent kobject.  Every case
+ * drives that lookup through plain device registration.
+ */
+
+#include <kunit/resource.h>
+#include <kunit/test.h>
+
+#include <linux/device.h>
+#include <linux/kernfs.h>
+#include <linux/kobject.h>
+#include <linux/slab.h>
+#include <linux/sysfs.h>
+
+static void glue_dev_release(struct device *dev)
+{
+	kfree(dev);
+}
+
+static void glue_dev_unregister(void *data)
+{
+	device_unregister(data);
+}
+
+static void glue_root_unregister(void *data)
+{
+	root_device_unregister(data);
+}
+
+static void glue_class_destroy(void *data)
+{
+	class_destroy(data);
+}
+
+/* Does a child named @name exist under @parent? */
+static bool glue_child_visible(struct kobject *parent, const char *name)
+{
+	struct kernfs_node *kn = kernfs_find_and_get(parent->sd, name);
+	bool found = kn;
+
+	kernfs_put(kn);
+	return found;
+}
+
+/* Freed by glue_dev_release() when the last reference drops. */
+static struct device *glue_dev_alloc(struct kunit *test,
+				     struct device *parent,
+				     const struct class *class,
+				     const char *name)
+{
+	struct device *dev;
+	int ret;
+
+	dev = kzalloc_obj(*dev);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, dev);
+
+	device_initialize(dev);
+	dev->parent = parent;
+	dev->class = class;
+	dev->release = glue_dev_release;
+
+	ret = dev_set_name(dev, "%s", name);
+	if (ret)
+		put_device(dev);
+	KUNIT_ASSERT_EQ(test, ret, 0);
+	return dev;
+}
+
+/* Register a class device; unregistered again by a deferred kunit action. */
+static struct device *glue_dev_add(struct kunit *test, struct device *parent,
+				   const struct class *class,
+				   const char *name)
+{
+	struct device *dev = glue_dev_alloc(test, parent, class, name);
+	int ret;
+
+	ret = device_add(dev);
+	if (ret)
+		put_device(dev);
+	KUNIT_ASSERT_EQ(test, ret, 0);
+	KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test,
+							glue_dev_unregister,
+							dev), 0);
+	return dev;
+}
+
+static struct class *glue_class(struct kunit *test, const char *name)
+{
+	struct class *class = class_create(name);
+
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, class);
+	KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test,
+							glue_class_destroy,
+							class), 0);
+	return class;
+}
+
+static struct device *glue_root(struct kunit *test, const char *name)
+{
+	struct device *root = root_device_register(name);
+
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, root);
+	KUNIT_ASSERT_EQ(test, kunit_add_action_or_reset(test,
+							glue_root_unregister,
+							root), 0);
+	return root;
+}
+
+/* Siblings share one glue dir: the second lookup has to hit the first. */
+static void glue_test_reuse(struct kunit *test)
+{
+	struct device *root, *dev_a, *dev_b;
+	struct kobject *glue;
+	struct class *class;
+
+	class = glue_class(test, "glue_kunit_reuse");
+	root = glue_root(test, "glue_kunit_reuse_root");
+
+	dev_a = glue_dev_add(test, root, class, "reuseA");
+	dev_b = glue_dev_add(test, root, class, "reuseB");
+
+	glue = dev_a->kobj.parent;
+	KUNIT_ASSERT_NOT_NULL(test, glue);
+	KUNIT_EXPECT_PTR_NE(test, glue, &root->kobj);
+	KUNIT_EXPECT_PTR_EQ(test, glue, dev_b->kobj.parent);
+	KUNIT_EXPECT_TRUE(test, glue_child_visible(glue, "reuseA"));
+	KUNIT_EXPECT_TRUE(test, glue_child_visible(glue, "reuseB"));
+}
+
+/* Two classes below one parent: each consults only its own class's index. */
+static void glue_test_cross_class(struct kunit *test)
+{
+	struct class *class_a, *class_b;
+	struct device *root, *dev_a, *dev_b;
+
+	class_a = glue_class(test, "glue_kunit_xclass_a");
+	class_b = glue_class(test, "glue_kunit_xclass_b");
+	root = glue_root(test, "glue_kunit_xclass_root");
+
+	dev_a = glue_dev_add(test, root, class_a, "xclassA");
+	dev_b = glue_dev_add(test, root, class_b, "xclassB");
+
+	KUNIT_ASSERT_NOT_NULL(test, dev_a->kobj.parent);
+	KUNIT_EXPECT_PTR_NE(test, dev_a->kobj.parent, dev_b->kobj.parent);
+}
+
+/* Reap retires the index entry: the next add must not find the freed dir. */
+static void glue_test_reap_recreate(struct kunit *test)
+{
+	struct device *root, *dev;
+	struct class *class;
+
+	class = glue_class(test, "glue_kunit_reap");
+	root = glue_root(test, "glue_kunit_reap_root");
+
+	dev = glue_dev_add(test, root, class, "reap0");
+	KUNIT_EXPECT_TRUE(test, glue_child_visible(&root->kobj,
+						   "glue_kunit_reap"));
+
+	/* last child gone: the glue dir goes with it ... */
+	kunit_release_action(test, glue_dev_unregister, dev);
+	KUNIT_EXPECT_FALSE(test, glue_child_visible(&root->kobj,
+						    "glue_kunit_reap"));
+
+	/* ... and a further registration recreates it */
+	dev = glue_dev_add(test, root, class, "reap1");
+	KUNIT_ASSERT_NOT_NULL(test, dev->kobj.parent);
+	KUNIT_EXPECT_TRUE(test, glue_child_visible(dev->kobj.parent, "reap1"));
+}
+
+static ssize_t glue_kunit_collide_show(struct device *dev,
+				       struct device_attribute *attr,
+				       char *buf)
+{
+	return sysfs_emit(buf, "\n");
+}
+static DEVICE_ATTR_RO(glue_kunit_collide);
+
+/*
+ * A same-named child that is not a glue dir must never be taken for one:
+ * the colliding add fails on the duplicate sysfs create and leaves
+ * nothing stale, so the same registration succeeds once the file is gone.
+ */
+static void glue_test_name_collision(struct kunit *test)
+{
+	struct device *root, *dev;
+	struct kernfs_node *kn;
+	struct class *class;
+
+	class = glue_class(test, "glue_kunit_collide");
+	root = glue_root(test, "glue_kunit_collide_root");
+
+	KUNIT_ASSERT_EQ(test,
+			device_create_file(root,
+					   &dev_attr_glue_kunit_collide), 0);
+
+	dev = glue_dev_alloc(test, root, class, "collide0");
+	KUNIT_ASSERT_EQ(test, device_add(dev), -EEXIST);
+	put_device(dev);
+
+	/* the same-named child is still the attribute file */
+	kn = kernfs_find_and_get(root->kobj.sd, "glue_kunit_collide");
+	KUNIT_ASSERT_NOT_NULL(test, kn);
+	KUNIT_EXPECT_EQ(test, kernfs_type(kn), KERNFS_FILE);
+	kernfs_put(kn);
+
+	device_remove_file(root, &dev_attr_glue_kunit_collide);
+	dev = glue_dev_add(test, root, class, "collide0");
+
+	/* and it lands in a glue dir of the class's name, not on the root */
+	KUNIT_ASSERT_NOT_NULL(test, dev->kobj.parent);
+	KUNIT_EXPECT_PTR_NE(test, dev->kobj.parent, &root->kobj);
+	KUNIT_EXPECT_STREQ(test, kobject_name(dev->kobj.parent), class->name);
+	KUNIT_EXPECT_PTR_EQ(test, dev->kobj.parent->parent, &root->kobj);
+}
+
+/*
+ * A parentless class device is keyed on the shared "virtual" kobject
+ * rather than on a device, so two of them still share one glue dir.
+ */
+static void glue_test_virtual_parent(struct kunit *test)
+{
+	struct device *dev_a, *dev_b;
+	struct kobject *glue;
+	struct class *class;
+
+	class = glue_class(test, "glue_kunit_virtual");
+
+	dev_a = glue_dev_add(test, NULL, class, "virtA");
+	dev_b = glue_dev_add(test, NULL, class, "virtB");
+
+	glue = dev_a->kobj.parent;
+	KUNIT_ASSERT_NOT_NULL(test, glue);
+	KUNIT_EXPECT_PTR_EQ(test, glue, dev_b->kobj.parent);
+	KUNIT_ASSERT_NOT_NULL(test, glue->parent);
+	KUNIT_EXPECT_STREQ(test, kobject_name(glue->parent), "virtual");
+}
+
+/*
+ * Every class's parentless devices share the one "virtual" key; two
+ * classes below it must still land in dirs of their own class's name.
+ */
+static void glue_test_virtual_cross_class(struct kunit *test)
+{
+	struct class *class_a, *class_b;
+	struct device *dev_a, *dev_b;
+	struct kobject *glue_a, *glue_b;
+
+	class_a = glue_class(test, "glue_kunit_virtxc_a");
+	class_b = glue_class(test, "glue_kunit_virtxc_b");
+
+	dev_a = glue_dev_add(test, NULL, class_a, "virtxcA");
+	dev_b = glue_dev_add(test, NULL, class_b, "virtxcB");
+
+	glue_a = dev_a->kobj.parent;
+	glue_b = dev_b->kobj.parent;
+	KUNIT_ASSERT_NOT_NULL(test, glue_a);
+	KUNIT_ASSERT_NOT_NULL(test, glue_b);
+
+	KUNIT_EXPECT_PTR_NE(test, glue_a, glue_b);
+	KUNIT_EXPECT_STREQ(test, kobject_name(glue_a), class_a->name);
+	KUNIT_EXPECT_STREQ(test, kobject_name(glue_b), class_b->name);
+
+	/* both dirs hang off the one shared key */
+	KUNIT_ASSERT_NOT_NULL(test, glue_a->parent);
+	KUNIT_EXPECT_PTR_EQ(test, glue_a->parent, glue_b->parent);
+	KUNIT_EXPECT_STREQ(test, kobject_name(glue_a->parent), "virtual");
+}
+
+#define GLUE_MANY_PARENTS 256
+
+/*
+ * With hundreds of parents indexed, every lookup must resolve the dir
+ * anchored at its own parent -- the case that falsifies the key comparison.
+ */
+static void glue_test_many_parents(struct kunit *test)
+{
+	struct device **roots, **devs;
+	struct class *class;
+	char name[32];
+	int i;
+
+	class = glue_class(test, "glue_kunit_many");
+
+	roots = kunit_kcalloc(test, GLUE_MANY_PARENTS, sizeof(*roots),
+			      GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, roots);
+	devs = kunit_kcalloc(test, GLUE_MANY_PARENTS, sizeof(*devs),
+			     GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, devs);
+
+	for (i = 0; i < GLUE_MANY_PARENTS; i++) {
+		snprintf(name, sizeof(name), "glue_kunit_many_root%d", i);
+		roots[i] = glue_root(test, name);
+		snprintf(name, sizeof(name), "many%d", i);
+		devs[i] = glue_dev_add(test, roots[i], class, name);
+	}
+
+	/* Distinctness via anchoring: a kobject has one parent. */
+	for (i = 0; i < GLUE_MANY_PARENTS; i++) {
+		struct kobject *glue = devs[i]->kobj.parent;
+
+		KUNIT_ASSERT_NOT_NULL(test, glue);
+		KUNIT_EXPECT_PTR_EQ(test, glue->parent, &roots[i]->kobj);
+	}
+}
+
+/*
+ * device_move() looks up against the new parent and must leave the old
+ * parent's entry behind as a valid hit: the old glue dir is never reaped
+ * (base behaviour), so a later device under the old parent reuses it.
+ */
+static void glue_test_device_move(struct kunit *test)
+{
+	struct device *root_a, *root_b, *dev, *dev_a2;
+	struct kobject *old_glue, *new_glue;
+	struct class *class;
+
+	class = glue_class(test, "glue_kunit_move");
+	root_a = glue_root(test, "glue_kunit_move_rootA");
+	root_b = glue_root(test, "glue_kunit_move_rootB");
+
+	dev = glue_dev_add(test, root_a, class, "move0");
+	old_glue = dev->kobj.parent;
+	KUNIT_ASSERT_NOT_NULL(test, old_glue);
+
+	KUNIT_ASSERT_EQ(test, device_move(dev, root_b, DPM_ORDER_NONE), 0);
+
+	new_glue = dev->kobj.parent;
+	KUNIT_ASSERT_NOT_NULL(test, new_glue);
+	KUNIT_EXPECT_PTR_NE(test, new_glue, old_glue);
+	KUNIT_EXPECT_PTR_EQ(test, new_glue->parent, &root_b->kobj);
+	KUNIT_EXPECT_TRUE(test, glue_child_visible(new_glue, "move0"));
+
+	/* nothing reaps the old dir on the success path: it is still there */
+	KUNIT_EXPECT_TRUE(test, glue_child_visible(&root_a->kobj,
+						   "glue_kunit_move"));
+
+	/* and it must still be the hit for the old parent */
+	dev_a2 = glue_dev_add(test, root_a, class, "move1");
+	KUNIT_EXPECT_PTR_EQ(test, dev_a2->kobj.parent, old_glue);
+}
+
+/*
+ * Removing one of two siblings must not retire the shared glue dir:
+ * the survivor keeps it visible, and a further sibling still reuses
+ * it -- a spurious unindex would fail that add on a duplicate create.
+ */
+static void glue_test_no_reap(struct kunit *test)
+{
+	struct device *root, *dev_a, *dev_b, *dev_c;
+	struct kobject *glue;
+	struct class *class;
+
+	class = glue_class(test, "glue_kunit_noreap");
+	root = glue_root(test, "glue_kunit_noreap_root");
+
+	dev_a = glue_dev_add(test, root, class, "noreapA");
+	dev_b = glue_dev_add(test, root, class, "noreapB");
+	glue = dev_b->kobj.parent;
+	KUNIT_ASSERT_NOT_NULL(test, glue);
+
+	kunit_release_action(test, glue_dev_unregister, dev_a);
+
+	KUNIT_EXPECT_TRUE(test, glue_child_visible(&root->kobj,
+						   "glue_kunit_noreap"));
+	KUNIT_EXPECT_PTR_EQ(test, dev_b->kobj.parent, glue);
+	KUNIT_EXPECT_TRUE(test, glue_child_visible(glue, "noreapB"));
+
+	dev_c = glue_dev_add(test, root, class, "noreapC");
+	KUNIT_EXPECT_PTR_EQ(test, dev_c->kobj.parent, glue);
+}
+
+/*
+ * Static so that unregistering under a live device frees only the
+ * driver-core generation, not the struct class the device points at.
+ */
+static const struct class glue_gone_class_a = {
+	.name = "glue_kunit_gone_a",
+};
+
+static const struct class glue_gone_class_b = {
+	.name = "glue_kunit_gone_b",
+};
+
+static void glue_static_class_unregister(void *data)
+{
+	class_unregister(data);
+}
+
+/*
+ * Unregistering a class under a live device is tolerated API misuse: the
+ * device's glue dir can no longer be reaped and is deliberately leaked
+ * (class_dir, kernfs node, and parent reference), here as in the base
+ * tree; kmemleak reports all three on every run.
+ *
+ * The case pins the index's placement: a dead generation's tree dies
+ * with its subsys_private, so the successor class starts on an empty
+ * tree and cannot resolve the leaked dir.  A global index would put the
+ * dead entry back in its search path.
+ */
+static void glue_test_class_gone(struct kunit *test)
+{
+	struct device *root, *dev_a, *dev_b;
+	struct kobject *glue_a;
+
+	KUNIT_ASSERT_EQ(test, class_register(&glue_gone_class_a), 0);
+	KUNIT_ASSERT_EQ(test,
+			kunit_add_action_or_reset(test,
+						  glue_static_class_unregister,
+						  (void *)&glue_gone_class_a),
+			0);
+	root = glue_root(test, "glue_kunit_gone_root");
+
+	dev_a = glue_dev_add(test, root, &glue_gone_class_a, "goneA");
+	glue_a = dev_a->kobj.parent;
+	KUNIT_ASSERT_NOT_NULL(test, glue_a);
+
+	/* the misuse: the class goes away under a live device */
+	kunit_release_action(test, glue_static_class_unregister,
+			     (void *)&glue_gone_class_a);
+
+	KUNIT_ASSERT_EQ(test, class_register(&glue_gone_class_b), 0);
+	KUNIT_ASSERT_EQ(test,
+			kunit_add_action_or_reset(test,
+						  glue_static_class_unregister,
+						  (void *)&glue_gone_class_b),
+			0);
+
+	dev_b = glue_dev_add(test, root, &glue_gone_class_b, "goneB");
+	KUNIT_ASSERT_NOT_NULL(test, dev_b->kobj.parent);
+	KUNIT_EXPECT_PTR_NE(test, dev_b->kobj.parent, glue_a);
+	KUNIT_EXPECT_PTR_NE(test, dev_b->kobj.parent, &root->kobj);
+	KUNIT_EXPECT_STREQ(test, kobject_name(dev_b->kobj.parent),
+			   glue_gone_class_b.name);
+	KUNIT_EXPECT_TRUE(test, glue_child_visible(dev_b->kobj.parent,
+						   "goneB"));
+}
+
+static struct kunit_case glue_dir_tests[] = {
+	KUNIT_CASE(glue_test_reuse),
+	KUNIT_CASE(glue_test_cross_class),
+	KUNIT_CASE(glue_test_reap_recreate),
+	KUNIT_CASE(glue_test_name_collision),
+	KUNIT_CASE(glue_test_virtual_parent),
+	KUNIT_CASE(glue_test_virtual_cross_class),
+	KUNIT_CASE(glue_test_many_parents),
+	KUNIT_CASE(glue_test_device_move),
+	KUNIT_CASE(glue_test_no_reap),
+	KUNIT_CASE(glue_test_class_gone),
+	{}
+};
+
+static struct kunit_suite glue_dir_test_suite = {
+	.name = "glue_dir",
+	.test_cases = glue_dir_tests,
+};
+
+kunit_test_suite(glue_dir_test_suite);
+
+MODULE_DESCRIPTION("KUnit tests for class glue directories");
+MODULE_LICENSE("GPL");
-- 
2.47.3

[PATCH 3/3] kernfs: batch inode ID allocation per CPU

[Pavol Sakac]

Every kernfs node allocates an inode ID from its root's IDR under the
per-root kernfs_idr_lock (per-root since commit cec59c440a05c ("kernfs:
switch global kernfs_idr_lock to per-fs lock")). sysfs is a single root,
so parallel creation funnels every node through one lock, and a
registration storm acquires it once per created node: even
registrations in disjoint subtrees contend on that one spinlock.

Let opted-in roots reserve a batch of IDs as NULL IDR entries and cache
them per CPU, so node creation pops one and installs itself with
idr_replace() under rcu_read_lock() instead of taking the shared lock.

Each call stays within the documented IDR contract: idr_replace() may run
under the RCU read lock concurrently with the idr_alloc() and
idr_remove() other CPUs do under the lock, and installing into an entry
reserved as NULL is the reserve-then-replace pattern of
Documentation/core-api/idr.rst. The property relied on beyond that is
concurrent idr_replace() on distinct reserved IDs, the normal mode here
since each reserved ID goes to exactly one caller; it holds because
replacing a reserved slot (NULL, IDR_FREE clear) is a single
rcu_assign_pointer() store and nothing else, and calculate_count()
returns 0 for such a slot, so the transition is count-neutral, writes no
shared radix-tree state, and cannot interact with a concurrent replace of
a distinct slot even within one radix-tree node.

Refill failure falls back to the locked path. An opted-in root must not
also ask for KERNFS_ROOT_SUPPORT_EXPORTOP, whose fhandle by-id lookups
must not meet an ino that is allocated but owned by no node. Enable
batching for sysfs: with batching, a registration storm takes
kernfs_idr_lock once per KERNFS_INO_BATCH-sized batch of nodes
instead of once per node.

Add root_device_ino_uniqueness_test to the root-device-devm KUnit
suite: it registers devices from several tasks at once, each crossing a
batch refill, and requires the IDs of nodes that are all alive at the
same time to be distinct.

Assisted-by: LLM
Signed-off-by: Pavol Sakac <sakacpav@amazon.de>
---
 drivers/base/test/root-device-test.c | 215 +++++++++++++++++++++++++++
 fs/kernfs/dir.c                      | 122 +++++++++++++++
 fs/kernfs/kernfs-internal.h          |   3 +
 fs/sysfs/mount.c                     |   3 +-
 include/linux/kernfs.h               |  13 ++
 5 files changed, 355 insertions(+), 1 deletion(-)

diff --git a/drivers/base/test/root-device-test.c b/drivers/base/test/root-device-test.c
index 9aea23c9123e..f3370d53f9d4 100644
--- a/drivers/base/test/root-device-test.c
+++ b/drivers/base/test/root-device-test.c
@@ -3,7 +3,12 @@
 
 #include <kunit/resource.h>
 
+#include <linux/completion.h>
+#include <linux/cpumask.h>
 #include <linux/device.h>
+#include <linux/kernfs.h>
+#include <linux/kthread.h>
+#include <linux/sort.h>
 
 #define DEVICE_NAME "test"
 
@@ -93,9 +98,219 @@ static void root_device_devm_register_get_unregister_with_devm_test(struct kunit
 	KUNIT_EXPECT_GT(test, ret, 0);
 }
 
+#if IS_ENABLED(CONFIG_SYSFS)
+/*
+ * Inode IDs may be handed out from per-CPU batches (KERNFS_ROOT_INO_BATCH on
+ * the sysfs root); one task's loop only draws from the CPU it runs on and
+ * would pass unchanged if batching were inert.  Hence several tasks at
+ * once.  Per task: more than INO_BATCH IDs for every online CPU, so a
+ * refill is crossed even if a task migrates across all of them, and never
+ * a whole number of batches, so the run also ends mid-batch.
+ */
+#define INO_THREADS		4
+#define INO_BATCH		16	/* KERNFS_INO_BATCH (kernfs-private) */
+#define INO_MIN_PER_THREAD	200
+/* one worker's registrations scale with the CPU count: ~0.5ms each on 4 CPUs */
+#define INO_TIMEOUT_BASE_MS	5000
+#define INO_TIMEOUT_PER_DEV_MS	10
+
+struct ino_worker {
+	int idx;
+	int nr;				/* devices to register */
+	int created;			/* devices actually registered */
+	int err;			/* first registration error */
+	bool joined;			/* completion was waited out */
+	struct device **devs;
+	ino_t *inos;
+	struct completion done;
+};
+
+struct ino_test_ctx {
+	struct ino_worker worker[INO_THREADS];
+	int spawned;
+};
+
+static int root_device_ino_worker(void *data)
+{
+	struct ino_worker *w = data;
+	int i;
+
+	for (i = 0; i < w->nr; i++) {
+		char name[32];
+		struct device *dev;
+
+		snprintf(name, sizeof(name), DEVICE_NAME "-ino-%d-%d",
+			 w->idx, i);
+		dev = root_device_register(name);
+		if (IS_ERR(dev)) {
+			w->err = PTR_ERR(dev);
+			break;
+		}
+		w->devs[i] = dev;
+		w->inos[i] = dev->kobj.sd ? kernfs_ino(dev->kobj.sd) : 0;
+		w->created = i + 1;
+	}
+
+	/*
+	 * complete_all(), not complete(): this is waited on twice, once by
+	 * the test and once by the teardown action, and a plain completion
+	 * is consumed by the first waiter.
+	 */
+	complete_all(&w->done);
+	return 0;
+}
+
+/*
+ * Registered before the first worker is spawned, so it also runs if the test
+ * aborts: join every worker, then undo what it did.
+ */
+static void root_device_ino_teardown(void *data)
+{
+	struct ino_test_ctx *ctx = data;
+	int t, i;
+
+	for (t = 0; t < ctx->spawned; t++)
+		wait_for_completion(&ctx->worker[t].done);
+
+	for (t = 0; t < INO_THREADS; t++)
+		for (i = 0; i < ctx->worker[t].created; i++)
+			root_device_unregister(ctx->worker[t].devs[i]);
+}
+
+static int ino_cmp(const void *a, const void *b)
+{
+	ino_t x = *(const ino_t *)a;
+	ino_t y = *(const ino_t *)b;
+
+	if (x < y)
+		return -1;
+	return x > y;
+}
+
+static void root_device_ino_uniqueness_test(struct kunit *test)
+{
+	int zeros = 0, dups = 0, descents = 0, ids = 0;
+	struct ino_test_ctx *ctx;
+	bool results_valid = true;
+	unsigned long timeout;
+	struct device **devs;
+	ino_t *inos, *sorted;
+	int nr, total, t, i;
+
+	nr = INO_BATCH * num_online_cpus() + INO_BATCH / 2;
+	if (nr < INO_MIN_PER_THREAD)
+		nr = INO_MIN_PER_THREAD;
+	total = INO_THREADS * nr;
+	timeout = msecs_to_jiffies(INO_TIMEOUT_BASE_MS +
+				   nr * INO_TIMEOUT_PER_DEV_MS);
+
+	ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL);
+	devs = kunit_kcalloc(test, total, sizeof(*devs), GFP_KERNEL);
+	inos = kunit_kcalloc(test, total, sizeof(*inos), GFP_KERNEL);
+	sorted = kunit_kcalloc(test, total, sizeof(*sorted), GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, devs);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, inos);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, sorted);
+
+	for (t = 0; t < INO_THREADS; t++) {
+		struct ino_worker *w = &ctx->worker[t];
+
+		w->idx = t;
+		w->nr = nr;
+		w->devs = devs + t * nr;
+		w->inos = inos + t * nr;
+		init_completion(&w->done);
+	}
+
+	KUNIT_ASSERT_EQ(test,
+			kunit_add_action_or_reset(test,
+						  root_device_ino_teardown,
+						  ctx), 0);
+
+	for (t = 0; t < INO_THREADS; t++) {
+		struct task_struct *task;
+
+		task = kthread_run(root_device_ino_worker, &ctx->worker[t],
+				   "root_dev_ino%d", t);
+		if (IS_ERR(task))
+			break;
+		ctx->spawned++;
+	}
+	KUNIT_EXPECT_EQ(test, ctx->spawned, INO_THREADS);
+	if (ctx->spawned != INO_THREADS)
+		results_valid = false;
+
+	/*
+	 * complete_all() is a worker's last act, so only a join that succeeds
+	 * orders its stores before the reads below; a worker that timed out
+	 * may still be writing its slice.  The teardown action joins those
+	 * unbounded, before any of this memory can be freed.
+	 */
+	for (t = 0; t < ctx->spawned; t++) {
+		struct ino_worker *w = &ctx->worker[t];
+
+		if (wait_for_completion_timeout(&w->done, timeout) > 0) {
+			w->joined = true;
+		} else {
+			KUNIT_FAIL(test, "worker %d timed out", t);
+			results_valid = false;
+		}
+	}
+
+	for (t = 0; t < ctx->spawned; t++) {
+		struct ino_worker *w = &ctx->worker[t];
+
+		if (!w->joined)
+			continue;
+
+		KUNIT_EXPECT_EQ(test, w->err, 0);
+		KUNIT_EXPECT_EQ(test, w->created, nr);
+		if (w->err || w->created != nr)
+			results_valid = false;
+
+		for (i = 0; i < w->created; i++) {
+			if (!w->inos[i])
+				zeros++;
+			/*
+			 * Batches are consumed from the top: IDs step down
+			 * within one, while the locked path only ascends, so
+			 * one descent is evidence the cache was in use.
+			 */
+			if (i && w->inos[i] < w->inos[i - 1])
+				descents++;
+			sorted[ids++] = w->inos[i];
+		}
+	}
+	KUNIT_EXPECT_EQ(test, zeros, 0);
+
+	/* every ID belongs to a node that is still alive here */
+	sort(sorted, ids, sizeof(*sorted), ino_cmp, NULL);
+	for (i = 1; i < ids; i++)
+		if (sorted[i] == sorted[i - 1])
+			dups++;
+	KUNIT_EXPECT_EQ(test, dups, 0);
+	if (zeros || dups)
+		results_valid = false;
+
+	/*
+	 * The batch cache is best-effort: when its allocation failed, every
+	 * ID above came from the locked fallback and no descent can have been
+	 * observed.  Make that visibly unavailable coverage rather than a
+	 * PASS that never exercised batching -- but only when everything
+	 * above held, since kunit_skip() would overwrite a recorded failure.
+	 */
+	if (IS_ENABLED(CONFIG_SMP) && results_valid && !descents)
+		kunit_skip(test, "sysfs inode batching fast path was not observed");
+}
+#endif
+
 static struct kunit_case root_device_devm_tests[] = {
 	KUNIT_CASE(root_device_devm_register_unregister_test),
 	KUNIT_CASE(root_device_devm_register_get_unregister_with_devm_test),
+#if IS_ENABLED(CONFIG_SYSFS)
+	KUNIT_CASE(root_device_ino_uniqueness_test),
+#endif
 	{}
 };
 
diff --git a/fs/kernfs/dir.c b/fs/kernfs/dir.c
index d68bce0b0b41..1938edd39eff 100644
--- a/fs/kernfs/dir.c
+++ b/fs/kernfs/dir.c
@@ -14,6 +14,7 @@
 #include <linux/slab.h>
 #include <linux/security.h>
 #include <linux/hash.h>
+#include <linux/local_lock.h>
 #include <linux/ns_common.h>
 
 #include "kernfs-internal.h"
@@ -28,6 +29,96 @@
 static DEFINE_SPINLOCK(kernfs_pr_cont_lock);
 static char kernfs_pr_cont_buf[PATH_MAX];	/* protected by pr_cont_lock */
 
+/*
+ * Cached IDs remain reserved as NULL IDR entries until they are consumed or
+ * the owning root is destroyed.
+ */
+#define KERNFS_INO_BATCH	16
+
+struct kernfs_ino_cache {
+	local_lock_t lock;
+	int count;
+	u64 id[KERNFS_INO_BATCH];
+};
+
+static int kernfs_ino_cache_alloc(struct kernfs_root *root)
+{
+	struct kernfs_ino_cache __percpu *cache;
+	int cpu;
+
+	cache = alloc_percpu(struct kernfs_ino_cache);
+	if (!cache)
+		return -ENOMEM;
+
+	for_each_possible_cpu(cpu) {
+		struct kernfs_ino_cache *c = per_cpu_ptr(cache, cpu);
+
+		local_lock_init(&c->lock);
+		c->count = 0;
+	}
+
+	root->ino_cache = cache;
+	return 0;
+}
+
+static void kernfs_ino_cache_free(struct kernfs_root *root)
+{
+	free_percpu(root->ino_cache);
+}
+
+/*
+ * Refill this CPU's cache in place: the free space is computed under
+ * the same local lock that guards the pop, so every reserved ID lands
+ * in the cache and none is ever returned. Lock ordering is
+ * one-directional: idr_preload() returns holding the radix-tree
+ * preload local_lock, the ino_cache local_lock nests inside it, and
+ * root->kernfs_idr_lock nests inside that. idr_alloc_cyclic(GFP_ATOMIC)
+ * does not sleep and is legal under both.
+ */
+static u64 kernfs_ino_batch_refill_and_get(struct kernfs_root *root)
+{
+	struct kernfs_ino_cache *c;
+	u64 id = 0;
+
+	idr_preload(GFP_KERNEL);
+	local_lock(&root->ino_cache->lock);
+	c = this_cpu_ptr(root->ino_cache);
+	spin_lock(&root->kernfs_idr_lock);
+	while (c->count < KERNFS_INO_BATCH) {
+		int ino = idr_alloc_cyclic(&root->ino_idr, NULL, 1, 0,
+					   GFP_ATOMIC);
+
+		if (ino < 0)
+			break;
+		if (ino < root->last_id_lowbits)
+			root->id_highbits++;
+		root->last_id_lowbits = ino;
+		c->id[c->count++] = (u64)root->id_highbits << 32 | ino;
+	}
+	spin_unlock(&root->kernfs_idr_lock);
+	if (c->count)
+		id = c->id[--c->count];
+	local_unlock(&root->ino_cache->lock);
+	idr_preload_end();
+	return id;
+}
+
+static u64 kernfs_ino_batch_get(struct kernfs_root *root)
+{
+	struct kernfs_ino_cache *c;
+	u64 id = 0;
+
+	local_lock(&root->ino_cache->lock);
+	c = this_cpu_ptr(root->ino_cache);
+	if (c->count > 0)
+		id = c->id[--c->count];
+	local_unlock(&root->ino_cache->lock);
+
+	if (!id)
+		id = kernfs_ino_batch_refill_and_get(root);
+	return id;
+}
+
 #define rb_to_kn(X) rb_entry((X), struct kernfs_node, rb)
 
 static bool __kernfs_active(struct kernfs_node *kn)
@@ -625,6 +716,7 @@ void kernfs_put(struct kernfs_node *kn)
 			goto repeat;
 	} else {
 		/* just released the root kn, free @root too */
+		kernfs_ino_cache_free(root);
 		idr_destroy(&root->ino_idr);
 		simple_xattr_cache_cleanup(&root->xa_cache);
 		kfree_rcu(root, rcu);
@@ -668,6 +760,28 @@ static struct kernfs_node *__kernfs_new_node(struct kernfs_root *root,
 	if (!kn)
 		goto err_out1;
 
+	if (root->ino_cache) {
+		u64 id = kernfs_ino_batch_get(root);
+
+		if (likely(id)) {
+			kn->id = id;
+			/*
+			 * The reservation cleared IDR_FREE, so this
+			 * is one count-neutral rcu_assign_pointer()
+			 * into a slot handed to exactly one caller:
+			 * nothing can concurrently remove or replace
+			 * it.  Publication matches the locked
+			 * idr_alloc() below -- count==0 makes ID
+			 * lookups refuse @kn until initialized.
+			 */
+			rcu_read_lock();
+			WARN_ON_ONCE(idr_replace(&root->ino_idr, kn,
+						 (u32)kernfs_ino(kn)));
+			rcu_read_unlock();
+			goto ino_done;
+		}
+	}
+
 	idr_preload(GFP_KERNEL);
 	spin_lock(&root->kernfs_idr_lock);
 	ret = idr_alloc_cyclic(&root->ino_idr, kn, 1, 0, GFP_ATOMIC);
@@ -682,6 +796,7 @@ static struct kernfs_node *__kernfs_new_node(struct kernfs_root *root,
 
 	kn->id = (u64)id_highbits << 32 | ret;
 
+ ino_done:
 	atomic_set(&kn->count, 1);
 	atomic_set(&kn->active, KN_DEACTIVATED_BIAS);
 	RB_CLEAR_NODE(&kn->rb);
@@ -1058,6 +1173,13 @@ struct kernfs_root *kernfs_create_root(struct kernfs_syscall_ops *scops,
 
 	root->syscall_ops = scops;
 	root->flags = flags;
+	/*
+	 * Batching is best-effort: without the cache every node takes the
+	 * locked path, so an allocation failure only needs to be visible.
+	 */
+	if (IS_ENABLED(CONFIG_SMP) && (flags & KERNFS_ROOT_INO_BATCH) &&
+	    kernfs_ino_cache_alloc(root))
+		pr_warn_once("kernfs: inode ID batching unavailable, using the locked allocation path\n");
 	root->kn = kn;
 	init_waitqueue_head(&root->deactivate_waitq);
 
diff --git a/fs/kernfs/kernfs-internal.h b/fs/kernfs/kernfs-internal.h
index aa784b540b36..6e12233731e6 100644
--- a/fs/kernfs/kernfs-internal.h
+++ b/fs/kernfs/kernfs-internal.h
@@ -30,6 +30,8 @@ struct kernfs_iattrs {
 	struct simple_xattr_limits xattr_limits;
 };
 
+struct kernfs_ino_cache;
+
 struct kernfs_root {
 	/* published fields */
 	struct kernfs_node	*kn;
@@ -40,6 +42,7 @@ struct kernfs_root {
 	spinlock_t		kernfs_idr_lock;	/* root->ino_idr */
 	u32			last_id_lowbits;
 	u32			id_highbits;
+	struct kernfs_ino_cache __percpu *ino_cache;
 	struct kernfs_syscall_ops *syscall_ops;
 
 	/* list of kernfs_super_info of this root, protected by kernfs_rwsem */
diff --git a/fs/sysfs/mount.c b/fs/sysfs/mount.c
index 88c10823fcaf..86288c5e34e8 100644
--- a/fs/sysfs/mount.c
+++ b/fs/sysfs/mount.c
@@ -86,7 +86,8 @@ int __init sysfs_init(void)
 {
 	int err;
 
-	sysfs_root = kernfs_create_root(NULL, KERNFS_ROOT_EXTRA_OPEN_PERM_CHECK,
+	sysfs_root = kernfs_create_root(NULL, KERNFS_ROOT_EXTRA_OPEN_PERM_CHECK |
+					KERNFS_ROOT_INO_BATCH,
 					NULL);
 	if (IS_ERR(sysfs_root))
 		return PTR_ERR(sysfs_root);
diff --git a/include/linux/kernfs.h b/include/linux/kernfs.h
index 351a5101c862..6440882b7d58 100644
--- a/include/linux/kernfs.h
+++ b/include/linux/kernfs.h
@@ -156,6 +156,19 @@ enum kernfs_root_flag {
 	 * Renames must not change the parent node.
 	 */
 	KERNFS_ROOT_INVARIANT_PARENT		= 0x0010,
+
+	/*
+	 * Reserve inode numbers for this root in per-CPU batches, taking the
+	 * root's ID allocation lock out of the per-node creation path.
+	 * Best-effort: if the cache cannot be allocated the root falls back to
+	 * the locked path.  Costs up to KERNFS_INO_BATCH - 1
+	 * reserved-but-unused IDs per possible CPU per opted-in root, held as
+	 * NULL IDR entries until the root's IDR is destroyed.  Must not be
+	 * combined with KERNFS_ROOT_SUPPORT_EXPORTOP: reserved IDs are
+	 * allocated inos with no node, which the fhandle by-id lookup must not
+	 * resolve.
+	 */
+	KERNFS_ROOT_INO_BATCH			= 0x0020,
 };
 
 /* type-specific structures for kernfs_node union members */
-- 
2.47.3

Re: [PATCH 2/3] driver core: Index class glue directories by parent kobject

[Andy Shevchenko]

On Fri, Sep 11, 2026 at 07:16:04PM +0200, Pavol Sakac wrote:
> get_device_parent() finds a parent's glue directory by walking the
> class's glue_dirs kset list under gdp_mutex, so one per parent, as
> vfio-dev needs per SR-IOV VF, is quadratic.
> 
> Index them by parent kobject in an rbtree embedded in the class's
> subsys_private, which dies with the kset list it indexes, so no
> per-entry class check is needed: two classes below one parent are told
> apart by tree selection. The key is a kobject because a parentless
> class device hangs off the shared "virtual" kobject, referenced by the
> glue dir while indexed. The rb_node lives in struct class_dir, adding
> no allocation and no failure mode.
> 
> gdp_mutex serializes the index, as it has glue dir lookup/create/remove
> since commit 77d3d7c1d561f
> ("driver-core: fix race condition in get_device_parent()") and
> commit e4a60d1390609
> ("sysfs: driver core: Fix glue dir race condition by gdp_mutex").

> A kernfs name lookup in the parent's directory needs no new state, but
> takes the kernfs root rwsem under gdp_mutex, behind the writes
> concurrent sysfs directory creation generates.
> 
> A KUnit suite covers the index: reuse of one parent's glue directory,
> two classes below one parent, the parentless "virtual" cases, reap and
> recreate, name collision, many parents, device_move(), and class
> teardown.

...

> +++ b/drivers/base/test/.kunitconfig
> @@ -1,2 +1,3 @@
>  CONFIG_KUNIT=y
>  CONFIG_DM_KUNIT_TEST=y
> +CONFIG_GLUE_DIR_KUNIT_TEST=y

Is this test case is so important that it must *always* run?

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH 1/3] kernfs: activate nodes while linking them

[Andy Shevchenko]

On Fri, Sep 11, 2026 at 07:16:03PM +0200, Pavol Sakac wrote:
> kernfs_add_one() links a new node, releases kernfs_rwsem, and takes it
> again through kernfs_activate(). A new node has no descendants, so the
> second hold only activates that node.
> 
> Activate ordinary nodes before releasing the linking hold, removing a
> second write-side acquisition and the linked-but-inactive interval.
> KERNFS_ROOT_CREATE_DEACTIVATED roots retain explicit subtree activation.
> 
> Removing the unlock-to-lock pair also removes its publication ordering.
> The in-tree lockless ID lookups use CREATE_DEACTIVATED roots and retain
> kernfs_activate(); unsynchronized callers cannot rely on observing a
> node.

...

> +static void kernfs_activate_one(struct kernfs_node *kn);
> +

Can we avoid adding forward declarations like this? Or is there circular
dependencies? If not, add another patch that simply moves the existing code
upper, so this patch won't need a forward declaration.

-- 
With Best Regards,
Andy Shevchenko