Re: [RFC PATCH 00/14] driver core: defer per-VF sysfs creation for fast SR-IOV bring-up

[Greg Kroah-Hartman <gregkh@linuxfoundation.org>]

On Thu, Jul 02, 2026 at 07:40:19PM +0200, Pavol Sakac wrote:
> Virtualization hardware keeps increasing in CPU-core and VF density.
> Kernel is trending towards preserving VFs using LUO across a kexec
> which will put SR-IOV in live-update hotpath. VFs must be re-added to
> the sysfs tree, along with its supporting devices (e.g. vfio,
> iommu_groups). On production hardware a single VF can consume ~80 kernfs
> nodes. With thousands of VFs on modern hardware, hundreds of thousands
> of kernfs nodes need to be created, which can add 100ms+ to the boot
> time and to guest downtime.

Hundreds of thousands?  You have a hundred sysfs files per device?

> Proposed here is a PoC of deferred materialization of sysfs files until
> first access to avoid cost of kernfs node creation in hot path. In the
> reproducer below using a synthetic VF to isolate the sysfs overhead,
> the per-device sysfs-creation time is reduced by ~74%. It exploits the
> expected access pattern of hypervisors, where only a subset of device
> files need to be accessed to attach a VF to a VM on the hot path.

Ick, that access pattern could change with other users, this feels very
fragile.

> To allow lazy init scheme, a minimal set of files and directories is
> built per device: a device directory resolves directly to the device's
> kobject and is used to materialize attributes at access time. The
> access is trapped in kernfs at the dir-read or path-walk stage and
> depending on the trigger, either the full directory or a single file is
> materialized.
> 
> The changes are two logically distinct parts:
> 
>   1. Refactor of hardcoded device_add / iommu_group sysfs attribute
>      creation to a table-driven form to allow walkability of all
>      attributes and single point of definition for attributes
>      regardless of lazy opt-in to avoid duplicate definitions and
>      associated maintainability issues with it.

This is probably a good idea anyway.  Want to just send this as a series
to start with, so we can take that?

>   2. Lazy-init infra to front-run access to device files at lookup/
>      readdir time to trigger materialization. Opt-in per device: PCI VFs
>      and VFIO devices opt in via device_set_sysfs_lazy(); iommu_group
>      opts in at the kobject level.

This is going to get complex.  Where exactly is the bottleneck?  I'm
guessing that systems with that many devices also have many hundreds of
CPUs, right?  Are we hitting a single lock somewhere?  There have been
changes in kernfs in the past to split up the locks to be more
fine-grained, perhaps that needs to continue to solve this?

> As a PoC, the code does not yet follow this logical structure in the
> commit sequence and likely does not yet cover all corner cases.
> 
> Synthetic reproducer measurements
> ---------------------------------
>   devices   per-dev time   total time       kernfs nodes (base -> lazy)
>   -------   ------------   --------------   ----------------------------
>      500    31.8->8.3 us    15.9-> 4.1 ms     36,417 ->  3,922  (72.8->7.8/dev)
>     1000    33.2->8.7 us    33.2-> 8.7 ms     73,427 ->  8,432  (73.4->8.4/dev)
>     2500    32.8->9.1 us    82.0->22.7 ms    184,421 -> 21,925  (73.8->8.8/dev)
>     5000    33.0->8.7 us   165.1->43.5 ms    369,401 -> 44,406  (73.9->8.9/dev)
> 
> The reproducer is synthetic: an in-kernel module registers N unbound
> platform devices (60 attributes / 4 groups, ~74 kernfs nodes each; no
> config space, BARs, or probe) in one timed burst that models SR-IOV
> enablement. It runs as a two-kernel A/B from one vanilla 7.1-rc2 tree
> (eager baseline vs patched, plus a patched/opt-in-OFF arm), 20 runs per
> point. Measures only sysfs-creation slice.

I sure hope sr-iov devices are NOT platform devices.  If so, please go
fix that up first :)

> Deferral removes ~74% of the sysfs-creation time and ~88% of the kernfs
> nodes. At 5000 devices that is 121.6 ms removed (165.1 -> 43.5 ms) and
> ~325,000 fewer kernfs nodes (369,401 -> 44,406). The table-driven rework
> adds a +7..+11% eager-path time overhead to non-opted devices.

You're just deferring this work to happen later, so when is that
"later"?  And what about multi-threaded probing of the bus, doesn't that
speed stuff up too?

> Available as a docker image that orchestrates builds in qemu guest and
> prints results as a table (x86_64 Linux host with /dev/kvm assumed):
> 
>    docker run --device /dev/kvm ghcr.io/pavsa/linux-lazy-sysfs-vf-bench:7.1-rc2
> 
> Assumptions / Trade-offs
> ------------------------
> The saving depends on the access pattern of hypervisors, and on no
> component walking the whole device tree on the hot path to avoid full
> materialization. Both are expected to hold on optimized hypervisors.

Hah, as if.  Many userspace tools, once a device is notified, walk the
whole tree of it to read the attributes.  You are going to be fighting
that problem constantly...

> An actual production hypervisor was observed in local testing to
> materialize an additional 5-10% of total nodes on average on the hot
> path.
> 
> Overhead: +7..+11% eager-path time overhead applies to every non-opted
> device going through device_add() as all struct devices traverse the
> walker and can't be gated under a config option.
> The lazy opt-in infra can be gated under config option to have zero
> impact if disabled.

Again, where exactly is the time being spent?  What lock is "hot"?

> Final words
> -----------
> This is an RFC to first get feedback on the decisions taken at driver
> core + kernfs level before engaging maintainers of other subsystems
> if we want to pursue this further or pivot to something else.
> 
> The code is at a proof-of-concept quality written with AI assistance
> with some known limitations with main target to gather potential
> savings and initiate conversation.
> 
> Feedback appreciated for:
> - table-driven attribute refactor targeting common device add path /
>   suggestions for more efficient/less invasive approach

What's wrong with the normal way attributes are defined and allocated?
They should all be tables today, and the driver core creating them when
needed, no individual driver should ever be doing that on its own.

thanks,

greg k-h