Re: [PATCH v6 07/31] vfio/pci: Detect CXL devices and load the CXL provider on demand

[Alex Williamson <alex@shazbot.org>]

On Fri, 16 Oct 2026 21:48:14 +0530
<mhonap@nvidia.com> wrote:

> From: Manish Honap <mhonap@nvidia.com>
> 
> A CXL device needs the vfio-cxl callbacks, but pulling vfio-cxl and the
> CXL core in unconditionally would bloat every vfio-pci setup. Detect a
> CXL Type-2 device at bind and request_module("vfio-cxl") only then, then
> hand the device to the registered ops.
> 
> Each bound CXL device pins the provider through
> vfio_pci_get_cxl_ops() (try_module_get) and drops it with
> vfio_pci_put_cxl_ops() at release, so vfio-cxl can unload once no CXL
> device is bound.
> 
> If the provider is absent the device is bound without the CXL
> extensions. A built-in provider whose initcall has not run yet is waited
> for with -EPROBE_DEFER; the deferred-probe machinery will retry the bind
> once the provider registers.
> 
> The provider .init() can return -EPROBE_DEFER when the device is not
> ready yet, for example when its CXL port has not enumerated. Propagate
> it so the bind is retried later. Any other init failure is non-fatal: a
> CXL Type-2 device is also a valid vfio-pci device, so log the error, drop
> the provider reference, leave cxl_ops NULL and continue without CXL
> support.
> 
> Assisted-by: LLM
> Signed-off-by: Manish Honap <mhonap@nvidia.com>
> ---
>  drivers/vfio/pci/vfio_pci_core.c | 101 +++++++++++++++++++++++++++++++
>  include/linux/vfio_pci_core.h    |   3 +
>  2 files changed, 104 insertions(+)
> 
> diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
> index 2cc5dd20396c..e50b36751d9a 100644
> --- a/drivers/vfio/pci/vfio_pci_core.c
> +++ b/drivers/vfio/pci/vfio_pci_core.c
> @@ -2202,6 +2202,101 @@ void vfio_pci_core_unregister_cxl_ops(const struct vfio_cxl_ops *ops)
>  }
>  EXPORT_SYMBOL_GPL(vfio_pci_core_unregister_cxl_ops);
>  
> +static const struct vfio_cxl_ops *vfio_pci_get_cxl_ops(void)
> +{
> +	guard(rwsem_read)(&vfio_pci_cxl_ops_rwsem);
> +
> +	if (vfio_pci_cxl_ops && try_module_get(vfio_pci_cxl_ops->owner))
> +		return vfio_pci_cxl_ops;
> +
> +	return NULL;
> +}
> +
> +static void vfio_pci_put_cxl_ops(const struct vfio_cxl_ops *ops)
> +{
> +	module_put(ops->owner);
> +}
> +
> +/*
> + * A CXL Type-2 device advertises both CXL.cache and CXL.mem in its CXL DVSEC.
> + * pcie_is_cxl() is also true for Type-1 (cache only) and Type-3 (mem only)
> + * devices, which the vfio-cxl provider does not handle, so confirm the Type-2
> + * identity before engaging it.
> + */

Probably another AI comment, it's only describing what the function is
obviously doing.  Drop it.

> +static bool vfio_pci_is_cxl_type2(struct pci_dev *pdev)
> +{
> +	u16 dvsec, cap;
> +
> +	if (!pcie_is_cxl(pdev))
> +		return false;
> +
> +	dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
> +					  PCI_DVSEC_CXL_DEVICE);
> +	if (!dvsec)
> +		return false;
> +
> +	if (pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap))
> +		return false;
> +
> +	return (cap & PCI_DVSEC_CXL_CACHE_CAPABLE) &&
> +		(cap & PCI_DVSEC_CXL_MEM_CAPABLE);
> +}
> +
> +/*
> + * Load vfio-cxl on demand for a CXL Type-2 device and hand the device to its
> + * ops. If the provider is absent the device is bound without the CXL
> + * extensions; a built-in provider whose initcall has not run yet is waited for with
> + * -EPROBE_DEFER.
> + */
> +static int vfio_pci_core_cxl_init(struct vfio_pci_core_device *vdev)
> +{
> +	const struct vfio_cxl_ops *ops;
> +	int ret;
> +
> +	if (!vfio_pci_is_cxl_type2(vdev->pdev))
> +		return 0;
> +
> +	request_module("vfio-cxl");

A local sashiko run questions:
---
[Severity: High]
Could request_module() in vfio_pci_core_cxl_init(), in
 drivers/vfio/pci/vfio_pci_core.c, deadlock when a matching Type-2 device
is probed asynchronously and vfio-cxl is an unloaded module whose own
async_probe setting remains false?

Asynchronous probing can be selected with vfio-pci's async_probe=1 module
argument or driver_async_probe=vfio-pci. The resulting wait cycle is:

    __driver_attach_async_helper()
        vfio_pci_core_cxl_init()
            request_module("vfio-cxl")
                waits for modprobe to exit

    do_init_module() for vfio-cxl
        async_synchronize_full()
            waits for the outstanding vfio-pci probe

In kernel/module/main.c, do_init_module() performs this synchronization
after the provider initializer returns. Successfully registering the
operations therefore does not break the cycle. The provider's async_probe
setting is independent of vfio-pci's setting.

The warning in __request_module(), in kernel/module/kmod.c, does not
abort loading. Dispatch through pci_call_probe(), in
 drivers/pci/pci-driver.c, also preserves the cycle because the originating
async callback waits in flush_work(). The global async wait has no timeout.

The final series retains this synchronous request. Could provider loading
be moved outside the outstanding async probe, with binding deferred until
the operations are available?
---

AIUI, this could be resolved if we do the request_module() on bind and
the CXL init on open, but CXL expects that we use the devm interfaces
that have a bind->release lifetime.  But then...

> +	ops = vfio_pci_get_cxl_ops();
> +	if (!ops)
> +		return IS_BUILTIN(CONFIG_VFIO_CXL) ? -EPROBE_DEFER : 0;
> +
> +	ret = ops->init(vdev);
> +	/*
> +	 * A provider that is not ready yet (for example its CXL port has not
> +	 * enumerated) returns -EPROBE_DEFER. Propagate it so the bind retries
> +	 * rather than continuing without CXL support.
> +	 */
> +	if (ret == -EPROBE_DEFER) {
> +		vfio_pci_put_cxl_ops(ops);
> +		return ret;
> +	}
> +	/*
> +	 * Any other failure is non-fatal: a CXL device that cannot be brought
> +	 * up as Type-2 still works without the CXL extensions, so log and
> +	 * continue rather than failing the bind.
> +	 */
> +	if (ret) {
> +		pci_warn(vdev->pdev,
> +			 "CXL init failed (%d), continuing without CXL support\n",
> +			 ret);
> +		vfio_pci_put_cxl_ops(ops);
> +		return 0;
> +	}
> +
> +	vdev->cxl_ops = ops;
> +	return 0;
> +}
> +
> +static void vfio_pci_core_cxl_release(struct vfio_pci_core_device *vdev)
> +{
> +	if (!vdev->cxl_ops)
> +		return;
> +
> +	vdev->cxl_ops->release(vdev);
> +	vfio_pci_put_cxl_ops(vdev->cxl_ops);
> +}
> +
>  int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
>  {
>  	struct vfio_pci_core_device *vdev =
> @@ -2223,6 +2318,10 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
>  	init_rwsem(&vdev->memory_lock);
>  	xa_init(&vdev->ctx);
>  
> +	ret = vfio_pci_core_cxl_init(vdev);
> +	if (ret)
> +		return ret;
> +
>  	return 0;
>  }
>  EXPORT_SYMBOL_GPL(vfio_pci_core_init_dev);
> @@ -2232,6 +2331,8 @@ void vfio_pci_core_release_dev(struct vfio_device *core_vdev)
>  	struct vfio_pci_core_device *vdev =
>  		container_of(core_vdev, struct vfio_pci_core_device, vdev);
>  
> +	vfio_pci_core_cxl_release(vdev);

Sashiko has a doozy regarding the resulting lifecycle teardown:

---
[Severity: Critical]
Can vfio_pci_core_release_dev(), in drivers/vfio/pci/vfio_pci_core.c,
invoke provider teardown after device-managed cleanup has freed vdev->cxl
when final object destruction is delayed during unbind?

In the final series, vfio_cxl_create_memdev(), in
 drivers/vfio/pci/cxl/vfio_cxl_core.c, allocates the provider state through
devm_cxl_dev_state_create() against pdev->dev. This release hook instead
runs at final vfio object destruction, which can follow driver removal
and device-managed cleanup.

Device-cdev close and unbind can interleave as follows:

    vfio_device_fops_release()
        drops the last registration reference
        completes the unregister wait

    vfio_pci_remove()
        finishes unregistering
        drops its vfio device reference
        returns while the cdev still retains the vfio object

    device_unbind_cleanup()
        devres_release_all() frees the provider state

    __fput()
        cdev_put() drops the remaining parent reference
        vfio_pci_core_release_dev() calls provider teardown

Enabling delayed kobject release provides another path without an open
file. The module pin preserves the provider callback text, but does not
preserve its device-managed state.

vfio_cxl_release_device(), in drivers/vfio/pci/cxl/vfio_cxl_core.c, then
reads the freed cxl->hpa_range and cxl->hdm to release the reservation and
free the decoder snapshot. Reuse of that allocation can cause an incorrect
reservation release or an invalid free. The final series retains both the
managed allocation and this final-release hook.

Could provider teardown run before device-managed cleanup, or could the
provider state remain allocated through final vfio release?
---

So perhaps trying to shoehorn CXL's devm lifecycle into vfio-pci is
biting us anyway.  Thanks,

Alex