[PATCH v6 00/31] vfio/pci: Add CXL Type-2 device passthrough support

[<mhonap@nvidia.com>]

From: Manish Honap <mhonap@nvidia.com>

Hello CXL Community,

This series adds VFIO passthrough for CXL Type-2 accelerators. The guest
drives its own virtual HDM decoder and can reset the device. The host
owns the physical decoder and the host physical address the memory lands
at. The guest only picks a guest physical address. This series targets a
single, non-interleaved, firmware-committed endpoint decoder.


Base and dependencies
---------------------

  Base: Linux 7.3-rc5 (72d3fcf802c4), plus:
  - Richard's zero-sized HDM decoder series (v10) [1].
  - Srirangan's cxl_reset series (v15) [2]. It caches the endpoint
    decoder settings in pdev->hdm at enumeration and implements CXL Reset
    as a PCI reset method in drivers/pci/cxl.c.

  Patch 4 of this series changes drivers/pci/cxl.c from [2]. v15 exports
  no reset entry that a module can call, and its reset method falls
  through to FLR when it cannot run, so patch 4 adds an entry for a
  driver that holds the HDM range and exports the helpers vfio-cxl needs.

  Runtime requirement on arm64: CXL Reset invalidates the CPU caches for
  the HDM range, which needs a registered cache maintenance driver. The
  series was tested with the SMCCC cache clean/invalidate provider (v3)
  [3] and firmware that implements the SMCCC cache operations; v4 [4] has
  since been posted. This is not a build dependency. Without a provider
  the device binds as plain vfio-pci, without the CXL extensions.


Changes since v5
----------------

v5 [5] kept the v4 provider model. v6 keeps it too and folds the v5
review, the rebase onto cxl_reset v15, and fixes found by testing on
hardware.

Base and dependency
  - Rebased from cxl_reset v12 onto v15. v15 moved the reset into
    drivers/pci/cxl.c, so the v5 patch that added
    cxl_reset_dvsec_sequence() in drivers/cxl/core/resource.c is replaced
    by patch 4: pci_cxl_reset_owned() runs the reset for the driver that
    holds the HDM range and returns its failure to that driver. A reset
    that cannot run fails instead of falling through to FLR.
  - Bind requires that the host can run CXL Reset for the device,
    including the CPU cache invalidation. Otherwise the device binds
    without CXL support, so no guest gets device memory that the host
    cannot clear on the next reset.

Structural
  - Patches 07 and 08 are folded into 06 (now patch 7).
  - The excluded-range patches 09 and 10 are now three: per-BAR sorted
    lists with the read/write path and MSI-X moved onto them, the sparse
    mmap capability, and refusing ioeventfds and dma-buf exports over an
    excluded range.
  - The HDM region patch is split into read/write (21) and mmap (22).
  - New vfio-pci-core patches: one try-reset helper for every reset point
    (15), revoke and re-arm of one device-specific region with its
    dma-bufs (16), and provider hooks across low power (17).
  - New PCI patch: bus and slot reset are refused below a CXL port with
    Secondary Bus Reset (SBR) masked (5).

Behavioral
  - Low power is supported and the D0 pin is gone. vfio-cxl takes the
    HDM region out of use before the device leaves D0 and restores the
    decoder when it returns.
  - The HDM region and its dma-bufs follow the committed decoder, not
    Memory Space Enable. vfio-pci-core no longer knows about HDM: each
    dma-buf records the region it was exported from, and vfio-cxl revokes
    and re-arms its own region around reset, hot reset and low power.
  - With SBR masked, VFIO_DEVICE_PCI_HOT_RESET fails instead of reporting
    a reset that never reached the device. With SBR unmasked, reset_done
    restores the decoder and the ioctl fails if the restore fails.
  - A failed CXL Reset returns its error and leaves the HDM region out of
    use until a later CXL Reset succeeds. A hot reset or a return from
    low power in between does not reopen it. The first reset at open
    fails the open only on -EAGAIN.
  - AER and DPC recovery take the HDM region out of use, and put it back
    only if it was in use before the error and the decoder is still
    committed.
  - Reads of the decoder register region fail with -EIO while Memory
    Space is disabled, as BAR reads do.
  - HDM region read and write copy through a page-sized kernel buffer, so
    memory_lock is not held across a user copy.
  - The guest's Initiate CXL Reset doorbell runs the real reset and
    reports Reset Complete or Reset Error in the shadow Status2.
  - Without a provider, vfio-pci still guards the CXL DVSEC of a Type-2
    function: Lock and Range writes are dropped, and Initiate CXL Reset
    runs as a vfio reset.
  - The CXL DVSEC is matched by offset range, so a DVSEC right before or
    after it in config space no longer bypasses the virtualization or
    indexes past the permission map.
  - CXL specification references are updated to CXL r4.0.

Found in hardware testing
    vfio-cxl now waits for Memory_Active, up to the Memory_Active_Timeout
    the device reports, before it puts the HDM region back. It also waits
    at bind, since a reset just before the bind clears it too.
  - The bind-time copy of the HDM state used a helper that asserts the
    device lock. The PCI core can run probe in a worker on the device's
    NUMA node, so lockdep warned at every bind. Patch 4 adds
    pci_cxl_hdm_snapshot(), which needs no device lock.
  - The disable_cxl module parameter was never read. It now takes effect.


Reviewer feedback addressed
---------------------------

The v5 [5] thread has the full discussion. This is where each comment
landed.

Alex Williamson
  - Provider load and its failure handling are one patch (7), and the
    messages say "continuing without CXL support".
  - The excluded-range list is per BAR, sorted, and rejects overlaps; the
    ROM is left alone; ioeventfds and dma-buf exports over an excluded
    range are refused (8 to 10).
  - cxl_dvsec_len had no user in v5 (11). It now has one: the CXL DVSEC
    is matched by offset range.
  - vfio-pci exports vfio_pci_core_reset_prepare() and
    vfio_pci_core_reset_done(), and calls them from its own error
    handlers instead of reaching into drvdata (13).
  - Low power is supported instead of pinning D0 (17, 21).
  - vfio_pci_core_unregister_dev_region() is gone. vfio-pci-core frees
    every dynamic region on a failed open as well as on close (12), so
    vfio-cxl has nothing to unwind.
  - hdm_valid is set at bind and changed only by reset, hot reset and
    low power.
  - vfio_pci_zap_bars() covers only the BARs again; a device-specific
    region is revoked by its provider (16).
  - The decoder register read takes memory_lock and checks Memory Space
    (24).
  - Hot reset with SBR masked and unmasked (5, 26).
  - Reset is a mandatory provider op behind one try-reset helper (15),
    with a success-oriented flow in vfio-cxl (29). .hdm_active is gone.

Jonathan Cameron
  - Patch 1 keeps the Reviewed-by tag.
  - A BAR-owning driver records one owned resource; a second call warns
    and fails with -EBUSY (2).
  - The uapi header uses __GENMASK() and _BITUL() (3).
  - CXL Reset writes back and invalidates the CPU caches for the HDM
    range before and after the reset (4).
  - Allocator and style changes in 9, 11 and 12, and the devres group
    factored into a helper (19).

Richard Cheng
  - The provider module reference is pinned per bound device (7).
  - ioeventfds and dma-buf exports over an excluded range are refused
    (10).
  - A dma-buf of the HDM region is exported with no_peer2peer set (28).
  - The PCI state is saved before the reset at open (15).

Gregory Price
  - The documentation is reworked (30) as per review comments.


Patch order
-----------

The tree builds at every commit. Five groups:

  Part 1  CXL core and PCI (patches 1 to 5)
    - Split the BAR block request and ioremap helpers, and let a
      BAR-owning driver own the component register block, so the HDM
      and RAS sub-blocks are left for vfio-cxl.
    - Move the component register defines to include/uapi/cxl/cxl_regs.h
      so a VMM and the selftest can use them.
    - Let the driver that owns the HDM range run CXL Reset.
    - Refuse bus and slot reset below a CXL port with SBR masked.

  Part 2  vfio-pci-core (patches 6 to 17)
    - The CXL provider ops registration and on-demand provider load.
    - The generic excluded-range list, its sparse mmap capability, and
      the ioeventfd and dma-buf checks.
    - The CXL DVSEC virtualization, the open and close hooks, the reset
      hooks, and the disable_cxl opt-out.
    - The try-reset helper, per-region revoke and re-arm, and the low
      power hooks.

  Part 3  vfio-cxl provider and HDM regions (patches 18 to 28)
    - The vfio-cxl module, the CXL memdev created at bind, and ownership
      of the whole component BAR.
    - The HDM memory region (read/write, then mmap), memory-failure
      containment for it, and the read-only decoder register region.
    - Excluding the decoder block from the direct BAR, the decoder
      restore after a hot reset, the device and decoder geometry
      capability, and the dma-buf export.

  Part 4  vfio-cxl reset (patch 29)
    - Run CXL Reset at every vfio reset point.

  Part 5  Documentation and selftests (patches 30 and 31)


Subsystem boundary
------------------

vfio-pci-core does not implement CXL registers. vfio-cxl is a separate
module that registers a struct vfio_cxl_ops at init. vfio-pci-core loads
it on demand for a CXL device (request_module() after pcie_is_cxl()) and
pins it per bound device. If a modular provider is missing or fails to
load, the device is driven as plain vfio-pci. The bind only defers for a
built-in provider whose initcall has not run yet.

vfio-pci-core provides generic pieces that a provider calls: the
excluded-range list, revoke and re-arm of one device-specific region,
the try-reset helper and the low power hooks. The CXL-specific work stays
behind the ops. The CXL DVSEC virtualization is the one CXL piece in
vfio-pci-core, because it lives in the config space permission hooks.


Memory ownership
----------------

The host resolves the host physical address once, at bind, from the
committed decoder. vfio-cxl creates the memdev at bind and tears it down
at unbind. It claims the HPA range with request_mem_region_exclusive(),
so nothing, /dev/mem included, can map a conflicting cacheable alias.

The guest programs a guest physical address into a virtual decoder that
the VMM emulates. It never reaches the physical decoder registers: those
are read through a read-only region and are excluded from the direct BAR
mapping, so the guest cannot move the host physical window.


HDM region access
-----------------

The HDM region carries the coherent device memory. A VMM mmaps it and
maps it into the guest through stage-2. The fault handler inserts the
host PFNs, including PMD-sized mappings when the mapping is aligned, only
while the committed decoder is in place and the media is ready.
Otherwise the fault gets SIGBUS. Region read and write check the same
state and return -EIO while the region is out of use. The range has no
struct page and is registered with the memory-failure machinery, so a
memory error is contained to a SIGBUS.


DMA and iommufd
---------------

Some accelerators, for example a GPU, send ATS-translated DMA to their
own HDM, so that range must be in the guest IOAS. IOMMU_IOAS_MAP cannot
pin a range with no struct page. VFIO_DEVICE_FEATURE_DMA_BUF on the HDM
region returns a dma-buf fd that iommufd maps with IOMMU_IOAS_MAP_FILE.
vfio-cxl revokes the dma-buf whenever it takes the region out of use.
iommufd then unmaps the range, and the VMM maps the same dma-buf again
once the region is back.


Reset
-----

An FLR does not reset CXL.cache or CXL.mem state and does not clear
device memory (CXL r4.0 sec 9.5), so it cannot separate one user of the
device from the next. vfio-cxl runs CXL Reset with Memory Clear at every
vfio reset point: open and close, VFIO_DEVICE_RESET, a guest FLR through
config space, and the guest's Initiate CXL Reset doorbell. Under
memory_lock, with the HDM mmap zapped and the dma-buf revoked, the host
invalidates the CPU caches for the HDM range, resets the function,
restores the decoder and the PCI config state, and invalidates the
caches again. vfio-cxl then waits for Memory_Active before it puts the
region back.

A CXL port masks SBR by default (CXL r4.0 sec 8.1.5.2). The PCI core now
refuses a bus or slot reset below such a port, so the vfio hot reset
ioctls fail and a VMM uses VFIO_DEVICE_RESET. If the port has SBR
unmasked, the hot reset decommits the decoder, and vfio-cxl restores it
in reset_done with cxl_restore_state_after_pci_reset(), the helper the
PCI core's cxl_bus reset method uses.


Low power
---------

The device may enter D3hot or runtime low power while it is assigned.
Low power loses the committed decoder, so vfio-cxl takes the HDM region
out of use first and restores the decoder when the device is back in D0.
Device memory is not saved or restored across low power.


Deferred
--------

  - Switched, multi-decoder and interleaved topologies stay rejected at
    bind.
  - Decoders that firmware did not commit are not supported.


Validation
----------

  - Each patch builds with W=1 (drivers/cxl, drivers/pci and
    drivers/vfio/pci, x86_64, VFIO_CXL=m).
  - scripts/checkpatch.pl --strict --codespell reports no errors or
    warnings. Patch 16 has one CHECK, a missing blank line between two
    inline stubs, kept to match the other stubs in vfio_pci_priv.h.
  - The selftest, tools/testing/selftests/vfio/vfio_cxl_type2_test.c, has
    27 tests: the device flag and regions, the HDM mmap and huge fault,
    the decoder register region and its -EIO gate, the dma-buf map with
    IOMMU_IOAS_MAP_FILE, region read and write across pages, the DVSEC
    reset doorbell, and the decoder after a D3hot cycle. All 27 pass on
    an arm64 host with 64K pages and lockdep (PROVE_LOCKING) enabled.
  - A guest with the device assigned booted, ran CUDA and unified memory
    workloads, and reset the device twice through guest CXL Reset through
    guest sysfs reset file and by using the CTRL2_INIT_CXL_RST in the
    guest. (used with QEMU vfio-cxl support v2 [6]).


AI assistance disclosure
------------------------

This series was developed with substantial AI assistance (Anthropic
Claude, through the Claude Code CLI). It was used for design trade-offs,
multi-agent regression and adversarial review of the v5 feedback using
review-core.md, the port to cxl_reset v15, analysis of the hardware
failures, and drafting the selftests and the documentation.
Following Documentation/process/coding-assistants.rst, each patch
carries an Assisted-by: LLM tag. I ran the tests on hardware, have
reviewed every change, and take full DCO responsibility for the series.


References
----------

[1] [PATCH v10 0/3] Support zero-sized HDM decoders
https://lore.kernel.org/all/20260914090858.19181-1-icheng@nvidia.com/

[2] [PATCH v15 00/16] PCI/CXL: Add CXL reset support for Type 2 devices
https://lore.kernel.org/all/20261011021422.3428136-1-smadhavan@nvidia.com/

[3] [PATCH v3 0/2] Add SMCCC cache clean/invalidate provider
https://lore.kernel.org/all/20260910082142.1522865-1-smadhavan@nvidia.com/

[4] [PATCH v4 0/2] Add SMCCC cache clean/invalidate provider
https://lore.kernel.org/all/20261013070205.3566865-1-smadhavan@nvidia.com/

[5] [PATCH v5 0/27] vfio/pci: Add CXL Type-2 device passthrough support
https://lore.kernel.org/all/20260916183540.3813685-1-mhonap@nvidia.com/

[6] [PATCH v2 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci
https://lore.kernel.org/linux-cxl/20260916184412.3825713-1-mhonap@nvidia.com/

Manish Honap (31):
  cxl/regs: Split the BAR block request and ioremap helpers
  cxl/regs: Let a BAR-owning driver own the component register block
  cxl: Move component register defines to uapi/cxl/cxl_regs.h
  PCI/CXL: Let the driver that owns the HDM range run CXL Reset
  PCI: Refuse bus and slot reset below a CXL port with SBR masked
  vfio/pci: Add the CXL provider ops registration interface
  vfio/pci: Detect CXL devices and load the CXL provider on demand
  vfio/pci: Add a generic excluded-range list
  vfio/pci: Advertise the mmap areas around excluded ranges
  vfio/pci: Refuse ioeventfds and dma-buf exports over excluded ranges
  vfio/pci: Virtualize the CXL DVSEC in vfio_pci_config.c
  vfio/pci: Call the CXL open and close hooks around device use
  vfio/pci: Bracket PCI resets with the CXL reset hooks
  vfio/pci: Provide an opt-out for the CXL Type-2 extensions
  vfio/pci: Route every reset point through a try-reset helper
  vfio/pci: Revoke and re-arm one device-specific region
  vfio/pci: Call the device provider across low-power transitions
  vfio/cxl: Add the vfio-cxl provider module skeleton
  vfio/cxl: Create the CXL memdev and set media ready at bind
  vfio/cxl: Own the whole component register BAR
  vfio/cxl: Expose the HDM memory region to the guest
  vfio/cxl: Allow mmap of the HDM memory region
  vfio/cxl: Contain HDM memory errors with memory_failure()
  vfio/cxl: Expose the HDM decoder registers read-only to the guest
  vfio/cxl: Exclude the HDM decoder registers from direct BAR access
  vfio/cxl: Restore the HDM decoder after a hot reset
  vfio/cxl: Describe the CXL device and decoder geometry to userspace
  vfio/cxl: Export the HDM memory region as a dma-buf
  vfio/cxl: Run the CXL reset at the vfio reset points
  Documentation: vfio-pci: Document CXL Type-2 device passthrough
  selftests/vfio: Add CXL Type-2 passthrough tests

 Documentation/driver-api/index.rst            |    1 +
 Documentation/driver-api/vfio-pci-cxl.rst     |  312 +++++
 MAINTAINERS                                   |   10 +
 drivers/cxl/core/hdm_state.c                  |   22 +-
 drivers/cxl/core/regs.c                       |   64 +-
 drivers/cxl/cxl.h                             |   47 +-
 drivers/pci/cxl.c                             |  136 +-
 drivers/pci/pci.c                             |   31 +-
 drivers/pci/pci.h                             |   18 +-
 drivers/vfio/pci/Kconfig                      |    2 +
 drivers/vfio/pci/Makefile                     |    2 +
 drivers/vfio/pci/cxl/Kconfig                  |   11 +
 drivers/vfio/pci/cxl/Makefile                 |    3 +
 drivers/vfio/pci/cxl/vfio_cxl_core.c          | 1124 +++++++++++++++++
 drivers/vfio/pci/vfio_pci.c                   |   52 +-
 drivers/vfio/pci/vfio_pci_config.c            |  225 +++-
 drivers/vfio/pci/vfio_pci_core.c              |  747 ++++++++++-
 drivers/vfio/pci/vfio_pci_dmabuf.c            |  183 ++-
 drivers/vfio/pci/vfio_pci_priv.h              |   20 +
 drivers/vfio/pci/vfio_pci_rdwr.c              |   76 +-
 include/cxl/cxl.h                             |    2 +
 include/cxl/hdm.h                             |    3 +-
 include/cxl/pci.h                             |   42 +
 include/linux/vfio_pci_core.h                 |   92 ++
 include/uapi/cxl/cxl_regs.h                   |   56 +
 include/uapi/linux/vfio.h                     |   24 +
 tools/testing/selftests/vfio/Makefile         |    1 +
 .../vfio/lib/include/libvfio/iommu.h          |    3 +
 tools/testing/selftests/vfio/lib/iommu.c      |   29 +
 .../selftests/vfio/lib/vfio_pci_device.c      |   57 +-
 .../selftests/vfio/vfio_cxl_type2_test.c      | 1095 ++++++++++++++++
 31 files changed, 4306 insertions(+), 184 deletions(-)
 create mode 100644 Documentation/driver-api/vfio-pci-cxl.rst
 create mode 100644 drivers/vfio/pci/cxl/Kconfig
 create mode 100644 drivers/vfio/pci/cxl/Makefile
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_core.c
 create mode 100644 include/uapi/cxl/cxl_regs.h
 create mode 100644 tools/testing/selftests/vfio/vfio_cxl_type2_test.c

-- 
2.53.0