[PATCH v6 00/31] vfio/pci: Add CXL Type-2 device passthrough support

[PATCH v6 00/31] vfio/pci: Add CXL Type-2 device passthrough support

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Hello CXL Community,

This series adds VFIO passthrough for CXL Type-2 accelerators. The guest
drives its own virtual HDM decoder and can reset the device. The host
owns the physical decoder and the host physical address the memory lands
at. The guest only picks a guest physical address. This series targets a
single, non-interleaved, firmware-committed endpoint decoder.


Base and dependencies
---------------------

  Base: Linux 7.3-rc5 (72d3fcf802c4), plus:
  - Richard's zero-sized HDM decoder series (v10) [1].
  - Srirangan's cxl_reset series (v15) [2]. It caches the endpoint
    decoder settings in pdev->hdm at enumeration and implements CXL Reset
    as a PCI reset method in drivers/pci/cxl.c.

  Patch 4 of this series changes drivers/pci/cxl.c from [2]. v15 exports
  no reset entry that a module can call, and its reset method falls
  through to FLR when it cannot run, so patch 4 adds an entry for a
  driver that holds the HDM range and exports the helpers vfio-cxl needs.

  Runtime requirement on arm64: CXL Reset invalidates the CPU caches for
  the HDM range, which needs a registered cache maintenance driver. The
  series was tested with the SMCCC cache clean/invalidate provider (v3)
  [3] and firmware that implements the SMCCC cache operations; v4 [4] has
  since been posted. This is not a build dependency. Without a provider
  the device binds as plain vfio-pci, without the CXL extensions.


Changes since v5
----------------

v5 [5] kept the v4 provider model. v6 keeps it too and folds the v5
review, the rebase onto cxl_reset v15, and fixes found by testing on
hardware.

Base and dependency
  - Rebased from cxl_reset v12 onto v15. v15 moved the reset into
    drivers/pci/cxl.c, so the v5 patch that added
    cxl_reset_dvsec_sequence() in drivers/cxl/core/resource.c is replaced
    by patch 4: pci_cxl_reset_owned() runs the reset for the driver that
    holds the HDM range and returns its failure to that driver. A reset
    that cannot run fails instead of falling through to FLR.
  - Bind requires that the host can run CXL Reset for the device,
    including the CPU cache invalidation. Otherwise the device binds
    without CXL support, so no guest gets device memory that the host
    cannot clear on the next reset.

Structural
  - Patches 07 and 08 are folded into 06 (now patch 7).
  - The excluded-range patches 09 and 10 are now three: per-BAR sorted
    lists with the read/write path and MSI-X moved onto them, the sparse
    mmap capability, and refusing ioeventfds and dma-buf exports over an
    excluded range.
  - The HDM region patch is split into read/write (21) and mmap (22).
  - New vfio-pci-core patches: one try-reset helper for every reset point
    (15), revoke and re-arm of one device-specific region with its
    dma-bufs (16), and provider hooks across low power (17).
  - New PCI patch: bus and slot reset are refused below a CXL port with
    Secondary Bus Reset (SBR) masked (5).

Behavioral
  - Low power is supported and the D0 pin is gone. vfio-cxl takes the
    HDM region out of use before the device leaves D0 and restores the
    decoder when it returns.
  - The HDM region and its dma-bufs follow the committed decoder, not
    Memory Space Enable. vfio-pci-core no longer knows about HDM: each
    dma-buf records the region it was exported from, and vfio-cxl revokes
    and re-arms its own region around reset, hot reset and low power.
  - With SBR masked, VFIO_DEVICE_PCI_HOT_RESET fails instead of reporting
    a reset that never reached the device. With SBR unmasked, reset_done
    restores the decoder and the ioctl fails if the restore fails.
  - A failed CXL Reset returns its error and leaves the HDM region out of
    use until a later CXL Reset succeeds. A hot reset or a return from
    low power in between does not reopen it. The first reset at open
    fails the open only on -EAGAIN.
  - AER and DPC recovery take the HDM region out of use, and put it back
    only if it was in use before the error and the decoder is still
    committed.
  - Reads of the decoder register region fail with -EIO while Memory
    Space is disabled, as BAR reads do.
  - HDM region read and write copy through a page-sized kernel buffer, so
    memory_lock is not held across a user copy.
  - The guest's Initiate CXL Reset doorbell runs the real reset and
    reports Reset Complete or Reset Error in the shadow Status2.
  - Without a provider, vfio-pci still guards the CXL DVSEC of a Type-2
    function: Lock and Range writes are dropped, and Initiate CXL Reset
    runs as a vfio reset.
  - The CXL DVSEC is matched by offset range, so a DVSEC right before or
    after it in config space no longer bypasses the virtualization or
    indexes past the permission map.
  - CXL specification references are updated to CXL r4.0.

Found in hardware testing
    vfio-cxl now waits for Memory_Active, up to the Memory_Active_Timeout
    the device reports, before it puts the HDM region back. It also waits
    at bind, since a reset just before the bind clears it too.
  - The bind-time copy of the HDM state used a helper that asserts the
    device lock. The PCI core can run probe in a worker on the device's
    NUMA node, so lockdep warned at every bind. Patch 4 adds
    pci_cxl_hdm_snapshot(), which needs no device lock.
  - The disable_cxl module parameter was never read. It now takes effect.


Reviewer feedback addressed
---------------------------

The v5 [5] thread has the full discussion. This is where each comment
landed.

Alex Williamson
  - Provider load and its failure handling are one patch (7), and the
    messages say "continuing without CXL support".
  - The excluded-range list is per BAR, sorted, and rejects overlaps; the
    ROM is left alone; ioeventfds and dma-buf exports over an excluded
    range are refused (8 to 10).
  - cxl_dvsec_len had no user in v5 (11). It now has one: the CXL DVSEC
    is matched by offset range.
  - vfio-pci exports vfio_pci_core_reset_prepare() and
    vfio_pci_core_reset_done(), and calls them from its own error
    handlers instead of reaching into drvdata (13).
  - Low power is supported instead of pinning D0 (17, 21).
  - vfio_pci_core_unregister_dev_region() is gone. vfio-pci-core frees
    every dynamic region on a failed open as well as on close (12), so
    vfio-cxl has nothing to unwind.
  - hdm_valid is set at bind and changed only by reset, hot reset and
    low power.
  - vfio_pci_zap_bars() covers only the BARs again; a device-specific
    region is revoked by its provider (16).
  - The decoder register read takes memory_lock and checks Memory Space
    (24).
  - Hot reset with SBR masked and unmasked (5, 26).
  - Reset is a mandatory provider op behind one try-reset helper (15),
    with a success-oriented flow in vfio-cxl (29). .hdm_active is gone.

Jonathan Cameron
  - Patch 1 keeps the Reviewed-by tag.
  - A BAR-owning driver records one owned resource; a second call warns
    and fails with -EBUSY (2).
  - The uapi header uses __GENMASK() and _BITUL() (3).
  - CXL Reset writes back and invalidates the CPU caches for the HDM
    range before and after the reset (4).
  - Allocator and style changes in 9, 11 and 12, and the devres group
    factored into a helper (19).

Richard Cheng
  - The provider module reference is pinned per bound device (7).
  - ioeventfds and dma-buf exports over an excluded range are refused
    (10).
  - A dma-buf of the HDM region is exported with no_peer2peer set (28).
  - The PCI state is saved before the reset at open (15).

Gregory Price
  - The documentation is reworked (30) as per review comments.


Patch order
-----------

The tree builds at every commit. Five groups:

  Part 1  CXL core and PCI (patches 1 to 5)
    - Split the BAR block request and ioremap helpers, and let a
      BAR-owning driver own the component register block, so the HDM
      and RAS sub-blocks are left for vfio-cxl.
    - Move the component register defines to include/uapi/cxl/cxl_regs.h
      so a VMM and the selftest can use them.
    - Let the driver that owns the HDM range run CXL Reset.
    - Refuse bus and slot reset below a CXL port with SBR masked.

  Part 2  vfio-pci-core (patches 6 to 17)
    - The CXL provider ops registration and on-demand provider load.
    - The generic excluded-range list, its sparse mmap capability, and
      the ioeventfd and dma-buf checks.
    - The CXL DVSEC virtualization, the open and close hooks, the reset
      hooks, and the disable_cxl opt-out.
    - The try-reset helper, per-region revoke and re-arm, and the low
      power hooks.

  Part 3  vfio-cxl provider and HDM regions (patches 18 to 28)
    - The vfio-cxl module, the CXL memdev created at bind, and ownership
      of the whole component BAR.
    - The HDM memory region (read/write, then mmap), memory-failure
      containment for it, and the read-only decoder register region.
    - Excluding the decoder block from the direct BAR, the decoder
      restore after a hot reset, the device and decoder geometry
      capability, and the dma-buf export.

  Part 4  vfio-cxl reset (patch 29)
    - Run CXL Reset at every vfio reset point.

  Part 5  Documentation and selftests (patches 30 and 31)


Subsystem boundary
------------------

vfio-pci-core does not implement CXL registers. vfio-cxl is a separate
module that registers a struct vfio_cxl_ops at init. vfio-pci-core loads
it on demand for a CXL device (request_module() after pcie_is_cxl()) and
pins it per bound device. If a modular provider is missing or fails to
load, the device is driven as plain vfio-pci. The bind only defers for a
built-in provider whose initcall has not run yet.

vfio-pci-core provides generic pieces that a provider calls: the
excluded-range list, revoke and re-arm of one device-specific region,
the try-reset helper and the low power hooks. The CXL-specific work stays
behind the ops. The CXL DVSEC virtualization is the one CXL piece in
vfio-pci-core, because it lives in the config space permission hooks.


Memory ownership
----------------

The host resolves the host physical address once, at bind, from the
committed decoder. vfio-cxl creates the memdev at bind and tears it down
at unbind. It claims the HPA range with request_mem_region_exclusive(),
so nothing, /dev/mem included, can map a conflicting cacheable alias.

The guest programs a guest physical address into a virtual decoder that
the VMM emulates. It never reaches the physical decoder registers: those
are read through a read-only region and are excluded from the direct BAR
mapping, so the guest cannot move the host physical window.


HDM region access
-----------------

The HDM region carries the coherent device memory. A VMM mmaps it and
maps it into the guest through stage-2. The fault handler inserts the
host PFNs, including PMD-sized mappings when the mapping is aligned, only
while the committed decoder is in place and the media is ready.
Otherwise the fault gets SIGBUS. Region read and write check the same
state and return -EIO while the region is out of use. The range has no
struct page and is registered with the memory-failure machinery, so a
memory error is contained to a SIGBUS.


DMA and iommufd
---------------

Some accelerators, for example a GPU, send ATS-translated DMA to their
own HDM, so that range must be in the guest IOAS. IOMMU_IOAS_MAP cannot
pin a range with no struct page. VFIO_DEVICE_FEATURE_DMA_BUF on the HDM
region returns a dma-buf fd that iommufd maps with IOMMU_IOAS_MAP_FILE.
vfio-cxl revokes the dma-buf whenever it takes the region out of use.
iommufd then unmaps the range, and the VMM maps the same dma-buf again
once the region is back.


Reset
-----

An FLR does not reset CXL.cache or CXL.mem state and does not clear
device memory (CXL r4.0 sec 9.5), so it cannot separate one user of the
device from the next. vfio-cxl runs CXL Reset with Memory Clear at every
vfio reset point: open and close, VFIO_DEVICE_RESET, a guest FLR through
config space, and the guest's Initiate CXL Reset doorbell. Under
memory_lock, with the HDM mmap zapped and the dma-buf revoked, the host
invalidates the CPU caches for the HDM range, resets the function,
restores the decoder and the PCI config state, and invalidates the
caches again. vfio-cxl then waits for Memory_Active before it puts the
region back.

A CXL port masks SBR by default (CXL r4.0 sec 8.1.5.2). The PCI core now
refuses a bus or slot reset below such a port, so the vfio hot reset
ioctls fail and a VMM uses VFIO_DEVICE_RESET. If the port has SBR
unmasked, the hot reset decommits the decoder, and vfio-cxl restores it
in reset_done with cxl_restore_state_after_pci_reset(), the helper the
PCI core's cxl_bus reset method uses.


Low power
---------

The device may enter D3hot or runtime low power while it is assigned.
Low power loses the committed decoder, so vfio-cxl takes the HDM region
out of use first and restores the decoder when the device is back in D0.
Device memory is not saved or restored across low power.


Deferred
--------

  - Switched, multi-decoder and interleaved topologies stay rejected at
    bind.
  - Decoders that firmware did not commit are not supported.


Validation
----------

  - Each patch builds with W=1 (drivers/cxl, drivers/pci and
    drivers/vfio/pci, x86_64, VFIO_CXL=m).
  - scripts/checkpatch.pl --strict --codespell reports no errors or
    warnings. Patch 16 has one CHECK, a missing blank line between two
    inline stubs, kept to match the other stubs in vfio_pci_priv.h.
  - The selftest, tools/testing/selftests/vfio/vfio_cxl_type2_test.c, has
    27 tests: the device flag and regions, the HDM mmap and huge fault,
    the decoder register region and its -EIO gate, the dma-buf map with
    IOMMU_IOAS_MAP_FILE, region read and write across pages, the DVSEC
    reset doorbell, and the decoder after a D3hot cycle. All 27 pass on
    an arm64 host with 64K pages and lockdep (PROVE_LOCKING) enabled.
  - A guest with the device assigned booted, ran CUDA and unified memory
    workloads, and reset the device twice through guest CXL Reset through
    guest sysfs reset file and by using the CTRL2_INIT_CXL_RST in the
    guest. (used with QEMU vfio-cxl support v2 [6]).


AI assistance disclosure
------------------------

This series was developed with substantial AI assistance (Anthropic
Claude, through the Claude Code CLI). It was used for design trade-offs,
multi-agent regression and adversarial review of the v5 feedback using
review-core.md, the port to cxl_reset v15, analysis of the hardware
failures, and drafting the selftests and the documentation.
Following Documentation/process/coding-assistants.rst, each patch
carries an Assisted-by: LLM tag. I ran the tests on hardware, have
reviewed every change, and take full DCO responsibility for the series.


References
----------

[1] [PATCH v10 0/3] Support zero-sized HDM decoders
https://lore.kernel.org/all/20260914090858.19181-1-icheng@nvidia.com/

[2] [PATCH v15 00/16] PCI/CXL: Add CXL reset support for Type 2 devices
https://lore.kernel.org/all/20261011021422.3428136-1-smadhavan@nvidia.com/

[3] [PATCH v3 0/2] Add SMCCC cache clean/invalidate provider
https://lore.kernel.org/all/20260910082142.1522865-1-smadhavan@nvidia.com/

[4] [PATCH v4 0/2] Add SMCCC cache clean/invalidate provider
https://lore.kernel.org/all/20261013070205.3566865-1-smadhavan@nvidia.com/

[5] [PATCH v5 0/27] vfio/pci: Add CXL Type-2 device passthrough support
https://lore.kernel.org/all/20260916183540.3813685-1-mhonap@nvidia.com/

[6] [PATCH v2 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci
https://lore.kernel.org/linux-cxl/20260916184412.3825713-1-mhonap@nvidia.com/

Manish Honap (31):
  cxl/regs: Split the BAR block request and ioremap helpers
  cxl/regs: Let a BAR-owning driver own the component register block
  cxl: Move component register defines to uapi/cxl/cxl_regs.h
  PCI/CXL: Let the driver that owns the HDM range run CXL Reset
  PCI: Refuse bus and slot reset below a CXL port with SBR masked
  vfio/pci: Add the CXL provider ops registration interface
  vfio/pci: Detect CXL devices and load the CXL provider on demand
  vfio/pci: Add a generic excluded-range list
  vfio/pci: Advertise the mmap areas around excluded ranges
  vfio/pci: Refuse ioeventfds and dma-buf exports over excluded ranges
  vfio/pci: Virtualize the CXL DVSEC in vfio_pci_config.c
  vfio/pci: Call the CXL open and close hooks around device use
  vfio/pci: Bracket PCI resets with the CXL reset hooks
  vfio/pci: Provide an opt-out for the CXL Type-2 extensions
  vfio/pci: Route every reset point through a try-reset helper
  vfio/pci: Revoke and re-arm one device-specific region
  vfio/pci: Call the device provider across low-power transitions
  vfio/cxl: Add the vfio-cxl provider module skeleton
  vfio/cxl: Create the CXL memdev and set media ready at bind
  vfio/cxl: Own the whole component register BAR
  vfio/cxl: Expose the HDM memory region to the guest
  vfio/cxl: Allow mmap of the HDM memory region
  vfio/cxl: Contain HDM memory errors with memory_failure()
  vfio/cxl: Expose the HDM decoder registers read-only to the guest
  vfio/cxl: Exclude the HDM decoder registers from direct BAR access
  vfio/cxl: Restore the HDM decoder after a hot reset
  vfio/cxl: Describe the CXL device and decoder geometry to userspace
  vfio/cxl: Export the HDM memory region as a dma-buf
  vfio/cxl: Run the CXL reset at the vfio reset points
  Documentation: vfio-pci: Document CXL Type-2 device passthrough
  selftests/vfio: Add CXL Type-2 passthrough tests

 Documentation/driver-api/index.rst            |    1 +
 Documentation/driver-api/vfio-pci-cxl.rst     |  312 +++++
 MAINTAINERS                                   |   10 +
 drivers/cxl/core/hdm_state.c                  |   22 +-
 drivers/cxl/core/regs.c                       |   64 +-
 drivers/cxl/cxl.h                             |   47 +-
 drivers/pci/cxl.c                             |  136 +-
 drivers/pci/pci.c                             |   31 +-
 drivers/pci/pci.h                             |   18 +-
 drivers/vfio/pci/Kconfig                      |    2 +
 drivers/vfio/pci/Makefile                     |    2 +
 drivers/vfio/pci/cxl/Kconfig                  |   11 +
 drivers/vfio/pci/cxl/Makefile                 |    3 +
 drivers/vfio/pci/cxl/vfio_cxl_core.c          | 1124 +++++++++++++++++
 drivers/vfio/pci/vfio_pci.c                   |   52 +-
 drivers/vfio/pci/vfio_pci_config.c            |  225 +++-
 drivers/vfio/pci/vfio_pci_core.c              |  747 ++++++++++-
 drivers/vfio/pci/vfio_pci_dmabuf.c            |  183 ++-
 drivers/vfio/pci/vfio_pci_priv.h              |   20 +
 drivers/vfio/pci/vfio_pci_rdwr.c              |   76 +-
 include/cxl/cxl.h                             |    2 +
 include/cxl/hdm.h                             |    3 +-
 include/cxl/pci.h                             |   42 +
 include/linux/vfio_pci_core.h                 |   92 ++
 include/uapi/cxl/cxl_regs.h                   |   56 +
 include/uapi/linux/vfio.h                     |   24 +
 tools/testing/selftests/vfio/Makefile         |    1 +
 .../vfio/lib/include/libvfio/iommu.h          |    3 +
 tools/testing/selftests/vfio/lib/iommu.c      |   29 +
 .../selftests/vfio/lib/vfio_pci_device.c      |   57 +-
 .../selftests/vfio/vfio_cxl_type2_test.c      | 1095 ++++++++++++++++
 31 files changed, 4306 insertions(+), 184 deletions(-)
 create mode 100644 Documentation/driver-api/vfio-pci-cxl.rst
 create mode 100644 drivers/vfio/pci/cxl/Kconfig
 create mode 100644 drivers/vfio/pci/cxl/Makefile
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_core.c
 create mode 100644 include/uapi/cxl/cxl_regs.h
 create mode 100644 tools/testing/selftests/vfio/vfio_cxl_type2_test.c

-- 
2.53.0

[PATCH v6 01/31] cxl/regs: Split the BAR block request and ioremap helpers

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

devm_cxl_iomap_block() claims a register block with
devm_request_mem_region() and then maps it with devm_ioremap().

A driver that already owns the parent BAR needs to skip the sub-block
request while still mapping it, so the request and the map must be
callable independently.

Split them into devm_cxl_request_block() and devm_cxl_ioremap_block().
devm_cxl_iomap_block() keeps calling both in turn, so there is no
functional change.

Assisted-by: LLM
Reviewed-by: Jonathan Cameron <jonathan.cameron@oss.qualcomm.com>
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/cxl/core/regs.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
index ba1f942d20e1..9235520b2f4e 100644
--- a/drivers/cxl/core/regs.c
+++ b/drivers/cxl/core/regs.c
@@ -177,6 +177,20 @@ void cxl_probe_device_regs(struct device *dev, void __iomem *base,
 }
 EXPORT_SYMBOL_NS_GPL(cxl_probe_device_regs, "CXL");
 
+static struct resource *devm_cxl_request_block(struct device *dev,
+					       resource_size_t addr,
+					       resource_size_t length)
+{
+	return devm_request_mem_region(dev, addr, length, dev_name(dev));
+}
+
+static void __iomem *devm_cxl_ioremap_block(struct device *dev,
+					    resource_size_t addr,
+					    resource_size_t length)
+{
+	return devm_ioremap(dev, addr, length);
+}
+
 void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,
 				   resource_size_t length)
 {
@@ -186,7 +200,7 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,
 	if (WARN_ON_ONCE(addr == CXL_RESOURCE_NONE))
 		return NULL;
 
-	res = devm_request_mem_region(dev, addr, length, dev_name(dev));
+	res = devm_cxl_request_block(dev, addr, length);
 	if (!res) {
 		resource_size_t end = addr + length - 1;
 
@@ -194,7 +208,7 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,
 		return NULL;
 	}
 
-	ret_val = devm_ioremap(dev, addr, length);
+	ret_val = devm_cxl_ioremap_block(dev, addr, length);
 	if (!ret_val)
 		dev_err(dev, "Failed to map region %pr\n", res);
 
-- 
2.53.0

[PATCH v6 02/31] cxl/regs: Let a BAR-owning driver own the component register block

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

cxl_map_component_regs() claims each mapped sub-block with
devm_request_mem_region(). A driver that requests the whole component
register BAR itself, such as vfio-cxl, would then collide with that
claim and fail to map the HDM decoder and RAS blocks.

Add cxl_reg_map_set_owned_resource() so such a driver records the
resource it already owns, the whole component register BAR, on the
register map.

Skip the sub-block request for a block inside that resource, and fail with
-ENXIO for a block outside it.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/cxl/core/regs.c | 46 +++++++++++++++++++++++++++++++++++++++--
 include/cxl/cxl.h       |  2 ++
 include/cxl/pci.h       |  3 +++
 3 files changed, 49 insertions(+), 2 deletions(-)

diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
index 9235520b2f4e..c3df99de92c3 100644
--- a/drivers/cxl/core/regs.c
+++ b/drivers/cxl/core/regs.c
@@ -216,6 +216,39 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,
 }
 EXPORT_SYMBOL_NS_GPL(devm_cxl_iomap_block, "CXL");
 
+/**
+ * cxl_reg_map_set_owned_resource() - record the BAR a driver owns
+ * @map: register map of the component register block
+ * @res: resource of the whole component register BAR, which the caller
+ *	requests itself, before or after this call
+ *
+ * A driver that requests the whole component register BAR itself, such as
+ * vfio-cxl, records that BAR here. cxl_map_component_regs() then maps the
+ * blocks inside @res without requesting them again and refuses any block
+ * outside it. A register map has at most one owned resource.
+ *
+ * Call this after cxl_pci_setup_regs() has filled @map: enumeration rewrites
+ * the whole map and drops a previously recorded owned resource.
+ *
+ * Return: 0 on success, -EBUSY if an owned resource is already set, or
+ * -EINVAL if @map has not been set up or its block lies outside @res.
+ */
+int cxl_reg_map_set_owned_resource(struct cxl_register_map *map,
+				   struct resource *res)
+{
+	struct resource block = DEFINE_RES_MEM(map->resource, map->max_size);
+
+	if (WARN_ON_ONCE(map->owned))
+		return -EBUSY;
+	if (WARN_ON_ONCE(map->resource == CXL_RESOURCE_NONE ||
+			 !resource_contains(res, &block)))
+		return -EINVAL;
+
+	map->owned = res;
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_reg_map_set_owned_resource, "CXL");
+
 int cxl_map_component_regs(const struct cxl_register_map *map,
 			   struct cxl_component_regs *regs,
 			   unsigned long map_mask)
@@ -234,6 +267,7 @@ int cxl_map_component_regs(const struct cxl_register_map *map,
 		struct mapinfo *mi = &mapinfo[i];
 		resource_size_t addr;
 		resource_size_t length;
+		struct resource res;
 
 		if (!mi->rmap->valid)
 			continue;
@@ -241,8 +275,16 @@ int cxl_map_component_regs(const struct cxl_register_map *map,
 			continue;
 		addr = map->resource + mi->rmap->offset;
 		length = mi->rmap->size;
-		*(mi->addr) = devm_cxl_iomap_block(host, addr, length);
-		if (!*(mi->addr))
+		res = DEFINE_RES_MEM(addr, length);
+
+		if (map->owned) {
+			if (!resource_contains(map->owned, &res))
+				return -ENXIO;
+			*mi->addr = devm_cxl_ioremap_block(host, addr, length);
+		} else {
+			*mi->addr = devm_cxl_iomap_block(host, addr, length);
+		}
+		if (!*mi->addr)
 			return -ENOMEM;
 	}
 
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
index 3e70f74685b6..4bd0fec4951f 100644
--- a/include/cxl/cxl.h
+++ b/include/cxl/cxl.h
@@ -153,6 +153,7 @@ struct cxl_pmu_reg_map {
  * @bar: PCI BAR containing the register block
  * @bar_offset: register block offset relative to @bar
  * @reg_type: see enum cxl_regloc_type
+ * @owned: driver-owned resource whose sub-blocks are mapped without re-requesting
  * @component_map: cxl_reg_map for component registers
  * @device_map: cxl_reg_maps for device registers
  * @pmu_map: cxl_reg_maps for CXL Performance Monitoring Units
@@ -165,6 +166,7 @@ struct cxl_register_map {
 	int bar;
 	resource_size_t bar_offset;
 	u8 reg_type;
+	struct resource *owned;
 	union {
 		struct cxl_component_reg_map component_map;
 		struct cxl_device_reg_map device_map;
diff --git a/include/cxl/pci.h b/include/cxl/pci.h
index b15d7e415e27..0b74819fea6d 100644
--- a/include/cxl/pci.h
+++ b/include/cxl/pci.h
@@ -18,9 +18,12 @@ enum cxl_regloc_type {
 
 struct cxl_register_map;
 struct pci_dev;
+struct resource;
 
 int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
 		       struct cxl_register_map *map);
+int cxl_reg_map_set_owned_resource(struct cxl_register_map *map,
+				   struct resource *res);
 int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
 		      struct cxl_register_map *map);
 int cxl_setup_regs(struct cxl_register_map *map);
-- 
2.53.0

[PATCH v6 03/31] cxl: Move component register defines to uapi/cxl/cxl_regs.h

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

The CXL component and HDM decoder register layout is spec-defined and is
needed by a userspace consumer that cannot include the private
drivers/cxl/cxl.h

The vfio-cxl selftest
(tools/testing/selftests/vfio/vfio_cxl_type2_test.c) builds against
installed uapi headers only. Move those defines to a new uapi header, the
way PCI register offsets live in uapi pci_regs.h, and include it from
drivers/cxl/cxl.h so existing in-kernel users are unchanged.

The section references in the moved comments are updated to CXL r4.0.

No functional change.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 MAINTAINERS                 |  1 +
 drivers/cxl/cxl.h           | 47 +------------------------------
 include/uapi/cxl/cxl_regs.h | 56 +++++++++++++++++++++++++++++++++++++
 3 files changed, 58 insertions(+), 46 deletions(-)
 create mode 100644 include/uapi/cxl/cxl_regs.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 08fc6efdb6cf..c3aef3dce68c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6575,6 +6575,7 @@ F:	Documentation/userspace-api/fwctl/fwctl-cxl.rst
 F:	drivers/cxl/
 F:	drivers/pci/cxl.c
 F:	include/cxl/
+F:	include/uapi/cxl/
 F:	include/uapi/linux/cxl_mem.h
 F:	tools/testing/cxl/
 
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index ad4e80be2bd7..b62074453866 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -13,6 +13,7 @@
 #include <linux/io.h>
 #include <linux/range.h>
 #include <cxl/cxl.h>
+#include <cxl/cxl_regs.h>
 
 extern const struct nvdimm_security_ops *cxl_security_ops;
 
@@ -24,52 +25,6 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
  * (port-driver, region-driver, nvdimm object-drivers... etc).
  */
 
-/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */
-#define CXL_COMPONENT_REG_BLOCK_SIZE SZ_64K
-
-/* CXL 2.0 8.2.5 CXL.cache and CXL.mem Registers*/
-#define CXL_CM_OFFSET 0x1000
-#define CXL_CM_CAP_HDR_OFFSET 0x0
-#define   CXL_CM_CAP_HDR_ID_MASK GENMASK(15, 0)
-#define     CM_CAP_HDR_CAP_ID 1
-#define   CXL_CM_CAP_HDR_VERSION_MASK GENMASK(19, 16)
-#define     CM_CAP_HDR_CAP_VERSION 1
-#define   CXL_CM_CAP_HDR_CACHE_MEM_VERSION_MASK GENMASK(23, 20)
-#define     CM_CAP_HDR_CACHE_MEM_VERSION 1
-#define   CXL_CM_CAP_HDR_ARRAY_SIZE_MASK GENMASK(31, 24)
-#define CXL_CM_CAP_PTR_MASK GENMASK(31, 20)
-
-#define   CXL_CM_CAP_CAP_ID_RAS 0x2
-#define   CXL_CM_CAP_CAP_ID_HDM 0x5
-#define   CXL_CM_CAP_CAP_HDM_VERSION 1
-
-/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
-#define CXL_HDM_DECODER_CAP_OFFSET 0x0
-#define   CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0)
-#define   CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4)
-#define   CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8)
-#define   CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9)
-#define   CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY BIT(11)
-#define   CXL_HDM_DECODER_INTERLEAVE_16_WAY BIT(12)
-#define CXL_HDM_DECODER_CTRL_OFFSET 0x4
-#define   CXL_HDM_DECODER_ENABLE BIT(1)
-#define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10)
-#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i) (0x20 * (i) + 0x14)
-#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i) (0x20 * (i) + 0x18)
-#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i) (0x20 * (i) + 0x1c)
-#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20)
-#define   CXL_HDM_DECODER0_CTRL_IG_MASK GENMASK(3, 0)
-#define   CXL_HDM_DECODER0_CTRL_IW_MASK GENMASK(7, 4)
-#define   CXL_HDM_DECODER0_CTRL_LOCK BIT(8)
-#define   CXL_HDM_DECODER0_CTRL_COMMIT BIT(9)
-#define   CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10)
-#define   CXL_HDM_DECODER0_CTRL_COMMIT_ERROR BIT(11)
-#define   CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12)
-#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
-#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
-#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
-#define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i)
-
 /* HDM decoder control register constants CXL 3.0 8.2.5.19.7 */
 #define CXL_DECODER_MIN_GRANULARITY 256
 #define CXL_DECODER_MAX_ENCODED_IG 6
diff --git a/include/uapi/cxl/cxl_regs.h b/include/uapi/cxl/cxl_regs.h
new file mode 100644
index 000000000000..276f9361c840
--- /dev/null
+++ b/include/uapi/cxl/cxl_regs.h
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/* Copyright (c) 2026 NVIDIA Corporation & Affiliates */
+
+#ifndef _UAPI_CXL_REGS_H_
+#define _UAPI_CXL_REGS_H_
+
+#include <linux/bits.h>
+#include <linux/const.h>
+
+/* CXL r4.0 sec 8.2.3 Component Register Layout and Definition */
+#define CXL_COMPONENT_REG_BLOCK_SIZE 0x10000
+
+/* CXL r4.0 sec 8.2.4 CXL.cache and CXL.mem Registers */
+#define CXL_CM_OFFSET 0x1000
+#define CXL_CM_CAP_HDR_OFFSET 0x0
+#define   CXL_CM_CAP_HDR_ID_MASK __GENMASK(15, 0)
+#define     CM_CAP_HDR_CAP_ID 1
+#define   CXL_CM_CAP_HDR_VERSION_MASK __GENMASK(19, 16)
+#define     CM_CAP_HDR_CAP_VERSION 1
+#define   CXL_CM_CAP_HDR_CACHE_MEM_VERSION_MASK __GENMASK(23, 20)
+#define     CM_CAP_HDR_CACHE_MEM_VERSION 1
+#define   CXL_CM_CAP_HDR_ARRAY_SIZE_MASK __GENMASK(31, 24)
+#define CXL_CM_CAP_PTR_MASK __GENMASK(31, 20)
+
+#define   CXL_CM_CAP_CAP_ID_RAS 0x2
+#define   CXL_CM_CAP_CAP_ID_HDM 0x5
+#define   CXL_CM_CAP_CAP_HDM_VERSION 1
+
+/* HDM decoders CXL r4.0 sec 8.2.4.20 CXL HDM Decoder Capability Structure */
+#define CXL_HDM_DECODER_CAP_OFFSET 0x0
+#define   CXL_HDM_DECODER_COUNT_MASK __GENMASK(3, 0)
+#define   CXL_HDM_DECODER_TARGET_COUNT_MASK __GENMASK(7, 4)
+#define   CXL_HDM_DECODER_INTERLEAVE_11_8 _BITUL(8)
+#define   CXL_HDM_DECODER_INTERLEAVE_14_12 _BITUL(9)
+#define   CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY _BITUL(11)
+#define   CXL_HDM_DECODER_INTERLEAVE_16_WAY _BITUL(12)
+#define CXL_HDM_DECODER_CTRL_OFFSET 0x4
+#define   CXL_HDM_DECODER_ENABLE _BITUL(1)
+#define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10)
+#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i) (0x20 * (i) + 0x14)
+#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i) (0x20 * (i) + 0x18)
+#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i) (0x20 * (i) + 0x1c)
+#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20)
+#define   CXL_HDM_DECODER0_CTRL_IG_MASK __GENMASK(3, 0)
+#define   CXL_HDM_DECODER0_CTRL_IW_MASK __GENMASK(7, 4)
+#define   CXL_HDM_DECODER0_CTRL_LOCK _BITUL(8)
+#define   CXL_HDM_DECODER0_CTRL_COMMIT _BITUL(9)
+#define   CXL_HDM_DECODER0_CTRL_COMMITTED _BITUL(10)
+#define   CXL_HDM_DECODER0_CTRL_COMMIT_ERROR _BITUL(11)
+#define   CXL_HDM_DECODER0_CTRL_HOSTONLY _BITUL(12)
+#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
+#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
+#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
+#define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i)
+
+#endif /* _UAPI_CXL_REGS_H_ */
-- 
2.53.0

[PATCH v6 04/31] PCI/CXL: Let the driver that owns the HDM range run CXL Reset

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

vfio-cxl passes a CXL Type-2 device to a guest and must run CXL Reset at
the vfio reset points. The cxl_reset PCI reset method does not fit that
caller:

- It requests every HPA range the device decodes. vfio-cxl holds that
  range with request_mem_region_exclusive() while it exposes the memory,
  so the request fails with -EBUSY.
- When the reset cannot run, the method returns -ENOTTY and the PCI core
  moves on to the next method. That is FLR, which leaves device memory
  intact for the next user.
- The snapshot and restore helpers that vfio-cxl needs after a bus reset
  and on return from low power are not exported.

Let cxl_hdm_quiesce_prepare() take the resource the caller already holds
and skip the request for ranges inside it. pci_cxl_reset_owned() accepts
only a resource marked busy and exclusive, as
request_mem_region_exclusive() leaves it. The CPU caches are still
written back and invalidated for every range before and after the reset,
as the reset method does.

Add:
- pci_cxl_reset_owned(): runs the reset method's sequence for a caller that
  holds the device lock and returns any failure to that caller

- pci_cxl_reset_capable(): checks the cached HDM state and the CPU cache
  invalidation that the reset-method probe skips.

- pci_cxl_hdm_snapshot(): copies the HDM state cached at enumeration
  without requiring the device lock. A driver needs this copy in probe,
  which the PCI core may run in a worker on the device's NUMA node, where
  device_lock_assert() would warn although the binding task holds the
  lock. pci_cxl_reset_prepare() becomes that copy plus the lock assertion.

Export:
- pci_cxl_reset_owned()
- pci_cxl_reset_capable()
- pci_cxl_hdm_snapshot()
- cxl_restore_state_after_pci_reset()

to the vfio-cxl module only, and declare them in include/cxl/pci.h.

CXL Reset always clears memory, so there is no caller Memory Clear choice.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/cxl/core/hdm_state.c |  22 ++++--
 drivers/pci/cxl.c            | 136 ++++++++++++++++++++++++++++-------
 drivers/pci/pci.h            |  18 +----
 include/cxl/hdm.h            |   3 +-
 include/cxl/pci.h            |  39 ++++++++++
 5 files changed, 171 insertions(+), 47 deletions(-)

diff --git a/drivers/cxl/core/hdm_state.c b/drivers/cxl/core/hdm_state.c
index 95daeda0ef80..9452047e3b97 100644
--- a/drivers/cxl/core/hdm_state.c
+++ b/drivers/cxl/core/hdm_state.c
@@ -267,7 +267,8 @@ static int cxl_hdm_ranges_collect(struct cxl_hdm_quiesce *q, struct device *dev,
 	return 0;
 }
 
-static int cxl_hdm_ranges_request(struct cxl_hdm_quiesce *q)
+static int cxl_hdm_ranges_request(struct cxl_hdm_quiesce *q,
+				  const struct resource *owned)
 {
 	struct cxl_hdm_range *range;
 
@@ -275,6 +276,12 @@ static int cxl_hdm_ranges_request(struct cxl_hdm_quiesce *q)
 
 	list_for_each_entry(range, &q->ranges, list) {
 		const struct range *hpa_range = &range->hpa_range;
+		struct resource res = DEFINE_RES_MEM(hpa_range->start,
+						     range->len);
+
+		/* The caller already holds this range exclusively. */
+		if (owned && resource_contains(owned, &res))
+			continue;
 
 		range->res = request_mem_region(hpa_range->start, range->len,
 						"cxl_reset");
@@ -316,15 +323,20 @@ static int cxl_hdm_ranges_invalidate(struct cxl_hdm_quiesce *q)
  * @dev: device for error messages
  * @q: quiesce state to initialize
  * @snapshot: HDM state from cxl_hdm_cache_snapshot()
+ * @owned: resource the caller holds exclusively over the ranges, or NULL
  *
  * Reserve every enabled system-physical range in @snapshot exclusively, then
- * write back and invalidate CPU caches for them. Call
- * cxl_hdm_quiesce_destroy() afterwards whether or not this succeeds.
+ * write back and invalidate CPU caches for them. A range inside @owned is
+ * not requested again; the caller's reservation already excludes other
+ * users.
+ *
+ * Call cxl_hdm_quiesce_destroy() afterwards whether or not this succeeds.
  *
  * Return: 0 on success, or a negative errno.
  */
 int cxl_hdm_quiesce_prepare(struct device *dev, struct cxl_hdm_quiesce *q,
-			    const struct cxl_hdm_info *snapshot)
+			    const struct cxl_hdm_info *snapshot,
+			    const struct resource *owned)
 {
 	int rc;
 
@@ -340,7 +352,7 @@ int cxl_hdm_quiesce_prepare(struct device *dev, struct cxl_hdm_quiesce *q,
 	if (rc)
 		return rc;
 
-	rc = cxl_hdm_ranges_request(q);
+	rc = cxl_hdm_ranges_request(q, owned);
 	if (rc)
 		return rc;
 
diff --git a/drivers/pci/cxl.c b/drivers/pci/cxl.c
index ed0b6f619c70..b4d467506c54 100644
--- a/drivers/pci/cxl.c
+++ b/drivers/pci/cxl.c
@@ -372,9 +372,32 @@ static int cxl_reset_save_restored_state(struct pci_dev *pdev, u16 command)
 	return rc;
 }
 
+/**
+ * pci_cxl_hdm_snapshot() - copy the HDM state cached at enumeration
+ * @pdev: CXL device
+ *
+ * The cache is read under cxl_rwsem.dpa, so the caller needs no device
+ * lock. A driver can call this from probe, which the PCI core may run in a
+ * worker on the device's NUMA node while the binding task holds the lock.
+ *
+ * Return: a copy to free with kfree(); NULL when @pdev has no cached HDM
+ * state; or an ERR_PTR() on failure.
+ */
+struct cxl_hdm_info *pci_cxl_hdm_snapshot(struct pci_dev *pdev)
+{
+	struct cxl_hdm_info *snapshot;
+
+	snapshot = cxl_hdm_cache_snapshot(&pdev->hdm);
+	if (IS_ERR(snapshot) && PTR_ERR(snapshot) == -ENXIO)
+		return NULL;
+
+	return snapshot;
+}
+EXPORT_SYMBOL_FOR_MODULES(pci_cxl_hdm_snapshot, "vfio-cxl");
+
 /**
  * pci_cxl_reset_prepare() - allocate HDM state storage ahead of a bus reset
- * @pdev: device about to be reset
+ * @pdev: device about to be reset, with its device lock held
  *
  * Copy the cached HDM state before the reset, so that restoring cannot fail
  * for lack of memory. The decoder settings are refreshed when the state is
@@ -386,14 +409,8 @@ static int cxl_reset_save_restored_state(struct pci_dev *pdev, u16 command)
  */
 struct cxl_hdm_info *pci_cxl_reset_prepare(struct pci_dev *pdev)
 {
-	struct cxl_hdm_info *snapshot;
-
 	device_lock_assert(&pdev->dev);
-	snapshot = cxl_hdm_cache_snapshot(&pdev->hdm);
-	if (IS_ERR(snapshot) && PTR_ERR(snapshot) == -ENXIO)
-		return NULL;
-
-	return snapshot;
+	return pci_cxl_hdm_snapshot(pdev);
 }
 
 /**
@@ -433,6 +450,7 @@ int cxl_restore_state_after_pci_reset(struct pci_dev *pdev,
 	cxl_region_quiesce_unlock();
 	return rc;
 }
+EXPORT_SYMBOL_FOR_MODULES(cxl_restore_state_after_pci_reset, "vfio-cxl");
 
 /*
  * CXL r4.0 sec 9.7.2 defines the reset completion timeout encodings.
@@ -642,7 +660,8 @@ static int __cxl_reset_execute(struct pci_dev *pdev, int dvsec, u16 cap,
 	return rc;
 }
 
-static int cxl_reset_quiesced(struct pci_dev *pdev, int dvsec, u16 cap)
+static int cxl_reset_quiesced(struct pci_dev *pdev, int dvsec, u16 cap,
+			      const struct resource *owned)
 	__must_hold(cxl_region_quiesce)
 {
 	struct cxl_hdm_info *snapshot __free(kfree) = NULL;
@@ -663,7 +682,7 @@ static int cxl_reset_quiesced(struct pci_dev *pdev, int dvsec, u16 cap)
 		return PTR_ERR(snapshot);
 	}
 
-	rc = cxl_hdm_quiesce_prepare(&pdev->dev, &q, snapshot);
+	rc = cxl_hdm_quiesce_prepare(&pdev->dev, &q, snapshot, owned);
 	if (rc) {
 		cxl_hdm_quiesce_destroy(&q);
 		return rc;
@@ -686,32 +705,40 @@ static int cxl_reset_quiesced(struct pci_dev *pdev, int dvsec, u16 cap)
 	return cxl_reset_save_restored_state(pdev, command);
 }
 
-static int cxl_reset_execute(struct pci_dev *pdev, int dvsec, u16 cap)
+static int cxl_reset_execute(struct pci_dev *pdev, int dvsec, u16 cap,
+			     const struct resource *owned)
 {
 	int rc;
 
+	/* Raw reset callers may not have saved the PCI state needed for restore. */
+	if (!pdev->state_saved) {
+		pci_err(pdev, "CXL reset requires saved PCI state\n");
+		return -EINVAL;
+	}
+
 	cxl_region_quiesce_lock();
-	rc = cxl_reset_quiesced(pdev, dvsec, cap);
+	rc = cxl_reset_quiesced(pdev, dvsec, cap, owned);
 	cxl_region_quiesce_unlock();
 
 	return rc;
 }
 
-int cxl_reset_function(struct pci_dev *pdev, bool probe)
+static int cxl_reset_check(struct pci_dev *pdev, bool probe, u16 *dvsec,
+			   u16 *cap)
 {
-	u16 dvsec, cap, ctrl;
+	u16 ctrl;
 	int rc;
 
-	rc = cxl_pci_device_dvsec(pdev, &dvsec, &cap);
+	rc = cxl_pci_device_dvsec(pdev, dvsec, cap);
 	if (rc)
 		return probe ? -ENOTTY : rc;
-	if (!(cap & PCI_DVSEC_CXL_CACHE_CAPABLE) ||
-	    !(cap & PCI_DVSEC_CXL_MEM_CAPABLE) ||
-	    !(cap & PCI_DVSEC_CXL_RST_CAPABLE) ||
-	    !(cap & PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE))
+	if (!(*cap & PCI_DVSEC_CXL_CACHE_CAPABLE) ||
+	    !(*cap & PCI_DVSEC_CXL_MEM_CAPABLE) ||
+	    !(*cap & PCI_DVSEC_CXL_RST_CAPABLE) ||
+	    !(*cap & PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE))
 		return -ENOTTY;
 
-	rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL, &ctrl);
+	rc = pci_read_config_word(pdev, *dvsec + PCI_DVSEC_CXL_CTRL, &ctrl);
 	if (rc)
 		return probe ? -ENOTTY : pcibios_err_to_errno(rc);
 	if (PCI_POSSIBLE_ERROR(ctrl))
@@ -737,11 +764,70 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)
 	if (!cpu_cache_has_invalidate_memregion())
 		return -ENOTTY;
 
-	/* Raw reset callers may not have saved the PCI state needed for restore. */
-	if (!pdev->state_saved) {
-		pci_err(pdev, "CXL reset requires saved PCI state\n");
+	return 0;
+}
+
+int cxl_reset_function(struct pci_dev *pdev, bool probe)
+{
+	u16 dvsec, cap;
+	int rc;
+
+	rc = cxl_reset_check(pdev, probe, &dvsec, &cap);
+	if (rc || probe)
+		return rc;
+
+	return cxl_reset_execute(pdev, dvsec, cap, NULL);
+}
+
+/**
+ * pci_cxl_reset_capable() - check whether CXL Reset can run on a device now
+ * @pdev: CXL device
+ *
+ * Unlike the reset-method probe, this also checks the cached HDM state and
+ * the CPU cache invalidation that the reset needs.
+ *
+ * Return: true when pci_cxl_reset_owned() can attempt the reset.
+ */
+bool pci_cxl_reset_capable(struct pci_dev *pdev)
+{
+	u16 dvsec, cap;
+
+	return !cxl_reset_check(pdev, false, &dvsec, &cap);
+}
+EXPORT_SYMBOL_FOR_MODULES(pci_cxl_reset_capable, "vfio-cxl");
+
+/**
+ * pci_cxl_reset_owned() - CXL Reset for the driver that owns the HDM range
+ * @pdev: CXL device, with its device lock held and PCI state saved
+ * @owned: resource the caller reserved over the device's HPA range with
+ *	   request_mem_region_exclusive(), or NULL
+ *
+ * Run the same sequence as the cxl_reset method. Ranges inside @owned are not
+ * requested again, so a driver that keeps the HPA range reserved while it
+ * exposes the memory can reset the device. The CPU caches are still
+ * invalidated for every range before and after the reset.
+ *
+ * Return: 0 on success, or a negative errno. Unlike the reset method, a
+ * device that cannot be reset returns an error to the caller instead of
+ * falling through to another reset method. -EINVAL if @owned is not a busy,
+ * exclusive reservation.
+ */
+int pci_cxl_reset_owned(struct pci_dev *pdev, const struct resource *owned)
+{
+	const unsigned long held = IORESOURCE_BUSY | IORESOURCE_EXCLUSIVE;
+	u16 dvsec, cap;
+	int rc;
+
+	device_lock_assert(&pdev->dev);
+
+	/* Only a busy, exclusive reservation keeps other users off the range. */
+	if (owned && WARN_ON_ONCE((owned->flags & held) != held))
 		return -EINVAL;
-	}
 
-	return cxl_reset_execute(pdev, dvsec, cap);
+	rc = cxl_reset_check(pdev, false, &dvsec, &cap);
+	if (rc)
+		return rc;
+
+	return cxl_reset_execute(pdev, dvsec, cap, owned);
 }
+EXPORT_SYMBOL_FOR_MODULES(pci_cxl_reset_owned, "vfio-cxl");
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 476e8eefafb2..b1fa4ec7ae75 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -730,14 +730,12 @@ static inline void pci_doe_destroy(struct pci_dev *pdev) { }
 static inline void pci_doe_disconnected(struct pci_dev *pdev) { }
 #endif
 
-struct cxl_hdm_info;
+#include <cxl/pci.h>
+
 #ifdef CONFIG_CXL_RESET
 void pci_cxl_hdm_cache_init(struct pci_dev *pdev);
 void pci_cxl_hdm_cache_release(struct pci_dev *pdev);
 int cxl_reset_function(struct pci_dev *pdev, bool probe);
-struct cxl_hdm_info *pci_cxl_reset_prepare(struct pci_dev *pdev);
-int cxl_restore_state_after_pci_reset(struct pci_dev *pdev,
-				      struct cxl_hdm_info *snapshot);
 #else
 static inline void pci_cxl_hdm_cache_init(struct pci_dev *pdev) { }
 static inline void pci_cxl_hdm_cache_release(struct pci_dev *pdev) { }
@@ -745,18 +743,6 @@ static inline int cxl_reset_function(struct pci_dev *pdev, bool probe)
 {
 	return -ENOTTY;
 }
-
-static inline struct cxl_hdm_info *pci_cxl_reset_prepare(struct pci_dev *pdev)
-{
-	return NULL;
-}
-
-static inline int
-cxl_restore_state_after_pci_reset(struct pci_dev *pdev,
-				  struct cxl_hdm_info *snapshot)
-{
-	return 0;
-}
 #endif
 
 #ifdef CONFIG_PCI_NPEM
diff --git a/include/cxl/hdm.h b/include/cxl/hdm.h
index 02cad2d74329..fa3b5845a33c 100644
--- a/include/cxl/hdm.h
+++ b/include/cxl/hdm.h
@@ -96,7 +96,8 @@ struct cxl_hdm_quiesce {
 void cxl_region_quiesce_lock(void);
 void cxl_region_quiesce_unlock(void);
 int cxl_hdm_quiesce_prepare(struct device *dev, struct cxl_hdm_quiesce *q,
-			    const struct cxl_hdm_info *snapshot)
+			    const struct cxl_hdm_info *snapshot,
+			    const struct resource *owned)
 	__must_hold(cxl_region_quiesce);
 int cxl_hdm_quiesce_flush(struct cxl_hdm_quiesce *q)
 	__must_hold(cxl_region_quiesce);
diff --git a/include/cxl/pci.h b/include/cxl/pci.h
index 0b74819fea6d..64efecb44c1f 100644
--- a/include/cxl/pci.h
+++ b/include/cxl/pci.h
@@ -4,6 +4,7 @@
 #ifndef __CXL_CXL_PCI_H__
 #define __CXL_CXL_PCI_H__
 
+#include <linux/errno.h>
 #include <linux/types.h>
 
 /* Register Block Identifier (RBI) */
@@ -27,6 +28,44 @@ int cxl_reg_map_set_owned_resource(struct cxl_register_map *map,
 int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
 		      struct cxl_register_map *map);
 int cxl_setup_regs(struct cxl_register_map *map);
+
+struct cxl_hdm_info;
+#ifdef CONFIG_CXL_RESET
+struct cxl_hdm_info *pci_cxl_hdm_snapshot(struct pci_dev *pdev);
+struct cxl_hdm_info *pci_cxl_reset_prepare(struct pci_dev *pdev);
+int cxl_restore_state_after_pci_reset(struct pci_dev *pdev,
+				      struct cxl_hdm_info *snapshot);
+bool pci_cxl_reset_capable(struct pci_dev *pdev);
+int pci_cxl_reset_owned(struct pci_dev *pdev, const struct resource *owned);
+#else
+static inline struct cxl_hdm_info *pci_cxl_hdm_snapshot(struct pci_dev *pdev)
+{
+	return NULL;
+}
+
+static inline struct cxl_hdm_info *pci_cxl_reset_prepare(struct pci_dev *pdev)
+{
+	return NULL;
+}
+
+static inline int
+cxl_restore_state_after_pci_reset(struct pci_dev *pdev,
+				  struct cxl_hdm_info *snapshot)
+{
+	return 0;
+}
+
+static inline bool pci_cxl_reset_capable(struct pci_dev *pdev)
+{
+	return false;
+}
+
+static inline int pci_cxl_reset_owned(struct pci_dev *pdev,
+				      const struct resource *owned)
+{
+	return -ENOTTY;
+}
+#endif
 /*
  * CXL r4.0 sec 8.1.3.1: a CXL.mem capable device reports one or two HDM
  * ranges. 00b is illegal when Mem_Capable is set and 11b is reserved.
-- 
2.53.0

[PATCH v6 05/31] PCI: Refuse bus and slot reset below a CXL port with SBR masked

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A CXL port ignores the Secondary Bus Reset bit in Bridge Control unless
Unmask SBR is set in its CXL Port DVSEC (CXL r4.0 sec 8.1.5.2).
pci_reset_bus_function() already returns -ENOTTY in that case.
pci_reset_bus(), pci_probe_reset_bus() and the slot variants do not, so
they report success for a reset that reached no device below the port.

vfio-pci uses these for VFIO_DEVICE_GET_PCI_HOT_RESET_INFO and
VFIO_DEVICE_PCI_HOT_RESET. A user is then told a hot reset happened while
a CXL device below a masked port kept its state.

Return -ENOTTY from pci_probe_reset_slot() and pci_probe_reset_bus()
when the bridge above the bus is a CXL port with SBR masked.
pci_reset_bus() and pci_try_reset_bridge() probe before they reset, so
they refuse too.

Users see this in three places:
- the vfio hot reset ioctls fail (vfio reports -ENODEV when neither probe
  succeeds),
- the reset_subordinate sysfs attribute returns -ENOTTY,
- vfio-pci groups a device in a hotplug slot below such a port by bus
  instead of by slot.

Every other pci_reset_bus() caller, including the vfio bus reset at
close, is refused the same way. With one device per CXL port both
groupings have the same members.

Leave pci_bus_error_reset() as it is. AER and host bridge error recovery
call it without saving or restoring state and only map its result to
recovered or disconnect. Failing it below a masked port would turn
recovery of every CXL device there, memory expanders included, into a
disconnect; that is a CXL error handling decision, not part of this fix.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/pci/pci.c | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index cecf8e681e69..87fe9de81acf 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -4984,6 +4984,20 @@ static bool cxl_sbr_masked(struct pci_dev *dev)
 	return true;
 }
 
+/*
+ * A secondary bus reset below a CXL port with SBR masked does not reach the
+ * devices on the bus, so a bus or slot reset there would report success for
+ * a reset that did not happen. The probes and the save/restore resets that
+ * pci_reset_bus() and reset_subordinate use refuse it; error recovery
+ * (pci_bus_error_reset()) keeps its existing behavior.
+ */
+static bool pci_bus_cxl_sbr_masked(struct pci_bus *bus)
+{
+	struct pci_dev *bridge = bus->self;
+
+	return bridge && pcie_is_cxl(bridge) && cxl_sbr_masked(bridge);
+}
+
 static int pci_reset_bus_function(struct pci_dev *dev, bool probe)
 {
 	struct pci_dev *bridge = pci_upstream_bridge(dev);
@@ -5615,6 +5629,9 @@ static int pci_slot_reset(struct pci_slot *slot, bool probe)
  */
 int pci_probe_reset_slot(struct pci_slot *slot)
 {
+	if (slot && pci_bus_cxl_sbr_masked(slot->bus))
+		return -ENOTTY;
+
 	return pci_slot_reset(slot, PCI_RESET_PROBE);
 }
 EXPORT_SYMBOL_GPL(pci_probe_reset_slot);
@@ -5638,7 +5655,7 @@ static int pci_try_reset_slot(struct pci_slot *slot)
 {
 	int rc;
 
-	rc = pci_slot_reset(slot, PCI_RESET_PROBE);
+	rc = pci_probe_reset_slot(slot);
 	if (rc)
 		return rc;
 
@@ -5685,7 +5702,7 @@ static int pci_try_reset_bus(struct pci_bus *bus)
 {
 	int rc;
 
-	rc = pci_bus_reset(bus, PCI_RESET_PROBE);
+	rc = pci_probe_reset_bus(bus);
 	if (rc)
 		return rc;
 
@@ -5726,8 +5743,13 @@ static int pci_reset_bridge(struct pci_dev *bridge, bool restore)
 	if (list_empty(&bus->slots))
 		goto bus_reset;
 
+	/*
+	 * Probe without the CXL SBR mask check: error recovery keeps its
+	 * reset path below a masked CXL port, and pci_try_reset_slot() and
+	 * pci_try_reset_bus() refuse the restore path there.
+	 */
 	list_for_each_entry(slot, &bus->slots, list)
-		if (pci_probe_reset_slot(slot))
+		if (pci_slot_reset(slot, PCI_RESET_PROBE))
 			goto bus_reset;
 
 	list_for_each_entry(slot, &bus->slots, list) {
@@ -5773,6 +5795,9 @@ int pci_try_reset_bridge(struct pci_dev *bridge)
  */
 int pci_probe_reset_bus(struct pci_bus *bus)
 {
+	if (pci_bus_cxl_sbr_masked(bus))
+		return -ENOTTY;
+
 	return pci_bus_reset(bus, PCI_RESET_PROBE);
 }
 EXPORT_SYMBOL_GPL(pci_probe_reset_bus);
-- 
2.53.0

[PATCH v6 06/31] vfio/pci: Add the CXL provider ops registration interface

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

vfio-pci-core must stay free of any CXL header dependency, so CXL Type-2
handling lives in a separate vfio-cxl module that plugs in a set of
callbacks.

Add the registration interface:
- vfio-cxl registers a single struct vfio_cxl_ops at module_init
- vfio-pci-core stores it under an rwsem.

A second registration is refused with -EBUSY.

The owner field will be used to pin vfio-cxl for the lifetime of each
bound CXL device.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci_core.c | 24 ++++++++++++++++++++++++
 include/linux/vfio_pci_core.h    | 10 ++++++++++
 2 files changed, 34 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 6757054e9d87..2cc5dd20396c 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -2178,6 +2178,30 @@ static void vfio_pci_vga_uninit(struct vfio_pci_core_device *vdev)
 					      VGA_RSRC_LEGACY_MEM);
 }
 
+static const struct vfio_cxl_ops *vfio_pci_cxl_ops;
+static DECLARE_RWSEM(vfio_pci_cxl_ops_rwsem);
+
+int vfio_pci_core_register_cxl_ops(const struct vfio_cxl_ops *ops)
+{
+	guard(rwsem_write)(&vfio_pci_cxl_ops_rwsem);
+
+	if (vfio_pci_cxl_ops)
+		return -EBUSY;
+
+	vfio_pci_cxl_ops = ops;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_register_cxl_ops);
+
+void vfio_pci_core_unregister_cxl_ops(const struct vfio_cxl_ops *ops)
+{
+	guard(rwsem_write)(&vfio_pci_cxl_ops_rwsem);
+
+	if (vfio_pci_cxl_ops == ops)
+		vfio_pci_cxl_ops = NULL;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_unregister_cxl_ops);
+
 int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
 {
 	struct vfio_pci_core_device *vdev =
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 9a1674c152aa..9fe0d1a3a370 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -66,6 +66,16 @@ struct vfio_pci_device_ops {
 			       size_t nr_ranges);
 };
 
+struct vfio_cxl_ops {
+	int	(*init)(struct vfio_pci_core_device *vdev);
+	void	(*release)(struct vfio_pci_core_device *vdev);
+	/* Pinned per bound CXL device so vfio-cxl cannot unload under usage */
+	struct module *owner;
+};
+
+int vfio_pci_core_register_cxl_ops(const struct vfio_cxl_ops *ops);
+void vfio_pci_core_unregister_cxl_ops(const struct vfio_cxl_ops *ops);
+
 #if IS_ENABLED(CONFIG_VFIO_PCI_DMABUF)
 int vfio_pci_core_fill_phys_vec(struct phys_vec *phys_vec,
 				struct vfio_region_dma_range *dma_ranges,
-- 
2.53.0

[PATCH v6 07/31] vfio/pci: Detect CXL devices and load the CXL provider on demand

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A CXL device needs the vfio-cxl callbacks, but pulling vfio-cxl and the
CXL core in unconditionally would bloat every vfio-pci setup. Detect a
CXL Type-2 device at bind and request_module("vfio-cxl") only then, then
hand the device to the registered ops.

Each bound CXL device pins the provider through
vfio_pci_get_cxl_ops() (try_module_get) and drops it with
vfio_pci_put_cxl_ops() at release, so vfio-cxl can unload once no CXL
device is bound.

If the provider is absent the device is bound without the CXL
extensions. A built-in provider whose initcall has not run yet is waited
for with -EPROBE_DEFER; the deferred-probe machinery will retry the bind
once the provider registers.

The provider .init() can return -EPROBE_DEFER when the device is not
ready yet, for example when its CXL port has not enumerated. Propagate
it so the bind is retried later. Any other init failure is non-fatal: a
CXL Type-2 device is also a valid vfio-pci device, so log the error, drop
the provider reference, leave cxl_ops NULL and continue without CXL
support.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci_core.c | 101 +++++++++++++++++++++++++++++++
 include/linux/vfio_pci_core.h    |   3 +
 2 files changed, 104 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 2cc5dd20396c..e50b36751d9a 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -2202,6 +2202,101 @@ void vfio_pci_core_unregister_cxl_ops(const struct vfio_cxl_ops *ops)
 }
 EXPORT_SYMBOL_GPL(vfio_pci_core_unregister_cxl_ops);
 
+static const struct vfio_cxl_ops *vfio_pci_get_cxl_ops(void)
+{
+	guard(rwsem_read)(&vfio_pci_cxl_ops_rwsem);
+
+	if (vfio_pci_cxl_ops && try_module_get(vfio_pci_cxl_ops->owner))
+		return vfio_pci_cxl_ops;
+
+	return NULL;
+}
+
+static void vfio_pci_put_cxl_ops(const struct vfio_cxl_ops *ops)
+{
+	module_put(ops->owner);
+}
+
+/*
+ * A CXL Type-2 device advertises both CXL.cache and CXL.mem in its CXL DVSEC.
+ * pcie_is_cxl() is also true for Type-1 (cache only) and Type-3 (mem only)
+ * devices, which the vfio-cxl provider does not handle, so confirm the Type-2
+ * identity before engaging it.
+ */
+static bool vfio_pci_is_cxl_type2(struct pci_dev *pdev)
+{
+	u16 dvsec, cap;
+
+	if (!pcie_is_cxl(pdev))
+		return false;
+
+	dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_DEVICE);
+	if (!dvsec)
+		return false;
+
+	if (pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap))
+		return false;
+
+	return (cap & PCI_DVSEC_CXL_CACHE_CAPABLE) &&
+		(cap & PCI_DVSEC_CXL_MEM_CAPABLE);
+}
+
+/*
+ * Load vfio-cxl on demand for a CXL Type-2 device and hand the device to its
+ * ops. If the provider is absent the device is bound without the CXL
+ * extensions; a built-in provider whose initcall has not run yet is waited for with
+ * -EPROBE_DEFER.
+ */
+static int vfio_pci_core_cxl_init(struct vfio_pci_core_device *vdev)
+{
+	const struct vfio_cxl_ops *ops;
+	int ret;
+
+	if (!vfio_pci_is_cxl_type2(vdev->pdev))
+		return 0;
+
+	request_module("vfio-cxl");
+	ops = vfio_pci_get_cxl_ops();
+	if (!ops)
+		return IS_BUILTIN(CONFIG_VFIO_CXL) ? -EPROBE_DEFER : 0;
+
+	ret = ops->init(vdev);
+	/*
+	 * A provider that is not ready yet (for example its CXL port has not
+	 * enumerated) returns -EPROBE_DEFER. Propagate it so the bind retries
+	 * rather than continuing without CXL support.
+	 */
+	if (ret == -EPROBE_DEFER) {
+		vfio_pci_put_cxl_ops(ops);
+		return ret;
+	}
+	/*
+	 * Any other failure is non-fatal: a CXL device that cannot be brought
+	 * up as Type-2 still works without the CXL extensions, so log and
+	 * continue rather than failing the bind.
+	 */
+	if (ret) {
+		pci_warn(vdev->pdev,
+			 "CXL init failed (%d), continuing without CXL support\n",
+			 ret);
+		vfio_pci_put_cxl_ops(ops);
+		return 0;
+	}
+
+	vdev->cxl_ops = ops;
+	return 0;
+}
+
+static void vfio_pci_core_cxl_release(struct vfio_pci_core_device *vdev)
+{
+	if (!vdev->cxl_ops)
+		return;
+
+	vdev->cxl_ops->release(vdev);
+	vfio_pci_put_cxl_ops(vdev->cxl_ops);
+}
+
 int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
 {
 	struct vfio_pci_core_device *vdev =
@@ -2223,6 +2318,10 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
 	init_rwsem(&vdev->memory_lock);
 	xa_init(&vdev->ctx);
 
+	ret = vfio_pci_core_cxl_init(vdev);
+	if (ret)
+		return ret;
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(vfio_pci_core_init_dev);
@@ -2232,6 +2331,8 @@ void vfio_pci_core_release_dev(struct vfio_device *core_vdev)
 	struct vfio_pci_core_device *vdev =
 		container_of(core_vdev, struct vfio_pci_core_device, vdev);
 
+	vfio_pci_core_cxl_release(vdev);
+
 	mutex_destroy(&vdev->igate);
 	mutex_destroy(&vdev->ioeventfds_lock);
 	kfree(vdev->region);
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 9fe0d1a3a370..7f3a2bcb5830 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -29,6 +29,7 @@ struct vfio_pci_core_device;
 struct vfio_pci_region;
 struct p2pdma_provider;
 struct dma_buf_attachment;
+struct vfio_cxl_state;
 
 struct vfio_pci_eventfd {
 	struct eventfd_ctx	*ctx;
@@ -109,6 +110,8 @@ struct vfio_pci_core_device {
 	struct vfio_device	vdev;
 	struct pci_dev		*pdev;
 	const struct vfio_pci_device_ops *pci_ops;
+	const struct vfio_cxl_ops *cxl_ops;
+	struct vfio_cxl_state   *cxl;
 	void __iomem		*barmap[PCI_STD_NUM_BARS];
 	bool			bar_mmap_supported[PCI_STD_NUM_BARS];
 	/* Flags modified at runtime - dedicated storage unit */
-- 
2.53.0

[PATCH v6 08/31] vfio/pci: Add a generic excluded-range list

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A provider such as vfio-cxl needs to keep sub-ranges of a BAR off the
direct guest path and serve them through its own trap. vfio-pci-core
already has one such window, the MSI-X table, open-coded in
vfio_pci_bar_rw().

Add vfio_pci_core_add_excluded_range(). Each BAR has its own list,
sorted by start, and an insert that overlaps a range on the same BAR or
ends past the BAR fails with -EINVAL. Ranges are added only while the
device is being opened and are freed in vfio_pci_core_disable(), so
readers on an open device take no lock.

The MSI-X table is virtualized in vfio_pci_bar_rw() by an open-coded
x_start/x_end window that fills reads with -1 and drops writes. Register
the MSI-X table as a read and write excluded range when the MSI-X
capability is parsed in vfio_pci_core_enable(), and walk the BAR's
ranges in order in vfio_pci_bar_rw(), so an access that spans several
ranges is handled in one pass. Only the part of the table inside its BAR
is registered, so a device whose MSI-X capability points outside the BAR
still opens, as before. The ROM BAR keeps its trailing -1 window,
outside the list.

No behavioral change for an MSI-X table in a standard BAR: an access to
it still reads -1 and drops writes.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci_core.c | 97 ++++++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_priv.h |  8 +++
 drivers/vfio/pci/vfio_pci_rdwr.c | 67 +++++++++++++++++++---
 include/linux/vfio_pci_core.h    | 20 +++++++
 4 files changed, 185 insertions(+), 7 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index e50b36751d9a..bee3d1b4d8d2 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -588,6 +588,8 @@ static const struct dev_pm_ops vfio_pci_core_pm_ops = {
 			   NULL)
 };
 
+static void vfio_pci_free_excluded_ranges(struct vfio_pci_core_device *vdev);
+
 int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 {
 	struct pci_dev *pdev = vdev->pdev;
@@ -644,6 +646,7 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 
 	msix_pos = pdev->msix_cap;
 	if (msix_pos) {
+		u64 bar_len = 0;
 		u16 flags;
 		u32 table;
 
@@ -654,6 +657,30 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 		vdev->msix_offset = table & PCI_MSIX_TABLE_OFFSET;
 		vdev->msix_size = ((flags & PCI_MSIX_FLAGS_QSIZE) + 1) * 16;
 		vdev->has_dyn_msix = pci_msix_can_alloc_dyn(pdev);
+
+		/*
+		 * Virtualize the MSI-X table through the excluded-range list:
+		 * reads fill -1 and writes are dropped so the guest never
+		 * reaches the hardware table directly. BAR access never goes
+		 * past the BAR, so only the part of the table inside the BAR
+		 * needs a range; a table placed outside its BAR needs none.
+		 */
+		if (vdev->msix_bar < PCI_STD_NUM_BARS)
+			bar_len = pci_resource_len(pdev, vdev->msix_bar);
+		if (vdev->msix_offset < bar_len) {
+			u64 len = min_t(u64, vdev->msix_size,
+					bar_len - vdev->msix_offset);
+
+			ret = vfio_pci_core_add_excluded_range(vdev, vdev->msix_bar,
+							       vdev->msix_offset, len,
+							       VFIO_PCI_EXCLUDE_READ |
+							       VFIO_PCI_EXCLUDE_WRITE);
+			if (ret) {
+				vfio_pci_free_excluded_ranges(vdev);
+				vfio_config_free(vdev);
+				goto out_free_zdev;
+			}
+		}
 	} else {
 		vdev->msix_bar = 0xFF;
 		vdev->has_dyn_msix = false;
@@ -740,6 +767,7 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
 	vdev->region = NULL; /* don't krealloc a freed pointer */
 
 	vfio_config_free(vdev);
+	vfio_pci_free_excluded_ranges(vdev);
 
 	for (i = 0; i < PCI_STD_NUM_BARS; i++) {
 		bar = i + PCI_STD_RESOURCES;
@@ -1012,6 +1040,72 @@ static int msix_mmappable_cap(struct vfio_pci_core_device *vdev,
 	return vfio_info_add_capability(caps, &header, sizeof(header));
 }
 
+/**
+ * vfio_pci_core_add_excluded_range() - keep a BAR sub-range off direct access
+ * @vdev: vfio PCI device being opened
+ * @bar: BAR index, below PCI_STD_NUM_BARS
+ * @start: offset of the range in @bar
+ * @size: length of the range in bytes
+ * @flags: VFIO_PCI_EXCLUDE_* paths the range is excluded from
+ *
+ * Call only while the device is being opened, from vfio_pci_core_enable() or
+ * a provider's open hook; vfio_pci_core_disable() frees the ranges. Ranges on
+ * one BAR must not overlap and must lie inside the BAR.
+ *
+ * Return: 0 on success, -EINVAL for an invalid or overlapping range or one
+ * that ends past the BAR, or -ENOMEM.
+ */
+int vfio_pci_core_add_excluded_range(struct vfio_pci_core_device *vdev, int bar,
+				     u64 start, u64 size, u32 flags)
+{
+	const u32 valid = VFIO_PCI_EXCLUDE_MMAP | VFIO_PCI_EXCLUDE_READ |
+			  VFIO_PCI_EXCLUDE_WRITE;
+	struct vfio_pci_excluded_range *range, *pos;
+	struct list_head *head;
+
+	if (bar < 0 || bar >= PCI_STD_NUM_BARS || !size ||
+	    start + size < start || !flags || (flags & ~valid))
+		return -EINVAL;
+	if (start + size > pci_resource_len(vdev->pdev, bar))
+		return -EINVAL;
+
+	/* Keep the list sorted by start; find the first range after @start. */
+	head = &vdev->excluded_ranges[bar];
+	list_for_each_entry(pos, head, entry) {
+		if (start < pos->start + pos->size && start + size > pos->start)
+			return -EINVAL;
+		if (pos->start > start)
+			break;
+	}
+
+	range = kmalloc_obj(*range);
+	if (!range)
+		return -ENOMEM;
+
+	*range = (struct vfio_pci_excluded_range) {
+		.start = start,
+		.size = size,
+		.flags = flags,
+	};
+	list_add_tail(&range->entry, &pos->entry);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_add_excluded_range);
+
+static void vfio_pci_free_excluded_ranges(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_excluded_range *range, *tmp;
+
+	for (int bar = 0; bar < PCI_STD_NUM_BARS; bar++) {
+		list_for_each_entry_safe(range, tmp, &vdev->excluded_ranges[bar],
+					 entry) {
+			list_del(&range->entry);
+			kfree(range);
+		}
+	}
+}
+
 int vfio_pci_core_register_dev_region(struct vfio_pci_core_device *vdev,
 				      unsigned int type, unsigned int subtype,
 				      const struct vfio_pci_regops *ops,
@@ -2317,6 +2411,8 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
 	INIT_LIST_HEAD(&vdev->dmabufs);
 	init_rwsem(&vdev->memory_lock);
 	xa_init(&vdev->ctx);
+	for (int bar = 0; bar < PCI_STD_NUM_BARS; bar++)
+		INIT_LIST_HEAD(&vdev->excluded_ranges[bar]);
 
 	ret = vfio_pci_core_cxl_init(vdev);
 	if (ret)
@@ -2332,6 +2428,7 @@ void vfio_pci_core_release_dev(struct vfio_device *core_vdev)
 		container_of(core_vdev, struct vfio_pci_core_device, vdev);
 
 	vfio_pci_core_cxl_release(vdev);
+	vfio_pci_free_excluded_ranges(vdev);
 
 	mutex_destroy(&vdev->igate);
 	mutex_destroy(&vdev->ioeventfds_lock);
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index 4e7162234a2e..53544e0cbedf 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -44,6 +44,14 @@ ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev,
 ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 			size_t count, loff_t *ppos, bool iswrite);
 
+/* A BAR sub-range kept off direct access; see vfio_pci_core_add_excluded_range() */
+struct vfio_pci_excluded_range {
+	struct list_head	entry;
+	u64			start;
+	u64			size;
+	u32			flags;
+};
+
 #ifdef CONFIG_VFIO_PCI_VGA
 ssize_t vfio_pci_vga_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 			size_t count, loff_t *ppos, bool iswrite);
diff --git a/drivers/vfio/pci/vfio_pci_rdwr.c b/drivers/vfio/pci/vfio_pci_rdwr.c
index 7f14dd46de17..93823c6391f6 100644
--- a/drivers/vfio/pci/vfio_pci_rdwr.c
+++ b/drivers/vfio/pci/vfio_pci_rdwr.c
@@ -198,6 +198,57 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
 }
 EXPORT_SYMBOL_GPL(vfio_pci_core_do_io_rw);
 
+/*
+ * Access @count bytes of @bar at @pos, walking the BAR's excluded ranges in
+ * order: a range excluded for this direction reads as -1 and drops writes,
+ * and the bytes around it reach the device. The MSI-X table is one such
+ * range, and a BAR can hold several, for example the MSI-X table and a CXL
+ * HDM decoder block.
+ */
+static ssize_t vfio_pci_bar_do_io_rw(struct vfio_pci_core_device *vdev,
+				     int bar, bool test_mem, void __iomem *io,
+				     char __user *buf, loff_t pos,
+				     size_t count, bool iswrite,
+				     enum vfio_pci_io_width max_width)
+{
+	u32 want = iswrite ? VFIO_PCI_EXCLUDE_WRITE : VFIO_PCI_EXCLUDE_READ;
+	struct vfio_pci_excluded_range *range;
+	ssize_t done = 0;
+	ssize_t ret;
+
+	list_for_each_entry(range, &vdev->excluded_ranges[bar], entry) {
+		u64 end = range->start + range->size;
+		size_t chunk;
+
+		if (!(range->flags & want) || end <= pos)
+			continue;
+		if (range->start >= pos + count)
+			break;
+
+		/* Up to the end of this range; do_io_rw() fills it. */
+		chunk = min_t(u64, count, end - pos);
+		ret = vfio_pci_core_do_io_rw(vdev, test_mem, io, buf, pos, chunk,
+					     range->start, end, iswrite,
+					     max_width);
+		if (ret < 0)
+			return done ?: ret;
+
+		done += ret;
+		pos += ret;
+		buf += ret;
+		count -= ret;
+		if (!count)
+			return done;
+	}
+
+	ret = vfio_pci_core_do_io_rw(vdev, test_mem, io, buf, pos, count, 0, 0,
+				     iswrite, max_width);
+	if (ret < 0)
+		return done ?: ret;
+
+	return done + ret;
+}
+
 ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 			size_t count, loff_t *ppos, bool iswrite)
 {
@@ -256,13 +307,15 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 		}
 	}
 
-	if (bar == vdev->msix_bar) {
-		x_start = vdev->msix_offset;
-		x_end = vdev->msix_offset + vdev->msix_size;
-	}
-
-	done = vfio_pci_core_do_io_rw(vdev, res->flags & IORESOURCE_MEM, io, buf, pos,
-				      count, x_start, x_end, iswrite, max_width);
+	if (bar == PCI_ROM_RESOURCE)
+		done = vfio_pci_core_do_io_rw(vdev, res->flags & IORESOURCE_MEM,
+					      io, buf, pos, count, x_start,
+					      x_end, iswrite, max_width);
+	else
+		done = vfio_pci_bar_do_io_rw(vdev, bar,
+					     res->flags & IORESOURCE_MEM, io,
+					     buf, pos, count, iswrite,
+					     max_width);
 
 	if (done >= 0)
 		*ppos += done;
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 7f3a2bcb5830..70acb03d466e 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -162,6 +162,13 @@ struct vfio_pci_core_device {
 	struct notifier_block	nb;
 	struct rw_semaphore	memory_lock;
 	struct list_head	dmabufs;
+	/*
+	 * Excluded BAR sub-ranges, one list per BAR, sorted by start and not
+	 * overlapping. The lists are filled only while the device is being
+	 * opened and emptied in vfio_pci_core_disable(); every reader runs on
+	 * an open device in between, so readers take no lock.
+	 */
+	struct list_head	excluded_ranges[PCI_STD_NUM_BARS];
 };
 
 enum vfio_pci_io_width {
@@ -172,6 +179,19 @@ enum vfio_pci_io_width {
 };
 
 /* Will be exported for vfio pci drivers usage */
+/*
+ * A provider can keep a BAR sub-range off the direct guest path, reached only
+ * through its own trap. The flags select which paths are excluded: mmap, and
+ * region read and write (an excluded read fills -1, an excluded write is
+ * dropped).
+ */
+#define VFIO_PCI_EXCLUDE_MMAP	BIT(0)
+#define VFIO_PCI_EXCLUDE_READ	BIT(1)
+#define VFIO_PCI_EXCLUDE_WRITE	BIT(2)
+
+int vfio_pci_core_add_excluded_range(struct vfio_pci_core_device *vdev, int bar,
+				     u64 start, u64 size, u32 flags);
+
 int vfio_pci_core_register_dev_region(struct vfio_pci_core_device *vdev,
 				      unsigned int type, unsigned int subtype,
 				      const struct vfio_pci_regops *ops,
-- 
2.53.0

[PATCH v6 09/31] vfio/pci: Advertise the mmap areas around excluded ranges

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A range excluded from mmap must not be reachable through an mmap of its
BAR. Refuse an mmap that overlaps such a range, and advertise the BAR
with a sparse mmap capability that lists the areas between the excluded
ranges, so a VMM maps those and traps the rest.

mmap is page granular, so each range is rounded out to whole pages, and
ranges that meet after rounding form one hole. The sorted list gives the
areas in two walks: the first counts them and the second fills them.

A BAR without mmap-excluded ranges gets no new capability. The MSI-X
table is excluded from read and write only, so the MSI-X BAR is
unchanged.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci_core.c | 91 ++++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index bee3d1b4d8d2..f099a294af34 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1106,6 +1106,88 @@ static void vfio_pci_free_excluded_ranges(struct vfio_pci_core_device *vdev)
 	}
 }
 
+/* True when [start, start + len) on @bar overlaps a range with any of @flags. */
+static bool vfio_pci_bar_range_excluded(struct vfio_pci_core_device *vdev,
+					int bar, u64 start, u64 len, u32 flags)
+{
+	struct vfio_pci_excluded_range *range;
+
+	/* Only BARs have excluded ranges; a device-specific region has none. */
+	if (bar >= PCI_STD_NUM_BARS)
+		return false;
+
+	list_for_each_entry(range, &vdev->excluded_ranges[bar], entry) {
+		if (range->start >= start + len)
+			break;
+		if ((range->flags & flags) &&
+		    start < range->start + range->size)
+			return true;
+	}
+
+	return false;
+}
+
+/*
+ * Advertise a BAR with mmap-excluded ranges as the mmap-able areas between
+ * them. mmap is page granular, so each range is rounded out to whole pages;
+ * ranges that meet after rounding merge into one hole. The first walk counts
+ * the areas and the second fills them.
+ */
+static int vfio_pci_excluded_sparse_cap(struct vfio_pci_core_device *vdev,
+					int bar, struct vfio_info_cap *caps)
+{
+	struct vfio_region_info_cap_sparse_mmap *sparse __free(kfree) = NULL;
+	u64 bar_len = pci_resource_len(vdev->pdev, bar);
+	struct vfio_pci_excluded_range *range;
+	u32 nr_areas = 0;
+	bool holes = false;
+	u64 pos = 0;
+	u32 i = 0;
+
+	list_for_each_entry(range, &vdev->excluded_ranges[bar], entry) {
+		if (!(range->flags & VFIO_PCI_EXCLUDE_MMAP))
+			continue;
+		holes = true;
+		if (ALIGN_DOWN(range->start, PAGE_SIZE) > pos)
+			nr_areas++;
+		pos = max(pos, ALIGN(range->start + range->size, PAGE_SIZE));
+	}
+	if (!holes)
+		return 0;
+	if (pos < bar_len)
+		nr_areas++;
+
+	sparse = kzalloc_flex(*sparse, areas, nr_areas);
+	if (!sparse)
+		return -ENOMEM;
+
+	sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
+	sparse->header.version = 1;
+	sparse->nr_areas = nr_areas;
+
+	pos = 0;
+	list_for_each_entry(range, &vdev->excluded_ranges[bar], entry) {
+		u64 hole = ALIGN_DOWN(range->start, PAGE_SIZE);
+
+		if (!(range->flags & VFIO_PCI_EXCLUDE_MMAP))
+			continue;
+		if (hole > pos)
+			sparse->areas[i++] = (struct vfio_region_sparse_mmap_area) {
+				.offset = pos,
+				.size = hole - pos,
+			};
+		pos = max(pos, ALIGN(range->start + range->size, PAGE_SIZE));
+	}
+	if (pos < bar_len)
+		sparse->areas[i] = (struct vfio_region_sparse_mmap_area) {
+			.offset = pos,
+			.size = bar_len - pos,
+		};
+
+	return vfio_info_add_capability(caps, &sparse->header,
+					struct_size(sparse, areas, nr_areas));
+}
+
 int vfio_pci_core_register_dev_region(struct vfio_pci_core_device *vdev,
 				      unsigned int type, unsigned int subtype,
 				      const struct vfio_pci_regops *ops,
@@ -1254,6 +1336,10 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
 				if (ret)
 					return ret;
 			}
+			ret = vfio_pci_excluded_sparse_cap(vdev, info->index,
+							   caps);
+			if (ret)
+				return ret;
 		}
 
 		break;
@@ -1947,6 +2033,11 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma
 	if (req_start + req_len > phys_len)
 		return -EINVAL;
 
+	/* An excluded sub-range is reachable only through its trap, not mmap. */
+	if (vfio_pci_bar_range_excluded(vdev, index, req_start, req_len,
+					VFIO_PCI_EXCLUDE_MMAP))
+		return -EINVAL;
+
 	/*
 	 * Ensure the BAR resource region is reserved for use.
 	 */
-- 
2.53.0

[PATCH v6 10/31] vfio/pci: Refuse ioeventfds and dma-buf exports over excluded ranges

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

vfio_pci_bar_rw() keeps the excluded ranges of a BAR off the device, but
two other paths reach BAR MMIO without it.

An ioeventfd writes the BAR directly when it fires, through
vfio_pci_core_iowriteN(). Refuse an ioeventfd whose range overlaps a
write-excluded range, as one over the MSI-X table is already refused.

A dma-buf export of a BAR range lets an importer map it for DMA, which
reaches a range that the BAR keeps off mmap. Refuse an export whose
ranges overlap an mmap-excluded range, so the dma-buf follows the same
rule as mmap.

Suggested-by: Alex Williamson <alex@shazbot.org>
Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci_core.c   |  4 ++--
 drivers/vfio/pci/vfio_pci_dmabuf.c | 14 ++++++++++++++
 drivers/vfio/pci/vfio_pci_priv.h   |  3 +++
 drivers/vfio/pci/vfio_pci_rdwr.c   |  9 +++++++++
 4 files changed, 28 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index f099a294af34..211fff6045e2 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1107,8 +1107,8 @@ static void vfio_pci_free_excluded_ranges(struct vfio_pci_core_device *vdev)
 }
 
 /* True when [start, start + len) on @bar overlaps a range with any of @flags. */
-static bool vfio_pci_bar_range_excluded(struct vfio_pci_core_device *vdev,
-					int bar, u64 start, u64 len, u32 flags)
+bool vfio_pci_bar_range_excluded(struct vfio_pci_core_device *vdev, int bar,
+				 u64 start, u64 len, u32 flags)
 {
 	struct vfio_pci_excluded_range *range;
 
diff --git a/drivers/vfio/pci/vfio_pci_dmabuf.c b/drivers/vfio/pci/vfio_pci_dmabuf.c
index c16f460c01d6..8606c15ab2fe 100644
--- a/drivers/vfio/pci/vfio_pci_dmabuf.c
+++ b/drivers/vfio/pci/vfio_pci_dmabuf.c
@@ -260,6 +260,20 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 	if (ret)
 		goto err_free_ranges;
 
+	/*
+	 * A DMA mapping built from the dma-buf would reach a range that the
+	 * BAR keeps off mmap, so refuse such a range here as mmap does.
+	 */
+	for (u32 i = 0; i < get_dma_buf.nr_ranges; i++) {
+		if (vfio_pci_bar_range_excluded(vdev, get_dma_buf.region_index,
+						dma_ranges[i].offset,
+						dma_ranges[i].length,
+						VFIO_PCI_EXCLUDE_MMAP)) {
+			ret = -EINVAL;
+			goto err_free_ranges;
+		}
+	}
+
 	priv = kzalloc_obj(*priv);
 	if (!priv) {
 		ret = -ENOMEM;
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index 53544e0cbedf..09a68c99b43c 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -52,6 +52,9 @@ struct vfio_pci_excluded_range {
 	u32			flags;
 };
 
+bool vfio_pci_bar_range_excluded(struct vfio_pci_core_device *vdev, int bar,
+				 u64 start, u64 len, u32 flags);
+
 #ifdef CONFIG_VFIO_PCI_VGA
 ssize_t vfio_pci_vga_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 			size_t count, loff_t *ppos, bool iswrite);
diff --git a/drivers/vfio/pci/vfio_pci_rdwr.c b/drivers/vfio/pci/vfio_pci_rdwr.c
index 93823c6391f6..ef098f0b1401 100644
--- a/drivers/vfio/pci/vfio_pci_rdwr.c
+++ b/drivers/vfio/pci/vfio_pci_rdwr.c
@@ -476,6 +476,15 @@ int vfio_pci_ioeventfd(struct vfio_pci_core_device *vdev, loff_t offset,
 	      pos >= vdev->msix_offset + vdev->msix_size))
 		return -EINVAL;
 
+	/*
+	 * The ioeventfd fire path writes the BAR directly, without the
+	 * excluded-range filter of vfio_pci_bar_rw(), so refuse a write-excluded
+	 * range here.
+	 */
+	if (vfio_pci_bar_range_excluded(vdev, bar, pos, count,
+					VFIO_PCI_EXCLUDE_WRITE))
+		return -EINVAL;
+
 	if (count == 8)
 		return -EINVAL;
 
-- 
2.53.0

[PATCH v6 11/31] vfio/pci: Virtualize the CXL DVSEC in vfio_pci_config.c

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A CXL Type-2 device is reprogrammable through its CXL DVSEC: a guest could
set Config Lock, toggle CXL.cache and CXL.mem enable, or rewrite the HDM
range registers that govern host memory decode. Virtualize the DVSEC so
the guest sees a shadow it cannot use to reprogram the hardware.

Build a per-device cxl_perm permission map, modeled on msi_perm. When
the device is bound through the CXL provider (e.g. vfio-cxl), the whole
CXL DVSEC is served from the vconfig shadow; only Control and Control2
are guest programmable, while Capability, Status, Lock and the Range
registers keep their firmware snapshot. A vendor DVSEC on the same
device is unaffected: the map is selected by the CXL DVSEC's range,
since every DVSEC has the same ID in the config map and two DVSECs can
be adjacent.

A Type-2 function bound without the provider (the provider is absent,
its init failed, or the extensions are disabled) gets a guard map
instead. The DVSEC reads from hardware, Control and Control2 stay
writable and the RW1C status bits stay clearable, but writes to Lock and
the Range registers are dropped, since they outlive the user. Initiate
CXL Reset uses the same doorbell; a cache write-back goes to the device.

Control2 carries the CXL reset and cache write-back-invalidate initiate
bits as self-clearing doorbells. With the provider, vfio never forwards
them to hardware, so a custom writefn synthesizes their completion in
the shadow: the initiate bit self-clears and the matching Status2 bit
(Cache Invalid or Reset Done) is set, so a guest that writes back its
cache (CXL r4.0 sec 9.6) and then runs CXL Reset (sec 9.7.2) progresses
instead of timing out. Cache Invalid is reported only while Disable
Caching is set, as the spec requires. The host performs the real cache
write-back and reset at the vfio reset points. Until a later patch in
this series runs the real reset from the doorbell, Initiate CXL Reset
completes in the shadow only.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci_config.c | 206 ++++++++++++++++++++++++++++-
 include/linux/vfio_pci_core.h      |   3 +
 2 files changed, 205 insertions(+), 4 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
index 9914f3ac69ae..1fc463540211 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -1747,6 +1747,194 @@ static const struct pci_device_id known_bogus_vf_intx_pin[] = {
  * for each area requiring emulated bits, but the array of pointers
  * would be comparable in size (at least for standard config space).
  */
+static int vfio_cxl_dvsec_write(struct vfio_pci_core_device *vdev, int pos,
+				int count, struct perm_bits *perm,
+				int offset, __le32 val)
+{
+	u16 start = vdev->cxl_dvsec;
+	__le16 *pctrl2, *pstatus2;
+	u16 ctrl2, status2;
+
+	count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
+	if (count < 0)
+		return count;
+
+	/* Only Control2 carries the self-clearing doorbells. */
+	if (offset > PCI_DVSEC_CXL_CTRL2 ||
+	    offset + count <= PCI_DVSEC_CXL_CTRL2)
+		return count;
+
+	pctrl2 = (__le16 *)&vdev->vconfig[start + PCI_DVSEC_CXL_CTRL2];
+	pstatus2 = (__le16 *)&vdev->vconfig[start + PCI_DVSEC_CXL_STATUS2];
+	ctrl2 = le16_to_cpu(*pctrl2);
+	status2 = le16_to_cpu(*pstatus2);
+
+	/*
+	 * INIT_CACHE_WBI and INIT_CXL_RST are self-clearing doorbells. With a
+	 * provider neither reaches hardware: synthesize their completion in the
+	 * shadow so the guest poll finishes, clearing the initiate bit and
+	 * setting the matching Status2 completion. Cache Invalid is set only
+	 * while caching is disabled and reads 0 otherwise (CXL r4.0 sec 8.1.3.5,
+	 * Table 8-9). Under the guard map INIT_CACHE_WBI and Cache Invalid go to
+	 * the device; only INIT_CXL_RST and its completion are virtualized.
+	 */
+	if (ctrl2 & PCI_DVSEC_CXL_INIT_CACHE_WBI) {
+		ctrl2 &= ~PCI_DVSEC_CXL_INIT_CACHE_WBI;
+		if (ctrl2 & PCI_DVSEC_CXL_DISABLE_CACHING)
+			status2 |= PCI_DVSEC_CXL_CACHE_INV;
+	}
+	if (!(ctrl2 & PCI_DVSEC_CXL_DISABLE_CACHING))
+		status2 &= ~PCI_DVSEC_CXL_CACHE_INV;
+	if (ctrl2 & PCI_DVSEC_CXL_INIT_CXL_RST) {
+		ctrl2 &= ~PCI_DVSEC_CXL_INIT_CXL_RST;
+		status2 &= ~PCI_DVSEC_CXL_RST_ERR;
+		status2 |= PCI_DVSEC_CXL_RST_DONE;
+	}
+
+	*pctrl2 = cpu_to_le16(ctrl2);
+	*pstatus2 = cpu_to_le16(status2);
+
+	return count;
+}
+
+static int init_cxl_dvsec_perm(struct perm_bits *perm, int len)
+{
+	if (alloc_perm_bits(perm, len))
+		return -ENOMEM;
+
+	perm->writefn = vfio_cxl_dvsec_write;
+
+	/* Serve the whole CXL DVSEC from the shadow. */
+	for (int i = 0; i < len; i++)
+		p_setb(perm, i, (u8)ALL_VIRT, NO_WRITE);
+
+	/*
+	 * Control and Control2 are guest programmable; Capability, Status,
+	 * Lock and the Range registers keep their firmware snapshot, so the
+	 * guest cannot set Config Lock or rewrite the capability and ranges.
+	 */
+	p_setw(perm, PCI_DVSEC_CXL_CTRL, (u16)ALL_VIRT, (u16)ALL_WRITE);
+	p_setw(perm, PCI_DVSEC_CXL_CTRL2, (u16)ALL_VIRT, (u16)ALL_WRITE);
+
+	return 0;
+}
+
+/*
+ * Without a CXL provider the device is bound without the CXL extensions, so
+ * the DVSEC is read from hardware, Control and Control2 stay writable, and
+ * the RW1C status bits in Status and Status2 (for example Viral_Status) can
+ * still be cleared. Drop the writes whose effect outlives the user: Config
+ * Lock and the Range registers. Initiate CXL Reset goes through the same
+ * doorbell as with a provider, so the reset runs at a vfio reset point under
+ * memory_lock.
+ */
+static int init_cxl_dvsec_guard_perm(struct perm_bits *perm, int len)
+{
+	if (alloc_perm_bits(perm, len))
+		return -ENOMEM;
+
+	perm->writefn = vfio_cxl_dvsec_write;
+
+	/* Extended capability header: ID and next pointer from vconfig. */
+	p_setd(perm, 0, ALL_VIRT, NO_WRITE);
+
+	p_setw(perm, PCI_DVSEC_CXL_CTRL, NO_VIRT, (u16)ALL_WRITE);
+	p_setw(perm, PCI_DVSEC_CXL_CTRL2, PCI_DVSEC_CXL_INIT_CXL_RST,
+	       (u16)ALL_WRITE);
+	/* Status (offset 0Eh, between Control and Control2): RW1C to the device. */
+	p_setw(perm, PCI_DVSEC_CXL_CTRL + 2, NO_VIRT, (u16)ALL_WRITE);
+	p_setw(perm, PCI_DVSEC_CXL_STATUS2,
+	       PCI_DVSEC_CXL_RST_DONE | PCI_DVSEC_CXL_RST_ERR,
+	       (u16)~(PCI_DVSEC_CXL_RST_DONE | PCI_DVSEC_CXL_RST_ERR));
+
+	return 0;
+}
+
+/* Virtualize the CXL DVSEC so a guest cannot reprogram the device through it. */
+static int vfio_cxl_dvsec_init(struct vfio_pci_core_device *vdev)
+{
+	struct pci_dev *pdev = vdev->pdev;
+	u32 dword;
+	u16 dvsec, cap;
+	int len, ret;
+
+	dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_DEVICE);
+	if (!dvsec)
+		return 0;
+
+	ret = pci_read_config_dword(pdev, dvsec + PCI_DVSEC_HEADER1, &dword);
+	if (ret)
+		return pcibios_err_to_errno(ret);
+	len = PCI_DVSEC_HEADER1_LEN(dword);
+
+	/*
+	 * The virtualization writes fixed DVSEC offsets up to Status2 (the reset
+	 * doorbell stamps it). A device that reports a shorter DVSEC is not a
+	 * usable Type-2 function; leave its DVSEC unvirtualized rather than
+	 * index the device-length-sized perm allocation past its end.
+	 */
+	if (len < PCI_DVSEC_CXL_STATUS2 + 2)
+		return 0;
+
+	/*
+	 * Without a provider, guard only a CXL Type-2 function, with the same
+	 * test vfio-pci-core uses to load the provider.
+	 */
+	if (!vdev->cxl_ops) {
+		if (!pcie_is_cxl(pdev))
+			return 0;
+		ret = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap);
+		if (ret)
+			return pcibios_err_to_errno(ret);
+		if (!(cap & PCI_DVSEC_CXL_CACHE_CAPABLE) ||
+		    !(cap & PCI_DVSEC_CXL_MEM_CAPABLE))
+			return 0;
+	}
+
+	vdev->cxl_perm = kmalloc_obj(*vdev->cxl_perm, GFP_KERNEL_ACCOUNT);
+	if (!vdev->cxl_perm)
+		return -ENOMEM;
+
+	if (vdev->cxl_ops)
+		ret = init_cxl_dvsec_perm(vdev->cxl_perm, len);
+	else
+		ret = init_cxl_dvsec_guard_perm(vdev->cxl_perm, len);
+	if (ret) {
+		kfree(vdev->cxl_perm);
+		vdev->cxl_perm = NULL;
+		return ret;
+	}
+
+	vdev->cxl_dvsec = dvsec;
+	vdev->cxl_dvsec_len = len;
+
+	return 0;
+}
+
+static void vfio_cxl_dvsec_exit(struct vfio_pci_core_device *vdev)
+{
+	if (!vdev->cxl_perm)
+		return;
+
+	free_perm_bits(vdev->cxl_perm);
+	kfree(vdev->cxl_perm);
+	vdev->cxl_perm = NULL;
+	vdev->cxl_dvsec = 0;
+	vdev->cxl_dvsec_len = 0;
+}
+
+/*
+ * Every DVSEC carries the same ID in pci_config_map, so vfio_find_cap_start()
+ * can walk from one DVSEC into an adjacent one. Match the CXL DVSEC by range.
+ */
+static bool vfio_cxl_dvsec_contains(struct vfio_pci_core_device *vdev,
+				    loff_t pos)
+{
+	return vdev->cxl_perm && pos >= vdev->cxl_dvsec &&
+	       pos < vdev->cxl_dvsec + vdev->cxl_dvsec_len;
+}
+
 int vfio_config_init(struct vfio_pci_core_device *vdev)
 {
 	struct pci_dev *pdev = vdev->pdev;
@@ -1842,6 +2030,10 @@ int vfio_config_init(struct vfio_pci_core_device *vdev)
 	if (ret)
 		goto out;
 
+	ret = vfio_cxl_dvsec_init(vdev);
+	if (ret)
+		goto out;
+
 	return 0;
 
 out:
@@ -1863,6 +2055,7 @@ void vfio_config_free(struct vfio_pci_core_device *vdev)
 		kfree(vdev->msi_perm);
 		vdev->msi_perm = NULL;
 	}
+	vfio_cxl_dvsec_exit(vdev);
 }
 
 /*
@@ -1926,12 +2119,17 @@ ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev,
 			 * of the extended capability list.  Use default, ro
 			 * access, which will virtualize the id and next values.
 			 */
-			if (cap_id > PCI_EXT_CAP_ID_MAX)
+			cap_start = vfio_find_cap_start(vdev, *ppos);
+
+			if (cap_id > PCI_EXT_CAP_ID_MAX) {
 				perm = &direct_ro_perms;
-			else
+			} else if (cap_id == PCI_EXT_CAP_ID_DVSEC &&
+				   vfio_cxl_dvsec_contains(vdev, *ppos)) {
+				perm = vdev->cxl_perm;
+				cap_start = vdev->cxl_dvsec;
+			} else {
 				perm = &ecap_perms[cap_id];
-
-			cap_start = vfio_find_cap_start(vdev, *ppos);
+			}
 		} else {
 			WARN_ON(cap_id > PCI_CAP_ID_MAX);
 
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 70acb03d466e..fd1ad0baf470 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -120,6 +120,9 @@ struct vfio_pci_core_device {
 	u8			*pci_config_map;
 	u8			*vconfig;
 	struct perm_bits	*msi_perm;
+	struct perm_bits	*cxl_perm;
+	u16			cxl_dvsec;
+	u16			cxl_dvsec_len;
 	spinlock_t		irqlock;
 	struct mutex		igate;
 	struct xarray		ctx;
-- 
2.53.0

[PATCH v6 12/31] vfio/pci: Call the CXL open and close hooks around device use

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

vfio-pci-core frees device regions and excluded ranges on close and on a
failed open, so a CXL provider must set them up at each open.

Add open and close to the CXL ops and call them from the common enable
and close paths. A failed open unwinds the enable in reverse order:
release the device regions, unmap the BARs, drop the excluded ranges,
then free the virtual config.

Factor the device region release and the BAR unmap out of
vfio_pci_core_disable() into helpers so the enable failure path and the
disable path share them.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci_core.c | 57 +++++++++++++++++++++++---------
 include/linux/vfio_pci_core.h    |  2 ++
 2 files changed, 44 insertions(+), 15 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 211fff6045e2..3d4e01eea7b1 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -590,6 +590,31 @@ static const struct dev_pm_ops vfio_pci_core_pm_ops = {
 
 static void vfio_pci_free_excluded_ranges(struct vfio_pci_core_device *vdev);
 
+static void vfio_pci_core_unmap_bars(struct vfio_pci_core_device *vdev)
+{
+	struct pci_dev *pdev = vdev->pdev;
+
+	for (int i = 0; i < PCI_STD_NUM_BARS; i++) {
+		int bar = i + PCI_STD_RESOURCES;
+
+		if (IS_ERR_OR_NULL(vdev->barmap[bar]))
+			continue;
+		pci_iounmap(pdev, vdev->barmap[bar]);
+		pci_release_selected_regions(pdev, 1 << bar);
+		vdev->barmap[bar] = NULL;
+	}
+}
+
+static void vfio_pci_core_free_dev_regions(struct vfio_pci_core_device *vdev)
+{
+	for (int i = 0; i < vdev->num_regions; i++)
+		vdev->region[i].ops->release(vdev, &vdev->region[i]);
+
+	vdev->num_regions = 0;
+	kfree(vdev->region);
+	vdev->region = NULL; /* don't krealloc a freed pointer */
+}
+
 int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 {
 	struct pci_dev *pdev = vdev->pdev;
@@ -691,8 +716,20 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 
 	vfio_pci_core_map_bars(vdev);
 
+	if (vdev->cxl_ops) {
+		ret = vdev->cxl_ops->open_device(vdev);
+		if (ret)
+			goto out_free_config;
+	}
+
 	return 0;
 
+out_free_config:
+	/* A failed first open never reaches vfio_pci_core_disable(). */
+	vfio_pci_core_free_dev_regions(vdev);
+	vfio_pci_core_unmap_bars(vdev);
+	vfio_pci_free_excluded_ranges(vdev);
+	vfio_config_free(vdev);
 out_free_zdev:
 	vfio_pci_zdev_close_device(vdev);
 out_free_state:
@@ -713,7 +750,6 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
 	struct pci_dev *pdev = vdev->pdev;
 	struct vfio_pci_dummy_resource *dummy_res, *tmp;
 	struct vfio_pci_ioeventfd *ioeventfd, *ioeventfd_tmp;
-	int i, bar;
 
 	/* For needs_reset */
 	lockdep_assert_held(&vdev->vdev.dev_set->lock);
@@ -759,24 +795,12 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
 
 	vdev->virq_disabled = false;
 
-	for (i = 0; i < vdev->num_regions; i++)
-		vdev->region[i].ops->release(vdev, &vdev->region[i]);
-
-	vdev->num_regions = 0;
-	kfree(vdev->region);
-	vdev->region = NULL; /* don't krealloc a freed pointer */
+	vfio_pci_core_free_dev_regions(vdev);
 
 	vfio_config_free(vdev);
 	vfio_pci_free_excluded_ranges(vdev);
 
-	for (i = 0; i < PCI_STD_NUM_BARS; i++) {
-		bar = i + PCI_STD_RESOURCES;
-		if (IS_ERR_OR_NULL(vdev->barmap[bar]))
-			continue;
-		pci_iounmap(pdev, vdev->barmap[bar]);
-		pci_release_selected_regions(pdev, 1 << bar);
-		vdev->barmap[bar] = NULL;
-	}
+	vfio_pci_core_unmap_bars(vdev);
 
 	list_for_each_entry_safe(dummy_res, tmp,
 				 &vdev->dummy_resources_list, res_next) {
@@ -859,6 +883,9 @@ void vfio_pci_core_close_device(struct vfio_device *core_vdev)
 #endif
 	vfio_pci_dma_buf_cleanup(vdev);
 
+	if (vdev->cxl_ops)
+		vdev->cxl_ops->close_device(vdev);
+
 	vfio_pci_core_disable(vdev);
 
 	mutex_lock(&vdev->igate);
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index fd1ad0baf470..6520a2dcfd4a 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -70,6 +70,8 @@ struct vfio_pci_device_ops {
 struct vfio_cxl_ops {
 	int	(*init)(struct vfio_pci_core_device *vdev);
 	void	(*release)(struct vfio_pci_core_device *vdev);
+	int	(*open_device)(struct vfio_pci_core_device *vdev);
+	void	(*close_device)(struct vfio_pci_core_device *vdev);
 	/* Pinned per bound CXL device so vfio-cxl cannot unload under usage */
 	struct module *owner;
 };
-- 
2.53.0

[PATCH v6 13/31] vfio/pci: Bracket PCI resets with the CXL reset hooks

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A CXL Type-2 device must quiesce its coherent link before any reset and
restore its HDM decoder state afterward, or a reset can leave the device
mastering the bus over decoders that are no longer valid. This has to
happen around every reset the PCI core drives for the device, not only
the ones VFIO issues directly, and around AER and DPC recovery, which
can reset the link without the reset callbacks.

Add:
- reset_prepare
- reset_done
- an optional error_resume to the CXL ops

Export:
- vfio_pci_core_reset_prepare()
- vfio_pci_core_reset_done()
- vfio_pci_core_error_detected()
- vfio_pci_core_error_resume()
which take the vfio device.

vfio-pci calls them from its own pci_error_handlers, which resolve the
driver data it owns. A variant driver that binds CXL devices calls them
from its own handlers the same way; a driver without reset_prepare,
reset_done, error_detected and resume callbacks gets no CXL support.

The provider hooks run with memory_lock held for write: vfio-pci-core
already holds it across its own hot reset, and
vfio_pci_core_reset_prepare() takes it for a reset started anywhere else
until vfio_pci_core_reset_done(). The hot reset records its task, so a
reset that another task starts meanwhile waits for memory_lock instead
of skipping it. pci_dev_lock() blocks user config access, which a
PCI_COMMAND write can be waiting for while it holds memory_lock, so
reset_prepare lets config access through while it waits. reset_done
returns an error when the provider could not restore its device, and
VFIO_DEVICE_PCI_HOT_RESET returns that error; for any other reset the
provider logs it.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci.c      |  39 +++++++-
 drivers/vfio/pci/vfio_pci_core.c | 154 +++++++++++++++++++++++++++++++
 include/linux/vfio_pci_core.h    |  22 +++++
 3 files changed, 214 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index 830369ff878d..286244059569 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -223,13 +223,50 @@ static const struct pci_device_id vfio_pci_table[] = {
 
 MODULE_DEVICE_TABLE(pci, vfio_pci_table);
 
+static void vfio_pci_reset_prepare(struct pci_dev *pdev)
+{
+	struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
+
+	vfio_pci_core_reset_prepare(vdev);
+}
+
+static void vfio_pci_reset_done(struct pci_dev *pdev)
+{
+	struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
+
+	vfio_pci_core_reset_done(vdev);
+}
+
+static pci_ers_result_t vfio_pci_error_detected(struct pci_dev *pdev,
+						pci_channel_state_t state)
+{
+	struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
+
+	vfio_pci_core_error_detected(vdev);
+	return vfio_pci_core_aer_err_detected(pdev, state);
+}
+
+static void vfio_pci_error_resume(struct pci_dev *pdev)
+{
+	struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev);
+
+	vfio_pci_core_error_resume(vdev);
+}
+
+static const struct pci_error_handlers vfio_pci_err_handlers = {
+	.error_detected = vfio_pci_error_detected,
+	.reset_prepare = vfio_pci_reset_prepare,
+	.reset_done = vfio_pci_reset_done,
+	.resume = vfio_pci_error_resume,
+};
+
 static struct pci_driver vfio_pci_driver = {
 	.name			= "vfio-pci",
 	.id_table		= vfio_pci_table,
 	.probe			= vfio_pci_probe,
 	.remove			= vfio_pci_remove,
 	.sriov_configure	= vfio_pci_sriov_configure,
-	.err_handler		= &vfio_pci_core_err_handlers,
+	.err_handler		= &vfio_pci_err_handlers,
 	.driver_managed_dma	= true,
 };
 
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 3d4e01eea7b1..f1e1ec36eec7 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -2462,6 +2462,7 @@ static bool vfio_pci_is_cxl_type2(struct pci_dev *pdev)
  */
 static int vfio_pci_core_cxl_init(struct vfio_pci_core_device *vdev)
 {
+	const struct pci_error_handlers *err_handler;
 	const struct vfio_cxl_ops *ops;
 	int ret;
 
@@ -2473,6 +2474,23 @@ static int vfio_pci_core_cxl_init(struct vfio_pci_core_device *vdev)
 	if (!ops)
 		return IS_BUILTIN(CONFIG_VFIO_CXL) ? -EPROBE_DEFER : 0;
 
+	/*
+	 * The provider must see every PCI core reset of the device and every
+	 * AER or DPC recovery. A driver without reset_prepare, reset_done,
+	 * error_detected and resume callbacks gets no CXL support; the driver's
+	 * callbacks must call vfio_pci_core_reset_prepare(), _reset_done(),
+	 * _error_detected() and _error_resume().
+	 */
+	err_handler = vdev->pdev->driver ? vdev->pdev->driver->err_handler : NULL;
+	if (!err_handler || !err_handler->reset_prepare ||
+	    !err_handler->reset_done || !err_handler->error_detected ||
+	    !err_handler->resume) {
+		pci_info(vdev->pdev,
+			 "driver lacks PCI reset or error recovery hooks, continuing without CXL support\n");
+		vfio_pci_put_cxl_ops(ops);
+		return 0;
+	}
+
 	ret = ops->init(vdev);
 	/*
 	 * A provider that is not ready yet (for example its CXL port has not
@@ -2759,6 +2777,128 @@ int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev,
 }
 EXPORT_SYMBOL_GPL(vfio_pci_core_sriov_configure);
 
+/*
+ * Take memory_lock for a reset that the PCI core runs on a locked device.
+ * pci_dev_lock() also blocks user config access, and vfio_basic_config_write()
+ * holds memory_lock while its pci_user_*_config() calls wait for that block
+ * to clear. Waiting for memory_lock with config access blocked would hang
+ * both tasks, so let user config access through while waiting: the PCI core
+ * calls reset_prepare before it saves or touches the device. The caller holds
+ * the device lock, so a block found set is the caller's own.
+ */
+static void vfio_pci_reset_lock_memory(struct vfio_pci_core_device *vdev)
+{
+	struct pci_dev *pdev = vdev->pdev;
+
+	if (down_write_trylock(&vdev->memory_lock))
+		return;
+
+	/* The caller did not block config access: nothing to release. */
+	if (pci_cfg_access_trylock(pdev)) {
+		pci_cfg_access_unlock(pdev);
+		down_write(&vdev->memory_lock);
+		return;
+	}
+
+	pci_cfg_access_unlock(pdev);
+	down_write(&vdev->memory_lock);
+	pci_cfg_access_lock(pdev);
+}
+
+/**
+ * vfio_pci_core_reset_prepare() - prepare a device for a PCI core reset
+ * @vdev: vfio PCI device about to be reset
+ *
+ * Call from the driver's pci_error_handlers.reset_prepare callback.
+ */
+void vfio_pci_core_reset_prepare(struct vfio_pci_core_device *vdev)
+{
+	if (!vdev->cxl_ops)
+		return;
+
+	/*
+	 * vfio-pci-core holds memory_lock across its own hot reset. Hold it
+	 * across a reset started anywhere else too, until reset_done. Test
+	 * the owner, not just a flag: another task can reset the device while
+	 * a hot reset holds memory_lock and has not yet locked the device.
+	 */
+	if (READ_ONCE(vdev->hot_reset_task) != current)
+		vfio_pci_reset_lock_memory(vdev);
+
+	vdev->cxl_ops->reset_prepare(vdev);
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_reset_prepare);
+
+/**
+ * vfio_pci_core_reset_done() - restore a device after a PCI core reset
+ * @vdev: vfio PCI device that was reset
+ *
+ * Call from the driver's pci_error_handlers.reset_done callback.
+ */
+void vfio_pci_core_reset_done(struct vfio_pci_core_device *vdev)
+{
+	int ret;
+
+	if (!vdev->cxl_ops)
+		return;
+
+	ret = vdev->cxl_ops->reset_done(vdev);
+
+	/*
+	 * hot_reset_task cannot change since reset_prepare: setting it needs
+	 * memory_lock, which this task or the hot reset task holds.
+	 */
+	if (READ_ONCE(vdev->hot_reset_task) == current) {
+		/* VFIO_DEVICE_PCI_HOT_RESET returns the error. */
+		vdev->reset_done_err = ret;
+		return;
+	}
+
+	/*
+	 * Nobody waits for the result of a reset started elsewhere. A provider
+	 * that failed has logged it and keeps its device memory out of use
+	 * until a later reset restores the device.
+	 */
+	up_write(&vdev->memory_lock);
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_reset_done);
+
+/**
+ * vfio_pci_core_error_detected() - prepare a device for AER or DPC recovery
+ * @vdev: vfio PCI device the recovery reports an error on
+ *
+ * Call from the driver's pci_error_handlers.error_detected callback. The
+ * recovery may reset the device without reset_prepare and reset_done, so a
+ * provider takes its device memory out of use here.
+ */
+void vfio_pci_core_error_detected(struct vfio_pci_core_device *vdev)
+{
+	if (!vdev->cxl_ops)
+		return;
+
+	guard(rwsem_write)(&vdev->memory_lock);
+	vdev->cxl_ops->reset_prepare(vdev);
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_error_detected);
+
+/**
+ * vfio_pci_core_error_resume() - resume a device after AER or DPC recovery
+ * @vdev: vfio PCI device the recovery resumes
+ *
+ * Call from the driver's pci_error_handlers.resume callback. A provider puts
+ * its device memory back if the device still decodes it; otherwise the memory
+ * stays out of use until the user resets the device.
+ */
+void vfio_pci_core_error_resume(struct vfio_pci_core_device *vdev)
+{
+	if (!vdev->cxl_ops || !vdev->cxl_ops->error_resume)
+		return;
+
+	guard(rwsem_write)(&vdev->memory_lock);
+	vdev->cxl_ops->error_resume(vdev);
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_error_resume);
+
 const struct pci_error_handlers vfio_pci_core_err_handlers = {
 	.error_detected = vfio_pci_core_aer_err_detected,
 };
@@ -2915,6 +3055,8 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 			ret = -EBUSY;
 			break;
 		}
+		WRITE_ONCE(vdev->hot_reset_task, current);
+		vdev->reset_done_err = 0;
 
 		vfio_pci_dma_buf_move(vdev, true);
 		vfio_pci_zap_bars(vdev);
@@ -2940,6 +3082,17 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 
 	ret = pci_reset_bus(pdev);
 
+	/* A provider that could not restore its device fails the hot reset. */
+	if (!ret) {
+		list_for_each_entry(vdev, &dev_set->device_list,
+				    vdev.dev_set_list) {
+			if (vdev->reset_done_err) {
+				ret = vdev->reset_done_err;
+				break;
+			}
+		}
+	}
+
 	vdev = list_last_entry(&dev_set->device_list,
 			       struct vfio_pci_core_device, vdev.dev_set_list);
 
@@ -2948,6 +3101,7 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 					 vdev.dev_set_list) {
 		if (vdev->vdev.open_count && __vfio_pci_memory_enabled(vdev))
 			vfio_pci_dma_buf_move(vdev, false);
+		WRITE_ONCE(vdev->hot_reset_task, NULL);
 		up_write(&vdev->memory_lock);
 	}
 
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 6520a2dcfd4a..881e51c42414 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -72,6 +72,21 @@ struct vfio_cxl_ops {
 	void	(*release)(struct vfio_pci_core_device *vdev);
 	int	(*open_device)(struct vfio_pci_core_device *vdev);
 	void	(*close_device)(struct vfio_pci_core_device *vdev);
+	/*
+	 * Optional. Called with memory_lock held for write when AER or DPC
+	 * recovery resumes the device, after reset_prepare was called for the
+	 * error. Config space is as the recovery left it.
+	 */
+	void	(*error_resume)(struct vfio_pci_core_device *vdev);
+	/*
+	 * Called around a PCI core reset of the device with memory_lock held
+	 * for write; reset_prepare also when AER or DPC recovery reports an
+	 * error. reset_done returns an error if the provider could not
+	 * restore the device; VFIO_DEVICE_PCI_HOT_RESET returns it, and for
+	 * any other reset the provider logs it.
+	 */
+	void	(*reset_prepare)(struct vfio_pci_core_device *vdev);
+	int	(*reset_done)(struct vfio_pci_core_device *vdev);
 	/* Pinned per bound CXL device so vfio-cxl cannot unload under usage */
 	struct module *owner;
 };
@@ -166,6 +181,9 @@ struct vfio_pci_core_device {
 	struct vfio_pci_core_device	*sriov_pf_core_dev;
 	struct notifier_block	nb;
 	struct rw_semaphore	memory_lock;
+	/* Task running VFIO_DEVICE_PCI_HOT_RESET while it holds memory_lock */
+	struct task_struct	*hot_reset_task;
+	int			reset_done_err;
 	struct list_head	dmabufs;
 	/*
 	 * Excluded BAR sub-ranges, one list per BAR, sorted by start and not
@@ -207,6 +225,10 @@ void vfio_pci_core_release_dev(struct vfio_device *core_vdev);
 int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev);
 void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev);
 extern const struct pci_error_handlers vfio_pci_core_err_handlers;
+void vfio_pci_core_reset_prepare(struct vfio_pci_core_device *vdev);
+void vfio_pci_core_reset_done(struct vfio_pci_core_device *vdev);
+void vfio_pci_core_error_detected(struct vfio_pci_core_device *vdev);
+void vfio_pci_core_error_resume(struct vfio_pci_core_device *vdev);
 int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev,
 				  int nr_virtfn);
 long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
-- 
2.53.0

[PATCH v6 14/31] vfio/pci: Provide an opt-out for the CXL Type-2 extensions

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Let an admin or a variant driver keep the CXL Type-2 extensions out of
the path and bind the device without them.

Add a vfio_pci.disable_cxl module parameter, and let a variant driver set
vdev->disable_cxl in its own init callback for per-device control. The
parameter is built only when the CXL provider is configured.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci.c      | 13 +++++++++++++
 drivers/vfio/pci/vfio_pci_core.c |  2 +-
 include/linux/vfio_pci_core.h    |  1 +
 3 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index 286244059569..0ea349fb8d4a 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -60,6 +60,16 @@ static bool disable_denylist;
 module_param(disable_denylist, bool, 0444);
 MODULE_PARM_DESC(disable_denylist, "Disable use of device denylist. Disabling the denylist allows binding to devices with known errata that may lead to exploitable stability or security issues when accessed by untrusted users.");
 
+#if IS_ENABLED(CONFIG_VFIO_CXL)
+/*
+ * A variant driver may instead set vdev->disable_cxl in its own init callback
+ * for per-device control without this module-wide parameter.
+ */
+static bool disable_cxl;
+module_param(disable_cxl, bool, 0644);
+MODULE_PARM_DESC(disable_cxl, "Disable CXL Type-2 extensions for all devices bound to vfio-pci.");
+#endif
+
 static bool vfio_pci_dev_in_denylist(struct pci_dev *pdev)
 {
 	switch (pdev->vendor) {
@@ -142,6 +152,9 @@ static int vfio_pci_init_dev(struct vfio_device *core_vdev)
 #ifdef CONFIG_VFIO_PCI_VGA
 	vdev->disable_vga = disable_vga;
 #endif
+#if IS_ENABLED(CONFIG_VFIO_CXL)
+	vdev->disable_cxl = disable_cxl;
+#endif
 
 	return vfio_pci_core_init_dev(core_vdev);
 }
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index f1e1ec36eec7..1e0ac28b7564 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -2466,7 +2466,7 @@ static int vfio_pci_core_cxl_init(struct vfio_pci_core_device *vdev)
 	const struct vfio_cxl_ops *ops;
 	int ret;
 
-	if (!vfio_pci_is_cxl_type2(vdev->pdev))
+	if (vdev->disable_cxl || !vfio_pci_is_cxl_type2(vdev->pdev))
 		return 0;
 
 	request_module("vfio-cxl");
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 881e51c42414..991a3fcd84eb 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -162,6 +162,7 @@ struct vfio_pci_core_device {
 	bool			disable_idle_d3:1;
 	bool			nointxmask:1;
 	bool			disable_vga:1;
+	bool			disable_cxl:1;
 	/* Flags modified at runtime - dedicated storage unit */
 	bool			needs_reset;
 	bool			pm_intx_masked;
-- 
2.53.0

[PATCH v6 15/31] vfio/pci: Route every reset point through a try-reset helper

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A provider such as vfio-cxl must reset its device itself: a CXL Type-2
function must not take an FLR, and its reset changes what the provider
lets the fault path map. vfio-pci-core resets a function at five points:
enable, close, VFIO_DEVICE_RESET, a virtualized PCIe or AF FLR, and the
guest's CXL Reset doorbell in the virtualized CXL DVSEC.

Add a mandatory reset op to the provider ops, and refuse to register ops
without it. Add vfio_pci_try_reset_function(), which keeps the trylock
of pci_try_reset_function(): it returns -EAGAIN when the device lock is
contended, and otherwise runs the provider's reset with the device lock
and memory_lock held. The close path keeps its bridge and device trylock
and calls a locked variant. vfio_pci_try_reset_function() saves the PCI
state first, as pci_try_reset_function() does, for the provider's reset
to restore. The locked variant does not save: on close the reset must
restore the open-time state that vfio_pci_core_disable() has just
loaded. A failed save is returned and the reset is not attempted.

The enable path still fails only on -EAGAIN. A provider checks at bind
that it can reset the device, so reset_works is set for a provider
device whatever the result of the first reset.

The CXL Reset doorbell now runs the reset and reports its result in
Status2: Reset Error on failure, Reset Done on success.

Suggested-by: Alex Williamson <alex@shazbot.org>
Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci_config.c | 19 ++++++--
 drivers/vfio/pci/vfio_pci_core.c   | 71 ++++++++++++++++++++++++++++--
 drivers/vfio/pci/vfio_pci_priv.h   |  1 +
 include/linux/vfio_pci_core.h      |  7 +++
 4 files changed, 90 insertions(+), 8 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
index 1fc463540211..6fbece95cb9f 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -910,7 +910,7 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,
 		if (!ret && (cap & PCI_EXP_DEVCAP_FLR)) {
 			vfio_pci_zap_and_down_write_memory_lock(vdev);
 			vfio_pci_dma_buf_move(vdev, true);
-			pci_try_reset_function(vdev->pdev);
+			vfio_pci_try_reset_function(vdev);
 			if (__vfio_pci_memory_enabled(vdev))
 				vfio_pci_dma_buf_move(vdev, false);
 			up_write(&vdev->memory_lock);
@@ -995,7 +995,7 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,
 		if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP)) {
 			vfio_pci_zap_and_down_write_memory_lock(vdev);
 			vfio_pci_dma_buf_move(vdev, true);
-			pci_try_reset_function(vdev->pdev);
+			vfio_pci_try_reset_function(vdev);
 			if (__vfio_pci_memory_enabled(vdev))
 				vfio_pci_dma_buf_move(vdev, false);
 			up_write(&vdev->memory_lock);
@@ -1786,9 +1786,20 @@ static int vfio_cxl_dvsec_write(struct vfio_pci_core_device *vdev, int pos,
 	if (!(ctrl2 & PCI_DVSEC_CXL_DISABLE_CACHING))
 		status2 &= ~PCI_DVSEC_CXL_CACHE_INV;
 	if (ctrl2 & PCI_DVSEC_CXL_INIT_CXL_RST) {
+		int ret;
+
 		ctrl2 &= ~PCI_DVSEC_CXL_INIT_CXL_RST;
-		status2 &= ~PCI_DVSEC_CXL_RST_ERR;
-		status2 |= PCI_DVSEC_CXL_RST_DONE;
+
+		/* Run the real reset here and report its result in Status2. */
+		vfio_pci_zap_and_down_write_memory_lock(vdev);
+		vfio_pci_dma_buf_move(vdev, true);
+		ret = vfio_pci_try_reset_function(vdev);
+		if (__vfio_pci_memory_enabled(vdev))
+			vfio_pci_dma_buf_move(vdev, false);
+		up_write(&vdev->memory_lock);
+
+		status2 &= ~(PCI_DVSEC_CXL_RST_DONE | PCI_DVSEC_CXL_RST_ERR);
+		status2 |= ret ? PCI_DVSEC_CXL_RST_ERR : PCI_DVSEC_CXL_RST_DONE;
 	}
 
 	*pctrl2 = cpu_to_le16(ctrl2);
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 1e0ac28b7564..6676c561465b 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -605,6 +605,62 @@ static void vfio_pci_core_unmap_bars(struct vfio_pci_core_device *vdev)
 	}
 }
 
+/**
+ * vfio_pci_try_reset_function() - reset the function without blocking
+ * @vdev: vfio PCI device, with memory_lock held for write
+ *
+ * Reset through the PCI core, or let the provider reset a device it serves.
+ * The provider resets the device with its PCI device lock held, so take that
+ * lock here, without waiting, as pci_try_reset_function() does.
+ *
+ * Return: 0 on success, -EAGAIN when the device lock is contended, or the
+ * reset error. For a provider device a failed pci_save_state() is also
+ * returned, and the reset is not attempted: unlike pci_try_reset_function(),
+ * the provider's reset needs a valid saved state to restore.
+ */
+int vfio_pci_try_reset_function(struct vfio_pci_core_device *vdev)
+{
+	struct pci_dev *pdev = vdev->pdev;
+	int ret;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	if (!vdev->cxl_ops)
+		return pci_try_reset_function(pdev);
+
+	if (!pci_dev_trylock(pdev))
+		return -EAGAIN;
+
+	/*
+	 * The provider restores PCI state from the saved copy, so save the
+	 * current state first, as pci_try_reset_function() does. The locked
+	 * variant on the close path must not: it resets with the state that
+	 * vfio_pci_core_disable() has just loaded.
+	 */
+	ret = pci_save_state(pdev);
+	if (!ret)
+		ret = vdev->cxl_ops->reset(vdev);
+	pci_dev_unlock(pdev);
+
+	return ret;
+}
+
+/*
+ * As vfio_pci_try_reset_function(), for a caller that holds the device lock.
+ * Only the close path calls this. It takes memory_lock with the device lock
+ * held, so user config access is blocked meanwhile: the order a sysfs reset
+ * uses. That cannot deadlock against a PCI_COMMAND write holding memory_lock
+ * because no user remains at close.
+ */
+static int vfio_pci_reset_function_locked(struct vfio_pci_core_device *vdev)
+{
+	if (!vdev->cxl_ops)
+		return __pci_reset_function_locked(vdev->pdev);
+
+	guard(rwsem_write)(&vdev->memory_lock);
+	return vdev->cxl_ops->reset(vdev);
+}
+
 static void vfio_pci_core_free_dev_regions(struct vfio_pci_core_device *vdev)
 {
 	for (int i = 0; i < vdev->num_regions; i++)
@@ -636,11 +692,14 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 		goto out_power;
 
 	/* If reset fails because of the device lock, fail this path entirely */
-	ret = pci_try_reset_function(pdev);
+	down_write(&vdev->memory_lock);
+	ret = vfio_pci_try_reset_function(vdev);
+	up_write(&vdev->memory_lock);
 	if (ret == -EAGAIN)
 		goto out_disable_device;
 
-	vdev->reset_works = !ret;
+	/* A provider checks at bind that it can reset the device. */
+	vdev->reset_works = vdev->cxl_ops || !ret;
 	pci_save_state(pdev);
 	vdev->pci_saved_state = pci_store_saved_state(pdev);
 	if (!vdev->pci_saved_state)
@@ -846,7 +905,7 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
 		if (bridge && !pci_dev_trylock(bridge))
 			goto out_restore_state;
 		if (pci_dev_trylock(pdev)) {
-			if (!__pci_reset_function_locked(pdev))
+			if (!vfio_pci_reset_function_locked(vdev))
 				vdev->needs_reset = false;
 			pci_dev_unlock(pdev);
 		}
@@ -1541,7 +1600,7 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
 	vfio_pci_set_power_state(vdev, PCI_D0);
 
 	vfio_pci_dma_buf_move(vdev, true);
-	ret = pci_try_reset_function(vdev->pdev);
+	ret = vfio_pci_try_reset_function(vdev);
 	if (__vfio_pci_memory_enabled(vdev))
 		vfio_pci_dma_buf_move(vdev, false);
 	up_write(&vdev->memory_lock);
@@ -2397,6 +2456,10 @@ int vfio_pci_core_register_cxl_ops(const struct vfio_cxl_ops *ops)
 {
 	guard(rwsem_write)(&vfio_pci_cxl_ops_rwsem);
 
+	/* vfio-pci-core never resets a provider's device by itself. */
+	if (!ops->reset)
+		return -EINVAL;
+
 	if (vfio_pci_cxl_ops)
 		return -EBUSY;
 
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index 09a68c99b43c..dbede6e24fa9 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -80,6 +80,7 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev,
 			     pci_power_t state);
 
 void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev);
+int vfio_pci_try_reset_function(struct vfio_pci_core_device *vdev);
 u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev);
 void vfio_pci_memory_unlock_and_restore(struct vfio_pci_core_device *vdev,
 					u16 cmd);
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 991a3fcd84eb..cb9ba92b696a 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -87,6 +87,13 @@ struct vfio_cxl_ops {
 	 */
 	void	(*reset_prepare)(struct vfio_pci_core_device *vdev);
 	int	(*reset_done)(struct vfio_pci_core_device *vdev);
+	/*
+	 * Reset the function in place of the PCI core; mandatory. Called with
+	 * the PCI device lock and memory_lock (for write) held. The caller
+	 * provides the saved PCI state the reset restores: the try-reset path
+	 * saves it first, the close path keeps the open-time copy.
+	 */
+	int	(*reset)(struct vfio_pci_core_device *vdev);
 	/* Pinned per bound CXL device so vfio-cxl cannot unload under usage */
 	struct module *owner;
 };
-- 
2.53.0

[PATCH v6 16/31] vfio/pci: Revoke and re-arm one device-specific region

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

The BARs of a vfio PCI device follow the memory enable bit: clearing it
zaps their mmaps and revokes their dma-bufs, and setting it re-arms the
dma-bufs. A device-specific region does not have to follow that bit. The
HDM memory region of a CXL Type-2 device follows the committed HDM
decoder, which changes across reset, bus reset and low power, not with
Memory Space Enable. Other device-specific regions, such as nvgrace
coherent memory or the IGD OpRegion, do not depend on it either.

Record the region index of each dma-buf at export. vfio_pci_dma_buf_move()
now moves only the BAR dma-bufs, and vfio_pci_dma_buf_cleanup() still
drains every dma-buf of the device.

Export vfio_pci_core_revoke_region(), which zaps the user mmaps of one
device-specific region and revokes its dma-bufs, and
vfio_pci_core_rearm_region(), which re-arms them. A provider calls them
under memory_lock when the state its region depends on changes. Also
export __vfio_pci_vmf_insert_pfn(), the pfn insert of
vfio_pci_vmf_insert_pfn() without its memory enable and runtime PM
checks, for the fault handler of such a region. Export
vfio_pci_core_clear_memory_enable() for a provider whose failed
operation left Memory Space off in the device: it zaps the BAR mmaps,
revokes the BAR dma-bufs and clears vfio's record of the bit.

Suggested-by: Alex Williamson <alex@shazbot.org>
Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci_core.c   | 109 +++++++++++++++++++++++++++--
 drivers/vfio/pci/vfio_pci_dmabuf.c |  98 +++++++++++++++++---------
 drivers/vfio/pci/vfio_pci_priv.h   |   7 ++
 include/linux/vfio_pci_core.h      |   8 +++
 4 files changed, 180 insertions(+), 42 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 6676c561465b..9d0c22a4f3c6 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1982,6 +1982,77 @@ static void vfio_pci_zap_bars(struct vfio_pci_core_device *vdev)
 	unmap_mapping_range(core_vdev->inode->i_mapping, start, len, true);
 }
 
+/**
+ * vfio_pci_core_revoke_region() - take a device-specific region out of use
+ * @vdev: vfio PCI device, with memory_lock held for write
+ * @index: vfio region index of a device-specific region
+ *
+ * Zap the user mmaps of the region and revoke its dma-bufs, for a provider
+ * whose region depends on device state that is about to change. The BARs
+ * follow the memory enable bit instead; a device-specific region may not.
+ * A later fault on the region goes back to the region's own fault handler.
+ */
+void vfio_pci_core_revoke_region(struct vfio_pci_core_device *vdev,
+				 unsigned int index)
+{
+	struct vfio_pci_region *region;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	if (WARN_ON(index < VFIO_PCI_NUM_REGIONS ||
+		    index - VFIO_PCI_NUM_REGIONS >= vdev->num_regions))
+		return;
+
+	region = &vdev->region[index - VFIO_PCI_NUM_REGIONS];
+	unmap_mapping_range(vdev->vdev.inode->i_mapping,
+			    VFIO_PCI_INDEX_TO_OFFSET(index), region->size, true);
+	vfio_pci_dma_buf_move_region(vdev, index, true);
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_revoke_region);
+
+/**
+ * vfio_pci_core_rearm_region() - put a revoked device-specific region back
+ * @vdev: vfio PCI device, with memory_lock held for write
+ * @index: vfio region index passed to vfio_pci_core_revoke_region()
+ */
+void vfio_pci_core_rearm_region(struct vfio_pci_core_device *vdev,
+				unsigned int index)
+{
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	vfio_pci_dma_buf_move_region(vdev, index, false);
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_rearm_region);
+
+/**
+ * vfio_pci_core_clear_memory_enable() - follow a device that lost Memory Space
+ * @vdev: vfio PCI device, with memory_lock held for write
+ *
+ * For a provider whose failed operation left Memory Space disabled in the
+ * device. Zap the BAR mmaps, revoke the BAR dma-bufs and clear vfio's record
+ * of the Memory Space bit, as a user write that clears the bit would. The
+ * user's next Memory Space write enables it again.
+ */
+void vfio_pci_core_clear_memory_enable(struct vfio_pci_core_device *vdev)
+{
+	__le16 *virt_cmd;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	/* Closed, or not open yet: vfio_config_init() reads the device. */
+	if (!vdev->vconfig)
+		return;
+
+	virt_cmd = (__le16 *)&vdev->vconfig[PCI_COMMAND];
+	if (!(*virt_cmd & cpu_to_le16(PCI_COMMAND_MEMORY)))
+		return;
+
+	vfio_pci_zap_bars(vdev);
+	vfio_pci_dma_buf_move(vdev, true);
+	*virt_cmd &= cpu_to_le16(~PCI_COMMAND_MEMORY);
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_clear_memory_enable);
+
 void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev)
 {
 	down_write(&vdev->memory_lock);
@@ -2019,16 +2090,26 @@ static unsigned long vma_to_pfn(struct vm_area_struct *vma)
 	return (pci_resource_start(vdev->pdev, index) >> PAGE_SHIFT) + pgoff;
 }
 
-vm_fault_t vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,
-				   struct vm_fault *vmf,
-				   unsigned long pfn,
-				   unsigned int order)
+/**
+ * __vfio_pci_vmf_insert_pfn() - insert a pfn without the memory enable and
+ *	runtime PM checks
+ * @vdev: vfio PCI device, with memory_lock held for read
+ * @vmf: the fault being served
+ * @pfn: first pfn to insert
+ * @order: page order of the mapping
+ *
+ * For a device-specific region that does not follow the memory enable bit.
+ * The caller checks the state its region depends on under memory_lock, and
+ * revokes the region with vfio_pci_core_revoke_region() when it changes or
+ * before the device leaves D0.
+ */
+vm_fault_t __vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,
+				     struct vm_fault *vmf,
+				     unsigned long pfn,
+				     unsigned int order)
 {
 	lockdep_assert_held_read(&vdev->memory_lock);
 
-	if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev))
-		return VM_FAULT_SIGBUS;
-
 	if (!order)
 		return vmf_insert_pfn(vmf->vma, vmf->address, pfn);
 
@@ -2040,6 +2121,20 @@ vm_fault_t vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,
 
 	return VM_FAULT_FALLBACK;
 }
+EXPORT_SYMBOL_GPL(__vfio_pci_vmf_insert_pfn);
+
+vm_fault_t vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,
+				   struct vm_fault *vmf,
+				   unsigned long pfn,
+				   unsigned int order)
+{
+	lockdep_assert_held_read(&vdev->memory_lock);
+
+	if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev))
+		return VM_FAULT_SIGBUS;
+
+	return __vfio_pci_vmf_insert_pfn(vdev, vmf, pfn, order);
+}
 EXPORT_SYMBOL_GPL(vfio_pci_vmf_insert_pfn);
 
 static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
diff --git a/drivers/vfio/pci/vfio_pci_dmabuf.c b/drivers/vfio/pci/vfio_pci_dmabuf.c
index 8606c15ab2fe..d0f7ec5487de 100644
--- a/drivers/vfio/pci/vfio_pci_dmabuf.c
+++ b/drivers/vfio/pci/vfio_pci_dmabuf.c
@@ -19,6 +19,7 @@ struct vfio_pci_dma_buf {
 	u32 nr_ranges;
 	struct kref kref;
 	struct completion comp;
+	u32 region_index;
 	u8 revoked : 1;
 };
 
@@ -288,6 +289,7 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 	priv->vdev = vdev;
 	priv->nr_ranges = get_dma_buf.nr_ranges;
 	priv->size = length;
+	priv->region_index = get_dma_buf.region_index;
 	ret = vdev->pci_ops->get_dmabuf_phys(vdev, &priv->provider,
 					     get_dma_buf.region_index,
 					     priv->phys_vec, dma_ranges,
@@ -347,6 +349,49 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 	return ret;
 }
 
+static void vfio_pci_dma_buf_move_one(struct vfio_pci_dma_buf *priv,
+				      bool revoked)
+{
+	if (!get_file_active(&priv->dmabuf->file))
+		return;
+
+	if (priv->revoked != revoked) {
+		dma_resv_lock(priv->dmabuf->resv, NULL);
+		if (revoked)
+			priv->revoked = true;
+		dma_buf_invalidate_mappings(priv->dmabuf);
+		dma_resv_wait_timeout(priv->dmabuf->resv,
+				      DMA_RESV_USAGE_BOOKKEEP, false,
+				      MAX_SCHEDULE_TIMEOUT);
+		dma_resv_unlock(priv->dmabuf->resv);
+		if (revoked) {
+			kref_put(&priv->kref, vfio_pci_dma_buf_done);
+			wait_for_completion(&priv->comp);
+			/*
+			 * Re-arm the registered kref reference and the
+			 * completion so the post-revoke state matches the
+			 * post-creation state.  An un-revoke followed by a
+			 * new mapping needs the kref to be non-zero before
+			 * kref_get(), and vfio_pci_dma_buf_cleanup()
+			 * delegates its drain back through this revoke
+			 * path on a possibly-already-revoked dma-buf.
+			 */
+			kref_init(&priv->kref);
+			reinit_completion(&priv->comp);
+		} else {
+			dma_resv_lock(priv->dmabuf->resv, NULL);
+			priv->revoked = false;
+			dma_resv_unlock(priv->dmabuf->resv);
+		}
+	}
+	fput(priv->dmabuf->file);
+}
+
+/*
+ * Revoke or re-arm the dma-bufs of the BARs, which follow the memory enable
+ * bit. A device-specific region is moved by its provider through
+ * vfio_pci_dma_buf_move_region().
+ */
 void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, bool revoked)
 {
 	struct vfio_pci_dma_buf *priv;
@@ -354,41 +399,23 @@ void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, bool revoked)
 
 	lockdep_assert_held_write(&vdev->memory_lock);
 
-	list_for_each_entry_safe(priv, tmp, &vdev->dmabufs, dmabufs_elm) {
-		if (!get_file_active(&priv->dmabuf->file))
-			continue;
+	list_for_each_entry_safe(priv, tmp, &vdev->dmabufs, dmabufs_elm)
+		if (priv->region_index < VFIO_PCI_NUM_REGIONS)
+			vfio_pci_dma_buf_move_one(priv, revoked);
+}
 
-		if (priv->revoked != revoked) {
-			dma_resv_lock(priv->dmabuf->resv, NULL);
-			if (revoked)
-				priv->revoked = true;
-			dma_buf_invalidate_mappings(priv->dmabuf);
-			dma_resv_wait_timeout(priv->dmabuf->resv,
-					      DMA_RESV_USAGE_BOOKKEEP, false,
-					      MAX_SCHEDULE_TIMEOUT);
-			dma_resv_unlock(priv->dmabuf->resv);
-			if (revoked) {
-				kref_put(&priv->kref, vfio_pci_dma_buf_done);
-				wait_for_completion(&priv->comp);
-				/*
-				 * Re-arm the registered kref reference and the
-				 * completion so the post-revoke state matches the
-				 * post-creation state.  An un-revoke followed by a
-				 * new mapping needs the kref to be non-zero before
-				 * kref_get(), and vfio_pci_dma_buf_cleanup()
-				 * delegates its drain back through this revoke
-				 * path on a possibly-already-revoked dma-buf.
-				 */
-				kref_init(&priv->kref);
-				reinit_completion(&priv->comp);
-			} else {
-				dma_resv_lock(priv->dmabuf->resv, NULL);
-				priv->revoked = false;
-				dma_resv_unlock(priv->dmabuf->resv);
-			}
-		}
-		fput(priv->dmabuf->file);
-	}
+/* Revoke or re-arm the dma-bufs exported from one device-specific region. */
+void vfio_pci_dma_buf_move_region(struct vfio_pci_core_device *vdev,
+				  unsigned int index, bool revoked)
+{
+	struct vfio_pci_dma_buf *priv;
+	struct vfio_pci_dma_buf *tmp;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	list_for_each_entry_safe(priv, tmp, &vdev->dmabufs, dmabufs_elm)
+		if (priv->region_index == index)
+			vfio_pci_dma_buf_move_one(priv, revoked);
 }
 
 void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev)
@@ -405,7 +432,8 @@ void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev)
 	 * ready, so cleanup itself does not need to participate in kref
 	 * bookkeeping.
 	 */
-	vfio_pci_dma_buf_move(vdev, true);
+	list_for_each_entry_safe(priv, tmp, &vdev->dmabufs, dmabufs_elm)
+		vfio_pci_dma_buf_move_one(priv, true);
 
 	list_for_each_entry_safe(priv, tmp, &vdev->dmabufs, dmabufs_elm) {
 		if (!get_file_active(&priv->dmabuf->file))
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index dbede6e24fa9..23bccfbb7eb7 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -141,6 +141,8 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 				  size_t argsz);
 void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev);
 void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, bool revoked);
+void vfio_pci_dma_buf_move_region(struct vfio_pci_core_device *vdev,
+				  unsigned int index, bool revoked);
 #else
 static inline int
 vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
@@ -156,6 +158,11 @@ static inline void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev,
 					 bool revoked)
 {
 }
+static inline void
+vfio_pci_dma_buf_move_region(struct vfio_pci_core_device *vdev,
+			     unsigned int index, bool revoked)
+{
+}
 #endif
 
 #endif
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index cb9ba92b696a..7dd93ef103cb 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -227,6 +227,11 @@ int vfio_pci_core_register_dev_region(struct vfio_pci_core_device *vdev,
 				      unsigned int type, unsigned int subtype,
 				      const struct vfio_pci_regops *ops,
 				      size_t size, u32 flags, void *data);
+void vfio_pci_core_revoke_region(struct vfio_pci_core_device *vdev,
+				 unsigned int index);
+void vfio_pci_core_rearm_region(struct vfio_pci_core_device *vdev,
+				unsigned int index);
+void vfio_pci_core_clear_memory_enable(struct vfio_pci_core_device *vdev);
 void vfio_pci_core_close_device(struct vfio_device *core_vdev);
 int vfio_pci_core_init_dev(struct vfio_device *core_vdev);
 void vfio_pci_core_release_dev(struct vfio_device *core_vdev);
@@ -250,6 +255,9 @@ ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *buf,
 		size_t count, loff_t *ppos);
 ssize_t vfio_pci_core_write(struct vfio_device *core_vdev, const char __user *buf,
 		size_t count, loff_t *ppos);
+vm_fault_t __vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,
+				     struct vm_fault *vmf, unsigned long pfn,
+				     unsigned int order);
 vm_fault_t vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,
 				   struct vm_fault *vmf, unsigned long pfn,
 				   unsigned int order);
-- 
2.53.0

[PATCH v6 17/31] vfio/pci: Call the device provider across low-power transitions

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A provider may serve state that does not survive low power. A CXL Type-2
device loses its committed HDM decoder in D3hot or D3cold, and the HDM
memory region must stay out of use until the host restores the decoder.
Without a hook the provider can only keep the device in D0.

Add optional power_down and power_up ops. vfio-pci-core calls power_down
with memory_lock held for write before the device leaves D0: in
vfio_pci_set_power_state() for a lower state, and in runtime suspend
before the PCI core moves the device to low power. It calls power_up
with memory_lock held after the device is back in D0 with its config
space restored: after a guest PM control write returns the device to D0,
and in runtime resume. If a request for a lower state leaves the device
in D0, because the transition failed or a quirk keeps it there,
vfio-pci-core calls power_up at once.

The other paths that return a device to D0 either reset it afterwards
(close, VFIO_DEVICE_RESET and the hot reset ioctl), where the provider
restores its state, or tear it down (unregister).

Suggested-by: Alex Williamson <alex@shazbot.org>
Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci_config.c |  4 +++
 drivers/vfio/pci/vfio_pci_core.c   | 40 +++++++++++++++++++++++++++++-
 drivers/vfio/pci/vfio_pci_priv.h   |  1 +
 include/linux/vfio_pci_core.h      |  7 ++++++
 4 files changed, 51 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
index 6fbece95cb9f..ec556921ecd4 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -712,6 +712,8 @@ static int __init init_pci_cap_basic_perm(struct perm_bits *perm)
 static void vfio_lock_and_set_power_state(struct vfio_pci_core_device *vdev,
 					  pci_power_t state)
 {
+	pci_power_t prev = vdev->pdev->current_state;
+
 	if (state >= PCI_D3hot) {
 		vfio_pci_zap_and_down_write_memory_lock(vdev);
 		vfio_pci_dma_buf_move(vdev, true);
@@ -720,6 +722,8 @@ static void vfio_lock_and_set_power_state(struct vfio_pci_core_device *vdev,
 	}
 
 	vfio_pci_set_power_state(vdev, state);
+	if (state == PCI_D0 && prev > PCI_D0)
+		vfio_pci_provider_power_up(vdev);
 	if (__vfio_pci_memory_enabled(vdev))
 		vfio_pci_dma_buf_move(vdev, false);
 	up_write(&vdev->memory_lock);
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 9d0c22a4f3c6..be2acab625c0 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -318,10 +318,31 @@ static void vfio_pci_probe_power_state(struct vfio_pci_core_device *vdev)
  * by PM capability emulation and separately from pci_dev internal saved state
  * to avoid it being overwritten and consumed around other resets.
  */
+/*
+ * A provider whose device state does not survive low power is told before
+ * the device leaves D0 and after it is back in D0 with its config space
+ * restored. Both run with memory_lock held for write.
+ */
+static void vfio_pci_provider_power_down(struct vfio_pci_core_device *vdev)
+{
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	if (vdev->cxl_ops && vdev->cxl_ops->power_down)
+		vdev->cxl_ops->power_down(vdev);
+}
+
+void vfio_pci_provider_power_up(struct vfio_pci_core_device *vdev)
+{
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	if (vdev->cxl_ops && vdev->cxl_ops->power_up)
+		vdev->cxl_ops->power_up(vdev);
+}
+
 int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t state)
 {
 	struct pci_dev *pdev = vdev->pdev;
-	bool needs_restore = false, needs_save = false;
+	bool needs_restore = false, needs_save = false, powered_down = false;
 	int ret;
 
 	/* Prevent changing power state for PFs with VFs enabled */
@@ -329,6 +350,11 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat
 		lockdep_assert_held_write(&vdev->memory_lock);
 		if (vdev->sriov_active)
 			return -EBUSY;
+
+		if (pdev->current_state == PCI_D0) {
+			vfio_pci_provider_power_down(vdev);
+			powered_down = true;
+		}
 	}
 
 	if (vdev->needs_pm_restore) {
@@ -366,6 +392,14 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat
 		}
 	}
 
+	/*
+	 * The device stayed in D0: the transition failed, or a quirk keeps it
+	 * there. No later D0 transition will call power_up, so undo power_down
+	 * now.
+	 */
+	if (powered_down && pdev->current_state == PCI_D0)
+		vfio_pci_provider_power_up(vdev);
+
 	return ret;
 }
 
@@ -502,6 +536,8 @@ static int vfio_pci_core_runtime_suspend(struct device *dev)
 	 * (saved locally in 'vdev->pm_save').
 	 */
 	vfio_pci_set_power_state(vdev, PCI_D0);
+	/* The PCI core moves the device to low power once this returns. */
+	vfio_pci_provider_power_down(vdev);
 	up_write(&vdev->memory_lock);
 
 	/*
@@ -526,6 +562,8 @@ static int vfio_pci_core_runtime_resume(struct device *dev)
 	 * low power mode.
 	 */
 	down_write(&vdev->memory_lock);
+	/* The PCI core has restored the config space of the device. */
+	vfio_pci_provider_power_up(vdev);
 	if (vdev->pm_wake_eventfd_ctx) {
 		eventfd_signal(vdev->pm_wake_eventfd_ctx);
 		__vfio_pci_runtime_pm_exit(vdev);
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index 23bccfbb7eb7..411adc2ba379 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -78,6 +78,7 @@ void vfio_config_free(struct vfio_pci_core_device *vdev);
 
 int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev,
 			     pci_power_t state);
+void vfio_pci_provider_power_up(struct vfio_pci_core_device *vdev);
 
 void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev);
 int vfio_pci_try_reset_function(struct vfio_pci_core_device *vdev);
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 7dd93ef103cb..ec1660e0d6d2 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -94,6 +94,13 @@ struct vfio_cxl_ops {
 	 * saves it first, the close path keeps the open-time copy.
 	 */
 	int	(*reset)(struct vfio_pci_core_device *vdev);
+	/*
+	 * Optional. Called with memory_lock held for write before the device
+	 * leaves D0, and after it returns to D0 with its config space
+	 * restored. power_down may be called again while already powered down.
+	 */
+	void	(*power_down)(struct vfio_pci_core_device *vdev);
+	void	(*power_up)(struct vfio_pci_core_device *vdev);
 	/* Pinned per bound CXL device so vfio-cxl cannot unload under usage */
 	struct module *owner;
 };
-- 
2.53.0

[PATCH v6 18/31] vfio/cxl: Add the vfio-cxl provider module skeleton

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Add the vfio-cxl module that vfio-pci-core loads on demand for a CXL
Type-2 device, registering its callbacks at module_init. Its reset op
runs the PCI core's reset methods until vfio-cxl serves the device
memory.

vfio-cxl is a feature of vfio-pci-core rather than a variant driver, so
its Kconfig sources above the variant drivers and its objects link right
after vfio-pci-core while still being loaded on demand:
request_module("vfio-cxl") finds it by module name.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 MAINTAINERS                          |  7 +++
 drivers/vfio/pci/Kconfig             |  2 +
 drivers/vfio/pci/Makefile            |  2 +
 drivers/vfio/pci/cxl/Kconfig         | 11 ++++
 drivers/vfio/pci/cxl/Makefile        |  3 +
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 82 ++++++++++++++++++++++++++++
 6 files changed, 107 insertions(+)
 create mode 100644 drivers/vfio/pci/cxl/Kconfig
 create mode 100644 drivers/vfio/pci/cxl/Makefile
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_core.c

diff --git a/MAINTAINERS b/MAINTAINERS
index c3aef3dce68c..8ac0a91ce647 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -28636,6 +28636,13 @@ L:	kvm@vger.kernel.org
 S:	Maintained
 F:	drivers/vfio/cdx/*
 
+VFIO CXL DRIVER
+M:	Manish Honap <mhonap@nvidia.com>
+L:	kvm@vger.kernel.org
+L:	linux-cxl@vger.kernel.org
+S:	Supported
+F:	drivers/vfio/pci/cxl/
+
 VFIO DRIVER
 M:	Alex Williamson <alex@shazbot.org>
 L:	kvm@vger.kernel.org
diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig
index 296bf01e185e..4cd6acd36053 100644
--- a/drivers/vfio/pci/Kconfig
+++ b/drivers/vfio/pci/Kconfig
@@ -58,6 +58,8 @@ config VFIO_PCI_ZDEV_KVM
 config VFIO_PCI_DMABUF
 	def_bool y if VFIO_PCI_CORE && PCI_P2PDMA && DMA_SHARED_BUFFER
 
+source "drivers/vfio/pci/cxl/Kconfig"
+
 source "drivers/vfio/pci/mlx5/Kconfig"
 
 source "drivers/vfio/pci/ism/Kconfig"
diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
index 6138f1bf241d..b65117d8d463 100644
--- a/drivers/vfio/pci/Makefile
+++ b/drivers/vfio/pci/Makefile
@@ -5,6 +5,8 @@ vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
 vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o
 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o
 
+obj-$(CONFIG_VFIO_CXL) += cxl/
+
 vfio-pci-y := vfio_pci.o
 vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o
 obj-$(CONFIG_VFIO_PCI) += vfio-pci.o
diff --git a/drivers/vfio/pci/cxl/Kconfig b/drivers/vfio/pci/cxl/Kconfig
new file mode 100644
index 000000000000..1392fa4ed15d
--- /dev/null
+++ b/drivers/vfio/pci/cxl/Kconfig
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config VFIO_CXL
+	tristate "VFIO support for CXL Type-2 devices"
+	depends on CXL_BUS && CXL_MEM
+	select VFIO_PCI_CORE
+	help
+	  Add CXL HDM handling to vfio-pci so a CXL Type-2 device, such as a
+	  CXL-attached accelerator, can be assigned to a virtual machine.
+	  vfio-pci loads this module on demand when it binds a CXL device.
+
+	  If you don't know what to do here, say N.
diff --git a/drivers/vfio/pci/cxl/Makefile b/drivers/vfio/pci/cxl/Makefile
new file mode 100644
index 000000000000..8501bf626108
--- /dev/null
+++ b/drivers/vfio/pci/cxl/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_VFIO_CXL) += vfio-cxl.o
+vfio-cxl-y := vfio_cxl_core.o
diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
new file mode 100644
index 000000000000..ee49d68d689c
--- /dev/null
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -0,0 +1,82 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * VFIO support for CXL Type-2 devices.
+ *
+ * Copyright (c) 2026 NVIDIA Corporation & Affiliates
+ */
+
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/vfio_pci_core.h>
+
+static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
+{
+	return 0;
+}
+
+static void vfio_cxl_release_device(struct vfio_pci_core_device *vdev)
+{
+}
+
+static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
+{
+	return 0;
+}
+
+static void vfio_cxl_close_device(struct vfio_pci_core_device *vdev)
+{
+}
+
+static void vfio_cxl_reset_prepare(struct vfio_pci_core_device *vdev)
+{
+}
+
+static int vfio_cxl_reset_done(struct vfio_pci_core_device *vdev)
+{
+	return 0;
+}
+
+/*
+ * Until vfio-cxl serves the device memory, reset through the PCI core's
+ * reset methods, which run CXL Reset ahead of FLR. Restore the PCI state
+ * here rather than through pci_reset_function_locked(): its reset_prepare
+ * and reset_done callbacks take memory_lock, which the caller already holds.
+ * The caller has saved the state, or on close loaded the open-time copy.
+ */
+static int vfio_cxl_reset(struct vfio_pci_core_device *vdev)
+{
+	struct pci_dev *pdev = vdev->pdev;
+	int ret;
+
+	ret = __pci_reset_function_locked(pdev);
+	pci_restore_state(pdev);
+
+	return ret;
+}
+
+static const struct vfio_cxl_ops vfio_cxl_ops = {
+	.init		= vfio_cxl_init_device,
+	.release	= vfio_cxl_release_device,
+	.open_device	= vfio_cxl_open_device,
+	.close_device	= vfio_cxl_close_device,
+	.reset_prepare	= vfio_cxl_reset_prepare,
+	.reset_done	= vfio_cxl_reset_done,
+	.reset		= vfio_cxl_reset,
+	.owner		= THIS_MODULE,
+};
+
+static int __init vfio_cxl_init(void)
+{
+	return vfio_pci_core_register_cxl_ops(&vfio_cxl_ops);
+}
+
+static void __exit vfio_cxl_exit(void)
+{
+	vfio_pci_core_unregister_cxl_ops(&vfio_cxl_ops);
+}
+
+module_init(vfio_cxl_init);
+module_exit(vfio_cxl_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("VFIO support for CXL Type-2 devices");
-- 
2.53.0

[PATCH v6 19/31] vfio/cxl: Create the CXL memdev and set media ready at bind

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

At bind, build the CXL memory device for the passed-through Type-2
accelerator so it joins the CXL topology and its HDM region resolves to a
host physical range. A Type-2 device has no mailbox, so there is no
media-ready register to poll: set media ready directly once the component
registers validate (mirroring drivers/net/ethernet/sfc/efx_cxl.c)

As per current vfio-cxl support, reject a device with:
- more than one HDM decoder
- interleaving enabled
- whose reset the host cannot service, including the CPU cache
  invalidation that the reset needs

The CXL-core allocations are grouped with devres, in a helper that
returns on the first error, so a failed bind unwinds them: a failed init
leaves the device bound without CXL support, so devm would otherwise hold
them until unbind.

The device is not kept in D0. Low power loses the committed HDM
decoder, and the host restores the decoder configuration when the device
returns to D0. The contents of device memory across low power are the
user's to keep.

Take the bind-time copy of the cached HDM state with
pci_cxl_hdm_snapshot(), which needs no device lock: probe may run in
a worker on the device's NUMA node that does not own the lock.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 135 +++++++++++++++++++++++++++
 1 file changed, 135 insertions(+)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index ee49d68d689c..ef1fc715020e 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -5,17 +5,151 @@
  * Copyright (c) 2026 NVIDIA Corporation & Affiliates
  */
 
+#include <linux/cleanup.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/range.h>
+#include <linux/slab.h>
 #include <linux/vfio_pci_core.h>
+#include <cxl/cxl.h>
+#include <cxl/hdm.h>
+#include <cxl/pci.h>
+
+/**
+ * struct vfio_cxl_state - per-device state for a vfio-cxl device
+ * @cxlds: CXL device state; kept first for devm_cxl_dev_state_create()
+ * @cxlmd: memory device joined to the CXL topology at bind
+ * @hpa_range: host physical range of the HDM region
+ * @hdm: copy of the HDM state cached at PCI enumeration, taken at bind
+ */
+struct vfio_cxl_state {
+	struct cxl_dev_state cxlds;
+	struct cxl_memdev *cxlmd;
+	struct range hpa_range;
+	struct cxl_hdm_info *hdm;
+};
+
+/*
+ * Create the CXL device state and join the device to the CXL topology. The
+ * caller groups these devres allocations, so return on the first error.
+ */
+static struct vfio_cxl_state *vfio_cxl_create_memdev(struct pci_dev *pdev,
+						     u64 hdm_size)
+{
+	struct vfio_cxl_state *cxl;
+	struct cxl_memdev *cxlmd;
+	u64 serial;
+	u16 dvsec;
+	int ret;
+
+	dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_DEVICE);
+	serial = pci_get_dsn(pdev);
+
+	cxl = devm_cxl_dev_state_create(&pdev->dev, CXL_DEVTYPE_DEVMEM, serial,
+					dvsec, struct vfio_cxl_state, cxlds,
+					false);
+	if (!cxl)
+		return ERR_PTR(-ENOMEM);
+
+	ret = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_COMPONENT,
+				 &cxl->cxlds.reg_map);
+	if (ret) {
+		pci_err(pdev, "vfio-cxl: no component registers\n");
+		return ERR_PTR(ret);
+	}
+
+	if (!cxl->cxlds.reg_map.component_map.hdm_decoder.valid) {
+		pci_err(pdev, "vfio-cxl: HDM decoder registers not found\n");
+		return ERR_PTR(-ENODEV);
+	}
+
+	/*
+	 * A Type-2 accelerator has no mailbox and no media-ready register, so
+	 * set media ready directly.
+	 */
+	cxl->cxlds.media_ready = true;
+
+	ret = cxl_set_capacity(&cxl->cxlds, hdm_size);
+	if (ret)
+		return ERR_PTR(ret);
+
+	cxlmd = devm_cxl_probe_mem(&cxl->cxlds, &cxl->hpa_range);
+	if (IS_ERR(cxlmd))
+		return ERR_CAST(cxlmd);
+
+	cxl->cxlmd = cxlmd;
+	return cxl;
+}
 
 static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
 {
+	struct pci_dev *pdev = vdev->pdev;
+	struct cxl_hdm_info *hdm __free(kfree) = NULL;
+	struct vfio_cxl_state *cxl;
+	u64 hdm_size;
+
+	/*
+	 * pdev->hdm is cached at PCI enumeration, before any driver binds, so a
+	 * device without it has no usable HDM decoder. Continue without CXL
+	 * support rather than defer the bind forever. The cache belongs to the
+	 * CXL core, so work from a copy. Probe may run in a worker on the
+	 * device's NUMA node that does not own the device lock, so take the
+	 * copy that needs none.
+	 */
+	hdm = pci_cxl_hdm_snapshot(pdev);
+	if (IS_ERR(hdm))
+		return PTR_ERR(hdm);
+	if (!hdm)
+		return -ENODEV;
+
+	/* The guest drives one virtual decoder; multiple are unsupported. */
+	if (hdm->decoder_count != 1)
+		return -EOPNOTSUPP;
+
+	/* An interleaved decoder cannot be mapped 1:1 to the guest. */
+	if (hdm->settings[0].config.interleave_ways != 1)
+		return -EOPNOTSUPP;
+
+	/*
+	 * The guest drives resets through the CXL Device DVSEC and polls the
+	 * shadow for completion. If the host cannot service a function-scoped
+	 * CXL reset, that request could never complete, so refuse the device
+	 * rather than advertise a reset the guest would poll on forever. The
+	 * check includes the CPU cache invalidation that the reset needs.
+	 */
+	if (!pci_cxl_reset_capable(pdev))
+		return -EOPNOTSUPP;
+
+	hdm_size = range_len(&hdm->settings[0].config.hpa_range);
+	if (!hdm_size)
+		return -ENXIO;
+
+	/*
+	 * Group the CXL-core allocations so a failure unwinds them here. A
+	 * failed init leaves the device bound without CXL support, so devm
+	 * would otherwise hold them until unbind.
+	 */
+	if (!devres_open_group(&pdev->dev, NULL, GFP_KERNEL))
+		return -ENOMEM;
+
+	cxl = vfio_cxl_create_memdev(pdev, hdm_size);
+	if (IS_ERR(cxl)) {
+		devres_release_group(&pdev->dev, NULL);
+		return PTR_ERR(cxl);
+	}
+	devres_close_group(&pdev->dev, NULL);
+
+	cxl->hdm = no_free_ptr(hdm);
+	vdev->cxl = cxl;
+
 	return 0;
 }
 
 static void vfio_cxl_release_device(struct vfio_pci_core_device *vdev)
 {
+	kfree(vdev->cxl->hdm);
+	vdev->cxl = NULL;
 }
 
 static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
@@ -80,3 +214,4 @@ module_exit(vfio_cxl_exit);
 
 MODULE_LICENSE("GPL");
 MODULE_DESCRIPTION("VFIO support for CXL Type-2 devices");
+MODULE_IMPORT_NS("CXL");
-- 
2.53.0

[PATCH v6 20/31] vfio/cxl: Own the whole component register BAR

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

devm_cxl_probe_mem() maps the component registers through the CXL core,
which by default claims each HDM and RAS sub-block. That collides with
the full-BAR request vfio-pci-core makes when the guest opens the device.

After the component registers are discovered, declare the containing BAR
owned with cxl_reg_map_set_owned_resource(). The core then ioremaps the
sub-blocks without requesting them, so vfio-pci-core owns the whole BAR
and the requests do not collide.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index ef1fc715020e..052be27bc46f 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -33,8 +33,9 @@ struct vfio_cxl_state {
  * Create the CXL device state and join the device to the CXL topology. The
  * caller groups these devres allocations, so return on the first error.
  */
-static struct vfio_cxl_state *vfio_cxl_create_memdev(struct pci_dev *pdev,
-						     u64 hdm_size)
+static struct vfio_cxl_state *
+vfio_cxl_create_memdev(struct pci_dev *pdev, const struct cxl_hdm_info *hdm,
+		       u64 hdm_size)
 {
 	struct vfio_cxl_state *cxl;
 	struct cxl_memdev *cxlmd;
@@ -59,6 +60,17 @@ static struct vfio_cxl_state *vfio_cxl_create_memdev(struct pci_dev *pdev,
 		return ERR_PTR(ret);
 	}
 
+	/*
+	 * vfio-pci-core requests the whole component-register BAR when the
+	 * guest opens the device. Declare that BAR owned so the CXL core
+	 * ioremaps the HDM and RAS sub-blocks without claiming them, and the
+	 * full-BAR request does not collide.
+	 */
+	ret = cxl_reg_map_set_owned_resource(&cxl->cxlds.reg_map,
+					     pci_resource_n(pdev, hdm->hdm_bar));
+	if (ret)
+		return ERR_PTR(ret);
+
 	if (!cxl->cxlds.reg_map.component_map.hdm_decoder.valid) {
 		pci_err(pdev, "vfio-cxl: HDM decoder registers not found\n");
 		return ERR_PTR(-ENODEV);
@@ -133,7 +145,7 @@ static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
 	if (!devres_open_group(&pdev->dev, NULL, GFP_KERNEL))
 		return -ENOMEM;
 
-	cxl = vfio_cxl_create_memdev(pdev, hdm_size);
+	cxl = vfio_cxl_create_memdev(pdev, hdm, hdm_size);
 	if (IS_ERR(cxl)) {
 		devres_release_group(&pdev->dev, NULL);
 		return PTR_ERR(cxl);
-- 
2.53.0

[PATCH v6 21/31] vfio/cxl: Expose the HDM memory region to the guest

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A CXL Type-2 guest maps the device HDM memory to use its coherent
CXL.mem. The HDM memory is a host physical range with no struct page,
so the guest and KVM need a write-back mapping of it.

Register the range as a region under VFIO_REGION_TYPE_PCI_VENDOR_TYPE
with the CXL vendor id rather than a bespoke region type, per open
because vfio-pci-core tears down all dynamic regions on close and on a
failed open. This patch serves read and write on the region fd; mmap
follows.

Own the resolved host physical range exclusively (IORESOURCE_EXCLUSIVE)
so nothing, /dev/mem included, can map a conflicting cacheable alias
that would fault the host once the range is mapped write-back. Release
it in the provider .release op rather than through a devm action on the
PCI device, which could run after vfio-cxl has unloaded. vfio-cxl now
holds the range, so its reset op runs CXL Reset with
pci_cxl_reset_owned().

Host access to the range is safe only while the committed decoder is in
place. Track that in hdm_valid, set at bind once the decoder is verified
committed. Low power loses the decoder, so close the gate in power_down
and restore the decoder with cxl_restore_state_after_pci_reset() in
power_up, reopening the gate on success.

Device memory can stay unavailable for seconds after the decoder is back:
the device clears Memory_Active in DVSEC CXL Range 1 Size Low until the
memory is usable (CXL r4.0 sec 8.1.3.8.2).

At bind, wait up to the Memory_Active_Timeout the device reports for
Memory_Active, since a reset just before the bind also clears it. A device
whose memory then reads active brings it up by itself, and power_up waits
for Memory_Active, up to the same timeout, before it reopens the gate.

A device whose memory stays inactive relies on its driver and is not
waited for.

CXL.mem decode follows the committed decoder and Mem_Enable, not Memory
Space Enable (CXL r4.0 sec 8.2.4.20.2, 8.2.4.20.7 and 8.1.3.2), so the gate
does not check the Command register. Read and write copy through a
page-sized kernel buffer, so memory_lock is never held across a user copy.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 285 ++++++++++++++++++++++++++-
 include/uapi/linux/vfio.h            |   4 +
 2 files changed, 279 insertions(+), 10 deletions(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 052be27bc46f..e6e633902ecd 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -5,11 +5,15 @@
  * Copyright (c) 2026 NVIDIA Corporation & Affiliates
  */
 
+#include <linux/bitfield.h>
 #include <linux/cleanup.h>
+#include <linux/delay.h>
+#include <linux/io.h>
 #include <linux/module.h>
 #include <linux/pci.h>
 #include <linux/range.h>
 #include <linux/slab.h>
+#include <linux/uaccess.h>
 #include <linux/vfio_pci_core.h>
 #include <cxl/cxl.h>
 #include <cxl/hdm.h>
@@ -20,13 +24,114 @@
  * @cxlds: CXL device state; kept first for devm_cxl_dev_state_create()
  * @cxlmd: memory device joined to the CXL topology at bind
  * @hpa_range: host physical range of the HDM region
+ * @hpa_res: exclusive reservation of @hpa_range, held for the bind
  * @hdm: copy of the HDM state cached at PCI enumeration, taken at bind
+ * @hdm_valid: true when host CPU access to the HDM range is safe; under memory_lock
+ * @await_mem_active: the device reported its memory active at bind, so wait
+ *	for Memory_Active before host access reopens after a reset or low power
  */
 struct vfio_cxl_state {
 	struct cxl_dev_state cxlds;
 	struct cxl_memdev *cxlmd;
 	struct range hpa_range;
+	struct resource *hpa_res;
 	struct cxl_hdm_info *hdm;
+	bool hdm_valid;
+	bool await_mem_active;
+};
+
+/*
+ * Copy one chunk between the bounce buffer and the HDM range under
+ * memory_lock. The user copy stays outside the lock: a user buffer that is
+ * itself an mmap of this device faults through a handler that takes
+ * memory_lock for read, and a writer queued in between would deadlock the
+ * two readers.
+ */
+static int vfio_cxl_mem_copy(struct vfio_pci_core_device *vdev, void *bounce,
+			     u64 pos, size_t len, bool iswrite)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	void *mem;
+
+	guard(rwsem_read)(&vdev->memory_lock);
+
+	/*
+	 * Only touch the HDM range with the decoder in a known-good state, or
+	 * a host CPU access aborts as a fatal SError. CXL.mem decode follows
+	 * the committed decoder and Mem_Enable, not Memory Space Enable (CXL
+	 * r4.0 sec 8.2.4.20.2, 8.1.3.2).
+	 */
+	if (!cxl->hdm_valid || !cxl->cxlds.media_ready)
+		return -EIO;
+
+	mem = memremap(cxl->hpa_range.start + pos, len, MEMREMAP_WB);
+	if (!mem)
+		return -ENOMEM;
+	if (iswrite)
+		memcpy(mem, bounce, len);
+	else
+		memcpy(bounce, mem, len);
+	memunmap(mem);
+
+	return 0;
+}
+
+static ssize_t vfio_cxl_mem_rw(struct vfio_pci_core_device *vdev,
+			       char __user *buf, size_t count, loff_t *ppos,
+			       bool iswrite)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	u64 pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	ssize_t done = 0;
+	void *bounce;
+	int ret = 0;
+
+	if (pos >= range_len(&cxl->hpa_range))
+		return -EINVAL;
+	count = min_t(u64, count, range_len(&cxl->hpa_range) - pos);
+
+	bounce = kmalloc(min_t(size_t, count, PAGE_SIZE), GFP_KERNEL);
+	if (!bounce)
+		return -ENOMEM;
+
+	while (count) {
+		size_t len = min_t(size_t, count, PAGE_SIZE);
+
+		if (iswrite && copy_from_user(bounce, buf + done, len)) {
+			ret = -EFAULT;
+			break;
+		}
+		ret = vfio_cxl_mem_copy(vdev, bounce, pos + done, len, iswrite);
+		if (ret)
+			break;
+		if (!iswrite && copy_to_user(buf + done, bounce, len)) {
+			ret = -EFAULT;
+			break;
+		}
+		done += len;
+		count -= len;
+	}
+	kfree(bounce);
+
+	if (!done)
+		return ret;
+
+	*ppos += done;
+	return done;
+}
+
+/*
+ * The CXL regions carry no per-region state (region->data is the shared,
+ * devm-managed vfio_cxl_state), so releasing a region is a no-op.
+ */
+static void vfio_cxl_region_release(struct vfio_pci_core_device *vdev,
+				    struct vfio_pci_region *region)
+{
+}
+
+static const struct vfio_pci_regops vfio_cxl_mem_regops = {
+	.rw = vfio_cxl_mem_rw,
+	.release = vfio_cxl_region_release,
 };
 
 /*
@@ -94,12 +199,75 @@ vfio_cxl_create_memdev(struct pci_dev *pdev, const struct cxl_hdm_info *hdm,
 	return cxl;
 }
 
+/* CXL r4.0 sec 8.1.3.8.2: Range 1 Size Low Memory_Active_Timeout encodings */
+static const unsigned int vfio_cxl_mem_active_timeout_s[] = { 1, 4, 16, 64, 256 };
+
+/*
+ * Poll Memory_Active in DVSEC CXL Range 1 Size Low (CXL r4.0 sec 8.1.3.8.2)
+ * for up to the Memory_Active_Timeout the device reports, which is stored in
+ * @timeout. Return 0 once it is set, -ETIMEDOUT, or -ENODEV if config space
+ * does not respond.
+ */
+static int vfio_cxl_poll_mem_active(struct pci_dev *pdev, u16 dvsec,
+				    unsigned int *timeout)
+{
+	int pos = dvsec + PCI_DVSEC_CXL_RANGE_SIZE_LOW(0);
+	unsigned long deadline;
+	u32 size_lo;
+
+	if (pci_read_config_dword(pdev, pos, &size_lo) ||
+	    PCI_POSSIBLE_ERROR(size_lo))
+		return -ENODEV;
+
+	*timeout = FIELD_GET(PCI_DVSEC_CXL_MEM_ACTIVE_TIMEOUT, size_lo);
+	*timeout = vfio_cxl_mem_active_timeout_s[min_t(unsigned int, *timeout,
+				ARRAY_SIZE(vfio_cxl_mem_active_timeout_s) - 1)];
+	deadline = jiffies + secs_to_jiffies(*timeout);
+
+	while (!(size_lo & PCI_DVSEC_CXL_MEM_ACTIVE)) {
+		if (time_after(jiffies, deadline))
+			return -ETIMEDOUT;
+		msleep(20);
+		if (pci_read_config_dword(pdev, pos, &size_lo) ||
+		    PCI_POSSIBLE_ERROR(size_lo))
+			return -ENODEV;
+	}
+
+	return 0;
+}
+
+/*
+ * A reset or a return from low power leaves device memory unavailable until
+ * the device sets Memory_Active. That can take seconds after the reset
+ * completes and the decoder is restored, and a host access to the HDM range
+ * before then can abort. Wait for it on a device that brings its memory up
+ * by itself; a device that relies on its driver for that was found at bind
+ * and is not waited for.
+ */
+static int vfio_cxl_await_mem_active(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	unsigned int timeout = 0;
+	int ret;
+
+	if (!cxl->await_mem_active)
+		return 0;
+
+	ret = vfio_cxl_poll_mem_active(vdev->pdev, cxl->hdm->dvsec, &timeout);
+	if (ret == -ETIMEDOUT)
+		pci_err(vdev->pdev, "vfio-cxl: device memory not active after %us\n",
+			timeout);
+	return ret;
+}
+
 static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
 {
 	struct pci_dev *pdev = vdev->pdev;
 	struct cxl_hdm_info *hdm __free(kfree) = NULL;
 	struct vfio_cxl_state *cxl;
+	unsigned int timeout = 0;
 	u64 hdm_size;
+	int ret;
 
 	/*
 	 * pdev->hdm is cached at PCI enumeration, before any driver binds, so a
@@ -150,9 +318,44 @@ static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
 		devres_release_group(&pdev->dev, NULL);
 		return PTR_ERR(cxl);
 	}
+
+	/*
+	 * Claim the range IORESOURCE_EXCLUSIVE so no conflicting cacheable
+	 * alias can fault the host once it is mapped write-back. Release it
+	 * explicitly in vfio_cxl_release_device() rather than through a devm
+	 * action on pdev->dev: the provider module ref is dropped as soon as
+	 * that .release op returns, so a pdev-devres callback could run after
+	 * vfio-cxl has unloaded and jump into freed module text.
+	 */
+	cxl->hpa_res = request_mem_region_exclusive(cxl->hpa_range.start,
+						    range_len(&cxl->hpa_range),
+						    "vfio-cxl-hdm");
+	if (!cxl->hpa_res) {
+		devres_release_group(&pdev->dev, NULL);
+		return -EBUSY;
+	}
 	devres_close_group(&pdev->dev, NULL);
 
 	cxl->hdm = no_free_ptr(hdm);
+	/*
+	 * Wait for Memory_Active after a reset only on a device that brings its
+	 * memory up by itself; see vfio_cxl_await_mem_active(). Memory_Active
+	 * also reads clear for seconds after a reset, for example one the host
+	 * ran just before this bind, so give the device its Memory_Active_Timeout
+	 * before deciding. A device that relies on its driver times out here
+	 * once.
+	 */
+	ret = vfio_cxl_poll_mem_active(pdev, cxl->hdm->dvsec, &timeout);
+	cxl->await_mem_active = !ret;
+	if (ret == -ETIMEDOUT)
+		pci_info(pdev, "vfio-cxl: device memory not active after %us, left to its driver\n",
+			 timeout);
+	/*
+	 * The decoder decodes a non-empty range, so it is committed, and host
+	 * access to the HDM range is safe. Reset, bus reset and low power
+	 * close this gate until the decoder is restored.
+	 */
+	cxl->hdm_valid = true;
 	vdev->cxl = cxl;
 
 	return 0;
@@ -160,13 +363,40 @@ static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
 
 static void vfio_cxl_release_device(struct vfio_pci_core_device *vdev)
 {
-	kfree(vdev->cxl->hdm);
+	struct vfio_cxl_state *cxl = vdev->cxl;
+
+	/*
+	 * Drop the exclusive HDM range claim before vfio_pci_core_cxl_release()
+	 * drops the provider module ref; see vfio_cxl_init_device().
+	 */
+	release_mem_region(cxl->hpa_range.start, range_len(&cxl->hpa_range));
+	kfree(cxl->hdm);
 	vdev->cxl = NULL;
 }
 
+static int vfio_cxl_add_region(struct vfio_pci_core_device *vdev, u32 subtype,
+			       const struct vfio_pci_regops *ops, size_t size,
+			       u32 flags)
+{
+	u32 type = VFIO_REGION_TYPE_PCI_VENDOR_TYPE | PCI_VENDOR_ID_CXL;
+
+	return vfio_pci_core_register_dev_region(vdev, type, subtype, ops,
+						 size, flags, vdev->cxl);
+}
+
 static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
 {
-	return 0;
+	struct vfio_cxl_state *cxl = vdev->cxl;
+
+	/*
+	 * vfio-pci-core frees all dynamic regions on close and on a failed
+	 * open, so register them here per open rather than at bind.
+	 */
+	return vfio_cxl_add_region(vdev, VFIO_REGION_SUBTYPE_CXL_MEM,
+				   &vfio_cxl_mem_regops,
+				   range_len(&cxl->hpa_range),
+				   VFIO_REGION_INFO_FLAG_READ |
+				   VFIO_REGION_INFO_FLAG_WRITE);
 }
 
 static void vfio_cxl_close_device(struct vfio_pci_core_device *vdev)
@@ -183,21 +413,54 @@ static int vfio_cxl_reset_done(struct vfio_pci_core_device *vdev)
 }
 
 /*
- * Until vfio-cxl serves the device memory, reset through the PCI core's
- * reset methods, which run CXL Reset ahead of FLR. Restore the PCI state
- * here rather than through pci_reset_function_locked(): its reset_prepare
- * and reset_done callbacks take memory_lock, which the caller already holds.
- * The caller has saved the state, or on close loaded the open-time copy.
+ * vfio-cxl holds the HDM range, so the PCI core's cxl_reset method would
+ * fail to request it. Run the same CXL Reset with that claim.
  */
 static int vfio_cxl_reset(struct vfio_pci_core_device *vdev)
 {
+	/*
+	 * The reset restores PCI state from the saved copy. The caller provides
+	 * it: the try-reset helper saves the current state, and the close path
+	 * keeps the open-time state that vfio_pci_core_disable() loaded.
+	 */
+	return pci_cxl_reset_owned(vdev->pdev, vdev->cxl->hpa_res);
+}
+
+/* Low power loses the committed decoder: close host access until restored. */
+static void vfio_cxl_power_down(struct vfio_pci_core_device *vdev)
+{
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	vdev->cxl->hdm_valid = false;
+}
+
+static void vfio_cxl_power_up(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
 	struct pci_dev *pdev = vdev->pdev;
 	int ret;
 
-	ret = __pci_reset_function_locked(pdev);
-	pci_restore_state(pdev);
+	lockdep_assert_held_write(&vdev->memory_lock);
 
-	return ret;
+	/*
+	 * Restore the decoder that low power lost. The restore needs the device
+	 * lock; if it is contended, keep host access closed until a reset
+	 * restores the decoder. The restore refreshes the decoders in cxl->hdm
+	 * from the cache; global and DVSEC control stay as taken at bind, when
+	 * pci_cxl_reset_capable() required HDM decode and Mem_Enable on.
+	 */
+	if (!pci_dev_trylock(pdev))
+		return;
+
+	ret = pci_save_state(pdev);
+	if (!ret)
+		ret = cxl_restore_state_after_pci_reset(pdev, cxl->hdm);
+	pci_dev_unlock(pdev);
+
+	if (ret)
+		pci_warn(pdev, "vfio-cxl: HDM decoder restore failed: %d\n", ret);
+	else if (!vfio_cxl_await_mem_active(vdev))
+		cxl->hdm_valid = true;
 }
 
 static const struct vfio_cxl_ops vfio_cxl_ops = {
@@ -208,6 +471,8 @@ static const struct vfio_cxl_ops vfio_cxl_ops = {
 	.reset_prepare	= vfio_cxl_reset_prepare,
 	.reset_done	= vfio_cxl_reset_done,
 	.reset		= vfio_cxl_reset,
+	.power_down	= vfio_cxl_power_down,
+	.power_up	= vfio_cxl_power_up,
 	.owner		= THIS_MODULE,
 };
 
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index e41437fa17ad..883b51521e6a 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -370,6 +370,10 @@ struct vfio_region_info_cap_type {
  */
 #define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD	(1)
 
+/* CXL Type-2 device (0x1e98) sub-types for VFIO_REGION_TYPE_PCI_VENDOR_TYPE */
+/* CXL.mem HDM region of a Type-2 device, read and write */
+#define VFIO_REGION_SUBTYPE_CXL_MEM		(1)
+
 /* sub-types for VFIO_REGION_TYPE_GFX */
 #define VFIO_REGION_SUBTYPE_GFX_EDID            (1)
 
-- 
2.53.0

[PATCH v6 22/31] vfio/cxl: Allow mmap of the HDM memory region

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A CXL Type-2 guest maps the device HDM memory to use its coherent
CXL.mem, so the guest and KVM need a write-back mapping of the HDM
memory region. Make the region mmap-able. The range has no struct page,
so the mapping is VM_PFNMAP and its fault handler inserts the pfn, at
huge-page order where the alignment allows.

The fault handler inserts a pfn only while the decoder is known good and
the media is ready, and returns SIGBUS otherwise. It inserts with
__vfio_pci_vmf_insert_pfn(), so a fault does not depend on memory
enable; low power already closes the gate through power_down. When the
gate closes, in low power and later in reset and bus reset, revoke the
region with vfio_pci_core_revoke_region() so no live mapping outlives
the decoder, and re-arm it once the decoder is restored. The BARs keep
following the memory enable bit; this region does not.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 131 +++++++++++++++++++++++++--
 include/uapi/linux/vfio.h            |   2 +-
 2 files changed, 123 insertions(+), 10 deletions(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index e6e633902ecd..977d0c9c747a 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -9,6 +9,7 @@
 #include <linux/cleanup.h>
 #include <linux/delay.h>
 #include <linux/io.h>
+#include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/pci.h>
 #include <linux/range.h>
@@ -29,6 +30,7 @@
  * @hdm_valid: true when host CPU access to the HDM range is safe; under memory_lock
  * @await_mem_active: the device reported its memory active at bind, so wait
  *	for Memory_Active before host access reopens after a reset or low power
+ * @mem_region_index: vfio region index of the HDM memory region while open, else 0
  */
 struct vfio_cxl_state {
 	struct cxl_dev_state cxlds;
@@ -38,8 +40,82 @@ struct vfio_cxl_state {
 	struct cxl_hdm_info *hdm;
 	bool hdm_valid;
 	bool await_mem_active;
+	unsigned int mem_region_index;
 };
 
+static unsigned long vfio_cxl_mem_pgoff(struct vm_area_struct *vma,
+					unsigned long addr)
+{
+	unsigned long mask = (1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1;
+
+	return (vma->vm_pgoff & mask) + ((addr - vma->vm_start) >> PAGE_SHIFT);
+}
+
+static vm_fault_t vfio_cxl_mem_huge_fault(struct vm_fault *vmf,
+					  unsigned int order)
+{
+	struct vm_area_struct *vma = vmf->vma;
+	struct vfio_pci_core_device *vdev = vma->vm_private_data;
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	unsigned long addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+	unsigned long pfn = PHYS_PFN(cxl->hpa_range.start) +
+			    vfio_cxl_mem_pgoff(vma, addr);
+	vm_fault_t ret = VM_FAULT_FALLBACK;
+
+	if (is_aligned_for_order(vma, addr, pfn, order)) {
+		scoped_guard(rwsem_read, &vdev->memory_lock) {
+			/*
+			 * Insert a PFN only for a known-good decoder whose
+			 * media is ready. CXL.mem decode follows the committed
+			 * decoder and Mem_Enable, not Memory Space Enable (CXL
+			 * r4.0 sec 8.2.4.20.2, 8.1.3.2), so skip the BAR memory
+			 * enable check; low power closes hdm_valid through
+			 * power_down.
+			 */
+			if (cxl->hdm_valid && cxl->cxlds.media_ready)
+				ret = __vfio_pci_vmf_insert_pfn(vdev, vmf, pfn,
+								order);
+			else
+				ret = VM_FAULT_SIGBUS;
+		}
+	}
+
+	return ret;
+}
+
+static vm_fault_t vfio_cxl_mem_fault(struct vm_fault *vmf)
+{
+	return vfio_cxl_mem_huge_fault(vmf, 0);
+}
+
+static const struct vm_operations_struct vfio_cxl_mem_vm_ops = {
+	.fault = vfio_cxl_mem_fault,
+#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP
+	.huge_fault = vfio_cxl_mem_huge_fault,
+#endif
+};
+
+static int vfio_cxl_mem_mmap(struct vfio_pci_core_device *vdev,
+			     struct vfio_pci_region *region,
+			     struct vm_area_struct *vma)
+{
+	unsigned long mask = (1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1;
+	u64 req_start = (vma->vm_pgoff & mask) << PAGE_SHIFT;
+	u64 req_len = vma->vm_end - vma->vm_start;
+
+	if (req_start + req_len > region->size)
+		return -EINVAL;
+
+	/*
+	 * CXL.mem is coherent memory, so leave the mapping write-back cacheable.
+	 */
+	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
+	vma->vm_ops = &vfio_cxl_mem_vm_ops;
+	vma->vm_private_data = vdev;
+
+	return 0;
+}
+
 /*
  * Copy one chunk between the bounce buffer and the HDM range under
  * memory_lock. The user copy stays outside the lock: a user buffer that is
@@ -131,6 +207,7 @@ static void vfio_cxl_region_release(struct vfio_pci_core_device *vdev,
 
 static const struct vfio_pci_regops vfio_cxl_mem_regops = {
 	.rw = vfio_cxl_mem_rw,
+	.mmap = vfio_cxl_mem_mmap,
 	.release = vfio_cxl_region_release,
 };
 
@@ -387,20 +464,31 @@ static int vfio_cxl_add_region(struct vfio_pci_core_device *vdev, u32 subtype,
 static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
 {
 	struct vfio_cxl_state *cxl = vdev->cxl;
+	int ret;
 
 	/*
 	 * vfio-pci-core frees all dynamic regions on close and on a failed
 	 * open, so register them here per open rather than at bind.
 	 */
-	return vfio_cxl_add_region(vdev, VFIO_REGION_SUBTYPE_CXL_MEM,
-				   &vfio_cxl_mem_regops,
-				   range_len(&cxl->hpa_range),
-				   VFIO_REGION_INFO_FLAG_READ |
-				   VFIO_REGION_INFO_FLAG_WRITE);
+	ret = vfio_cxl_add_region(vdev, VFIO_REGION_SUBTYPE_CXL_MEM,
+				  &vfio_cxl_mem_regops,
+				  range_len(&cxl->hpa_range),
+				  VFIO_REGION_INFO_FLAG_READ |
+				  VFIO_REGION_INFO_FLAG_WRITE |
+				  VFIO_REGION_INFO_FLAG_MMAP);
+	if (ret)
+		return ret;
+
+	/* Record where the HDM memory region landed, to revoke it later. */
+	cxl->mem_region_index = VFIO_PCI_NUM_REGIONS + vdev->num_regions - 1;
+
+	return 0;
 }
 
 static void vfio_cxl_close_device(struct vfio_pci_core_device *vdev)
 {
+	/* vfio-pci-core frees the regions after this returns. */
+	vdev->cxl->mem_region_index = 0;
 }
 
 static void vfio_cxl_reset_prepare(struct vfio_pci_core_device *vdev)
@@ -412,6 +500,33 @@ static int vfio_cxl_reset_done(struct vfio_pci_core_device *vdev)
 	return 0;
 }
 
+/*
+ * Take the HDM memory region out of use: zap its mmaps and revoke its
+ * dma-bufs. A fault then gets SIGBUS from vfio_cxl_mem_huge_fault() until
+ * hdm_valid is set again.
+ */
+static void vfio_cxl_revoke_hdm(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	cxl->hdm_valid = false;
+	if (cxl->mem_region_index)
+		vfio_pci_core_revoke_region(vdev, cxl->mem_region_index);
+}
+
+static void vfio_cxl_rearm_hdm(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	cxl->hdm_valid = true;
+	if (cxl->mem_region_index)
+		vfio_pci_core_rearm_region(vdev, cxl->mem_region_index);
+}
+
 /*
  * vfio-cxl holds the HDM range, so the PCI core's cxl_reset method would
  * fail to request it. Run the same CXL Reset with that claim.
@@ -429,9 +544,7 @@ static int vfio_cxl_reset(struct vfio_pci_core_device *vdev)
 /* Low power loses the committed decoder: close host access until restored. */
 static void vfio_cxl_power_down(struct vfio_pci_core_device *vdev)
 {
-	lockdep_assert_held_write(&vdev->memory_lock);
-
-	vdev->cxl->hdm_valid = false;
+	vfio_cxl_revoke_hdm(vdev);
 }
 
 static void vfio_cxl_power_up(struct vfio_pci_core_device *vdev)
@@ -460,7 +573,7 @@ static void vfio_cxl_power_up(struct vfio_pci_core_device *vdev)
 	if (ret)
 		pci_warn(pdev, "vfio-cxl: HDM decoder restore failed: %d\n", ret);
 	else if (!vfio_cxl_await_mem_active(vdev))
-		cxl->hdm_valid = true;
+		vfio_cxl_rearm_hdm(vdev);
 }
 
 static const struct vfio_cxl_ops vfio_cxl_ops = {
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 883b51521e6a..359fed40eb1b 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -371,7 +371,7 @@ struct vfio_region_info_cap_type {
 #define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD	(1)
 
 /* CXL Type-2 device (0x1e98) sub-types for VFIO_REGION_TYPE_PCI_VENDOR_TYPE */
-/* CXL.mem HDM region of a Type-2 device, read and write */
+/* CXL.mem HDM region of a Type-2 device, read, write and mmap */
 #define VFIO_REGION_SUBTYPE_CXL_MEM		(1)
 
 /* sub-types for VFIO_REGION_TYPE_GFX */
-- 
2.53.0

[PATCH v6 23/31] vfio/cxl: Contain HDM memory errors with memory_failure()

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

The HDM memory region is mmap-able, so an fd holder can fault its
struct-page-less device memory in from the host CPU. A memory error on
that range cannot be routed through the normal struct-page path and
would otherwise escalate to a fatal host SError.

Register the range with memory_failure() via
register_pfn_address_space() so such an error is instead contained to a
SIGBUS to each process that maps the poisoned page. Provide a
pfn-to-vma-offset callback so memory_failure() can map a poisoned pfn
back to the file offset of each user mapping; the region is a single
linear range at hpa_range.start, so recover the offset the same way the
fault handler derived the pfn.

register_pfn_address_space() returns -EOPNOTSUPP when
CONFIG_MEMORY_FAILURE is off, in which case expose the region without
containment.  Unregister the range on close.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 84 +++++++++++++++++++++++++++-
 1 file changed, 83 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 977d0c9c747a..f2f7ddfc80bb 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -9,6 +9,7 @@
 #include <linux/cleanup.h>
 #include <linux/delay.h>
 #include <linux/io.h>
+#include <linux/memory-failure.h>
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/pci.h>
@@ -27,6 +28,7 @@
  * @hpa_range: host physical range of the HDM region
  * @hpa_res: exclusive reservation of @hpa_range, held for the bind
  * @hdm: copy of the HDM state cached at PCI enumeration, taken at bind
+ * @hdm_pfn_space: HDM-region pfn range registered with memory_failure()
  * @hdm_valid: true when host CPU access to the HDM range is safe; under memory_lock
  * @await_mem_active: the device reported its memory active at bind, so wait
  *	for Memory_Active before host access reopens after a reset or low power
@@ -38,6 +40,7 @@ struct vfio_cxl_state {
 	struct range hpa_range;
 	struct resource *hpa_res;
 	struct cxl_hdm_info *hdm;
+	struct pfn_address_space hdm_pfn_space;
 	bool hdm_valid;
 	bool await_mem_active;
 	unsigned int mem_region_index;
@@ -211,6 +214,63 @@ static const struct vfio_pci_regops vfio_cxl_mem_regops = {
 	.release = vfio_cxl_region_release,
 };
 
+/*
+ * Map a poisoned HDM-region pfn back to the file offset of each user mapping so
+ * memory_failure() can find the mapping and signal its owner. The region is a
+ * single linear range at hpa_range.start; recover the per-vma file offset the
+ * same way the fault handler derived the pfn.
+ */
+static int vfio_cxl_pfn_to_vma_pgoff(struct vm_area_struct *vma,
+				     unsigned long pfn, pgoff_t *pgoff)
+{
+	struct vfio_pci_core_device *vdev;
+	struct vfio_cxl_state *cxl;
+	pgoff_t vma_off, pfn_off;
+	unsigned long start_pfn;
+
+	if (vma->vm_ops != &vfio_cxl_mem_vm_ops)
+		return -ENOENT;
+
+	vdev = vma->vm_private_data;
+	cxl = vdev->cxl;
+
+	start_pfn = PHYS_PFN(cxl->hpa_range.start);
+	if (pfn < start_pfn ||
+	    pfn >= start_pfn + (range_len(&cxl->hpa_range) >> PAGE_SHIFT))
+		return -EFAULT;
+
+	pfn_off = pfn - start_pfn;
+	vma_off = vma->vm_pgoff &
+		  ((1UL << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
+	/* Skip VMAs that do not map the pfn, e.g. a partial mmap of the region. */
+	if (pfn_off < vma_off || pfn_off - vma_off >= vma_pages(vma))
+		return -EFAULT;
+
+	*pgoff = vma->vm_pgoff + (pfn_off - vma_off);
+	return 0;
+}
+
+/*
+ * The HDM region is struct-page-less device memory, so a memory error on it
+ * cannot be routed through the normal page path. Register the range with
+ * memory_failure() so such an error is contained to a SIGBUS to each process
+ * that maps the poisoned page, instead of escalating to a host SError. The
+ * mapping itself is not torn down.
+ */
+static int vfio_cxl_register_pfn_space(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	unsigned long start_pfn = PHYS_PFN(cxl->hpa_range.start);
+
+	cxl->hdm_pfn_space.node.start = start_pfn;
+	cxl->hdm_pfn_space.node.last =
+		start_pfn + (range_len(&cxl->hpa_range) >> PAGE_SHIFT) - 1;
+	cxl->hdm_pfn_space.mapping = vdev->vdev.inode->i_mapping;
+	cxl->hdm_pfn_space.pfn_to_vma_pgoff = vfio_cxl_pfn_to_vma_pgoff;
+
+	return register_pfn_address_space(&cxl->hdm_pfn_space);
+}
+
 /*
  * Create the CXL device state and join the device to the CXL topology. The
  * caller groups these devres allocations, so return on the first error.
@@ -482,13 +542,35 @@ static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
 	/* Record where the HDM memory region landed, to revoke it later. */
 	cxl->mem_region_index = VFIO_PCI_NUM_REGIONS + vdev->num_regions - 1;
 
+	/*
+	 * The HDM region is advertised mmap-able, so a fd holder can fault its
+	 * struct-page-less device memory in from the host CPU. Register it with
+	 * memory_failure() to contain a memory error. -EOPNOTSUPP means
+	 * CONFIG_MEMORY_FAILURE is off, so run without containment.
+	 */
+	ret = vfio_cxl_register_pfn_space(vdev);
+	if (ret && ret != -EOPNOTSUPP)
+		goto err_index;
+
 	return 0;
+
+err_index:
+	/*
+	 * A failed open never reaches close_device, and vfio-pci-core frees
+	 * the regions: forget the index, or a later revoke of the HDM gate
+	 * (runtime suspend of the closed device) names a region that is gone.
+	 */
+	cxl->mem_region_index = 0;
+	return ret;
 }
 
 static void vfio_cxl_close_device(struct vfio_pci_core_device *vdev)
 {
+	struct vfio_cxl_state *cxl = vdev->cxl;
+
+	unregister_pfn_address_space(&cxl->hdm_pfn_space);
 	/* vfio-pci-core frees the regions after this returns. */
-	vdev->cxl->mem_region_index = 0;
+	cxl->mem_region_index = 0;
 }
 
 static void vfio_cxl_reset_prepare(struct vfio_pci_core_device *vdev)
-- 
2.53.0

[PATCH v6 24/31] vfio/cxl: Expose the HDM decoder registers read-only to the guest

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A CXL Type-2 guest reads the HDM decoder registers to learn the HDM
region it was handed. Those registers live in the component BAR that
vfio-pci owns.

Map the decoder block at bind: its location comes from the pdev->hdm
enumeration cache, and vfio-pci owns the BAR, so map it without
claiming the block and expose it as a second, read-only region under
VFIO_REGION_TYPE_PCI_VENDOR_TYPE with the CXL vendor id, registered per
open like the HDM memory region.

Serve reads live from the mapped block and absorb writes without
forwarding them to hardware. The block is BAR MMIO, so it decodes only
with Memory Space enabled: serve reads under memory_lock and fail them
with -EIO while memory is disabled, as vfio_pci_bar_rw() does. A failed
decoder restore leaves Memory Space off; clear vfio's record to match.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 123 ++++++++++++++++++++++++++-
 include/uapi/linux/vfio.h            |   2 +
 2 files changed, 123 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index f2f7ddfc80bb..4a8aaa765a98 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -29,6 +29,8 @@
  * @hpa_res: exclusive reservation of @hpa_range, held for the bind
  * @hdm: copy of the HDM state cached at PCI enumeration, taken at bind
  * @hdm_pfn_space: HDM-region pfn range registered with memory_failure()
+ * @hdm_regs: mapped HDM decoder registers, read live by the decoder region
+ * @hdm_len: length of the HDM decoder register block
  * @hdm_valid: true when host CPU access to the HDM range is safe; under memory_lock
  * @await_mem_active: the device reported its memory active at bind, so wait
  *	for Memory_Active before host access reopens after a reset or low power
@@ -41,6 +43,8 @@ struct vfio_cxl_state {
 	struct resource *hpa_res;
 	struct cxl_hdm_info *hdm;
 	struct pfn_address_space hdm_pfn_space;
+	void __iomem *hdm_regs;
+	u32 hdm_len;
 	bool hdm_valid;
 	bool await_mem_active;
 	unsigned int mem_region_index;
@@ -214,6 +218,76 @@ static const struct vfio_pci_regops vfio_cxl_mem_regops = {
 	.release = vfio_cxl_region_release,
 };
 
+static ssize_t vfio_cxl_comp_rw(struct vfio_pci_core_device *vdev,
+				char __user *buf, size_t count, loff_t *ppos,
+				bool iswrite)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	__le32 *tmp;
+
+	if (pos >= cxl->hdm_len)
+		return -EINVAL;
+
+	/*
+	 * Component registers allow only full-width accesses: a 32-bit
+	 * register as one 4-byte read or write (CXL r4.0 sec 8.2.3).
+	 */
+	if (pos % sizeof(u32) || count % sizeof(u32))
+		return -EINVAL;
+
+	count = min_t(size_t, count, cxl->hdm_len - pos);
+
+	/*
+	 * The physical decoder was committed before the guest saw the device
+	 * (firmware may also have locked it), so the guest never drives it:
+	 * absorb writes without forwarding them to hardware. The guest programs
+	 * a GPA that the VMM virtualizes; reads return the live registers, which
+	 * already report the decoder committed. BASE_LOW and BASE_HIGH carry the
+	 * host HPA, visible only to the trusted VMM that virtualizes it away from
+	 * the guest.
+	 */
+	if (iswrite) {
+		*ppos += count;
+		return count;
+	}
+
+	tmp = kmalloc(count, GFP_KERNEL);
+	if (!tmp)
+		return -ENOMEM;
+
+	/* The block is BAR MMIO, so it decodes only with memory enabled. */
+	scoped_guard(rwsem_read, &vdev->memory_lock) {
+		if (!__vfio_pci_memory_enabled(vdev)) {
+			kfree(tmp);
+			return -EIO;
+		}
+		/*
+		 * One readl() per register: memcpy_fromio() may split the
+		 * copy into byte or qword reads, and the decoder registers
+		 * take only aligned dword accesses. Return them in the
+		 * register byte order, as a raw copy would.
+		 */
+		for (size_t i = 0; i < count / sizeof(u32); i++)
+			tmp[i] = cpu_to_le32(readl(cxl->hdm_regs + pos +
+						   i * sizeof(u32)));
+	}
+
+	if (copy_to_user(buf, tmp, count)) {
+		kfree(tmp);
+		return -EFAULT;
+	}
+	kfree(tmp);
+
+	*ppos += count;
+	return count;
+}
+
+static const struct vfio_pci_regops vfio_cxl_comp_regops = {
+	.rw = vfio_cxl_comp_rw,
+	.release = vfio_cxl_region_release,
+};
+
 /*
  * Map a poisoned HDM-region pfn back to the file offset of each user mapping so
  * memory_failure() can find the mapping and signal its owner. The region is a
@@ -318,6 +392,20 @@ vfio_cxl_create_memdev(struct pci_dev *pdev, const struct cxl_hdm_info *hdm,
 		return ERR_PTR(-ENODEV);
 	}
 
+	/*
+	 * Map the HDM decoder registers so the decoder region can read them
+	 * live. The block location comes from the enumeration cache in
+	 * pdev->hdm, read through the copy in hdm; vfio-pci owns the BAR, so map
+	 * without claiming the block.
+	 */
+	cxl->hdm_regs = devm_ioremap(&pdev->dev,
+				     pci_resource_start(pdev, hdm->hdm_bar) +
+				     hdm->hdm_offset, hdm->hdm_size);
+	if (!cxl->hdm_regs)
+		return ERR_PTR(-ENOMEM);
+
+	cxl->hdm_len = hdm->hdm_size;
+
 	/*
 	 * A Type-2 accelerator has no mailbox and no media-ready register, so
 	 * set media ready directly.
@@ -542,6 +630,13 @@ static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
 	/* Record where the HDM memory region landed, to revoke it later. */
 	cxl->mem_region_index = VFIO_PCI_NUM_REGIONS + vdev->num_regions - 1;
 
+	ret = vfio_cxl_add_region(vdev, VFIO_REGION_SUBTYPE_CXL_COMP_REGS,
+				  &vfio_cxl_comp_regops, cxl->hdm_len,
+				  VFIO_REGION_INFO_FLAG_READ |
+				  VFIO_REGION_INFO_FLAG_WRITE);
+	if (ret)
+		goto err_index;
+
 	/*
 	 * The HDM region is advertised mmap-able, so a fd holder can fault its
 	 * struct-page-less device memory in from the host CPU. Register it with
@@ -609,6 +704,28 @@ static void vfio_cxl_rearm_hdm(struct vfio_pci_core_device *vdev)
 		vfio_pci_core_rearm_region(vdev, cxl->mem_region_index);
 }
 
+/*
+ * A CXL Reset or decoder restore that fails leaves Memory Space and Bus
+ * Master disabled (cxl_reset_save_disabled_state()). The user reads Command
+ * from the device, but vfio-pci-core gates BAR access, BAR mmap faults, BAR
+ * dma-bufs and the decoder register region on its own record of Memory
+ * Space. Match that record to the device, zapping the BAR mmaps and
+ * revoking the BAR dma-bufs as a user clear would; the user's next Memory
+ * Space write sets it again.
+ */
+static void vfio_cxl_sync_mem_enable(struct vfio_pci_core_device *vdev)
+{
+	u16 cmd;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	if (!pci_read_config_word(vdev->pdev, PCI_COMMAND, &cmd) &&
+	    !PCI_POSSIBLE_ERROR(cmd) && (cmd & PCI_COMMAND_MEMORY))
+		return;
+
+	vfio_pci_core_clear_memory_enable(vdev);
+}
+
 /*
  * vfio-cxl holds the HDM range, so the PCI core's cxl_reset method would
  * fail to request it. Run the same CXL Reset with that claim.
@@ -652,10 +769,12 @@ static void vfio_cxl_power_up(struct vfio_pci_core_device *vdev)
 		ret = cxl_restore_state_after_pci_reset(pdev, cxl->hdm);
 	pci_dev_unlock(pdev);
 
-	if (ret)
+	if (ret) {
 		pci_warn(pdev, "vfio-cxl: HDM decoder restore failed: %d\n", ret);
-	else if (!vfio_cxl_await_mem_active(vdev))
+		vfio_cxl_sync_mem_enable(vdev);
+	} else if (!vfio_cxl_await_mem_active(vdev)) {
 		vfio_cxl_rearm_hdm(vdev);
+	}
 }
 
 static const struct vfio_cxl_ops vfio_cxl_ops = {
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 359fed40eb1b..6cba3284b21e 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -373,6 +373,8 @@ struct vfio_region_info_cap_type {
 /* CXL Type-2 device (0x1e98) sub-types for VFIO_REGION_TYPE_PCI_VENDOR_TYPE */
 /* CXL.mem HDM region of a Type-2 device, read, write and mmap */
 #define VFIO_REGION_SUBTYPE_CXL_MEM		(1)
+/* CXL HDM decoder registers: read live, guest writes are absorbed */
+#define VFIO_REGION_SUBTYPE_CXL_COMP_REGS	(2)
 
 /* sub-types for VFIO_REGION_TYPE_GFX */
 #define VFIO_REGION_SUBTYPE_GFX_EDID            (1)
-- 
2.53.0

[PATCH v6 25/31] vfio/cxl: Exclude the HDM decoder registers from direct BAR access

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

The HDM decoder register block lives in the component BAR that vfio-pci
owns. The guest reads the decoder only through the trapped, read-only
decoder region; it must not reach the physical registers through the BAR
directly, and a write through the BAR could uncommit or move an unlocked
decoder that the host relies on.

Add the block to the generic excluded-range list with
VFIO_PCI_EXCLUDE_MMAP | VFIO_PCI_EXCLUDE_READ | VFIO_PCI_EXCLUDE_WRITE
similar to the MSI-X exclusion.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 4a8aaa765a98..68e898ee1985 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -614,6 +614,22 @@ static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
 	struct vfio_cxl_state *cxl = vdev->cxl;
 	int ret;
 
+	/*
+	 * Keep the HDM decoder block out of the guest's direct BAR access: the
+	 * guest reaches it only through the trapped decoder region, and a write
+	 * through the BAR could uncommit or move an unlocked decoder that the
+	 * host relies on. Exclude it from mmap, fill reads with -1 and drop
+	 * writes. The list is cleared on close, so re-add it per open.
+	 */
+	ret = vfio_pci_core_add_excluded_range(vdev, cxl->hdm->hdm_bar,
+					       cxl->hdm->hdm_offset,
+					       cxl->hdm_len,
+					       VFIO_PCI_EXCLUDE_MMAP |
+					       VFIO_PCI_EXCLUDE_READ |
+					       VFIO_PCI_EXCLUDE_WRITE);
+	if (ret)
+		return ret;
+
 	/*
 	 * vfio-pci-core frees all dynamic regions on close and on a failed
 	 * open, so register them here per open rather than at bind.
-- 
2.53.0

[PATCH v6 26/31] vfio/cxl: Restore the HDM decoder after a hot reset

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

The HDM memory fault inserts a PFN only when the decoder is in a
known-good state (cxl->hdm_valid), so the host CPU never faults the
coherent range into a disabled decoder.

VFIO_DEVICE_PCI_HOT_RESET drives a plain secondary bus reset rather than
the CXL reset sequence. A secondary bus reset only reaches a CXL
endpoint when the upstream port has SBR unmasked (CXL r4.0 sec 8.1.5.2).
With SBR masked the PCI core now refuses the bus reset, so the hot reset
fails instead of doing nothing. With SBR unmasked the reset is a Hot
Reset, which returns the decoder's RWL fields, Commit included, to their
defaults and decommits it (CXL r4.0 sec 8.2.4.20.7, Table 8-1).

Take the HDM memory region out of use in reset_prepare. In reset_done,
when the port has SBR unmasked and the decoder is no longer committed,
restore it with cxl_restore_state_after_pci_reset(), the helper the PCI
core's cxl_bus reset method uses, and put the region back. A decoder
that is still committed, after an FLR or a cxl_bus reset that restored
it itself, is left alone. If the restore fails, reset_done returns the
error, the hot reset ioctl fails with it, and the region stays out of
use until a reset succeeds. reset_done also runs after a reset method
that failed and after resets that do not reach the decoder; then put the
region back only if it was in use when reset_prepare ran. Before putting
the region back, reset_done waits for Memory_Active as power_up does, and
returns -ETIMEDOUT if the memory does not come back.

AER and DPC recovery can reset the link without reset_done. The HDM
region is taken out of use when the error is reported and put back when
recovery resumes the device only if it was in use before and the decoder
is still committed; recovery does not restore config space, so otherwise
the region stays out of use until the user resets the device. The
resume path also waits for Memory_Active before it puts the region back.

vfio_cxl_sbr_unmasked() mirrors the cxl_sbr_masked() check the PCI core
uses (drivers/pci/pci.c) for its own CXL bus reset. power_up now also
skips the restore when the decoder is still committed, as after a D3
request that left the device in D0.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 156 +++++++++++++++++++++++++--
 1 file changed, 147 insertions(+), 9 deletions(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 68e898ee1985..3cbb9fbc8328 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -18,6 +18,7 @@
 #include <linux/uaccess.h>
 #include <linux/vfio_pci_core.h>
 #include <cxl/cxl.h>
+#include <cxl/cxl_regs.h>
 #include <cxl/hdm.h>
 #include <cxl/pci.h>
 
@@ -35,6 +36,7 @@
  * @await_mem_active: the device reported its memory active at bind, so wait
  *	for Memory_Active before host access reopens after a reset or low power
  * @mem_region_index: vfio region index of the HDM memory region while open, else 0
+ * @hdm_open_at_prepare: @hdm_valid as reset_prepare found it; under memory_lock
  */
 struct vfio_cxl_state {
 	struct cxl_dev_state cxlds;
@@ -48,6 +50,7 @@ struct vfio_cxl_state {
 	bool hdm_valid;
 	bool await_mem_active;
 	unsigned int mem_region_index;
+	bool hdm_open_at_prepare;
 };
 
 static unsigned long vfio_cxl_mem_pgoff(struct vm_area_struct *vma,
@@ -684,15 +687,6 @@ static void vfio_cxl_close_device(struct vfio_pci_core_device *vdev)
 	cxl->mem_region_index = 0;
 }
 
-static void vfio_cxl_reset_prepare(struct vfio_pci_core_device *vdev)
-{
-}
-
-static int vfio_cxl_reset_done(struct vfio_pci_core_device *vdev)
-{
-	return 0;
-}
-
 /*
  * Take the HDM memory region out of use: zap its mmaps and revoke its
  * dma-bufs. A fault then gets SIGBUS from vfio_cxl_mem_huge_fault() until
@@ -756,6 +750,33 @@ static int vfio_cxl_reset(struct vfio_pci_core_device *vdev)
 	return pci_cxl_reset_owned(vdev->pdev, vdev->cxl->hpa_res);
 }
 
+/*
+ * Report whether the device still decodes its HDM range: memory decode on,
+ * CXL.mem enabled in the CXL Device DVSEC and the decoder committed. A
+ * secondary bus reset or a link down clears all three. Config space is read
+ * first; the decoder registers sit in a BAR and are read only while memory
+ * decode is on.
+ */
+static bool vfio_cxl_hdm_intact(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	struct pci_dev *pdev = vdev->pdev;
+	u16 cmd, ctrl;
+	u32 dctrl;
+
+	if (pci_read_config_word(pdev, PCI_COMMAND, &cmd) ||
+	    PCI_POSSIBLE_ERROR(cmd) || !(cmd & PCI_COMMAND_MEMORY))
+		return false;
+
+	if (pci_read_config_word(pdev, cxl->hdm->dvsec + PCI_DVSEC_CXL_CTRL,
+				 &ctrl) ||
+	    PCI_POSSIBLE_ERROR(ctrl) || !(ctrl & PCI_DVSEC_CXL_MEM_ENABLE))
+		return false;
+
+	dctrl = readl(cxl->hdm_regs + CXL_HDM_DECODER0_CTRL_OFFSET(0));
+	return dctrl != U32_MAX && (dctrl & CXL_HDM_DECODER0_CTRL_COMMITTED);
+}
+
 /* Low power loses the committed decoder: close host access until restored. */
 static void vfio_cxl_power_down(struct vfio_pci_core_device *vdev)
 {
@@ -770,6 +791,16 @@ static void vfio_cxl_power_up(struct vfio_pci_core_device *vdev)
 
 	lockdep_assert_held_write(&vdev->memory_lock);
 
+	/*
+	 * A D3 request that failed, or a quirk that kept the device in D0,
+	 * lost nothing: reopen without decommitting and recommitting.
+	 */
+	if (vfio_cxl_hdm_intact(vdev)) {
+		if (!vfio_cxl_await_mem_active(vdev))
+			vfio_cxl_rearm_hdm(vdev);
+		return;
+	}
+
 	/*
 	 * Restore the decoder that low power lost. The restore needs the device
 	 * lock; if it is contended, keep host access closed until a reset
@@ -793,6 +824,112 @@ static void vfio_cxl_power_up(struct vfio_pci_core_device *vdev)
 	}
 }
 
+/* A PCI core reset may clear the decoder: take the HDM region out of use. */
+static void vfio_cxl_reset_prepare(struct vfio_pci_core_device *vdev)
+{
+	vdev->cxl->hdm_open_at_prepare = vdev->cxl->hdm_valid;
+	vfio_cxl_revoke_hdm(vdev);
+}
+
+/*
+ * A secondary bus reset only reaches a CXL endpoint when the upstream port
+ * has SBR unmasked (CXL r4.0 sec 8.1.5.2); otherwise the Bridge Control SBR
+ * bit is ignored and the decoder is left intact. Mirrors the cxl_sbr_masked()
+ * check the PCI core uses for its own CXL bus reset.
+ */
+static bool vfio_cxl_sbr_unmasked(struct pci_dev *pdev)
+{
+	struct pci_dev *bridge = pci_upstream_bridge(pdev);
+	u16 dvsec, ctl;
+
+	if (!bridge)
+		return false;
+
+	dvsec = pci_find_dvsec_capability(bridge, PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_PORT);
+	if (!dvsec)
+		return false;
+
+	if (pci_read_config_word(bridge, dvsec + PCI_DVSEC_CXL_PORT_CTL, &ctl))
+		return false;
+
+	return ctl & PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR;
+}
+
+static int vfio_cxl_reset_done(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	struct pci_dev *pdev = vdev->pdev;
+	int ret;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	/*
+	 * VFIO_DEVICE_PCI_HOT_RESET drives a plain secondary bus reset, not the
+	 * CXL-aware cxl_reset_bus_function(). When the upstream port has SBR
+	 * unmasked the reset decommits the decoder. The PCI core has restored
+	 * config space by now; restore the decoder with the same helper its
+	 * cxl_bus reset method uses. A masked port refuses the bus reset. Other
+	 * resets (FLR, the cxl_reset and cxl_bus methods) leave the decoder
+	 * committed or restore it themselves; restore only a decoder that is
+	 * gone, so a committed one is not decommitted and committed again.
+	 */
+	if (vfio_cxl_sbr_unmasked(pdev) && !vfio_cxl_hdm_intact(vdev)) {
+		ret = pci_save_state(pdev);
+		if (!ret)
+			ret = cxl_restore_state_after_pci_reset(pdev, cxl->hdm);
+		if (ret) {
+			pci_warn(pdev, "vfio-cxl: HDM decoder restore failed: %d\n",
+				 ret);
+			vfio_cxl_sync_mem_enable(vdev);
+			return ret;
+		}
+	} else if (!cxl->hdm_open_at_prepare) {
+		/*
+		 * reset_done runs even when the reset method failed. Nothing
+		 * restored the decoder, and it was not known good before this
+		 * reset (a failed reset or a skipped restore closed the gate),
+		 * so keep host access closed.
+		 */
+		return 0;
+	}
+
+	ret = vfio_cxl_await_mem_active(vdev);
+	if (ret)
+		return ret;
+
+	vfio_cxl_rearm_hdm(vdev);
+	return 0;
+}
+
+/*
+ * AER or DPC recovery resumes the device without restoring its config space,
+ * so do not restore the decoder here. Put the HDM region back only if the
+ * device still decodes it; otherwise the user's reset restores it.
+ */
+static void vfio_cxl_error_resume(struct vfio_pci_core_device *vdev)
+{
+	struct pci_dev *pdev = vdev->pdev;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	/* Low power closed the region; vfio_cxl_power_up() reopens it. */
+	if (pdev->current_state != PCI_D0)
+		return;
+
+	/* The region was closed before the error; keep it that way. */
+	if (!vdev->cxl->hdm_open_at_prepare)
+		return;
+
+	if (!vfio_cxl_hdm_intact(vdev)) {
+		pci_warn(pdev, "vfio-cxl: HDM decoder lost in error recovery, reset the device to restore it\n");
+		return;
+	}
+
+	if (!vfio_cxl_await_mem_active(vdev))
+		vfio_cxl_rearm_hdm(vdev);
+}
+
 static const struct vfio_cxl_ops vfio_cxl_ops = {
 	.init		= vfio_cxl_init_device,
 	.release	= vfio_cxl_release_device,
@@ -800,6 +937,7 @@ static const struct vfio_cxl_ops vfio_cxl_ops = {
 	.close_device	= vfio_cxl_close_device,
 	.reset_prepare	= vfio_cxl_reset_prepare,
 	.reset_done	= vfio_cxl_reset_done,
+	.error_resume	= vfio_cxl_error_resume,
 	.reset		= vfio_cxl_reset,
 	.power_down	= vfio_cxl_power_down,
 	.power_up	= vfio_cxl_power_up,
-- 
2.53.0

[PATCH v6 27/31] vfio/cxl: Describe the CXL device and decoder geometry to userspace

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Advertise the assignment with a device-info flag so a VMM can tell a CXL
Type-2 device apart from a device bound without the CXL extensions.

On the trapped decoder register region, add a capability carrying the BAR
and offset of the HDM decoder block so the VMM can place the trapped
window where the guest expects to find it.

The size of the guest's memory window comes from the HDM memory region.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 16 ++++++++++++++++
 drivers/vfio/pci/vfio_pci_core.c     |  3 +++
 include/uapi/linux/vfio.h            | 18 ++++++++++++++++++
 3 files changed, 37 insertions(+)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 3cbb9fbc8328..9ef052a3745e 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -286,9 +286,25 @@ static ssize_t vfio_cxl_comp_rw(struct vfio_pci_core_device *vdev,
 	return count;
 }
 
+static int vfio_cxl_comp_add_capability(struct vfio_pci_core_device *vdev,
+					struct vfio_pci_region *region,
+					struct vfio_info_cap *caps)
+{
+	struct cxl_hdm_info *hdm = vdev->cxl->hdm;
+	struct vfio_region_info_cap_cxl_comp_regs cap = {
+		.header.id = VFIO_REGION_INFO_CAP_CXL_COMP_REGS,
+		.header.version = 1,
+		.bar = hdm->hdm_bar,
+		.offset = hdm->hdm_offset,
+	};
+
+	return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
+}
+
 static const struct vfio_pci_regops vfio_cxl_comp_regops = {
 	.rw = vfio_cxl_comp_rw,
 	.release = vfio_cxl_region_release,
+	.add_capability = vfio_cxl_comp_add_capability,
 };
 
 /*
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index be2acab625c0..0c4a3637619f 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1391,6 +1391,9 @@ static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev,
 	if (vdev->reset_works)
 		info.flags |= VFIO_DEVICE_FLAGS_RESET;
 
+	if (vdev->cxl_ops)
+		info.flags |= VFIO_DEVICE_FLAGS_CXL;
+
 	info.num_regions = VFIO_PCI_NUM_REGIONS + vdev->num_regions;
 	info.num_irqs = VFIO_PCI_NUM_IRQS;
 
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 6cba3284b21e..81985e68166a 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -215,6 +215,7 @@ struct vfio_device_info {
 #define VFIO_DEVICE_FLAGS_FSL_MC (1 << 6)	/* vfio-fsl-mc device */
 #define VFIO_DEVICE_FLAGS_CAPS	(1 << 7)	/* Info supports caps */
 #define VFIO_DEVICE_FLAGS_CDX	(1 << 8)	/* vfio-cdx device */
+#define VFIO_DEVICE_FLAGS_CXL	(1 << 9)	/* vfio-cxl device */
 	__u32	num_regions;	/* Max region index + 1 */
 	__u32	num_irqs;	/* Max IRQ index + 1 */
 	__u32   cap_offset;	/* Offset within info struct of first cap */
@@ -503,6 +504,23 @@ struct vfio_region_info_cap_nvlink2_lnkspd {
 	__u32 __pad;
 };
 
+/*
+ * Geometry of a CXL Type-2 device's HDM decoder registers, so a VMM can place
+ * the trapped component register window where the guest expects it. The trapped
+ * region spans the whole HDM decoder block (every decoder), not just decoder 0:
+ * a VMM reads the decoder count and each decoder's committed base from the block
+ * itself. Additional trapped component capabilities, such as CXL RAS, are
+ * exposed as their own region subtypes rather than by extending this cap.
+ */
+#define VFIO_REGION_INFO_CAP_CXL_COMP_REGS	6
+
+struct vfio_region_info_cap_cxl_comp_regs {
+	struct vfio_info_cap_header header;
+	__u32 bar;
+	__u32 __resv;
+	__aligned_u64 offset;
+};
+
 /**
  * VFIO_DEVICE_GET_IRQ_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 9,
  *				    struct vfio_irq_info)
-- 
2.53.0

[PATCH v6 28/31] vfio/cxl: Export the HDM memory region as a dma-buf

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A Type-2 accelerator issues ATS-translated DMA to addresses inside its
own HDM window, so that coherent host range must be present in the
guest's IOAS (the iommufd IOAS backing the nested SMMU stage-2). iommufd
maps a struct-page-less range only by fd, via IOMMU_IOAS_MAP_FILE over a
dma-buf; a userspace-VA IOMMU_IOAS_MAP of the HDM mmap is rejected
because the VMA is VM_IO | VM_PFNMAP. Without a dma-buf the range could
only be mapped through an out-of-tree PFNMAP work-around.

vfio-pci already exports BAR memory as a P2P dma-buf, but the exporter
is BAR-only: vfio_pci_core_feature_dma_buf() rejects any region index at
or above the ROM index, and vfio_pci_core_get_dmabuf_phys() resolves the
physical range from a PCI BAR. The HDM memory region is a dynamic
device-specific region, not a BAR.

Let a device-specific region reach the device's get_dmabuf_phys(): a
region index at or above VFIO_PCI_NUM_REGIONS skips the BAR-resource
check and is validated by the driver instead, bounded to the regions
that exist. Install a CXL-aware get_dmabuf_phys() in the vfio-cxl
provider that returns cxl->hpa_range for the HDM memory region and
delegates real BARs to the core, keeping the BAR path unchanged and the
core free of CXL knowledge.

The HDM window is coherent host memory with no p2pdma provider of its
own, so borrow BAR 0's, matching nvgrace-gpu's handling of its non-BAR
device memory. The iommufd importer does not consume the provider; the
scatterlist map path (real peer DMA) is left to a follow-up once
upstream grows a negotiated interconnect for coherent CXL memory. Mark a
device-specific dma-buf so vfio_pci_dma_buf_map() refuses that path,
which would translate the range with the borrowed provider's bus offset.

The HDM dma-buf belongs to its region, not to the BARs: a memory enable
change leaves it alone, and vfio-cxl revokes and re-arms it with the
region when the decoder state changes. A new dma-buf of a
device-specific region starts in the state its region reports through a
new optional revoked() region op, so an HDM export made with memory
disabled starts armed, and one made after a failed reset starts revoked.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 76 +++++++++++++++++++++++++++-
 drivers/vfio/pci/vfio_pci_dmabuf.c   | 73 +++++++++++++++++++++++---
 include/linux/vfio_pci_core.h        |  9 ++++
 3 files changed, 151 insertions(+), 7 deletions(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 9ef052a3745e..518f0b093652 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -13,6 +13,7 @@
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/pci-p2pdma.h>
 #include <linux/range.h>
 #include <linux/slab.h>
 #include <linux/uaccess.h>
@@ -206,6 +207,21 @@ static ssize_t vfio_cxl_mem_rw(struct vfio_pci_core_device *vdev,
 	return done;
 }
 
+/*
+ * The HDM region follows hdm_valid, not Memory Space Enable, so a dma-buf
+ * exported from it starts revoked while host access is unsafe and is
+ * re-armed with the region in vfio_cxl_rearm_hdm().
+ */
+static bool vfio_cxl_mem_revoked(struct vfio_pci_core_device *vdev,
+				 struct vfio_pci_region *region)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	return !cxl->hdm_valid || !cxl->cxlds.media_ready;
+}
+
 /*
  * The CXL regions carry no per-region state (region->data is the shared,
  * devm-managed vfio_cxl_state), so releasing a region is a no-op.
@@ -219,6 +235,7 @@ static const struct vfio_pci_regops vfio_cxl_mem_regops = {
 	.rw = vfio_cxl_mem_rw,
 	.mmap = vfio_cxl_mem_mmap,
 	.release = vfio_cxl_region_release,
+	.revoked = vfio_cxl_mem_revoked,
 };
 
 static ssize_t vfio_cxl_comp_rw(struct vfio_pci_core_device *vdev,
@@ -443,6 +460,50 @@ vfio_cxl_create_memdev(struct pci_dev *pdev, const struct cxl_hdm_info *hdm,
 	return cxl;
 }
 
+/*
+ * Resolve the physical range that backs a dma-buf export. The core exporter
+ * only knows BARs; teach it the HDM memory region so a guest IOAS can map the
+ * coherent window by fd (IOMMU_IOAS_MAP_FILE) instead of the removed PFNMAP
+ * work-around. Real BARs stay on the byte-identical core path.
+ */
+static int vfio_cxl_get_dmabuf_phys(struct vfio_pci_core_device *vdev,
+				    struct p2pdma_provider **provider,
+				    unsigned int region_index,
+				    struct phys_vec *phys_vec,
+				    struct vfio_region_dma_range *dma_ranges,
+				    size_t nr_ranges)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+
+	/* Real BARs go through the core P2P exporter unchanged. */
+	if (region_index < VFIO_PCI_NUM_REGIONS)
+		return vfio_pci_core_get_dmabuf_phys(vdev, provider,
+						     region_index, phys_vec,
+						     dma_ranges, nr_ranges);
+
+	/* Of the device regions, only the HDM memory window is exportable. */
+	if (region_index != cxl->mem_region_index)
+		return -EINVAL;
+
+	/*
+	 * The HDM window is coherent host memory, not BAR MMIO, so it has no
+	 * p2pdma provider of its own. Borrow BAR 0's: the P2P properties match
+	 * and the iommufd importer does not consume the provider. The sgt map
+	 * path (real peer DMA) is not supported for the HDM window.
+	 */
+	*provider = pcim_p2pdma_provider(vdev->pdev, 0);
+	if (!*provider)
+		return -EINVAL;
+
+	return vfio_pci_core_fill_phys_vec(phys_vec, dma_ranges, nr_ranges,
+					  cxl->hpa_range.start,
+					  range_len(&cxl->hpa_range));
+}
+
+static const struct vfio_pci_device_ops vfio_cxl_pci_dev_ops = {
+	.get_dmabuf_phys = vfio_cxl_get_dmabuf_phys,
+};
+
 /* CXL r4.0 sec 8.1.3.8.2: Range 1 Size Low Memory_Active_Timeout encodings */
 static const unsigned int vfio_cxl_mem_active_timeout_s[] = { 1, 4, 16, 64, 256 };
 
@@ -662,9 +723,22 @@ static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
 	if (ret)
 		return ret;
 
-	/* Record where the HDM memory region landed, to revoke it later. */
+	/*
+	 * Record where the HDM memory region landed, to revoke it and to
+	 * resolve its dma-buf export.
+	 */
 	cxl->mem_region_index = VFIO_PCI_NUM_REGIONS + vdev->num_regions - 1;
 
+	/*
+	 * Override the device ops so a dma-buf export of the HDM memory region
+	 * resolves to the coherent host range. This is done at open, not init:
+	 * vfio_pci_probe() resets pci_ops after vfio_alloc_device() returns, so
+	 * an override installed during init would be clobbered. Only a CXL device
+	 * reaches this hook (cxl_ops is set on init success), so a device bound
+	 * without CXL support keeps the core ops.
+	 */
+	vdev->pci_ops = &vfio_cxl_pci_dev_ops;
+
 	ret = vfio_cxl_add_region(vdev, VFIO_REGION_SUBTYPE_CXL_COMP_REGS,
 				  &vfio_cxl_comp_regops, cxl->hdm_len,
 				  VFIO_REGION_INFO_FLAG_READ |
diff --git a/drivers/vfio/pci/vfio_pci_dmabuf.c b/drivers/vfio/pci/vfio_pci_dmabuf.c
index d0f7ec5487de..237f7e055dd8 100644
--- a/drivers/vfio/pci/vfio_pci_dmabuf.c
+++ b/drivers/vfio/pci/vfio_pci_dmabuf.c
@@ -21,6 +21,13 @@ struct vfio_pci_dma_buf {
 	struct completion comp;
 	u32 region_index;
 	u8 revoked : 1;
+	/*
+	 * Set for a device-specific region (e.g. the CXL HDM window) that
+	 * borrows a BAR's p2pdma provider: mapping by physical address through
+	 * iommufd is safe, but a peer-DMA sgt map would translate with the
+	 * wrong provider bus offset.
+	 */
+	u8 no_peer2peer : 1;
 };
 
 static int vfio_pci_dma_buf_attach(struct dma_buf *dmabuf,
@@ -60,6 +67,15 @@ vfio_pci_dma_buf_map(struct dma_buf_attachment *attachment,
 	if (priv->revoked)
 		return ERR_PTR(-ENODEV);
 
+	/*
+	 * A borrowed-provider export can only be mapped by physical address
+	 * through iommufd, not as a peer-DMA sgt: dma_buf_phys_vec_to_sgt()
+	 * would translate the physical range with the borrowed provider's bus
+	 * offset and hand the peer a wrong address. Refuse it.
+	 */
+	if (priv->no_peer2peer)
+		return ERR_PTR(-EOPNOTSUPP);
+
 	ret = dma_buf_phys_vec_to_sgt(attachment, priv->provider,
 				      priv->phys_vec, priv->nr_ranges,
 				      priv->size, dir);
@@ -179,6 +195,15 @@ int vfio_pci_core_get_dmabuf_phys(struct vfio_pci_core_device *vdev,
 {
 	struct pci_dev *pdev = vdev->pdev;
 
+	/*
+	 * This resolver only handles PCI BARs. A device-specific region index
+	 * (>= PCI_STD_NUM_BARS) would index the per-BAR provider array
+	 * (p2p->mem[]) out of bounds in pcim_p2pdma_provider(), so reject it; a
+	 * driver that exports such a region installs its own get_dmabuf_phys.
+	 */
+	if (region_index >= PCI_STD_NUM_BARS)
+		return -EINVAL;
+
 	*provider = pcim_p2pdma_provider(pdev, region_index);
 	if (!*provider)
 		return -EINVAL;
@@ -219,6 +244,26 @@ static int validate_dmabuf_input(struct vfio_device_feature_dma_buf *dma_buf,
 	return 0;
 }
 
+/*
+ * A BAR dma-buf follows the memory enable bit. A device-specific region
+ * follows its own state, which its provider moves with
+ * vfio_pci_core_revoke_region() and vfio_pci_core_rearm_region(); start a
+ * new dma-buf in that state, since a memory enable change never moves it.
+ */
+static bool vfio_pci_dma_buf_start_revoked(struct vfio_pci_core_device *vdev,
+					   u32 index)
+{
+	struct vfio_pci_region *region;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	if (index < VFIO_PCI_NUM_REGIONS)
+		return !__vfio_pci_memory_enabled(vdev);
+
+	region = &vdev->region[index - VFIO_PCI_NUM_REGIONS];
+	return region->ops->revoked && region->ops->revoked(vdev, region);
+}
+
 int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 				  struct vfio_device_feature_dma_buf __user *arg,
 				  size_t argsz)
@@ -228,6 +273,7 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 	DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
 	struct vfio_pci_dma_buf *priv;
 	size_t length;
+	u32 index;
 	int ret;
 
 	if (!vdev->pci_ops || !vdev->pci_ops->get_dmabuf_phys)
@@ -244,13 +290,22 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 	if (!get_dma_buf.nr_ranges || get_dma_buf.flags)
 		return -EINVAL;
 
+	index = get_dma_buf.region_index;
+
 	/*
-	 * For PCI the region_index is the BAR number like everything
-	 * else.  Check that PCI resources have been claimed for it.
+	 * A fixed region index is the BAR number; only a BAR can be exported
+	 * and its PCI resource must be claimed. A device-specific region (index
+	 * >= VFIO_PCI_NUM_REGIONS) has no BAR resource and is validated by the
+	 * device's get_dmabuf_phys instead, but the index must name a region
+	 * that exists.
 	 */
-	if (get_dma_buf.region_index >= VFIO_PCI_ROM_REGION_INDEX ||
-	    IS_ERR(vfio_pci_core_get_iomap(vdev, get_dma_buf.region_index)))
+	if (index < VFIO_PCI_NUM_REGIONS) {
+		if (index >= VFIO_PCI_ROM_REGION_INDEX ||
+		    IS_ERR(vfio_pci_core_get_iomap(vdev, index)))
+			return -ENODEV;
+	} else if (index - VFIO_PCI_NUM_REGIONS >= vdev->num_regions) {
 		return -ENODEV;
+	}
 
 	dma_ranges = memdup_array_user(&arg->dma_ranges, get_dma_buf.nr_ranges,
 				       sizeof(*dma_ranges));
@@ -266,7 +321,7 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 	 * BAR keeps off mmap, so refuse such a range here as mmap does.
 	 */
 	for (u32 i = 0; i < get_dma_buf.nr_ranges; i++) {
-		if (vfio_pci_bar_range_excluded(vdev, get_dma_buf.region_index,
+		if (vfio_pci_bar_range_excluded(vdev, index,
 						dma_ranges[i].offset,
 						dma_ranges[i].length,
 						VFIO_PCI_EXCLUDE_MMAP)) {
@@ -290,6 +345,12 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 	priv->nr_ranges = get_dma_buf.nr_ranges;
 	priv->size = length;
 	priv->region_index = get_dma_buf.region_index;
+	/*
+	 * A device-specific region borrows a BAR's p2pdma provider, so it can
+	 * only be mapped by physical address (iommufd), never as a peer-DMA
+	 * sgt. See vfio_pci_dma_buf_map().
+	 */
+	priv->no_peer2peer = index >= VFIO_PCI_NUM_REGIONS;
 	ret = vdev->pci_ops->get_dmabuf_phys(vdev, &priv->provider,
 					     get_dma_buf.region_index,
 					     priv->phys_vec, dma_ranges,
@@ -323,7 +384,7 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 	INIT_LIST_HEAD(&priv->dmabufs_elm);
 	down_write(&vdev->memory_lock);
 	dma_resv_lock(priv->dmabuf->resv, NULL);
-	priv->revoked = !__vfio_pci_memory_enabled(vdev);
+	priv->revoked = vfio_pci_dma_buf_start_revoked(vdev, index);
 	list_add_tail(&priv->dmabufs_elm, &vdev->dmabufs);
 	dma_resv_unlock(priv->dmabuf->resv);
 	up_write(&vdev->memory_lock);
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index ec1660e0d6d2..6069a2db5dd6 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -47,6 +47,15 @@ struct vfio_pci_regops {
 	int	(*add_capability)(struct vfio_pci_core_device *vdev,
 				  struct vfio_pci_region *region,
 				  struct vfio_info_cap *caps);
+	/*
+	 * Optional, for a device-specific region exported as a dma-buf.
+	 * Called with memory_lock held for write; returns true while the
+	 * region is revoked, so a new dma-buf starts in the state that
+	 * vfio_pci_core_revoke_region() and vfio_pci_core_rearm_region()
+	 * maintain. Without it, the dma-buf starts armed.
+	 */
+	bool	(*revoked)(struct vfio_pci_core_device *vdev,
+			   struct vfio_pci_region *region);
 };
 
 struct vfio_pci_region {
-- 
2.53.0

[PATCH v6 29/31] vfio/cxl: Run the CXL reset at the vfio reset points

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A CXL Type-2 function must not take an FLR in place of CXL Reset: an FLR
leaves CXL.cache and CXL.mem state, the CXL DVSEC and the HDM contents
untouched (CXL r4.0 sec 9.5), so device memory and cache would carry one
user's data to the next. CXL Reset resets that state as well (sec 9.7).
The PCI core orders cxl_reset ahead of flr in pci_reset_fn_methods[],
but falls through to FLR when cxl_reset returns -ENOTTY.

vfio-pci-core routes its reset points through the provider reset op. In
vfio-cxl that op runs pci_cxl_reset_owned() with the HDM range claim.
The reset resets the function, clearing or randomizing volatile device
memory, and restores the HDM decoder and PCI config state, so it is a
complete replacement for pci_try_reset_function() on a CXL device. The
op runs with memory_lock and the PCI device lock held.

Take the HDM memory region out of use for the duration of the reset, so
a fault cannot insert a PFN into a decoder that is being torn down, and
put it back only once the reset has restored the decoder and the device
reports its memory active again. After CXL Reset with Memory Clear the
device can keep Memory_Active clear for seconds (about 3 s on the device
this was tested on), and a host access to the HDM range in that window
raised an SError. A reset whose memory does not come back within the
Memory_Active_Timeout fails like any other. A failed reset
returns its error and leaves the region out of use until a later CXL
Reset succeeds. Neither a decoder restore after low power nor a hot
reset reopens it in between, since device memory may still hold data the
reset was meant to clear. If a failed reset left CXL.cache or CXL.mem
disabled in the DVSEC Control register, which makes cxl_reset refuse the
device, put back the Control value taken at bind before the next reset.
A failed reset leaves Memory Space disabled; clear vfio-pci-core's
record of it to match. AER recovery does not reopen HDM after a failed
CXL Reset either.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 82 ++++++++++++++++++++++++++--
 1 file changed, 77 insertions(+), 5 deletions(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 518f0b093652..70794efd517b 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -38,6 +38,8 @@
  *	for Memory_Active before host access reopens after a reset or low power
  * @mem_region_index: vfio region index of the HDM memory region while open, else 0
  * @hdm_open_at_prepare: @hdm_valid as reset_prepare found it; under memory_lock
+ * @hdm_needs_reset: a CXL Reset failed, so only a CXL Reset that succeeds
+ *	reopens host access; under memory_lock
  */
 struct vfio_cxl_state {
 	struct cxl_dev_state cxlds;
@@ -52,6 +54,7 @@ struct vfio_cxl_state {
 	bool await_mem_active;
 	unsigned int mem_region_index;
 	bool hdm_open_at_prepare;
+	bool hdm_needs_reset;
 };
 
 static unsigned long vfio_cxl_mem_pgoff(struct vm_area_struct *vma,
@@ -827,17 +830,75 @@ static void vfio_cxl_sync_mem_enable(struct vfio_pci_core_device *vdev)
 }
 
 /*
- * vfio-cxl holds the HDM range, so the PCI core's cxl_reset method would
- * fail to request it. Run the same CXL Reset with that claim.
+ * cxl_reset refuses a device whose CXL DVSEC Control has Cache_Enable or
+ * Mem_Enable clear (-ENOTTY). A CXL Reset that failed after the device reset
+ * may leave Control at its reset value, so every later reset would refuse and
+ * host access would never reopen. Put back the Control value taken at bind,
+ * when pci_cxl_reset_capable() required both enables, before the reset.
+ */
+static void vfio_cxl_restore_dvsec_ctrl(struct vfio_pci_core_device *vdev)
+{
+	const u16 enables = PCI_DVSEC_CXL_CACHE_ENABLE | PCI_DVSEC_CXL_MEM_ENABLE;
+	const struct cxl_hdm_info *hdm = vdev->cxl->hdm;
+	struct pci_dev *pdev = vdev->pdev;
+	u16 ctrl;
+
+	if (!hdm->dvsec_ctrl_valid || (hdm->dvsec_ctrl & enables) != enables)
+		return;
+
+	if (pci_read_config_word(pdev, hdm->dvsec + PCI_DVSEC_CXL_CTRL, &ctrl) ||
+	    PCI_POSSIBLE_ERROR(ctrl) || (ctrl & enables) == enables)
+		return;
+
+	pci_warn(pdev, "vfio-cxl: restoring CXL DVSEC control %#06x (was %#06x)\n",
+		 hdm->dvsec_ctrl, ctrl);
+	pci_write_config_word(pdev, hdm->dvsec + PCI_DVSEC_CXL_CTRL,
+			      hdm->dvsec_ctrl);
+}
+
+/*
+ * Run CXL Reset in place of a PCI function reset. An FLR leaves CXL.mem
+ * state and HDM contents intact (CXL r4.0 sec 9.5), so the vfio reset points
+ * route here. The reset clears or randomizes volatile device memory and
+ * restores the HDM decoder. The caller holds memory_lock and the PCI device
+ * lock. vfio-cxl holds the HDM range, so the PCI core's cxl_reset method
+ * would fail to request it; run the same CXL Reset with that claim.
  */
 static int vfio_cxl_reset(struct vfio_pci_core_device *vdev)
 {
+	struct pci_dev *pdev = vdev->pdev;
+	int ret;
+
+	/* Host CPU access to the HDM range is unsafe until the decoder is back. */
+	vfio_cxl_revoke_hdm(vdev);
+
+	/* Only after a failure: see vfio_cxl_restore_dvsec_ctrl(). */
+	if (vdev->cxl->hdm_needs_reset)
+		vfio_cxl_restore_dvsec_ctrl(vdev);
+
 	/*
 	 * The reset restores PCI state from the saved copy. The caller provides
 	 * it: the try-reset helper saves the current state, and the close path
 	 * keeps the open-time state that vfio_pci_core_disable() loaded.
 	 */
-	return pci_cxl_reset_owned(vdev->pdev, vdev->cxl->hpa_res);
+	ret = pci_cxl_reset_owned(pdev, vdev->cxl->hpa_res);
+	if (!ret)
+		ret = vfio_cxl_await_mem_active(vdev);
+	if (ret) {
+		/*
+		 * Device memory may hold data the reset was meant to clear. Only
+		 * a later CXL Reset that succeeds reopens host access; a decoder
+		 * restore alone does not.
+		 */
+		vdev->cxl->hdm_needs_reset = true;
+		vfio_cxl_sync_mem_enable(vdev);
+		return ret;
+	}
+
+	vdev->cxl->hdm_needs_reset = false;
+	vfio_cxl_rearm_hdm(vdev);
+
+	return 0;
 }
 
 /*
@@ -881,6 +942,10 @@ static void vfio_cxl_power_up(struct vfio_pci_core_device *vdev)
 
 	lockdep_assert_held_write(&vdev->memory_lock);
 
+	/* After a failed CXL Reset, only a CXL Reset reopens host access. */
+	if (cxl->hdm_needs_reset)
+		return;
+
 	/*
 	 * A D3 request that failed, or a quirk that kept the device in D0,
 	 * lost nothing: reopen without decommitting and recommitting.
@@ -984,6 +1049,10 @@ static int vfio_cxl_reset_done(struct vfio_pci_core_device *vdev)
 		return 0;
 	}
 
+	/* After a failed CXL Reset, only a CXL Reset reopens host access. */
+	if (cxl->hdm_needs_reset)
+		return 0;
+
 	ret = vfio_cxl_await_mem_active(vdev);
 	if (ret)
 		return ret;
@@ -1007,8 +1076,11 @@ static void vfio_cxl_error_resume(struct vfio_pci_core_device *vdev)
 	if (pdev->current_state != PCI_D0)
 		return;
 
-	/* The region was closed before the error; keep it that way. */
-	if (!vdev->cxl->hdm_open_at_prepare)
+	/*
+	 * The region was closed before the error, or a CXL Reset failed while
+	 * recovery ran; keep it closed until a CXL Reset succeeds.
+	 */
+	if (!vdev->cxl->hdm_open_at_prepare || vdev->cxl->hdm_needs_reset)
 		return;
 
 	if (!vfio_cxl_hdm_intact(vdev)) {
-- 
2.53.0

[PATCH v6 30/31] Documentation: vfio-pci: Document CXL Type-2 device passthrough

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Describe the vfio-cxl provider module:
- Address model,
- Two regions,
- Live decoder region and the guest commit,
- DVSEC virtualization,
- The dma-buf export for iommufd stage-2 mapping,
- The reset contract, including the hot reset,
- Low power

so the supported topology and the kernel/VMM split are on record.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 Documentation/driver-api/index.rst        |   1 +
 Documentation/driver-api/vfio-pci-cxl.rst | 312 ++++++++++++++++++++++
 MAINTAINERS                               |   1 +
 3 files changed, 314 insertions(+)
 create mode 100644 Documentation/driver-api/vfio-pci-cxl.rst

diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst
index 6601a258690f..733ea1b64c72 100644
--- a/Documentation/driver-api/index.rst
+++ b/Documentation/driver-api/index.rst
@@ -46,6 +46,7 @@ of interest to most developers working on device drivers.
    uio-howto
    vfio-mediated-device
    vfio
+   vfio-pci-cxl
    vfio-pci-device-specific-driver-acceptance
 
 Bus-level documentation
diff --git a/Documentation/driver-api/vfio-pci-cxl.rst b/Documentation/driver-api/vfio-pci-cxl.rst
new file mode 100644
index 000000000000..fc47a8015f1f
--- /dev/null
+++ b/Documentation/driver-api/vfio-pci-cxl.rst
@@ -0,0 +1,312 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=======================================
+VFIO-PCI: CXL Type-2 device passthrough
+=======================================
+
+Overview
+--------
+
+A CXL Type-2 device is an accelerator (for example a GPU) that exposes
+host-managed device memory through an HDM decoder. vfio-pci alone passes
+the HDM decoder registers through as part of a BAR, only guards the CXL
+Device DVSEC against writes that outlive the user, and does not place the
+device memory at a guest-chosen address.
+
+The optional ``vfio-cxl`` module traps the decoder registers and exposes
+the device memory as a region that a virtual machine monitor (VMM) can
+place in guest memory. It is a provider for vfio-pci-core, not a separate
+PCI driver. vfio-pci-core keeps generic hooks and the CXL DVSEC
+virtualization, calls no CXL core function, and loads ``vfio-cxl`` when
+it binds a CXL device.
+
+Address model
+-------------
+
+The HDM memory is a coherent range of host physical addresses (HPA).
+Firmware commits the endpoint decoder for that range before the guest sees
+the device. vfio-cxl takes the range from the committed decoder and holds it
+for the bind lifetime. vfio-cxl does not know where the memory appears in
+guest physical address space (GPA).
+
+The VMM picks the GPA. It builds the guest's CXL Fixed Memory Window
+(CFMWS), so it decides the guest physical window the device can occupy. The
+guest then programs a virtual endpoint HDM decoder inside that window (see
+"Virtual decoders"). It never programs the physical decoder.
+
+There are two ways to back a guest decoder with device memory. With fixed
+placement, each guest address maps to one host address at a fixed offset.
+A host window at HPA [0x10000, 0x20000] presented to the guest at GPA
+[0x50000, 0x60000] resolves a guest decoder at GPA [0x58000, 0x60000] to
+HPA [0x18000, 0x20000]::
+
+  +-----------+------------------------+----------------------------------+
+  | host      | HPA [0x10000, 0x20000] | decoder committed by firmware    |
+  | CFMWS     | GPA [0x50000, 0x60000] | built by the VMM                 |
+  | vdecoder0 | GPA [0x58000, 0x60000] | backed by HPA [0x18000, 0x20000] |
+  +-----------+------------------------+----------------------------------+
+
+Without fixed placement, the host kernel would back each guest page with
+any free page of device memory when the guest first touches it. That suits
+memory whose position does not matter to the device, for example compressed
+memory::
+
+  +-----------+------------------------+---------------------+
+  | CFMWS     | GPA [0x50000, 0x60000] | built by the VMM    |
+  | vdecoder0 | GPA [0x58000, 0x60000] | no fixed host range |
+  +-----------+------------------------+---------------------+
+
+vfio-cxl implements fixed placement only. Firmware commits the physical
+decoder and vfio-cxl never moves it, so the device memory sits at one host
+range, and the VMM maps the whole range at one guest offset. Placement
+without a fixed offset is not implemented.
+
+Driver model
+------------
+
+vfio-pci binds the device. During bind,
+vfio-pci-core detects a CXL device (``pcie_is_cxl()``), loads ``vfio-cxl``
+with ``request_module()``, and calls the registered ``struct
+vfio_cxl_ops``. The module reference is pinned for the bind lifetime so
+``vfio-cxl`` cannot unload while a device is bound.
+
+At bind the provider creates the CXL memory device and takes ownership of
+the whole component-register BAR. A Type-2 function has no mailbox, so the
+provider marks the media ready directly. A non-CXL device, or a CXL device
+whose CXL setup fails, falls back to the ordinary vfio-pci paths. The
+failure is not fatal to the bind.
+
+A CXL Type-2 device bound without the provider (``vfio-cxl`` not built or
+not loadable, its setup failed, or ``disable_cxl`` is set) gets none of the
+protections described below. The HDM decoder registers in the component
+register BAR are directly accessible to the user, and a function reset uses
+the PCI core reset methods, which can fall back to a Function Level Reset
+(FLR). An FLR is not a CXL Reset and does not clear device memory. vfio-pci
+still guards the CXL DVSEC of such a device: writes to the Lock and Range
+registers are dropped, and Initiate CXL Reset runs as a vfio reset instead
+of reaching the device.
+
+The provider also requires that the host can reset the device with CXL Reset
+with Memory Clear, including the CPU cache invalidation for the HDM range
+that the reset performs before and after the reset. On arm64 that
+invalidation needs a registered cache maintenance driver, for example the
+SMCCC cache clean/invalidate provider with matching firmware. Without it the
+device is bound without the CXL extensions, so no guest receives device
+memory that the host could not clear on the next reset.
+
+Regions
+-------
+
+``vfio-cxl`` adds two regions under the PCI vendor-type region
+``VFIO_REGION_TYPE_PCI_VENDOR_TYPE`` for the CXL vendor (0x1e98):
+
+``VFIO_REGION_SUBTYPE_CXL_MEM``
+    The HDM memory region, backed by the fixed host physical range. It can
+    be mapped with mmap. The fault handler inserts the host PFNs, including
+    PMD-sized mappings when the mapping is aligned, but only while a host
+    CPU access to the range is safe: the committed decoder is in place and
+    the media is ready. Otherwise the fault takes ``SIGBUS``. CXL.mem
+    decode follows the committed decoder, not the PCI Memory Space Enable
+    bit, so this region does not follow the Command register. It is taken
+    out of use across reset, bus reset and low power instead: vfio-cxl zaps
+    its mmaps and revokes its dma-bufs, and a fault takes ``SIGBUS``. The
+    range has no struct page and is registered with the memory-failure
+    machinery, so a memory error is contained to a ``SIGBUS``. The VMM maps
+    this region into guest memory at a fixed guest offset (see
+    "Address model"), and can also export it as a dma-buf (see "DMA and
+    iommufd").
+
+``VFIO_REGION_SUBTYPE_CXL_COMP_REGS``
+    The HDM decoder registers. Access is read/write only (no mmap) and
+    must be dword aligned. A misaligned or out-of-range access returns
+    ``-EINVAL``. Reads are served live from the committed decoder, and fail
+    with ``-EIO`` while Memory Space is disabled, as BAR reads do. The
+    decoder was committed before bind, and firmware may also have locked
+    it, so the register block is read-only to the user and writes are
+    dropped. The region carries a ``VFIO_REGION_INFO_CAP_CXL_COMP_REGS``
+    capability that reports the component BAR and the offset of the decoder
+    block within it, so the VMM can place the trapped window where the guest
+    expects it.
+
+The decoder register range is also excluded from the direct component-BAR
+mmap and from BAR reads and writes through the device fd. A user write
+there could uncommit or move a decoder the host relies on, so reads return
+ones and writes are dropped. The rest of the component BAR is a normal
+vfio-pci BAR.
+
+Virtual decoders
+----------------
+
+The guest sees virtual endpoint HDM decoders. The VMM emulates them, and
+reads the physical decoder through the decoder-register region::
+
+  +-------------------+---------------------------------+-------------------+
+  | guest             | VMM                             | vfio-cxl          |
+  +-------------------+---------------------------------+-------------------+
+  | writes base, size | keeps them (a GPA)              |                   |
+  +-------------------+---------------------------------+-------------------+
+  | writes COMMIT     | reads the physical decoder      | live read returns |
+  |                   | from the region                 | COMMITTED         |
+  |                   | maps the HDM region into        |                   |
+  |                   | guest memory, reports COMMITTED |                   |
+  +-------------------+---------------------------------+-------------------+
+  | reads base        | returns the GPA                 |                   |
+  +-------------------+---------------------------------+-------------------+
+
+Writes to the region are dropped, so neither the guest nor the VMM can
+decommit or move the physical decoder. vfio-cxl commits it again only when
+a reset or low power has cleared it, with the settings taken at bind. Reads
+of the region come from the hardware.
+
+CXL Device DVSEC
+----------------
+
+vfio-pci-core virtualizes the CXL Device DVSEC body through its config-space
+permission hooks. Reads and writes inside the DVSEC body use a per-open
+shadow. A guest write stays in the shadow and does not reach hardware.
+Accesses outside the DVSEC body go to the device as usual.
+
+The self-clearing Control2 doorbells (Initiate CXL Reset and Initiate
+Cache Write-Back and Invalidate) are never forwarded to hardware. Initiate
+Cache Write-Back and Invalidate completes in the shadow at once. The host
+writes back and invalidates the device cache as part of CXL Reset.
+Initiate CXL Reset runs the real reset (see "Reset") and reports the result
+in the shadow Status2: Reset Complete on success, Reset Error on failure.
+
+DMA and iommufd
+---------------
+
+Some accelerators, for example a GPU, read and write their own HDM memory
+with DMA translated through Address Translation Services (ATS), using the
+same guest physical addresses as the vCPUs. For such a device the HDM range
+must be in the guest IOAS: the iommufd I/O address space that the VMM builds
+for the device, which translates the guest physical addresses the device uses
+to host physical addresses. A device that does not send DMA for its own
+memory through the IOMMU does not need the range in the IOAS.
+
+vfio-cxl claims the whole HDM range at bind with
+``request_mem_region_exclusive()``, and the host never onlines it as system
+RAM, so the range has no struct page. ``IOMMU_IOAS_MAP`` takes a VMM
+virtual address and pins the pages behind it, so it cannot map this range.
+Instead, ``VFIO_DEVICE_FEATURE_DMA_BUF`` on the HDM memory region returns a
+dma-buf fd, and ``IOMMU_IOAS_MAP_FILE`` maps that fd into the IOAS without
+a virtual address or a page pin.
+
+Two second-stage translations point at the HDM range. KVM's stage-2 maps
+guest physical addresses to it for the vCPUs, through the VMM's mmap of the
+HDM memory region. The IOAS maps the same guest physical addresses for the
+device's DMA, through the dma-buf. Across a reset or a low power transition
+the decoder that routes the range to the device goes away, so vfio-cxl
+removes both translations first: it zaps the mmap and revokes the dma-buf.
+Otherwise a CPU access or a DMA could target a range that nothing decodes.
+
+When the region is usable again, the CPU side comes back through the
+region's fault handler. The device side does not. iommufd unmaps the IOAS
+range when the dma-buf is revoked and does not map it again, so the VMM
+has to unmap that IOVA range and map the same dma-buf again with
+``IOMMU_IOAS_MAP_FILE``.
+
+Reset
+-----
+
+An FLR does not reset CXL.cache or CXL.mem state and does not clear device
+memory (CXL r4.0 sec 9.5), so it cannot isolate one user of the device from
+the next. FLR is therefore virtualized rather than passed to the device.
+vfio-cxl resets the device with CXL Reset through ``pci_cxl_reset_owned()``.
+The host invalidates the CPU caches for the HDM range, resets the function
+with Memory Clear, restores the HDM decoder and the PCI config state, and
+invalidates the CPU caches again. A failure is returned to the caller. The
+reset never falls back to FLR.
+
+A guest reset request, either Initiate CXL Reset in the DVSEC or a guest FLR
+through config space, is trapped and not forwarded. The real reset runs at
+the vfio reset points: the reset ioctl, the virtualized FLR and the Initiate
+CXL Reset doorbell. vfio-cxl zaps the HDM mapping, revokes the dma-buf and
+runs CXL Reset with Memory Clear, which clears or randomizes volatile device
+memory (CXL r4.0 sec 9.7). vfio-cxl then restores the decoder. If the reset
+fails, the ioctl returns the error and the HDM region stays out of use until
+a later CXL Reset through a vfio reset point succeeds. A hot reset or a
+return from low power in between does not reopen it.
+
+A CXL port masks Secondary Bus Reset by default (CXL r4.0 sec 8.1.5.2), and
+a masked SBR does not reach the endpoint. The PCI core refuses a bus or slot
+reset below such a port, so ``VFIO_DEVICE_GET_PCI_HOT_RESET_INFO`` and
+``VFIO_DEVICE_PCI_HOT_RESET`` fail for the device, and a VMM uses
+``VFIO_DEVICE_RESET`` instead. If the port has SBR unmasked, the hot reset
+decommits the decoder. vfio-cxl takes the HDM region out of use before the
+reset and restores the decoder afterwards with
+``cxl_restore_state_after_pci_reset()``, the helper the PCI core's
+``cxl_bus`` reset method uses, unless the decoder is still committed. If the
+restore fails, the hot reset ioctl returns the error. A reset that the host
+starts outside vfio, for example through the ``reset`` attribute in sysfs,
+is handled the same way. Its result is only logged, and on failure the HDM
+region stays out of use until a later reset succeeds.
+
+PCIe error recovery (AER and DPC) can reset the link without those
+callbacks. vfio-cxl takes the HDM region out of use when the error is
+reported. It puts the region back when recovery resumes the device only if
+the region was in use before the error, no CXL Reset failed meanwhile, and
+the decoder is still committed. Otherwise the region stays out of use until
+the user resets the device.
+
+Device memory can stay unavailable for a while after the decoder is back.
+The device clears Memory_Active in DVSEC CXL Range 1 Size Low until the
+memory is usable again (CXL r4.0 sec 8.1.3.8.2). At bind, vfio-cxl waits up
+to the Memory_Active_Timeout the device reports for Memory_Active, since a
+reset just before the bind also clears it. On a device whose memory then
+reads active, vfio-cxl keeps the HDM region out of use after a CXL Reset, a
+hot reset, AER or DPC recovery and a return from low power until
+Memory_Active is set, waiting up to the same timeout. A wait that times out
+leaves the region out of use, as a failed reset does. A device whose memory
+stays inactive at bind relies on its driver to bring it up and is not
+waited for.
+
+After a reset vfio-cxl restores the firmware-committed decoder from the
+settings taken at bind, so the geometry the VMM reads back is unchanged.
+CPU access comes back through the region's fault handler. Device DMA to
+HDM needs the VMM to map the dma-buf into the IOAS again (see "DMA and
+iommufd").
+
+Low power
+---------
+
+The device may enter D3hot or runtime low power while it is assigned. Low
+power loses the committed HDM decoder, so vfio-cxl takes the HDM region out
+of use before it enters low power. It restores the decoder configuration
+when the device returns to D0, after vfio-pci-core has restored its config
+space. If a low power request left the device in D0 and the decoder is
+still committed, the region is put back without a restore. vfio-cxl does
+not save or restore device memory across low power.
+
+UAPI
+----
+
+``VFIO_DEVICE_FLAGS_CXL``
+    Set in ``VFIO_DEVICE_GET_INFO`` flags for a CXL Type-2 device.
+
+``VFIO_REGION_TYPE_PCI_VENDOR_TYPE | 0x1e98``
+    With ``VFIO_REGION_SUBTYPE_CXL_MEM`` or
+    ``VFIO_REGION_SUBTYPE_CXL_COMP_REGS``. Reported through the region-info
+    ``VFIO_REGION_INFO_CAP_TYPE`` capability. Userspace finds each region by
+    scanning for the type and subtype.
+
+``VFIO_REGION_INFO_CAP_CXL_COMP_REGS``
+    On the component-register region, reports the component BAR index and
+    the decoder-block offset within it.
+
+``VFIO_DEVICE_FEATURE_DMA_BUF``
+    On the HDM memory region, returns a dma-buf fd for
+    ``IOMMU_IOAS_MAP_FILE``.
+
+The HDM decoder register layout is available to a VMM without a private
+kernel header via ``uapi/cxl/cxl_regs.h``.
+
+Scope
+-----
+
+This support covers a single, non-interleaved endpoint decoder on a
+directly attached device. Multi-decoder devices, interleave, and
+switch-attached topologies are not supported.
+
+A selftest, ``tools/testing/selftests/vfio/vfio_cxl_type2_test.c``,
+exercises the interfaces above on a bound device.
diff --git a/MAINTAINERS b/MAINTAINERS
index 8ac0a91ce647..9fc94cdf021c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -28641,6 +28641,7 @@ M:	Manish Honap <mhonap@nvidia.com>
 L:	kvm@vger.kernel.org
 L:	linux-cxl@vger.kernel.org
 S:	Supported
+F:	Documentation/driver-api/vfio-pci-cxl.rst
 F:	drivers/vfio/pci/cxl/
 
 VFIO DRIVER
-- 
2.53.0

[PATCH v6 31/31] selftests/vfio: Add CXL Type-2 passthrough tests

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Exercise the vfio-cxl contract on a bound Type-2 device: the CXL
device-info flag and the two vendor-type regions, the HDM memory mmap
(including a 2 MB huge fault), the read-only trapped decoder region
against the live host-committed decoder, and the guest's DVSEC reset
doorbell and manual HDM discovery.

Check that the decoder register region reads fail with EIO while Memory
Space is disabled, and that the decoder reads back committed and HDM
access works after low power entry and exit.

Map the HDM memory the way a VMM does: export it as a dma-buf with
VFIO_DEVICE_FEATURE_DMA_BUF and map the fd into the IOAS with
IOMMU_IOAS_MAP_FILE. The struct-page-less range cannot be pinned through
a userspace VA, so the by-fd path is the one that works. Skip when the
kernel has no CXL dma-buf exporter, so the test becomes a real pass once
that support is present. Check that a dma-buf exported with Memory Space
disabled maps once it is enabled, that read and write on the HDM region
span several pages and may target a mapping of the region itself, and
that a multi-dword read of the decoder registers matches single-dword
reads.

Also check that close restores the config state saved at open, and that
a D3hot to D0 cycle restores the decoder, also when the device stays in
D0. The sparse mmap test accepts zero areas when the page-rounded
decoder hole covers the whole BAR. The multi-chunk read and write test
is sized in pages of the running kernel.

Add a partial-mmap path to the vfio selftest library so the trapped
component BAR maps its sparse ranges, and an __iommu_map_file() helper for
the dma-buf mapping.

Assisted-by: LLM
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 MAINTAINERS                                   |    1 +
 tools/testing/selftests/vfio/Makefile         |    1 +
 .../vfio/lib/include/libvfio/iommu.h          |    3 +
 tools/testing/selftests/vfio/lib/iommu.c      |   29 +
 .../selftests/vfio/lib/vfio_pci_device.c      |   57 +-
 .../selftests/vfio/vfio_cxl_type2_test.c      | 1095 +++++++++++++++++
 6 files changed, 1183 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/vfio/vfio_cxl_type2_test.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 9fc94cdf021c..2ab5934aa3e6 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -28643,6 +28643,7 @@ L:	linux-cxl@vger.kernel.org
 S:	Supported
 F:	Documentation/driver-api/vfio-pci-cxl.rst
 F:	drivers/vfio/pci/cxl/
+F:	tools/testing/selftests/vfio/vfio_cxl_type2_test.c
 
 VFIO DRIVER
 M:	Alex Williamson <alex@shazbot.org>
diff --git a/tools/testing/selftests/vfio/Makefile b/tools/testing/selftests/vfio/Makefile
index 2c32c48db509..08f88e88cb4d 100644
--- a/tools/testing/selftests/vfio/Makefile
+++ b/tools/testing/selftests/vfio/Makefile
@@ -13,6 +13,7 @@ TEST_GEN_PROGS += vfio_pci_device_test
 TEST_GEN_PROGS += vfio_pci_device_init_perf_test
 TEST_GEN_PROGS += vfio_pci_driver_test
 TEST_GEN_PROGS += vfio_pci_sriov_uapi_test
+TEST_GEN_PROGS += vfio_cxl_type2_test
 
 TEST_FILES += scripts/cleanup.sh
 TEST_FILES += scripts/lib.sh
diff --git a/tools/testing/selftests/vfio/lib/include/libvfio/iommu.h b/tools/testing/selftests/vfio/lib/include/libvfio/iommu.h
index e9a3386a4719..77040fdb8848 100644
--- a/tools/testing/selftests/vfio/lib/include/libvfio/iommu.h
+++ b/tools/testing/selftests/vfio/lib/include/libvfio/iommu.h
@@ -42,6 +42,9 @@ static inline void iommu_map(struct iommu *iommu, struct dma_region *region)
 	VFIO_ASSERT_EQ(__iommu_map(iommu, region), 0);
 }
 
+int __iommu_map_file(struct iommu *iommu, int fd, u64 start, u64 length,
+		     iova_t iova);
+
 int __iommu_unmap(struct iommu *iommu, struct dma_region *region, u64 *unmapped);
 
 static inline void iommu_unmap(struct iommu *iommu, struct dma_region *region)
diff --git a/tools/testing/selftests/vfio/lib/iommu.c b/tools/testing/selftests/vfio/lib/iommu.c
index b6f3c5c84e01..9109cedd734c 100644
--- a/tools/testing/selftests/vfio/lib/iommu.c
+++ b/tools/testing/selftests/vfio/lib/iommu.c
@@ -149,6 +149,35 @@ int __iommu_map(struct iommu *iommu, struct dma_region *region)
 	return 0;
 }
 
+/*
+ * Map a range of a file (a memfd or a supported dma-buf, such as a VFIO PCI
+ * dma-buf from VFIO_DEVICE_FEATURE_DMA_BUF) into the IOAS by fd. This is an
+ * iommufd-only ioctl; the legacy VFIO container has no equivalent.
+ */
+int __iommu_map_file(struct iommu *iommu, int fd, u64 start, u64 length,
+		     iova_t iova)
+{
+	struct iommu_ioas_map_file args = {
+		.size = sizeof(args),
+		.flags = IOMMU_IOAS_MAP_READABLE |
+			 IOMMU_IOAS_MAP_WRITEABLE |
+			 IOMMU_IOAS_MAP_FIXED_IOVA,
+		.ioas_id = iommu->ioas_id,
+		.fd = fd,
+		.start = start,
+		.length = length,
+		.iova = iova,
+	};
+
+	if (!iommu->iommufd)
+		return -EINVAL;
+
+	if (ioctl(iommu->iommufd, IOMMU_IOAS_MAP_FILE, &args))
+		return -errno;
+
+	return 0;
+}
+
 static int __vfio_iommu_unmap(int fd, u64 iova, u64 size, u32 flags, u64 *unmapped)
 {
 	struct vfio_iommu_type1_dma_unmap args = {
diff --git a/tools/testing/selftests/vfio/lib/vfio_pci_device.c b/tools/testing/selftests/vfio/lib/vfio_pci_device.c
index 4063a0e2b3df..5f5ec3a70bd8 100644
--- a/tools/testing/selftests/vfio/lib/vfio_pci_device.c
+++ b/tools/testing/selftests/vfio/lib/vfio_pci_device.c
@@ -187,9 +187,31 @@ static void vfio_pci_region_get(struct vfio_pci_device *device, int index,
 	ioctl_assert(device->fd, VFIO_DEVICE_GET_REGION_INFO, info);
 }
 
+/* Return the sparse-mmap capability in @info, or NULL if the region has none. */
+static struct vfio_region_info_cap_sparse_mmap *
+vfio_pci_sparse_mmap_cap(struct vfio_region_info *info)
+{
+	struct vfio_info_cap_header *hdr;
+	u32 offset;
+
+	if (!(info->flags & VFIO_REGION_INFO_FLAG_CAPS))
+		return NULL;
+
+	for (offset = info->cap_offset; offset; offset = hdr->next) {
+		hdr = (void *)info + offset;
+		if (hdr->id == VFIO_REGION_INFO_CAP_SPARSE_MMAP)
+			return (struct vfio_region_info_cap_sparse_mmap *)hdr;
+	}
+
+	return NULL;
+}
+
 static void vfio_pci_bar_map(struct vfio_pci_device *device, int index)
 {
 	struct vfio_pci_bar *bar = &device->bars[index];
+	struct vfio_region_info_cap_sparse_mmap *sparse;
+	u8 infobuf[1024] = {};
+	struct vfio_region_info *info = (void *)infobuf;
 	size_t align, size;
 	int prot = 0;
 	void *vaddr;
@@ -217,9 +239,38 @@ static void vfio_pci_bar_map(struct vfio_pci_device *device, int index)
 	align = min_t(size_t, size, SZ_1G);
 
 	vaddr = mmap_reserve(size, align, 0);
-	bar->vaddr = mmap(vaddr, size, prot, MAP_SHARED | MAP_FIXED,
-			  device->fd, bar->info.offset);
-	VFIO_ASSERT_NE(bar->vaddr, MAP_FAILED);
+
+	/*
+	 * A BAR that is only partially mmappable, such as a CXL Type-2 component
+	 * BAR with the HDM decoder block trapped, advertises the mmappable
+	 * ranges through a sparse-mmap capability. Map each area within the
+	 * reservation and leave the excluded ranges unmapped; mapping the whole
+	 * BAR would be rejected.
+	 */
+	info->argsz = sizeof(infobuf);
+	info->index = index;
+	ioctl_assert(device->fd, VFIO_DEVICE_GET_REGION_INFO, info);
+	sparse = vfio_pci_sparse_mmap_cap(info);
+	if (sparse) {
+		u32 i;
+
+		bar->vaddr = vaddr;
+		for (i = 0; i < sparse->nr_areas; i++) {
+			void *p;
+
+			if (!sparse->areas[i].size)
+				continue;
+			p = mmap(vaddr + sparse->areas[i].offset,
+				 sparse->areas[i].size, prot,
+				 MAP_SHARED | MAP_FIXED, device->fd,
+				 bar->info.offset + sparse->areas[i].offset);
+			VFIO_ASSERT_NE(p, MAP_FAILED);
+		}
+	} else {
+		bar->vaddr = mmap(vaddr, size, prot, MAP_SHARED | MAP_FIXED,
+				  device->fd, bar->info.offset);
+		VFIO_ASSERT_NE(bar->vaddr, MAP_FAILED);
+	}
 
 	madvise(bar->vaddr, size, MADV_HUGEPAGE);
 }
diff --git a/tools/testing/selftests/vfio/vfio_cxl_type2_test.c b/tools/testing/selftests/vfio/vfio_cxl_type2_test.c
new file mode 100644
index 000000000000..990739b3d286
--- /dev/null
+++ b/tools/testing/selftests/vfio/vfio_cxl_type2_test.c
@@ -0,0 +1,1095 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * vfio_cxl_type2_test - corner-case tests for the vfio-cxl kernel contract.
+ *
+ * Exercises the user-visible surface the vfio-cxl module adds to a CXL Type-2
+ * device: the two VFIO regions (HDM memory and the trapped HDM decoder block),
+ * the component-register geometry capability, the dma-buf export of the HDM
+ * memory, and the guest decoder view.
+ *
+ * The host commits and locks the physical decoder before the guest sees the
+ * device, so the trapped block is served by a live read and a guest write is
+ * absorbed.
+ *
+ * Usage: ./vfio_cxl_type2_test <BDF>  (or export VFIO_SELFTESTS_BDF=<BDF>).
+ * The device must be bound to vfio-pci with the vfio-cxl module available.
+ *
+ * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.
+ */
+
+#include <errno.h>
+#include <fcntl.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+
+#include <linux/pci_regs.h>
+#include <linux/sizes.h>
+#include <linux/vfio.h>
+
+#include <cxl/cxl_regs.h>
+
+#include <libvfio.h>
+
+#include "kselftest_harness.h"
+
+#define PCI_DVSEC_VENDOR_ID_CXL		0x1e98
+#define PCI_DVSEC_ID_CXL_DEVICE		0x0000
+
+/* CXL r4.0 sec 8.1.9.1: Register Block Identifier of the component regs */
+#define CXL_REGLOC_RBI_COMPONENT	1
+
+/* Register Locator DVSEC block-1 field masks. */
+#define REG_LOCATOR_BIR_MASK		0x00000007
+#define REG_LOCATOR_BLOCK_ID_MASK	0x0000ff00
+#define REG_LOCATOR_BLOCK_OFF_LOW_MASK	0xffff0000
+
+/* vfio-pci region-offset packing is kernel-internal, not UAPI; define locally. */
+#ifndef VFIO_PCI_OFFSET_SHIFT
+#define VFIO_PCI_OFFSET_SHIFT		40
+#endif
+#ifndef VFIO_PCI_INDEX_TO_OFFSET
+#define VFIO_PCI_INDEX_TO_OFFSET(i)	((uint64_t)(i) << VFIO_PCI_OFFSET_SHIFT)
+#endif
+
+static const char *device_bdf;
+
+/* Locate a region-info capability by id inside a GET_REGION_INFO buffer. */
+static const struct vfio_info_cap_header *
+find_region_cap(const void *buf, size_t bufsz, uint16_t id)
+{
+	const struct vfio_region_info *ri = buf;
+	const struct vfio_info_cap_header *cap;
+	size_t off = ri->cap_offset;
+
+	while (off && off + sizeof(*cap) <= bufsz) {
+		cap = (const void *)((const char *)buf + off);
+		if (cap->id == id)
+			return cap;
+		off = cap->next;
+	}
+	return NULL;
+}
+
+/* Find a CXL region by subtype; returns the region index or -1, @buf left holding its info. */
+static int find_cxl_region(int fd, uint32_t nregions, uint32_t subtype,
+			   void *buf, size_t bufsz)
+{
+	uint32_t i;
+
+	for (i = 0; i < nregions; i++) {
+		struct vfio_region_info *ri = buf;
+		const struct vfio_region_info_cap_type *t;
+		const struct vfio_info_cap_header *hdr;
+
+		memset(buf, 0, bufsz);
+		ri->argsz = bufsz;
+		ri->index = i;
+		if (ioctl(fd, VFIO_DEVICE_GET_REGION_INFO, ri))
+			continue;
+		if (!(ri->flags & VFIO_REGION_INFO_FLAG_CAPS))
+			continue;
+
+		hdr = find_region_cap(buf, bufsz, VFIO_REGION_INFO_CAP_TYPE);
+		if (!hdr)
+			continue;
+		t = (const void *)hdr;
+		if (t->type == (VFIO_REGION_TYPE_PCI_VENDOR_TYPE |
+				PCI_DVSEC_VENDOR_ID_CXL) &&
+		    t->subtype == subtype)
+			return i;
+	}
+	return -1;
+}
+
+/* Walk the PCI extended capability list for the CXL Device DVSEC. */
+static uint16_t find_cxl_dvsec(struct vfio_pci_device *dev)
+{
+	uint16_t pos = PCI_CFG_SPACE_SIZE;
+	int iter = 0;
+
+	while (pos && iter++ < 64) {
+		uint32_t hdr = vfio_pci_config_readl(dev, pos);
+		uint16_t cap_id = hdr & 0xffff;
+		uint16_t next = (hdr >> 20) & 0xffc;
+		uint32_t h1, h2;
+
+		if (cap_id == PCI_EXT_CAP_ID_DVSEC) {
+			h1 = vfio_pci_config_readl(dev, pos + 4);
+			h2 = vfio_pci_config_readl(dev, pos + 8);
+			if ((h1 & 0xffff) == PCI_DVSEC_VENDOR_ID_CXL &&
+			    (h2 & 0xffff) == PCI_DVSEC_ID_CXL_DEVICE)
+				return pos;
+		}
+		pos = next;
+	}
+	return 0;
+}
+
+/* Read config space from sysfs, which is the hardware, not the vfio view. */
+static int sysfs_config_read(uint32_t off, void *buf, size_t len)
+{
+	char path[128];
+	ssize_t n;
+	int fd;
+
+	snprintf(path, sizeof(path), "/sys/bus/pci/devices/%s/config",
+		 device_bdf);
+	fd = open(path, O_RDONLY);
+	if (fd < 0)
+		return -errno;
+	n = pread(fd, buf, len, off);
+	close(fd);
+	return n == (ssize_t)len ? 0 : -EIO;
+}
+
+FIXTURE(vfio_cxl) {
+	struct iommu *iommu;
+	struct vfio_pci_device *dev;
+
+	int mem_idx;
+	uint64_t mem_size;
+	uint32_t mem_flags;
+	int comp_idx;
+	uint64_t comp_size;
+	uint32_t comp_bar;
+	uint64_t comp_offset;	/* HDM block offset within comp_bar */
+	uint64_t comp_off;	/* mmap/rw base offset of the comp region */
+	uint16_t dvsec;
+};
+
+FIXTURE_SETUP(vfio_cxl)
+{
+	uint8_t infobuf[512] = {};
+	struct vfio_device_info *info = (void *)infobuf;
+	const struct vfio_region_info_cap_cxl_comp_regs *geo;
+	const struct vfio_info_cap_header *hdr;
+	uint8_t rbuf[1024];
+	uint16_t cmd;
+
+	self->iommu = iommu_init(default_iommu_mode);
+	self->dev = vfio_pci_device_init(device_bdf, self->iommu);
+
+	info->argsz = sizeof(infobuf);
+	ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_INFO, info));
+
+	if (!(info->flags & VFIO_DEVICE_FLAGS_CXL))
+		SKIP(return, "not a CXL Type-2 device");
+
+	self->mem_idx = find_cxl_region(self->dev->fd, info->num_regions,
+					VFIO_REGION_SUBTYPE_CXL_MEM,
+					rbuf, sizeof(rbuf));
+	ASSERT_GE(self->mem_idx, 0);
+	self->mem_size = ((struct vfio_region_info *)rbuf)->size;
+	self->mem_flags = ((struct vfio_region_info *)rbuf)->flags;
+
+	self->comp_idx = find_cxl_region(self->dev->fd, info->num_regions,
+					 VFIO_REGION_SUBTYPE_CXL_COMP_REGS,
+					 rbuf, sizeof(rbuf));
+	ASSERT_GE(self->comp_idx, 0);
+	self->comp_size = ((struct vfio_region_info *)rbuf)->size;
+
+	/* The geometry cap rides on the component-register region. */
+	hdr = find_region_cap(rbuf, sizeof(rbuf),
+			      VFIO_REGION_INFO_CAP_CXL_COMP_REGS);
+	ASSERT_NE(NULL, hdr);
+	geo = (const void *)hdr;
+	self->comp_bar = geo->bar;
+	self->comp_offset = geo->offset;
+
+	self->comp_off = VFIO_PCI_INDEX_TO_OFFSET(self->comp_idx);
+	self->dvsec = find_cxl_dvsec(self->dev);
+
+	/* Enable PCI Memory-Space so the HDM mmap tests can touch the mapping. */
+	cmd = vfio_pci_config_readw(self->dev, PCI_COMMAND);
+	vfio_pci_config_writew(self->dev, PCI_COMMAND,
+			       cmd | PCI_COMMAND_MEMORY);
+}
+
+FIXTURE_TEARDOWN(vfio_cxl)
+{
+	vfio_pci_device_cleanup(self->dev);
+	iommu_cleanup(self->iommu);
+}
+
+/* GET_INFO advertises the flag and both CXL regions with a sane geometry cap. */
+TEST_F(vfio_cxl, device_is_cxl)
+{
+	ASSERT_NE(self->mem_idx, self->comp_idx);
+	ASSERT_GT(self->mem_size, 0);
+	ASSERT_GT(self->comp_size, 0);
+	ASSERT_LT(self->comp_bar, PCI_STD_NUM_BARS);
+	/* The HDM memory must advertise mmap; a VMM needs it for stage-2. */
+	ASSERT_NE(0, self->mem_flags & VFIO_REGION_INFO_FLAG_MMAP);
+}
+
+/*
+ * The component BAR carries the physical HDM decoder block, which vfio traps
+ * and excludes from mmap so the guest cannot reprogram it. The whole-BAR map
+ * must fail; the ranges around the excluded block must map.
+ */
+TEST_F(vfio_cxl, comp_bar_sparse_mmap)
+{
+	size_t page_size = getpagesize();
+	uint8_t rbuf[1024] = {};
+	struct vfio_region_info *ri = (void *)rbuf;
+	const struct vfio_region_info_cap_sparse_mmap *sm;
+	const struct vfio_info_cap_header *hdr;
+	uint64_t bar_off, decoder_page, hole_end;
+	bool covered;
+	void *map;
+	uint32_t i;
+
+	ri->argsz = sizeof(rbuf);
+	ri->index = self->comp_bar;
+	ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_REGION_INFO, ri));
+	ASSERT_NE(0, ri->flags & VFIO_REGION_INFO_FLAG_MMAP);
+	bar_off = ri->offset;
+
+	/* The trapped decoder block splits the BAR, so it must be sparse. */
+	hdr = find_region_cap(rbuf, sizeof(rbuf),
+			      VFIO_REGION_INFO_CAP_SPARSE_MMAP);
+	ASSERT_NE(NULL, hdr);
+	sm = (const void *)hdr;
+
+	/*
+	 * mmap is page granular, so the hole is the decoder block rounded out
+	 * to pages. With 64 KiB pages it can cover a small component BAR
+	 * entirely; then there is nothing left to map and no area.
+	 */
+	decoder_page = self->comp_offset & ~(uint64_t)(page_size - 1);
+	hole_end = (self->comp_offset + self->comp_size + page_size - 1) &
+		   ~(uint64_t)(page_size - 1);
+	covered = decoder_page == 0 && hole_end >= ri->size;
+	ASSERT_EQ(covered, sm->nr_areas == 0);
+
+	/* Mapping the whole BAR must fail: it covers the excluded block. */
+	map = mmap(NULL, ri->size, PROT_READ | PROT_WRITE, MAP_SHARED,
+		   self->dev->fd, bar_off);
+	ASSERT_EQ(MAP_FAILED, map);
+
+	/* Every advertised area is page aligned and must map. */
+	for (i = 0; i < sm->nr_areas; i++) {
+		uint64_t ao = sm->areas[i].offset;
+		uint64_t as = sm->areas[i].size;
+
+		if (!as)
+			continue;
+		ASSERT_EQ(0, ao & (page_size - 1));
+		ASSERT_EQ(0, as & (page_size - 1));
+
+		map = mmap(NULL, as, PROT_READ | PROT_WRITE, MAP_SHARED,
+			   self->dev->fd, bar_off + ao);
+		ASSERT_NE(MAP_FAILED, map);
+		ASSERT_EQ(0, munmap(map, as));
+	}
+
+	/* The page holding the decoder block must never be mmappable. */
+	map = mmap(NULL, page_size, PROT_READ | PROT_WRITE, MAP_SHARED,
+		   self->dev->fd, bar_off + decoder_page);
+	ASSERT_EQ(MAP_FAILED, map);
+}
+
+/* Basic HDM memory mmap read/write round-trip. */
+TEST_F(vfio_cxl, hdm_mem_mmap_rw)
+{
+	uint64_t off = VFIO_PCI_INDEX_TO_OFFSET(self->mem_idx);
+	uint32_t pattern = 0xdeadbeefU, readback = 0;
+	void *map;
+
+	if (self->mem_size < SZ_4K)
+		SKIP(return, "HDM memory < 4K");
+
+	map = mmap(NULL, SZ_4K, PROT_READ | PROT_WRITE, MAP_SHARED,
+		   self->dev->fd, off);
+	ASSERT_NE(MAP_FAILED, map);
+
+	memcpy(map, &pattern, sizeof(pattern));
+	memcpy(&readback, map, sizeof(readback));
+	ASSERT_EQ(pattern, readback);
+
+	ASSERT_EQ(0, munmap(map, SZ_4K));
+}
+
+/*
+ * read() and write() on the HDM region copy through a kernel buffer in page
+ * chunks: a transfer that spans several chunks and ends mid-page round-trips.
+ * Size it in pages of the running kernel, so a 64K-page kernel also crosses
+ * chunk boundaries. The zero-length read is answered by vfio-pci-core.
+ */
+TEST_F(vfio_cxl, hdm_mem_rw_multi_chunk)
+{
+	uint64_t off = VFIO_PCI_INDEX_TO_OFFSET(self->mem_idx);
+	size_t pg = getpagesize();
+	size_t len = 3 * pg + 100;
+	uint8_t *src, *dst;
+	size_t i;
+
+	if (self->mem_size < 4 * pg)
+		SKIP(return, "HDM memory < 4 pages");
+
+	src = malloc(len);
+	dst = calloc(1, len);
+	ASSERT_NE(NULL, src);
+	ASSERT_NE(NULL, dst);
+	for (i = 0; i < len; i++)
+		src[i] = i * 7 + 3;
+
+	ASSERT_EQ((ssize_t)len, pwrite(self->dev->fd, src, len, off + 64));
+	ASSERT_EQ((ssize_t)len, pread(self->dev->fd, dst, len, off + 64));
+	ASSERT_EQ(0, memcmp(src, dst, len));
+	ASSERT_EQ(0, pread(self->dev->fd, dst, 0, off));
+
+	free(src);
+	free(dst);
+}
+
+/*
+ * The user buffer of a read() may itself be an mmap of the HDM region. The
+ * copy to it faults the page in through the region's fault handler, which
+ * takes memory_lock; the read must not hold memory_lock across that copy.
+ * Run with lockdep to catch the recursive acquisition.
+ */
+TEST_F(vfio_cxl, hdm_mem_rw_into_own_mmap)
+{
+	uint64_t off = VFIO_PCI_INDEX_TO_OFFSET(self->mem_idx);
+	uint32_t pattern = 0xc0ffee11U, readback = 0;
+	uint8_t *map;
+
+	if (self->mem_size < 2 * SZ_4K)
+		SKIP(return, "HDM memory < 8K");
+
+	map = mmap(NULL, SZ_4K, PROT_READ | PROT_WRITE, MAP_SHARED,
+		   self->dev->fd, off);
+	ASSERT_NE(MAP_FAILED, map);
+
+	ASSERT_EQ((ssize_t)sizeof(pattern),
+		  pwrite(self->dev->fd, &pattern, sizeof(pattern),
+			 off + SZ_4K));
+	/* map is not faulted in yet: the copy to user faults it here. */
+	ASSERT_EQ((ssize_t)sizeof(pattern),
+		  pread(self->dev->fd, map, sizeof(pattern), off + SZ_4K));
+	memcpy(&readback, map, sizeof(readback));
+	ASSERT_EQ(pattern, readback);
+
+	ASSERT_EQ(0, munmap(map, SZ_4K));
+}
+
+/* A 2 MB-aligned HDM window mapped as a huge (PMD) fault. */
+TEST_F(vfio_cxl, hdm_mem_huge_mmap)
+{
+	uint64_t off = VFIO_PCI_INDEX_TO_OFFSET(self->mem_idx);
+	uint32_t pattern = 0x5a5a5a5aU, readback = 0;
+	void *map, *last;
+
+	if (self->mem_size < SZ_2M)
+		SKIP(return, "HDM memory < 2M");
+
+	map = mmap(NULL, SZ_2M, PROT_READ | PROT_WRITE, MAP_SHARED,
+		   self->dev->fd, off);
+	ASSERT_NE(MAP_FAILED, map);
+
+	last = (char *)map + SZ_2M - sizeof(pattern);
+	memcpy(last, &pattern, sizeof(pattern));
+	memcpy(&readback, last, sizeof(readback));
+	ASSERT_EQ(pattern, readback);
+
+	ASSERT_EQ(0, munmap(map, SZ_2M));
+}
+
+/*
+ * A VMM maps the HDM range into the guest IOAS by fd (the struct-page-less
+ * range cannot be pinned through a VA): export it as a dma-buf and map that fd
+ * with IOMMU_IOAS_MAP_FILE.
+ */
+TEST_F(vfio_cxl, hdm_mem_ioas_map)
+{
+	uint8_t buf[sizeof(struct vfio_device_feature) +
+		    sizeof(struct vfio_device_feature_dma_buf) +
+		    sizeof(struct vfio_region_dma_range)] = {};
+	struct vfio_device_feature *feat = (void *)buf;
+	struct vfio_device_feature_dma_buf *db = (void *)feat->data;
+	struct iova_allocator *iova_alloc;
+	int dmabuf_fd;
+	iova_t iova;
+	int ret;
+
+	if (!self->iommu->iommufd)
+		SKIP(return, "IOMMU_IOAS_MAP_FILE needs the iommufd backend");
+	if (self->mem_size < SZ_2M)
+		SKIP(return, "HDM memory < 2M");
+
+	feat->argsz = sizeof(buf);
+	feat->flags = VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_DMA_BUF;
+	db->region_index = self->mem_idx;
+	db->nr_ranges = 1;
+	db->dma_ranges[0].offset = 0;
+	db->dma_ranges[0].length = SZ_2M;
+
+	ret = ioctl(self->dev->fd, VFIO_DEVICE_FEATURE, feat);
+	if (ret < 0 && (errno == EINVAL || errno == EOPNOTSUPP))
+		SKIP(return, "kernel has no CXL dma-buf exporter");
+	ASSERT_GE(ret, 0);
+	dmabuf_fd = ret;
+
+	iova_alloc = iova_allocator_init(self->iommu);
+	iova = iova_allocator_alloc(iova_alloc, SZ_2M);
+
+	ASSERT_EQ(0, __iommu_map_file(self->iommu, dmabuf_fd, 0, SZ_2M, iova));
+	iommu_unmap_all(self->iommu);
+
+	iova_allocator_cleanup(iova_alloc);
+	ASSERT_EQ(0, close(dmabuf_fd));
+}
+
+/*
+ * The HDM dma-buf follows the HDM region, not Memory Space Enable: one
+ * exported while Memory Space is off starts armed and maps with
+ * IOMMU_IOAS_MAP_FILE once Memory Space is back on, without another export.
+ */
+TEST_F(vfio_cxl, hdm_dmabuf_export_memory_off)
+{
+	uint8_t buf[sizeof(struct vfio_device_feature) +
+		    sizeof(struct vfio_device_feature_dma_buf) +
+		    sizeof(struct vfio_region_dma_range)] = {};
+	struct vfio_device_feature *feat = (void *)buf;
+	struct vfio_device_feature_dma_buf *db = (void *)feat->data;
+	struct iova_allocator *iova_alloc;
+	int dmabuf_fd;
+	uint16_t cmd;
+	iova_t iova;
+
+	if (!self->iommu->iommufd)
+		SKIP(return, "IOMMU_IOAS_MAP_FILE needs the iommufd backend");
+	if (self->mem_size < SZ_2M)
+		SKIP(return, "HDM memory < 2M");
+
+	feat->argsz = sizeof(buf);
+	feat->flags = VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_DMA_BUF;
+	db->region_index = self->mem_idx;
+	db->nr_ranges = 1;
+	db->dma_ranges[0].offset = 0;
+	db->dma_ranges[0].length = SZ_2M;
+
+	cmd = vfio_pci_config_readw(self->dev, PCI_COMMAND);
+	vfio_pci_config_writew(self->dev, PCI_COMMAND,
+			       cmd & ~PCI_COMMAND_MEMORY);
+	dmabuf_fd = ioctl(self->dev->fd, VFIO_DEVICE_FEATURE, feat);
+	vfio_pci_config_writew(self->dev, PCI_COMMAND, cmd);
+	if (dmabuf_fd < 0 && (errno == EINVAL || errno == EOPNOTSUPP))
+		SKIP(return, "kernel has no CXL dma-buf exporter");
+	ASSERT_GE(dmabuf_fd, 0);
+
+	iova_alloc = iova_allocator_init(self->iommu);
+	iova = iova_allocator_alloc(iova_alloc, SZ_2M);
+
+	ASSERT_EQ(0, __iommu_map_file(self->iommu, dmabuf_fd, 0, SZ_2M, iova));
+	iommu_unmap_all(self->iommu);
+
+	iova_allocator_cleanup(iova_alloc);
+	ASSERT_EQ(0, close(dmabuf_fd));
+}
+
+/* The trapped block starts at the HDM decoder registers; CTRL 0 reads back. */
+TEST_F(vfio_cxl, comp_regs_hdm_read)
+{
+	uint64_t ctrl = self->comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0);
+	uint32_t val = 0;
+
+	ASSERT_GE(self->comp_size, CXL_HDM_DECODER0_CTRL_OFFSET(0) + 4);
+	ASSERT_EQ((ssize_t)sizeof(val),
+		  pread(self->dev->fd, &val, sizeof(val), ctrl));
+}
+
+/*
+ * The decoder block is BAR MMIO, so its region reads only with PCI Memory
+ * Space enabled; with it disabled the read fails with EIO.
+ */
+TEST_F(vfio_cxl, comp_regs_read_needs_memory)
+{
+	uint64_t ctrl = self->comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0);
+	uint16_t cmd;
+	uint32_t val;
+	ssize_t ret;
+
+	cmd = vfio_pci_config_readw(self->dev, PCI_COMMAND);
+	vfio_pci_config_writew(self->dev, PCI_COMMAND,
+			       cmd & ~PCI_COMMAND_MEMORY);
+	ret = pread(self->dev->fd, &val, sizeof(val), ctrl);
+	vfio_pci_config_writew(self->dev, PCI_COMMAND, cmd);
+
+	ASSERT_EQ(-1, ret);
+	ASSERT_EQ(EIO, errno);
+}
+
+/*
+ * A multi-dword read of the decoder block returns what the same registers
+ * return one dword at a time (decoder 0 BASE_LOW, BASE_HIGH, SIZE_LOW,
+ * SIZE_HIGH).
+ */
+TEST_F(vfio_cxl, comp_regs_read_multi_dword)
+{
+	uint64_t base = self->comp_off + CXL_HDM_DECODER0_BASE_LOW_OFFSET(0);
+	uint32_t block[4], one;
+	int i;
+
+	ASSERT_GE(self->comp_size,
+		  CXL_HDM_DECODER0_BASE_LOW_OFFSET(0) + sizeof(block));
+	ASSERT_EQ((ssize_t)sizeof(block),
+		  pread(self->dev->fd, block, sizeof(block), base));
+	for (i = 0; i < 4; i++) {
+		ASSERT_EQ((ssize_t)sizeof(one),
+			  pread(self->dev->fd, &one, sizeof(one),
+				base + i * sizeof(one)));
+		ASSERT_EQ(block[i], one);
+	}
+}
+
+/* The decoder registers only take aligned dword accesses. */
+TEST_F(vfio_cxl, comp_regs_reject_unaligned)
+{
+	uint32_t val = 0;
+	uint16_t half = 0;
+
+	ASSERT_EQ(-1, pread(self->dev->fd, &val, sizeof(val),
+			    self->comp_off + 1));
+	ASSERT_EQ(-1, pread(self->dev->fd, &half, sizeof(half),
+			    self->comp_off));
+}
+
+/* Accesses past the region end are rejected. */
+TEST_F(vfio_cxl, comp_regs_reject_out_of_range)
+{
+	uint32_t val = 0;
+
+	ASSERT_EQ(-1, pread(self->dev->fd, &val, sizeof(val),
+			    self->comp_off + self->comp_size));
+}
+
+/*
+ * Commit handshake: the host committed and locked the physical decoder, so the
+ * trapped block reads back COMMITTED and a guest decommit request is absorbed.
+ */
+TEST_F(vfio_cxl, hdm_commit_fsm)
+{
+	uint64_t ctrl = self->comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0);
+	uint32_t v, orig;
+
+	ASSERT_EQ((ssize_t)sizeof(orig),
+		  pread(self->dev->fd, &orig, sizeof(orig), ctrl));
+
+	if (!(orig & CXL_HDM_DECODER0_CTRL_COMMITTED))
+		SKIP(return, "HDM decoder 0 not committed by firmware");
+
+	/* A decommit request is absorbed; the decoder stays committed. */
+	v = orig & ~CXL_HDM_DECODER0_CTRL_COMMIT;
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pwrite(self->dev->fd, &v, sizeof(v), ctrl));
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pread(self->dev->fd, &v, sizeof(v), ctrl));
+	ASSERT_TRUE(v & CXL_HDM_DECODER0_CTRL_COMMITTED);
+}
+
+/*
+ * Lock on commit: a guest write of COMMIT | LOCK, and then of 0, is absorbed,
+ * so the decoder stays committed and its LOCK bit stays as firmware left it.
+ */
+TEST_F(vfio_cxl, hdm_lock_on_commit)
+{
+	uint64_t ctrl = self->comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0);
+	uint32_t before, v;
+
+	ASSERT_EQ((ssize_t)sizeof(before),
+		  pread(self->dev->fd, &before, sizeof(before), ctrl));
+	if (!(before & CXL_HDM_DECODER0_CTRL_LOCK))
+		TH_LOG("decoder 0 not locked by firmware; checking absorption only");
+
+	v = CXL_HDM_DECODER0_CTRL_COMMIT | CXL_HDM_DECODER0_CTRL_LOCK;
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pwrite(self->dev->fd, &v, sizeof(v), ctrl));
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pread(self->dev->fd, &v, sizeof(v), ctrl));
+	ASSERT_TRUE(v & CXL_HDM_DECODER0_CTRL_COMMITTED);
+	ASSERT_EQ(before & CXL_HDM_DECODER0_CTRL_LOCK,
+		  v & CXL_HDM_DECODER0_CTRL_LOCK);
+
+	/* Attempt to decommit the decoder; it must stay committed. */
+	v = 0;
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pwrite(self->dev->fd, &v, sizeof(v), ctrl));
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pread(self->dev->fd, &v, sizeof(v), ctrl));
+	ASSERT_TRUE(v & CXL_HDM_DECODER0_CTRL_COMMITTED);
+	ASSERT_EQ(before & CXL_HDM_DECODER0_CTRL_LOCK,
+		  v & CXL_HDM_DECODER0_CTRL_LOCK);
+}
+
+/*
+ * The committed decoder holds its base read-only: a guest write to the base is
+ * absorbed and reads back the host-committed value.
+ */
+TEST_F(vfio_cxl, hdm_base_write_absorbed)
+{
+	uint64_t lo_off = self->comp_off + CXL_HDM_DECODER0_BASE_LOW_OFFSET(0);
+	uint64_t ctrl_off = self->comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0);
+	uint32_t v = 0x30000000U;	/* 256 MB-aligned low base bits */
+	uint32_t rb = 0, orig = 0, ctrl = 0;
+
+	ASSERT_GE(self->comp_size, CXL_HDM_DECODER0_BASE_LOW_OFFSET(0) + 4);
+	ASSERT_EQ((ssize_t)sizeof(ctrl),
+		  pread(self->dev->fd, &ctrl, sizeof(ctrl), ctrl_off));
+	if (!(ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED))
+		SKIP(return, "HDM decoder 0 not committed by firmware");
+
+	ASSERT_EQ((ssize_t)sizeof(orig),
+		  pread(self->dev->fd, &orig, sizeof(orig), lo_off));
+
+	/* The write is absorbed; the base reads back unchanged. */
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pwrite(self->dev->fd, &v, sizeof(v), lo_off));
+	ASSERT_EQ((ssize_t)sizeof(rb),
+		  pread(self->dev->fd, &rb, sizeof(rb), lo_off));
+	ASSERT_EQ(orig, rb);
+}
+
+/*
+ * The CXL Device DVSEC body is virtualized by the kernel; a config read is
+ * served from the shadow.
+ */
+TEST_F(vfio_cxl, dvsec_body_read)
+{
+	uint32_t v;
+
+	if (!self->dvsec)
+		SKIP(return, "CXL Device DVSEC not found");
+
+	v = vfio_pci_config_readl(self->dev, self->dvsec + PCI_DVSEC_HEADER1);
+	ASSERT_NE(0xffffffffU, v);
+}
+
+/*
+ * Guest-initiated CXL reset: the DVSEC Initiate_CXL_Reset write runs the real
+ * reset, self-clears, and STATUS2 in the shadow reports the outcome.
+ */
+TEST_F(vfio_cxl, guest_cxl_reset)
+{
+	uint16_t cap, ctrl2, status2;
+
+	if (!self->dvsec)
+		SKIP(return, "CXL Device DVSEC not found");
+
+	cap = vfio_pci_config_readw(self->dev, self->dvsec + PCI_DVSEC_CXL_CAP);
+	if (!(cap & PCI_DVSEC_CXL_RST_CAPABLE))
+		SKIP(return, "device does not support CXL reset");
+
+	ctrl2 = vfio_pci_config_readw(self->dev,
+				      self->dvsec + PCI_DVSEC_CXL_CTRL2);
+	vfio_pci_config_writew(self->dev, self->dvsec + PCI_DVSEC_CXL_CTRL2,
+			       ctrl2 | PCI_DVSEC_CXL_INIT_CXL_RST);
+
+	/* Initiate_CXL_Reset self-clears once the sequence has run. */
+	ctrl2 = vfio_pci_config_readw(self->dev,
+				      self->dvsec + PCI_DVSEC_CXL_CTRL2);
+	ASSERT_FALSE(ctrl2 & PCI_DVSEC_CXL_INIT_CXL_RST);
+
+	/* STATUS2 reports the outcome; a completed reset sets RESET_COMPLETE. */
+	status2 = vfio_pci_config_readw(self->dev,
+					self->dvsec + PCI_DVSEC_CXL_STATUS2);
+	ASSERT_TRUE(status2 & PCI_DVSEC_CXL_RST_DONE);
+	ASSERT_FALSE(status2 & PCI_DVSEC_CXL_RST_ERR);
+}
+
+/*
+ * A real reset must run end to end, not just the shadow DVSEC path
+ * guest_cxl_reset covers: a CXL Type-2 device must advertise
+ * VFIO_DEVICE_FLAGS_RESET and complete a real VFIO_DEVICE_RESET.
+ */
+TEST_F(vfio_cxl, device_reset)
+{
+	uint8_t infobuf[512] = {};
+	struct vfio_device_info *info = (void *)infobuf;
+	int ret, retries = 20;
+
+	info->argsz = sizeof(infobuf);
+	ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_INFO, info));
+
+	ASSERT_NE(0, info->flags & VFIO_DEVICE_FLAGS_RESET);
+
+	/* Drive the real reset, retrying the transient device-lock contention. */
+	do {
+		ret = __vfio_pci_device_reset(self->dev);
+		if (ret == -EAGAIN)
+			usleep(10000);
+	} while (ret == -EAGAIN && retries-- > 0);
+	ASSERT_EQ(0, ret);
+}
+
+/*
+ * The component BAR is reachable by fd read except the trapped decoder block,
+ * which is served only through the comp-regs region; a VMM relies on this split.
+ */
+TEST_F(vfio_cxl, comp_bar_rdwr_split)
+{
+	uint64_t bar_off = VFIO_PCI_INDEX_TO_OFFSET(self->comp_bar);
+	uint32_t val;
+
+	/* A non-decoder dword of the BAR reads back through the fd. */
+	ASSERT_EQ((ssize_t)sizeof(val),
+		  pread(self->dev->fd, &val, sizeof(val), bar_off));
+
+	/*
+	 * The trapped decoder block is off-limits to raw BAR fd access: the read
+	 * returns all-ones, not the real decoder contents.
+	 */
+	val = 0;
+	ASSERT_EQ((ssize_t)sizeof(val),
+		  pread(self->dev->fd, &val, sizeof(val),
+			bar_off + self->comp_offset));
+	ASSERT_EQ(0xffffffffU, val);
+}
+
+/*
+ * Mirror the guest's HDM discovery, which does not use VFIO's geometry cap: it
+ * finds the range by walking the config-space DVSECs and the component-register
+ * array to the HDM decoder itself.
+ */
+TEST_F(vfio_cxl, guest_hdm_discovery)
+{
+	uint64_t bar_off = VFIO_PCI_INDEX_TO_OFFSET(self->comp_bar);
+	uint32_t reg_lo, reg_hi, cap_array, cap_count, hdr;
+	uint32_t bl, bh, sl, sh, ctrl;
+	uint64_t block_off, cm, hdm_off = 0, base, size;
+	uint16_t pos = PCI_CFG_SPACE_SIZE, regloc = 0, block1;
+	int iter = 0, bar, i;
+
+	/* 1. Find the CXL Register Locator DVSEC in config space. */
+	while (pos && iter++ < 64) {
+		uint32_t h = vfio_pci_config_readl(self->dev, pos);
+
+		if ((h & 0xffff) == PCI_EXT_CAP_ID_DVSEC) {
+			uint32_t h1 = vfio_pci_config_readl(self->dev, pos + 4);
+			uint32_t h2 = vfio_pci_config_readl(self->dev, pos + 8);
+
+			if ((h1 & 0xffff) == PCI_DVSEC_VENDOR_ID_CXL &&
+			    (h2 & 0xffff) == PCI_DVSEC_CXL_REG_LOCATOR) {
+				regloc = pos;
+				break;
+			}
+		}
+		pos = (h >> 20) & 0xffc;
+	}
+	ASSERT_NE(0, regloc);
+
+	/* 2. Take the component register block BAR and offset from block 1. */
+	block1 = regloc + PCI_DVSEC_CXL_REG_LOCATOR_BLOCK1;
+	reg_lo = vfio_pci_config_readl(self->dev, block1);
+	reg_hi = vfio_pci_config_readl(self->dev, block1 + 4);
+
+	ASSERT_EQ(CXL_REGLOC_RBI_COMPONENT,
+		  (reg_lo & REG_LOCATOR_BLOCK_ID_MASK) >> 8);
+	bar = reg_lo & REG_LOCATOR_BIR_MASK;
+	block_off = ((uint64_t)reg_hi << 32) |
+		    (reg_lo & REG_LOCATOR_BLOCK_OFF_LOW_MASK);
+
+	/* The DVSEC must name the same BAR the geometry cap reported. */
+	ASSERT_EQ(self->comp_bar, bar);
+
+	/* 3. Walk the CM capability array over the BAR to find the HDM cap. */
+	cm = block_off + CXL_CM_OFFSET;
+	ASSERT_EQ((ssize_t)sizeof(cap_array),
+		  pread(self->dev->fd, &cap_array, sizeof(cap_array),
+			bar_off + cm + CXL_CM_CAP_HDR_OFFSET));
+	ASSERT_EQ(CM_CAP_HDR_CAP_ID, cap_array & CXL_CM_CAP_HDR_ID_MASK);
+
+	cap_count = (cap_array & CXL_CM_CAP_HDR_ARRAY_SIZE_MASK) >> 24;
+	for (i = 1; i <= (int)cap_count; i++) {
+		ASSERT_EQ((ssize_t)sizeof(hdr),
+			  pread(self->dev->fd, &hdr, sizeof(hdr),
+				bar_off + cm + i * 4));
+		if ((hdr & CXL_CM_CAP_HDR_ID_MASK) == CXL_CM_CAP_CAP_ID_HDM) {
+			hdm_off = cm + ((hdr & CXL_CM_CAP_PTR_MASK) >> 20);
+			break;
+		}
+	}
+	ASSERT_NE(0, hdm_off);
+
+	/* The guest's manual walk must land on the decoder the kernel traps. */
+	ASSERT_EQ(self->comp_offset, hdm_off);
+
+	/* 4. Read decoder 0 through the trapped region and derive base/size. */
+	ASSERT_EQ((ssize_t)sizeof(bl),
+		  pread(self->dev->fd, &bl, sizeof(bl),
+			self->comp_off + CXL_HDM_DECODER0_BASE_LOW_OFFSET(0)));
+	ASSERT_EQ((ssize_t)sizeof(bh),
+		  pread(self->dev->fd, &bh, sizeof(bh),
+			self->comp_off + CXL_HDM_DECODER0_BASE_HIGH_OFFSET(0)));
+	ASSERT_EQ((ssize_t)sizeof(sl),
+		  pread(self->dev->fd, &sl, sizeof(sl),
+			self->comp_off + CXL_HDM_DECODER0_SIZE_LOW_OFFSET(0)));
+	ASSERT_EQ((ssize_t)sizeof(sh),
+		  pread(self->dev->fd, &sh, sizeof(sh),
+			self->comp_off + CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(0)));
+	ASSERT_EQ((ssize_t)sizeof(ctrl),
+		  pread(self->dev->fd, &ctrl, sizeof(ctrl),
+			self->comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0)));
+
+	base = ((uint64_t)bh << 32) | bl;
+	size = ((uint64_t)sh << 32) | sl;
+
+	/* A guest only accepts a committed decoder; the derived range must be non-empty. */
+	if (!(ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED))
+		SKIP(return, "HDM decoder 0 not committed by firmware");
+
+	ASSERT_GT(size, 0);
+	ASSERT_LT(base, base + size);
+}
+
+/*
+ * Tie the committed decoder's geometry to the HDM memory region a VMM hands the
+ * guest: confirm the mmap-able region covers exactly the decoder's range, then
+ * map the advertised base and touch it.
+ */
+TEST_F(vfio_cxl, hdm_mem_touch_committed_base)
+{
+	uint64_t mem_off = VFIO_PCI_INDEX_TO_OFFSET(self->mem_idx);
+	uint32_t pattern = 0xc0ffee11U, readback = 0;
+	uint32_t bl, bh, sl, sh, ctrl;
+	uint64_t base, size;
+	void *map;
+
+	ASSERT_EQ((ssize_t)sizeof(ctrl),
+		  pread(self->dev->fd, &ctrl, sizeof(ctrl),
+			self->comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0)));
+	if (!(ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED))
+		SKIP(return, "HDM decoder 0 not committed by firmware");
+
+	ASSERT_EQ((ssize_t)sizeof(bl),
+		  pread(self->dev->fd, &bl, sizeof(bl),
+			self->comp_off + CXL_HDM_DECODER0_BASE_LOW_OFFSET(0)));
+	ASSERT_EQ((ssize_t)sizeof(bh),
+		  pread(self->dev->fd, &bh, sizeof(bh),
+			self->comp_off + CXL_HDM_DECODER0_BASE_HIGH_OFFSET(0)));
+	ASSERT_EQ((ssize_t)sizeof(sl),
+		  pread(self->dev->fd, &sl, sizeof(sl),
+			self->comp_off + CXL_HDM_DECODER0_SIZE_LOW_OFFSET(0)));
+	ASSERT_EQ((ssize_t)sizeof(sh),
+		  pread(self->dev->fd, &sh, sizeof(sh),
+			self->comp_off + CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(0)));
+
+	base = ((uint64_t)bh << 32) | bl;
+	size = ((uint64_t)sh << 32) | sl;
+
+	ASSERT_GT(size, 0);
+	ASSERT_LT(base, base + size);
+	/* The mmap-able HDM region must cover exactly the committed decoder. */
+	ASSERT_EQ(size, self->mem_size);
+
+	if (self->mem_size < SZ_4K)
+		SKIP(return, "HDM memory < 4K");
+
+	/* Region offset 0 is the decoder's advertised base; map it and touch it. */
+	map = mmap(NULL, SZ_4K, PROT_READ | PROT_WRITE, MAP_SHARED,
+		   self->dev->fd, mem_off);
+	ASSERT_NE(MAP_FAILED, map);
+
+	memcpy(map, &pattern, sizeof(pattern));
+	memcpy(&readback, map, sizeof(readback));
+	ASSERT_EQ(pattern, readback);
+
+	ASSERT_EQ(0, munmap(map, SZ_4K));
+}
+
+/*
+ * The CXL Device DVSEC is served from a shadow: Control is guest-programmable
+ * but Capability keeps its firmware snapshot, so a guest cannot reprogram the
+ * device through the DVSEC.
+ */
+TEST_F(vfio_cxl, dvsec_write_virtualized)
+{
+	uint16_t cap, ctrl, v;
+
+	if (!self->dvsec)
+		SKIP(return, "CXL Device DVSEC not found");
+
+	/* A write to the read-only Capability register is absorbed. */
+	cap = vfio_pci_config_readw(self->dev, self->dvsec + PCI_DVSEC_CXL_CAP);
+	vfio_pci_config_writew(self->dev, self->dvsec + PCI_DVSEC_CXL_CAP,
+			       cap ^ 0xffff);
+	v = vfio_pci_config_readw(self->dev, self->dvsec + PCI_DVSEC_CXL_CAP);
+	ASSERT_EQ(cap, v);
+
+	/* Control is guest-programmable; a write reads back from the shadow. */
+	ctrl = vfio_pci_config_readw(self->dev, self->dvsec + PCI_DVSEC_CXL_CTRL);
+	vfio_pci_config_writew(self->dev, self->dvsec + PCI_DVSEC_CXL_CTRL,
+			       ctrl ^ 0x1);
+	v = vfio_pci_config_readw(self->dev, self->dvsec + PCI_DVSEC_CXL_CTRL);
+	ASSERT_EQ((uint16_t)(ctrl ^ 0x1), v);
+
+	/* The write stayed in the shadow: hardware Control is unchanged. */
+	if (!sysfs_config_read(self->dvsec + PCI_DVSEC_CXL_CTRL, &v, sizeof(v)))
+		ASSERT_EQ(ctrl, v);
+	vfio_pci_config_writew(self->dev, self->dvsec + PCI_DVSEC_CXL_CTRL, ctrl);
+}
+
+/*
+ * Region-info contract: the HDM memory region is readable, writable and
+ * mmappable; the trapped comp-regs region is fd-only and carries the geometry
+ * capability.
+ */
+TEST_F(vfio_cxl, region_flags)
+{
+	const uint32_t rw = VFIO_REGION_INFO_FLAG_READ |
+			    VFIO_REGION_INFO_FLAG_WRITE;
+	const struct vfio_info_cap_header *geo;
+	uint8_t rbuf[1024] = {};
+	struct vfio_region_info *ri = (void *)rbuf;
+
+	ri->argsz = sizeof(rbuf);
+	ri->index = self->mem_idx;
+	ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_REGION_INFO, ri));
+	ASSERT_EQ(rw | VFIO_REGION_INFO_FLAG_MMAP,
+		  ri->flags & (rw | VFIO_REGION_INFO_FLAG_MMAP));
+
+	memset(rbuf, 0, sizeof(rbuf));
+	ri->argsz = sizeof(rbuf);
+	ri->index = self->comp_idx;
+	ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_REGION_INFO, ri));
+	ASSERT_EQ(rw, ri->flags & rw);
+	ASSERT_EQ(0, ri->flags & VFIO_REGION_INFO_FLAG_MMAP);
+	geo = find_region_cap(rbuf, sizeof(rbuf),
+			      VFIO_REGION_INFO_CAP_CXL_COMP_REGS);
+	ASSERT_NE(NULL, geo);
+}
+
+/* True when decoder 0 reads back committed and the HDM region is readable. */
+static bool hdm_back_after(int fd, uint64_t comp_off, int mem_idx)
+{
+	uint64_t ctrl = comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0);
+	uint64_t mem = VFIO_PCI_INDEX_TO_OFFSET(mem_idx);
+	uint32_t val = 0;
+
+	if (pread(fd, &val, sizeof(val), ctrl) != sizeof(val))
+		return false;
+	if (!(val & CXL_HDM_DECODER0_CTRL_COMMITTED))
+		return false;
+
+	return pread(fd, &val, sizeof(val), mem) == sizeof(val);
+}
+
+/*
+ * Low power loses the committed decoder; the host restores it when the device
+ * returns to D0, so the decoder reads back committed and HDM access works.
+ */
+TEST_F(vfio_cxl, low_power_restores_decoder)
+{
+	struct vfio_device_feature feature = {
+		.argsz = sizeof(feature),
+		.flags = VFIO_DEVICE_FEATURE_SET |
+			 VFIO_DEVICE_FEATURE_LOW_POWER_ENTRY,
+	};
+
+	if (ioctl(self->dev->fd, VFIO_DEVICE_FEATURE, &feature))
+		SKIP(return, "low power entry not supported (%d)", errno);
+
+	feature.flags = VFIO_DEVICE_FEATURE_SET |
+			VFIO_DEVICE_FEATURE_LOW_POWER_EXIT;
+	ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_FEATURE, &feature));
+
+	ASSERT_TRUE(hdm_back_after(self->dev->fd, self->comp_off,
+				   self->mem_idx));
+}
+
+/* Read or write the physical config space through sysfs, bypassing vfio. */
+static int sysfs_config(bool wr, off_t off, void *buf, size_t len)
+{
+	char path[128];
+	ssize_t ret;
+	int fd;
+
+	snprintf(path, sizeof(path), "/sys/bus/pci/devices/%s/config",
+		 device_bdf);
+	fd = open(path, wr ? O_WRONLY : O_RDONLY);
+	if (fd < 0)
+		return -errno;
+	ret = wr ? pwrite(fd, buf, len, off) : pread(fd, buf, len, off);
+	close(fd);
+	return ret == (ssize_t)len ? 0 : -EIO;
+}
+
+/*
+ * The close path restores the config state saved at open: Memory Space on
+ * (pci_enable_device()) and Bus Master off (pci_clear_master()). The close
+ * time CXL Reset must not replace it with the post-use state, where
+ * vfio_pci_core_disable() has left only INTx Disable set.
+ */
+TEST(close_restores_open_time_command)
+{
+	struct vfio_device_info info = { .argsz = sizeof(info) };
+	struct vfio_pci_device *dev;
+	struct iommu *iommu;
+	uint16_t cmd = 0;
+	bool cxl;
+
+	iommu = iommu_init(default_iommu_mode);
+	dev = vfio_pci_device_init(device_bdf, iommu);
+	ASSERT_EQ(0, ioctl(dev->fd, VFIO_DEVICE_GET_INFO, &info));
+	cxl = info.flags & VFIO_DEVICE_FLAGS_CXL;
+	if (cxl)
+		vfio_pci_cmd_set(dev, PCI_COMMAND_MEMORY | PCI_COMMAND_MASTER);
+	vfio_pci_device_cleanup(dev);
+	iommu_cleanup(iommu);
+	if (!cxl)
+		SKIP(return, "not a CXL Type-2 device");
+
+	ASSERT_EQ(0, sysfs_config(false, PCI_COMMAND, &cmd, sizeof(cmd)));
+	ASSERT_NE(0, cmd & PCI_COMMAND_MEMORY);
+	ASSERT_EQ(0, cmd & PCI_COMMAND_MASTER);
+}
+
+/* Find a standard capability through the vfio config view. */
+static uint8_t find_std_cap(struct vfio_pci_device *dev, uint8_t id)
+{
+	uint8_t pos = vfio_pci_config_readb(dev, PCI_CAPABILITY_LIST) & ~3;
+	int iter = 0;
+
+	while (pos && iter++ < 48) {
+		if (vfio_pci_config_readb(dev, pos + PCI_CAP_LIST_ID) == id)
+			return pos;
+		pos = vfio_pci_config_readb(dev, pos + PCI_CAP_LIST_NEXT) & ~3;
+	}
+	return 0;
+}
+
+/*
+ * A D3hot write to the PM control register closes the HDM gate and the D0
+ * write reopens it with the decoder restored. If D3hot entry fails or a quirk
+ * keeps the device in D0, the gate must still reopen at once.
+ */
+TEST_F(vfio_cxl, pm_d3hot_restores_decoder)
+{
+	uint8_t pm = find_std_cap(self->dev, PCI_CAP_ID_PM);
+	uint16_t ctrl;
+
+	if (!pm)
+		SKIP(return, "no PCI PM capability");
+
+	ctrl = vfio_pci_config_readw(self->dev, pm + PCI_PM_CTRL);
+	vfio_pci_config_writew(self->dev, pm + PCI_PM_CTRL,
+			       (ctrl & ~PCI_PM_CTRL_STATE_MASK) | 3);
+	vfio_pci_config_writew(self->dev, pm + PCI_PM_CTRL,
+			       ctrl & ~PCI_PM_CTRL_STATE_MASK);
+
+	ASSERT_TRUE(hdm_back_after(self->dev->fd, self->comp_off,
+				   self->mem_idx));
+}
+
+int main(int argc, char *argv[])
+{
+	device_bdf = vfio_selftests_get_bdf(&argc, argv);
+	return test_harness_run(argc, argv);
+}
-- 
2.53.0