[PATCH v4 00/27] vfio/pci: Add CXL Type-2 device passthrough support

[PATCH v4 00/27] vfio/pci: Add CXL Type-2 device passthrough support

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

This series adds VFIO passthrough for CXL Type-2 accelerators. The guest
drives its own virtual HDM decoder and can reset the device. The host
owns the physical decoder and the host physical address the memory
lands at. The guest only picks a guest physical address. This series is targeted
towards a single, non-interleaved endpoint decoder.


Base and dependencies
---------------------

  Base: Linux Stable 7.2-rc6, plus the two dependencies below

  Depends-on:
    - cxl_set_capacity() and include/cxl/pci.h from Alejandro's
      "Type2 device basic support" v30 [1]
    - CXL reset core from Srirangan's cxl_reset series (v10) [2],
      which caches the decoder programming in pdev->hdm and keeps the HDM
      decoder and reset helpers in drivers/cxl/core/resource.c. Part 1 adds
      a function-scoped reset entry (cxl_reset_dvsec_sequence) on top, and
      Part 4 builds on it.


Changes since v3
----------------

v3 [3] kept the register virtualization in cxl-core, behind an opaque
cxl_passthrough handle that exported four helpers, with vfio-pci as a thin
transport. v4 turns that around. The register emulation lives in vfio-cxl, a
separate module vfio-pci-core loads on demand, and cxl-core keeps only the
reset entry and a few enabling helpers. The patch count grew from 11 to 27,
most of it the reset handling v3 left out of scope.


Reviewer feedback addressed
---------------------------

The v3 [3] thread has the full discussion; this is where each objection landed.

Dan Williams
  - Register emulation moved from cxl-core into vfio-cxl; the opaque
    cxl_passthrough handle and drivers/cxl/core/passthrough.c are gone.
  - The lock-on-commit FSM is fixed: a committed and locked decoder ignores
    further control writes until a reset, and the commit bit is not cleared
    while locked.
  - HDM mmap handles huge pages; the fault handler inserts the HPA pfn at the
    largest aligned order.
  - The BAR sub-block request is skipped through a flag on the cxl-core
    register map (a BAR-owning driver leaves the sub-blocks unclaimed), not a
    workaround on the vfio side.
  - HOST_FIRMWARE_COMMITTED is dropped; a present cap already implies a
    committed decoder, since bind fails without an active one.
  - cxl_get_hdm_info() is dropped in favour of reading pdev->hdm.
  - Style items are applied: the module lives under drivers/vfio/pci/cxl/, it
    reuses the register defines from pci_regs.h and cxl.h, and it cleans up
    with devm_add_action_or_reset().

Alex Williamson
  - vfio-pci does not statically depend on CXL; it loads vfio-cxl on demand
    and pins it per bound device.
  - The CXL state is sampled at open_device, not at bind, so a low-power
    transition cannot leave it stale.
  - The device info cap drops the region indices; regions are found by VFIO
    subtype, and the component-register geometry is on the comp-reg region
    info cap.
  - The DVSEC clip shim is replaced by a DVSEC readfn/writefn in ecap_perms
    that range-checks the offset against the stored DVSEC body.
  - vdev->cxl is a typed struct vfio_cxl_state pointer, not void *.
  - cxl_await_range_active() stays in cxl-core; vfio does not call or export
    it, and the cover no longer claims otherwise.

Dave Jiang
  - The base moved to Linux 7.2-rc6 (plus the two named dependencies), so the
    series applies cleanly with b4 (v3 failed on its stated base).

Richard Cheng
  - The host hang on real hardware is fixed. The HDM region is reached through
    an mmap fault that inserts the HPA pfn rather than a memremap(WB) kva, and
    the CXL state is acquired at open rather than at bind.


Patch order
-----------

The patches are ordered so the tree builds at every commit and each
change sits next to the code it depends on. Six groups:

  Part 1  CXL core (patches 1-5)
    - Register-map plumbing so a BAR-owning driver can leave the HDM/RAS
      sub-blocks unclaimed.
    - Component register defines moved to include/uapi/cxl/cxl_regs.h so a VMM
      can consume them.
    - Media readiness established in cxl_mem_probe() so a mailbox-less Type-2
      memdev created through devm_cxl_probe_mem() is not rejected.
    - A function-scoped reset entry for vfio-pci.

  Part 2  VFIO-CXL bring-up (patches 6-13)
    - The vfio-cxl module and its ops-provider hook into vfio-pci-core.
    - The CXL memory device created at bind.
    - A bind-time gate that rejects decoder topologies the emulation does not
      model.
    - Ownership of the whole component BAR.
    - A vfio-pci-core helper for a provider to exclude a BAR sub-range from
      mmap, read/write, and DMA-BUF export.

  Part 3  HDM region and register regions (patches 14-21)
    - The CXL region type, the open/close hooks, the DVSEC shadow and its
      virtualization.
    - The HDM region as an mmap-able region mapped write-back cacheable
      (CXL.mem is coherent) together with the trapped HDM decoder register
      snapshot.
    - The decoder block is kept off the direct BAR mapping.
    - Lock-on-commit decoder emulation, and the UAPI that describes a CXL device
      to userspace.
    - DVSEC virtualization and decoder emulation apply the CXL r4.0 write
      class field by field:
        - Control is programmable
        - Status is write-1-to-clear
        - Capability, Lock and the Range/Base/Size fields keep their firmware
          snapshot or their commit-gated state.
    - The CXL.mem-enable bit in the DVSEC control register is pinned on.
    - A guest store to the decoded window with CXL.mem disabled would fault the
      device and raise a host SError, so a guest request to clear the bit is
      ignored.

  Part 4  VFIO-CXL reset (patches 22-24)
    - HDM mapping revoke across reset and power transitions
    - Decoder snapshot refresh after reset
    - A guest-triggered CXL reset driven from the DVSEC.

  Part 5  CXL opt-out (patch 25)
    - A vfio_pci.disable_cxl module parameter and a per-device
      vdev->disable_cxl bit a variant driver sets in its own .init before
      it calls vfio_pci_core_init_dev().
    - Setting it later, in probe, is too late: the core .init has already
      attached the CXL provider. An admin or a variant driver can keep the CXL
      extensions out of the path and drive the device as plain vfio-pci.
    - Mirrors the disable_denylist opt-out.

  Part 6  Documentation and selftests (patches 26-27)


Subsystem boundary
------------------

vfio-pci-core does not implement CXL registers. vfio-cxl is a separate
module that registers a struct vfio_cxl_ops at init. vfio-pci-core loads
it on demand for a CXL device (request_module plus pcie_is_cxl) and pins
it per bound device. If a modular provider is missing or fails to load,
the device is driven as plain vfio-pci. The bind only defers for the
built-in initcall-order case, where request_module cannot help.

The core exposes only the primitives that have to live in core
(memory_lock, mapping revoke, dma-buf quiesce, and the BAR sub-range
exclusion). CXL-specific work stays behind the ops.


Memory ownership
----------------

The host resolves the host physical address once, at bind, through
devm_cxl_probe_mem(). The memdev is owned for the bind lifetime and torn
down at unbind. The HPA range is claimed IORESOURCE_EXCLUSIVE so no
mismatched cacheable alias can form, including one mapped through
/dev/mem.

The guest programs a guest physical address into a trapped virtual
decoder and polls a shadow for commit. It never reaches the physical
decoder registers. Those are served only through the trap and are
excluded from the direct BAR mapping, so the guest cannot move the host
physical window.


HDM region access
-----------------

The HDM region carries the coherent device memory. A VMM mmaps it and
maps it into the guest through stage-2; that is the access path. The
region advertises READ and WRITE so a VMM can derive an accessible (non
PROT_NONE) mmap protection from those flags, but fd read/write on the
region is rejected with -EIO.

The guest-facing mmap fault maps the pfn directly and does not hit that path.
The flags describe the mmap protection; they are not a promise of fd I/O.

The fault path only inserts the pfn while the decoder is in a known-good
restored state and the device has PCI Memory-Space enabled, so a host
access cannot reach a revoked or disabled decoder.


Reset
-----

A CXL Type-2 device is reset through its DVSEC sequence at every path
that can reset it, not through an FLR. A guest write of
Initiate_CXL_Reset in the CXL DVSEC is trapped, not forwarded. Under
memory_lock, with the HDM mapping revoked and the IOMMU mapping already
blocked, the host saves config, drops Bus Master, runs
cxl_reset_dvsec_sequence(), and restores config with Bus Master still
dropped until the outcome is known. Bus Master is re-enabled only after
a clean reset, so the function cannot DMA over decoders that were not
restored. The guest owns Mem_Clr_Enable. The outcome comes back through
DVSEC STATUS2 for the guest to poll.

The same DVSEC reset is the primary reset method at every vfio-pci-core
site. VM power-on (enable), power-off (close), the VFIO_DEVICE_RESET
ioctl, and a guest FLR emulated through config space all go through a
shared helper.

A bus hot reset rejects a multifunction CXL device. A single-function
one is quiesced through its DVSEC sequence before the secondary bus
reset. Every path revokes the HDM mapping first. The secondary bus reset
clears the physical decoder, so the decoder is restored and the snapshot
is re-sampled afterward, with Bus Master held dropped across the restore
so nothing can DMA over cleared decode. A restore failure is reported
through the reset ioctl; the bus reset's success is not returned in that
case.

A bound CXL Type-2 device is also kept out of idle D3. It is NoSoftRst-,
so a D3hot->D0 transition soft-resets the function and clears the
accelerator memory controller. That can corrupt the coherent CXL.mem the
decoder exposes, and only the accelerator's own driver re-initializes
it. vfio-cxl sets disable_idle_d3 so the host keeps the device in D0
while it is bound, rather than powering it down between opens.


Validation
----------

  - Each patch builds (drivers/cxl and drivers/vfio) and passes
    scripts/checkpatch.pl --codespell --strict with no errors, warnings, or
    checks.
  - The series applies in sequence on the stated base on top of stated
    dependencies.
  - Ran couple of rounds of masoncl/review-prompts on this series before posting.
  - Testing workaround (not part of this series):
     - The HDM region is a VM_PFNMAP RAM-device region, and mapping it into a
       guest IOMMU address space for device-side ATS currently returns
       -EOPNOTSUPP. The in-tree iommufd/VFIO map path does not follow a
       VM_PFNMAP range, so a DMA map of the CXL.mem range is refused (QEMU logs
       it as a non-fatal vfio_container_dma_map ... = -95 warning and continues).
     - To exercise the ATS path on hardware, an out-of-tree workaround that
       bypasses the PFNMAP restriction on the iommufd map path was applied so
       the CXL.mem range could be mapped into the stage-2 IOAS.
     - The selftests and the CPU/coherent guest access paths do not need it;
       only device-side ATS to CXL.mem does.
     - The intended upstream fix is a dma-buf based import of the CXL.mem range
       into iommufd. That will be a separate series.


Follow-on UAPI
--------------

- The trapped component-register region spans the whole HDM decoder block, not
  just decoder 0, so a multi-decoder device needs no new region or cap: a VMM
  reads the decoder count and each committed base from the block itself.
- The COMP_REGS geometry cap keeps a reserved field as a versioning anchor.
- Further trapped surfaces such as CXL RAS are planned as new CXL region
  subtypes (3+) rather than by extending this cap.
- The single committed, non-interleaved decoder is a bind-time policy in one
  place, not an ABI assumption, so in future multi-decoder and switched
  topologies relax only at that place.


Deferred
--------

  - Topology reach. Switched, multi-decoder, and interleaved decoders
    stay rejected at bind.
  - Non Firmware committed decoder system support.


References
----------

[1] [PATCH v30 0/2] Type2 device basic support
https://lore.kernel.org/all/20260629183727.51502-1-alejandro.lucero-palau@amd.com/

[2] [PATCH v10 00/12] PCI/CXL: Add CXL reset support for Type 2 devices
https://lore.kernel.org/linux-cxl/20260804192958.1823952-1-smadhavan@nvidia.com/

[3] Previous version of this patch series
[PATCH v3 00/11] vfio/pci: Add CXL Type-2 device passthrough support
https://lore.kernel.org/linux-cxl/20260625165407.1769572-1-mhonap@nvidia.com/


Manish Honap (27):
  cxl: Fix resource.c include path and export cxl_restore_hdm_after_pci_reset
  cxl/regs: Skip sub-block region request for BAR-owning drivers
  cxl: Move component register defines to uapi/cxl/cxl_regs.h
  cxl: Establish media readiness in cxl_mem_probe()
  cxl: Add a function-scoped reset entry for vfio-pci
  vfio/pci: Add CXL ops registration interface
  vfio/pci: Detect CXL devices and load vfio-cxl on demand
  vfio/cxl: Add the vfio-cxl module skeleton
  vfio/cxl: Create the CXL memory device at bind
  vfio/cxl: Reject unsupported decoder topologies at bind
  vfio/cxl: Own the whole component register BAR
  vfio/pci: Let a provider exclude a BAR sub-range from mmap
  vfio/pci: Refuse read/write to an excluded BAR sub-range
  vfio: Add CXL region type for the HDM region
  vfio/pci: Call CXL open and close hooks around device use
  vfio/cxl: Shadow the CXL DVSEC body at open
  vfio/cxl: Virtualize the CXL DVSEC
  vfio/cxl: Expose the HDM memory and trap the decoder registers
  vfio/cxl: Keep the HDM decoder block off the direct BAR mapping
  vfio/cxl: Emulate the HDM decoder commit handshake
  vfio/cxl: Describe the CXL device and decoder geometry to userspace
  vfio/cxl: Revoke the HDM mapping on reset and power transitions
  vfio/cxl: Refresh the decoder snapshot after a device reset
  vfio/cxl: Service a guest-triggered CXL reset
  vfio/pci: Provide an opt-out for the CXL Type-2 extensions
  Documentation: vfio-pci: Document CXL Type-2 device passthrough
  selftests/vfio: Add CXL Type-2 passthrough corner-case tests

 Documentation/driver-api/index.rst            |    1 +
 Documentation/driver-api/vfio-pci-cxl.rst     |  141 +++
 MAINTAINERS                                   |   10 +
 drivers/cxl/core/pci.c                        |   24 +-
 drivers/cxl/core/regs.c                       |   32 +-
 drivers/cxl/core/resource.c                   |   72 +-
 drivers/cxl/cxl.h                             |   47 +-
 drivers/cxl/mem.c                             |    9 +-
 drivers/cxl/pci.c                             |    4 +-
 drivers/vfio/pci/Kconfig                      |    2 +
 drivers/vfio/pci/Makefile                     |    2 +
 drivers/vfio/pci/cxl/Kconfig                  |   11 +
 drivers/vfio/pci/cxl/Makefile                 |    3 +
 drivers/vfio/pci/cxl/vfio_cxl_core.c          | 1036 +++++++++++++++++
 drivers/vfio/pci/vfio_pci.c                   |    9 +
 drivers/vfio/pci/vfio_pci_config.c            |   42 +-
 drivers/vfio/pci/vfio_pci_core.c              |  478 +++++++-
 drivers/vfio/pci/vfio_pci_dmabuf.c            |   13 +
 drivers/vfio/pci/vfio_pci_priv.h              |   35 +
 drivers/vfio/pci/vfio_pci_rdwr.c              |    8 +
 include/cxl/cxl.h                             |   15 +
 include/cxl/pci.h                             |    2 +-
 include/linux/vfio_pci_core.h                 |   37 +
 include/uapi/cxl/cxl_regs.h                   |   58 +
 include/uapi/linux/pci_regs.h                 |    1 +
 include/uapi/linux/vfio.h                     |   26 +
 tools/testing/selftests/vfio/Makefile         |    1 +
 .../selftests/vfio/lib/vfio_pci_device.c      |   57 +-
 .../selftests/vfio/vfio_cxl_type2_test.c      |  799 +++++++++++++
 29 files changed, 2877 insertions(+), 98 deletions(-)
 create mode 100644 Documentation/driver-api/vfio-pci-cxl.rst
 create mode 100644 drivers/vfio/pci/cxl/Kconfig
 create mode 100644 drivers/vfio/pci/cxl/Makefile
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_core.c
 create mode 100644 include/uapi/cxl/cxl_regs.h
 create mode 100644 tools/testing/selftests/vfio/vfio_cxl_type2_test.c


base-commit: 075b74841bd0065a3bda3440873c747938e69b68
-- 
2.25.1

[PATCH v4 01/27] cxl: Fix resource.c include path and export cxl_restore_hdm_after_pci_reset

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

The vfio-cxl Type-2 driver restores the endpoint HDM decoder after a
D3hot->D0 soft reset by calling cxl_restore_hdm_after_pci_reset(), so
export it to the CXL namespace. While here, correct the resource.c
include of the CXL PCI definitions from <cxlpci.h> to <cxl/pci.h>.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/cxl/core/resource.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/cxl/core/resource.c b/drivers/cxl/core/resource.c
index daa5e7fa2509..492ffe8e3576 100644
--- a/drivers/cxl/core/resource.c
+++ b/drivers/cxl/core/resource.c
@@ -16,7 +16,7 @@
 #include <linux/pci.h>
 #include <linux/slab.h>
 
-#include <cxlpci.h>
+#include <cxl/pci.h>
 
 #include "cxl.h"
 #include "core.h"
@@ -850,6 +850,7 @@ int cxl_restore_hdm_after_pci_reset(struct pci_dev *pdev)
 
 	return cxl_reset_save_restored_state(pdev, command);
 }
+EXPORT_SYMBOL_NS_GPL(cxl_restore_hdm_after_pci_reset, "CXL");
 
 static void cxl_hdm_range_context_init(struct cxl_hdm_range_context *ctx)
 {
-- 
2.25.1

[PATCH v4 02/27] cxl/regs: Skip sub-block region request for BAR-owning drivers

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

cxl_map_component_regs() claims each mapped sub-block with
devm_request_mem_region(). A driver that owns the whole component
register BAR, such as vfio-cxl, has already claimed the full BAR, so
the per-sub-block claim collides and the mapping fails.

Add a bar_owned parameter to cxl_pci_setup_regs() and record it in
skip_sub_bar_request on the register map. When set, cxl-core maps the
sub-block without requesting the region and leaves ownership with the
upper driver. cxl_pci passes false and keeps its existing claim,
preserving /dev/mem tooling access to the rest of the component space.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/cxl/core/pci.c  |  9 ++++++++-
 drivers/cxl/core/regs.c | 32 ++++++++++++++++++++++----------
 drivers/cxl/pci.c       |  4 ++--
 include/cxl/cxl.h       |  3 +++
 include/cxl/pci.h       |  2 +-
 5 files changed, 36 insertions(+), 14 deletions(-)

diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
index 9d807c1a002c..08d4c955137d 100644
--- a/drivers/cxl/core/pci.c
+++ b/drivers/cxl/core/pci.c
@@ -720,7 +720,7 @@ static int cxl_rcrb_get_comp_regs(struct pci_dev *pdev,
 }
 
 int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
-		       struct cxl_register_map *map)
+		       struct cxl_register_map *map, bool bar_owned)
 {
 	int rc;
 
@@ -750,6 +750,13 @@ int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
 		return rc;
 	}
 
+	/*
+	 * A caller that owns the whole register BAR (for example vfio-cxl)
+	 * maps the sub-blocks without claiming them, so the later
+	 * cxl_map_component_regs() does not collide with the full-BAR request.
+	 */
+	map->skip_sub_bar_request = bar_owned;
+
 	return cxl_setup_regs(map);
 }
 EXPORT_SYMBOL_NS_GPL(cxl_pci_setup_regs, "CXL");
diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
index 989e79383b99..90dd85f322f5 100644
--- a/drivers/cxl/core/regs.c
+++ b/drivers/cxl/core/regs.c
@@ -177,35 +177,47 @@ void cxl_probe_device_regs(struct device *dev, void __iomem *base,
 }
 EXPORT_SYMBOL_NS_GPL(cxl_probe_device_regs, "CXL");
 
-void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,
-				   resource_size_t length)
+static void __iomem *__cxl_iomap_block(struct device *dev, resource_size_t addr,
+				       resource_size_t length, bool request)
 {
+	resource_size_t end = addr + length - 1;
 	void __iomem *ret_val;
 	struct resource *res;
 
 	if (WARN_ON_ONCE(addr == CXL_RESOURCE_NONE))
 		return NULL;
 
-	res = devm_request_mem_region(dev, addr, length, dev_name(dev));
-	if (!res) {
-		resource_size_t end = addr + length - 1;
-
-		dev_err(dev, "Failed to request region %pa-%pa\n", &addr, &end);
-		return NULL;
+	/*
+	 * An owning driver (e.g. vfio-cxl) may already hold the whole BAR; a
+	 * sub-block request would then collide with that claim.
+	 */
+	if (request) {
+		res = devm_request_mem_region(dev, addr, length, dev_name(dev));
+		if (!res) {
+			dev_err(dev, "Failed to request region %pa-%pa\n", &addr, &end);
+			return NULL;
+		}
 	}
 
 	ret_val = devm_ioremap(dev, addr, length);
 	if (!ret_val)
-		dev_err(dev, "Failed to map region %pr\n", res);
+		dev_err(dev, "Failed to map region %pa-%pa\n", &addr, &end);
 
 	return ret_val;
 }
+
+void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,
+				   resource_size_t length)
+{
+	return __cxl_iomap_block(dev, addr, length, true);
+}
 EXPORT_SYMBOL_NS_GPL(devm_cxl_iomap_block, "CXL");
 
 int cxl_map_component_regs(const struct cxl_register_map *map,
 			   struct cxl_component_regs *regs,
 			   unsigned long map_mask)
 {
+	bool request = !map->skip_sub_bar_request;
 	struct device *host = map->host;
 	struct mapinfo {
 		const struct cxl_reg_map *rmap;
@@ -227,7 +239,7 @@ int cxl_map_component_regs(const struct cxl_register_map *map,
 			continue;
 		addr = map->resource + mi->rmap->offset;
 		length = mi->rmap->size;
-		*(mi->addr) = devm_cxl_iomap_block(host, addr, length);
+		*(mi->addr) = __cxl_iomap_block(host, addr, length, request);
 		if (!*(mi->addr))
 			return -ENOMEM;
 	}
diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index bb892dbfdd6d..ec1693c21f61 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -816,7 +816,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	cxlds->rcd = is_cxl_restricted(pdev);
 
-	rc = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_MEMDEV, &map);
+	rc = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_MEMDEV, &map, false);
 	if (rc)
 		return rc;
 
@@ -829,7 +829,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	 * still be useful for management functions so don't return an error.
 	 */
 	rc = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_COMPONENT,
-				&cxlds->reg_map);
+				&cxlds->reg_map, false);
 	if (rc)
 		dev_warn(&pdev->dev, "No component registers (%d)\n", rc);
 	else if (!cxlds->reg_map.component_map.ras.valid)
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
index efb8ac97bd92..f8e8fddba152 100644
--- a/include/cxl/cxl.h
+++ b/include/cxl/cxl.h
@@ -213,6 +213,8 @@ struct cxl_pmu_reg_map {
  * @resource: physical resource base of the register block
  * @max_size: maximum mapping size to perform register search
  * @reg_type: see enum cxl_regloc_type
+ * @skip_sub_bar_request: set via cxl_pci_setup_regs(); the owning driver holds
+ *			  the BAR, so skip the sub-block claim
  * @component_map: cxl_reg_map for component registers
  * @device_map: cxl_reg_maps for device registers
  * @pmu_map: cxl_reg_maps for CXL Performance Monitoring Units
@@ -223,6 +225,7 @@ struct cxl_register_map {
 	resource_size_t resource;
 	resource_size_t max_size;
 	u8 reg_type;
+	bool skip_sub_bar_request;
 	union {
 		struct cxl_component_reg_map component_map;
 		struct cxl_device_reg_map device_map;
diff --git a/include/cxl/pci.h b/include/cxl/pci.h
index 3e0000015871..81a32004a604 100644
--- a/include/cxl/pci.h
+++ b/include/cxl/pci.h
@@ -18,5 +18,5 @@ struct cxl_register_map;
 struct pci_dev;
 
 int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
-		       struct cxl_register_map *map);
+		       struct cxl_register_map *map, bool bar_owned);
 #endif
-- 
2.25.1

[PATCH v4 03/27] cxl: Move component register defines to uapi/cxl/cxl_regs.h

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

The CXL component and HDM decoder register layout is spec-defined and is
needed by consumers outside the CXL subsystem such as vfio-cxl, which
cannot include the private drivers/cxl/cxl.h. Move those defines to a new
uapi header, the way PCI register offsets live in uapi pci_regs.h, and
include it from drivers/cxl/cxl.h so existing users are unchanged.

No functional change; the GENMASK and BIT forms are written as plain
values as an uAPI header requirement.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 MAINTAINERS                 |  1 +
 drivers/cxl/cxl.h           | 47 +-----------------------------
 include/uapi/cxl/cxl_regs.h | 58 +++++++++++++++++++++++++++++++++++++
 3 files changed, 60 insertions(+), 46 deletions(-)
 create mode 100644 include/uapi/cxl/cxl_regs.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 5114e6db7307..c665bf8f1a28 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6528,6 +6528,7 @@ F:	Documentation/driver-api/cxl
 F:	Documentation/userspace-api/fwctl/fwctl-cxl.rst
 F:	drivers/cxl/
 F:	include/cxl/
+F:	include/uapi/cxl/
 F:	include/uapi/linux/cxl_mem.h
 F:	tools/testing/cxl/
 
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index c795cce0fcbc..2893b2427be1 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -13,6 +13,7 @@
 #include <linux/io.h>
 #include <linux/range.h>
 #include <cxl/cxl.h>
+#include <cxl/cxl_regs.h>
 
 extern const struct nvdimm_security_ops *cxl_security_ops;
 
@@ -24,52 +25,6 @@ extern const struct nvdimm_security_ops *cxl_security_ops;
  * (port-driver, region-driver, nvdimm object-drivers... etc).
  */
 
-/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */
-#define CXL_COMPONENT_REG_BLOCK_SIZE SZ_64K
-
-/* CXL 2.0 8.2.5 CXL.cache and CXL.mem Registers*/
-#define CXL_CM_OFFSET 0x1000
-#define CXL_CM_CAP_HDR_OFFSET 0x0
-#define   CXL_CM_CAP_HDR_ID_MASK GENMASK(15, 0)
-#define     CM_CAP_HDR_CAP_ID 1
-#define   CXL_CM_CAP_HDR_VERSION_MASK GENMASK(19, 16)
-#define     CM_CAP_HDR_CAP_VERSION 1
-#define   CXL_CM_CAP_HDR_CACHE_MEM_VERSION_MASK GENMASK(23, 20)
-#define     CM_CAP_HDR_CACHE_MEM_VERSION 1
-#define   CXL_CM_CAP_HDR_ARRAY_SIZE_MASK GENMASK(31, 24)
-#define CXL_CM_CAP_PTR_MASK GENMASK(31, 20)
-
-#define   CXL_CM_CAP_CAP_ID_RAS 0x2
-#define   CXL_CM_CAP_CAP_ID_HDM 0x5
-#define   CXL_CM_CAP_CAP_HDM_VERSION 1
-
-/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
-#define CXL_HDM_DECODER_CAP_OFFSET 0x0
-#define   CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0)
-#define   CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4)
-#define   CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8)
-#define   CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9)
-#define   CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY BIT(11)
-#define   CXL_HDM_DECODER_INTERLEAVE_16_WAY BIT(12)
-#define CXL_HDM_DECODER_CTRL_OFFSET 0x4
-#define   CXL_HDM_DECODER_ENABLE BIT(1)
-#define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10)
-#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i) (0x20 * (i) + 0x14)
-#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i) (0x20 * (i) + 0x18)
-#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i) (0x20 * (i) + 0x1c)
-#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20)
-#define   CXL_HDM_DECODER0_CTRL_IG_MASK GENMASK(3, 0)
-#define   CXL_HDM_DECODER0_CTRL_IW_MASK GENMASK(7, 4)
-#define   CXL_HDM_DECODER0_CTRL_LOCK BIT(8)
-#define   CXL_HDM_DECODER0_CTRL_COMMIT BIT(9)
-#define   CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10)
-#define   CXL_HDM_DECODER0_CTRL_COMMIT_ERROR BIT(11)
-#define   CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12)
-#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
-#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
-#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
-#define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i)
-
 /* HDM decoder control register constants CXL 3.0 8.2.5.19.7 */
 #define CXL_DECODER_MIN_GRANULARITY 256
 #define CXL_DECODER_MAX_ENCODED_IG 6
diff --git a/include/uapi/cxl/cxl_regs.h b/include/uapi/cxl/cxl_regs.h
new file mode 100644
index 000000000000..04bf790f0fdc
--- /dev/null
+++ b/include/uapi/cxl/cxl_regs.h
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/* Copyright (c) 2026 NVIDIA Corporation & Affiliates */
+
+#ifndef _UAPI_CXL_REGS_H_
+#define _UAPI_CXL_REGS_H_
+
+/*
+ * CXL Component Register layout from the CXL specification. Kept in uapi so a
+ * VMM can consume the register offsets without a kernel header dependency.
+ */
+
+/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */
+#define CXL_COMPONENT_REG_BLOCK_SIZE 0x10000
+
+/* CXL 2.0 8.2.5 CXL.cache and CXL.mem Registers */
+#define CXL_CM_OFFSET 0x1000
+#define CXL_CM_CAP_HDR_OFFSET 0x0
+#define   CXL_CM_CAP_HDR_ID_MASK 0xffff
+#define     CM_CAP_HDR_CAP_ID 1
+#define   CXL_CM_CAP_HDR_VERSION_MASK 0xf0000
+#define     CM_CAP_HDR_CAP_VERSION 1
+#define   CXL_CM_CAP_HDR_CACHE_MEM_VERSION_MASK 0xf00000
+#define     CM_CAP_HDR_CACHE_MEM_VERSION 1
+#define   CXL_CM_CAP_HDR_ARRAY_SIZE_MASK 0xff000000
+#define CXL_CM_CAP_PTR_MASK 0xfff00000
+
+#define   CXL_CM_CAP_CAP_ID_RAS 0x2
+#define   CXL_CM_CAP_CAP_ID_HDM 0x5
+#define   CXL_CM_CAP_CAP_HDM_VERSION 1
+
+/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */
+#define CXL_HDM_DECODER_CAP_OFFSET 0x0
+#define   CXL_HDM_DECODER_COUNT_MASK 0xf
+#define   CXL_HDM_DECODER_TARGET_COUNT_MASK 0xf0
+#define   CXL_HDM_DECODER_INTERLEAVE_11_8 0x100
+#define   CXL_HDM_DECODER_INTERLEAVE_14_12 0x200
+#define   CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY 0x800
+#define   CXL_HDM_DECODER_INTERLEAVE_16_WAY 0x1000
+#define CXL_HDM_DECODER_CTRL_OFFSET 0x4
+#define   CXL_HDM_DECODER_ENABLE 0x2
+#define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10)
+#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i) (0x20 * (i) + 0x14)
+#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i) (0x20 * (i) + 0x18)
+#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i) (0x20 * (i) + 0x1c)
+#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20)
+#define   CXL_HDM_DECODER0_CTRL_IG_MASK 0xf
+#define   CXL_HDM_DECODER0_CTRL_IW_MASK 0xf0
+#define   CXL_HDM_DECODER0_CTRL_LOCK 0x100
+#define   CXL_HDM_DECODER0_CTRL_COMMIT 0x200
+#define   CXL_HDM_DECODER0_CTRL_COMMITTED 0x400
+#define   CXL_HDM_DECODER0_CTRL_COMMIT_ERROR 0x800
+#define   CXL_HDM_DECODER0_CTRL_HOSTONLY 0x1000
+#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
+#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
+#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
+#define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i)
+
+#endif /* _UAPI_CXL_REGS_H_ */
-- 
2.25.1

[PATCH v4 04/27] cxl: Establish media readiness in cxl_mem_probe()

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

media_ready was only ever set by cxl_pci, in advance of registering a
memdev and with CXL Memory Device register assumptions. A consumer that
creates a memdev without cxl_pci, such as a mailbox-less Type-2
accelerator brought up through devm_cxl_probe_mem(), therefore handed
cxl_mem a device with media_ready still false, and cxl_mem_probe()
rejected it with -EBUSY. __devm_cxl_add_memdev() turns that into -ENXIO
back to the caller and the bind fails.

Move the readiness wait into cxl_mem_probe() so every memdev consumer
gets a ready resource regardless of how the memdev was created. When
media_ready is not already set, wait on the device's DVSEC
Mem_Info_Valid and Mem_Active bits and mark it ready. cxl_pci keeps
setting media_ready before it registers its memdev, so that path skips
the wait.

The CXL Memory Device register group is optional and many Type-2 devices
do not implement it, so cxl_await_media_ready() must not read the Memdev
Status register unless the group is mapped. Reading regs.memdev on a
device that lacks it would fault. Gate that read on regs.memdev; the
DVSEC bits already prove readiness for such devices.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/cxl/core/pci.c | 15 +++++++++++----
 drivers/cxl/mem.c      |  9 +++++++--
 2 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
index 08d4c955137d..9b372d5a1aa4 100644
--- a/drivers/cxl/core/pci.c
+++ b/drivers/cxl/core/pci.c
@@ -151,7 +151,6 @@ int cxl_await_media_ready(struct cxl_dev_state *cxlds)
 	struct pci_dev *pdev = to_pci_dev(cxlds->dev);
 	int d = cxlds->cxl_dvsec;
 	int rc, i, hdm_count;
-	u64 md_status;
 	u16 cap;
 
 	rc = pci_read_config_word(pdev,
@@ -172,9 +171,17 @@ int cxl_await_media_ready(struct cxl_dev_state *cxlds)
 			return rc;
 	}
 
-	md_status = readq(cxlds->regs.memdev + CXLMDEV_STATUS_OFFSET);
-	if (!CXLMDEV_READY(md_status))
-		return -EIO;
+	/*
+	 * It is possible some Type-2 devices (CXL_DEVTYPE_DEVMEM) do not
+	 * implement regs.memdev; only consult the Memdev Status register when
+	 * the group is actually present.
+	 */
+	if (cxlds->regs.memdev) {
+		u64 md_status = readq(cxlds->regs.memdev + CXLMDEV_STATUS_OFFSET);
+
+		if (!CXLMDEV_READY(md_status))
+			return -EIO;
+	}
 
 	return 0;
 }
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 798e5c369cfc..9c6e99b9124c 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -105,8 +105,13 @@ static int cxl_mem_probe(struct device *dev)
 	struct dentry *dentry;
 	int rc;
 
-	if (!cxlds->media_ready)
-		return -EBUSY;
+	if (!cxlds->media_ready) {
+		rc = cxl_await_media_ready(cxlds);
+		if (rc)
+			return rc;
+		cxlds->media_ready = true;
+		dev_dbg(dev, "CXL media ready\n");
+	}
 
 	/*
 	 * Someone is trying to reattach this device after it lost its port
-- 
2.25.1

[PATCH v4 05/27] cxl: Add a function-scoped reset entry for vfio-pci

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

vfio-pci needs to run a CXL reset on a passthroughed Type-2 device
without the host-memory handling in cxl_reset_function(): the memory
behind the decoder belongs to the guest.

Add cxl_reset_dvsec_sequence(), which runs the DVSEC reset and HDM decoder
restore under pci_dev_lock and cxl_rwsem, and cxl_reset_capable() to gate
it on a function-scoped reset.

Parameterize Memory Clear in cxl_reset_execute() so the guest owns the
choice; the host path keeps it disabled.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/cxl/core/resource.c | 69 ++++++++++++++++++++++++++++++++++---
 include/cxl/cxl.h           | 12 +++++++
 2 files changed, 76 insertions(+), 5 deletions(-)

diff --git a/drivers/cxl/core/resource.c b/drivers/cxl/core/resource.c
index 492ffe8e3576..cd15bd9171e6 100644
--- a/drivers/cxl/core/resource.c
+++ b/drivers/cxl/core/resource.c
@@ -1164,7 +1164,7 @@ static int cxl_reset_enable_cache(struct pci_dev *pdev, int dvsec)
 				     PCI_DVSEC_CXL_DISABLE_CACHING);
 }
 
-static int cxl_reset_initiate(struct pci_dev *pdev, int dvsec)
+static int cxl_reset_initiate(struct pci_dev *pdev, int dvsec, bool mem_clr_en)
 {
 	u16 ctrl2;
 	int rc;
@@ -1173,7 +1173,10 @@ static int cxl_reset_initiate(struct pci_dev *pdev, int dvsec)
 	if (rc)
 		return rc;
 
-	ctrl2 &= ~PCI_DVSEC_CXL_RST_MEM_CLR_EN;
+	if (mem_clr_en)
+		ctrl2 |= PCI_DVSEC_CXL_RST_MEM_CLR_EN;
+	else
+		ctrl2 &= ~PCI_DVSEC_CXL_RST_MEM_CLR_EN;
 	ctrl2 |= PCI_DVSEC_CXL_INIT_CXL_RST;
 	return cxl_reset_write_ctrl2(pdev, dvsec, ctrl2);
 }
@@ -1273,7 +1276,7 @@ static int cxl_reset_wait_done(struct pci_dev *pdev, int dvsec, u16 cap)
 }
 
 static int cxl_reset_execute(struct pci_dev *pdev, bool *target_prepared,
-			     int dvsec, u16 cap)
+			     int dvsec, u16 cap, bool mem_clr_en)
 {
 	int rc, rc2;
 
@@ -1283,7 +1286,7 @@ static int cxl_reset_execute(struct pci_dev *pdev, bool *target_prepared,
 
 	rc = cxl_pci_target_reset_prepare(pdev, target_prepared);
 	if (!rc)
-		rc = cxl_reset_initiate(pdev, dvsec);
+		rc = cxl_reset_initiate(pdev, dvsec, mem_clr_en);
 	if (!rc)
 		rc = cxl_reset_wait_done(pdev, dvsec, cap);
 
@@ -1322,7 +1325,8 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)
 	scoped_guard(rwsem_write, &cxl_rwsem.region) {
 		rc = cxl_hdm_ranges_prepare(&range_ctx, pdev);
 		if (!rc)
-			rc = cxl_reset_execute(pdev, &target_prepared, dvsec, cap);
+			rc = cxl_reset_execute(pdev, &target_prepared, dvsec,
+					       cap, false);
 		if (!rc) {
 			u16 command;
 
@@ -1340,3 +1344,58 @@ int cxl_reset_function(struct pci_dev *pdev, bool probe)
 	cxl_pci_target_reset_done(pdev, &target_prepared);
 	return rc;
 }
+
+/* True when a function-scoped CXL reset is available for @pdev. */
+bool cxl_reset_capable(struct pci_dev *pdev)
+{
+	u16 cap;
+
+	if (cxl_reset_dvsec(pdev, &cap) < 0)
+		return false;
+
+	if (pdev->multifunction)
+		return false;
+
+	return cxl_reset_hdm_available(pdev);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_reset_capable, "CXL");
+
+/*
+ * Run the DVSEC reset sequence and restore HDM state for a caller that owns
+ * device quiesce and PCI config save/restore, such as vfio-pci. The HDM range
+ * collection and CPU cache flush that cxl_reset_function() does for host-owned
+ * memory are skipped; that memory belongs to the guest here.
+ */
+int cxl_reset_dvsec_sequence(struct pci_dev *pdev, bool mem_clr_en)
+{
+	bool target_prepared = false;
+	int dvsec;
+	int rc;
+	u16 cap;
+
+	dvsec = cxl_reset_dvsec(pdev, &cap);
+	if (dvsec < 0)
+		return dvsec;
+
+	if (pdev->multifunction)
+		return -ENOTTY;
+
+	/*
+	 * Trylock rather than block: This follows the trylock convention of
+	 * pci_reset_bus().
+	 */
+	if (!pci_dev_trylock(pdev))
+		return -EBUSY;
+
+	scoped_guard(rwsem_write, &cxl_rwsem.region) {
+		rc = cxl_reset_execute(pdev, &target_prepared, dvsec, cap,
+				       mem_clr_en);
+		if (!rc)
+			rc = cxl_restore_hdm_after_pci_reset(pdev);
+	}
+
+	cxl_pci_target_reset_done(pdev, &target_prepared);
+	pci_dev_unlock(pdev);
+	return rc;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_reset_dvsec_sequence, "CXL");
diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
index f8e8fddba152..541ed6de75a6 100644
--- a/include/cxl/cxl.h
+++ b/include/cxl/cxl.h
@@ -164,6 +164,8 @@ void pci_cxl_hdm_init(struct pci_dev *pdev);
 void pci_cxl_hdm_release(struct pci_dev *pdev);
 int cxl_restore_hdm_after_pci_reset(struct pci_dev *pdev);
 int cxl_reset_function(struct pci_dev *pdev, bool probe);
+bool cxl_reset_capable(struct pci_dev *pdev);
+int cxl_reset_dvsec_sequence(struct pci_dev *pdev, bool mem_clr_en);
 #else
 static inline void pci_cxl_hdm_init(struct pci_dev *pdev)
 {
@@ -182,6 +184,16 @@ static inline int cxl_reset_function(struct pci_dev *pdev, bool probe)
 {
 	return -ENOTTY;
 }
+
+static inline bool cxl_reset_capable(struct pci_dev *pdev)
+{
+	return false;
+}
+
+static inline int cxl_reset_dvsec_sequence(struct pci_dev *pdev, bool mem_clr_en)
+{
+	return -ENOTTY;
+}
 #endif
 
 struct cxl_reg_map {
-- 
2.25.1

[PATCH v4 06/27] vfio/pci: Add CXL ops registration interface

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

vfio-pci-core must stay free of any CXL header dependency, so CXL
Type-2 handling lives in a separate vfio-cxl module that plugs in a set
of callbacks. Add the registration interface: vfio-cxl registers a
single struct vfio_cxl_ops at module_init, and vfio-pci-core stores it
under a mutex.

The owner field lets a later patch pin vfio-cxl for the lifetime of each
bound CXL device. No caller yet; the detection path is added next.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci_core.c | 27 +++++++++++++++++++++++++++
 include/linux/vfio_pci_core.h    | 10 ++++++++++
 2 files changed, 37 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 3f11a9624b9c..88e68d43af9a 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -2670,6 +2670,33 @@ static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set)
 	}
 }
 
+static const struct vfio_cxl_ops *vfio_pci_cxl_ops;
+static DEFINE_MUTEX(vfio_pci_cxl_ops_lock);
+
+int vfio_pci_core_register_cxl_ops(const struct vfio_cxl_ops *ops)
+{
+	int ret = 0;
+
+	mutex_lock(&vfio_pci_cxl_ops_lock);
+	if (vfio_pci_cxl_ops)
+		ret = -EBUSY;
+	else
+		vfio_pci_cxl_ops = ops;
+	mutex_unlock(&vfio_pci_cxl_ops_lock);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_register_cxl_ops);
+
+void vfio_pci_core_unregister_cxl_ops(const struct vfio_cxl_ops *ops)
+{
+	mutex_lock(&vfio_pci_cxl_ops_lock);
+	if (vfio_pci_cxl_ops == ops)
+		vfio_pci_cxl_ops = NULL;
+	mutex_unlock(&vfio_pci_cxl_ops_lock);
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_unregister_cxl_ops);
+
 static void vfio_pci_core_cleanup(void)
 {
 	vfio_pci_uninit_perm_bits();
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 9a1674c152aa..14753972e714 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -66,6 +66,16 @@ struct vfio_pci_device_ops {
 			       size_t nr_ranges);
 };
 
+struct vfio_cxl_ops {
+	int	(*init_device)(struct vfio_pci_core_device *vdev);
+	void	(*release_device)(struct vfio_pci_core_device *vdev);
+	/* Pinned per bound CXL device so vfio-cxl cannot unload under usage */
+	struct module *owner;
+};
+
+int vfio_pci_core_register_cxl_ops(const struct vfio_cxl_ops *ops);
+void vfio_pci_core_unregister_cxl_ops(const struct vfio_cxl_ops *ops);
+
 #if IS_ENABLED(CONFIG_VFIO_PCI_DMABUF)
 int vfio_pci_core_fill_phys_vec(struct phys_vec *phys_vec,
 				struct vfio_region_dma_range *dma_ranges,
-- 
2.25.1

[PATCH v4 07/27] vfio/pci: Detect CXL devices and load vfio-cxl on demand

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A CXL device needs the vfio-cxl callbacks, but pulling vfio-cxl and the
CXL core in unconditionally would bloat every vfio-pci setup. At bind,
detect a CXL device with pcie_is_cxl() and request_module("vfio-cxl")
only then, and hand the device to the registered ops.

Each bound CXL device pins vfio-cxl through try_module_get() and drops
the reference at release, so vfio-cxl can unload once no CXL device is
bound. If vfio-cxl is absent the device is driven as plain vfio-pci.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci_core.c | 81 ++++++++++++++++++++++++++++++--
 include/linux/vfio_pci_core.h    |  3 ++
 2 files changed, 81 insertions(+), 3 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 88e68d43af9a..0f9b5dfeea66 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -2176,6 +2176,44 @@ static void vfio_pci_vga_uninit(struct vfio_pci_core_device *vdev)
 					      VGA_RSRC_LEGACY_MEM);
 }
 
+static const struct vfio_cxl_ops *vfio_pci_cxl_ops;
+static DEFINE_MUTEX(vfio_pci_cxl_ops_lock);
+
+static const struct vfio_cxl_ops *vfio_pci_get_cxl_ops(void)
+{
+	const struct vfio_cxl_ops *ops;
+
+	mutex_lock(&vfio_pci_cxl_ops_lock);
+	ops = vfio_pci_cxl_ops;
+	if (ops && !try_module_get(ops->owner))
+		ops = NULL;
+	mutex_unlock(&vfio_pci_cxl_ops_lock);
+
+	return ops;
+}
+
+/*
+ * A CXL Type-2 device advertises both CXL.cache and CXL.mem in its CXL DVSEC.
+ * pcie_is_cxl() is also true for Type-1 (cache only) and Type-3 (mem only)
+ * devices, which the vfio-cxl provider does not handle, so confirm the Type-2
+ * identity before engaging it.
+ */
+static bool vfio_pci_is_cxl_type2(struct pci_dev *pdev)
+{
+	u16 dvsec, cap;
+
+	dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_DEVICE);
+	if (!dvsec)
+		return false;
+
+	if (pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap))
+		return false;
+
+	return (cap & PCI_DVSEC_CXL_CACHE_CAPABLE) &&
+	       (cap & PCI_DVSEC_CXL_MEM_CAPABLE);
+}
+
 int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
 {
 	struct vfio_pci_core_device *vdev =
@@ -2197,6 +2235,41 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
 	init_rwsem(&vdev->memory_lock);
 	xa_init(&vdev->ctx);
 
+	/*
+	 * Load vfio-cxl on demand for a CXL device. If it is absent, drive the
+	 * device as plain vfio-pci rather than failing the bind.
+	 */
+	if (pcie_is_cxl(vdev->pdev) && vfio_pci_is_cxl_type2(vdev->pdev)) {
+		const struct vfio_cxl_ops *ops;
+
+		request_module("vfio-cxl");
+		ops = vfio_pci_get_cxl_ops();
+		if (ops) {
+			ret = ops->init_device(vdev);
+			if (ret) {
+				module_put(ops->owner);
+				return ret;
+			}
+			vdev->cxl_ops = ops;
+			/*
+			 * Pin the device in D0 while bound rather than let
+			 * the host power it down between opens.
+			 */
+			vdev->disable_idle_d3 = true;
+		} else if (IS_BUILTIN(CONFIG_VFIO_CXL)) {
+			/*
+			 * Only DEFER for a built-in provider so the bind
+			 * retries once vfio-cxl registers its ops.
+			 * A modular provider was already loaded synchronously
+			 * by request_module() above, so if it is still absent
+			 * it is missing, blocked, or failed to init; drive the
+			 * device as plain vfio-pci then rather than defer the
+			 * bind forever.
+			 */
+			return -EPROBE_DEFER;
+		}
+	}
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(vfio_pci_core_init_dev);
@@ -2206,6 +2279,11 @@ void vfio_pci_core_release_dev(struct vfio_device *core_vdev)
 	struct vfio_pci_core_device *vdev =
 		container_of(core_vdev, struct vfio_pci_core_device, vdev);
 
+	if (vdev->cxl_ops) {
+		vdev->cxl_ops->release_device(vdev);
+		module_put(vdev->cxl_ops->owner);
+	}
+
 	mutex_destroy(&vdev->igate);
 	mutex_destroy(&vdev->ioeventfds_lock);
 	kfree(vdev->region);
@@ -2670,9 +2748,6 @@ static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set)
 	}
 }
 
-static const struct vfio_cxl_ops *vfio_pci_cxl_ops;
-static DEFINE_MUTEX(vfio_pci_cxl_ops_lock);
-
 int vfio_pci_core_register_cxl_ops(const struct vfio_cxl_ops *ops)
 {
 	int ret = 0;
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 14753972e714..117cd67995d8 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -29,6 +29,7 @@ struct vfio_pci_core_device;
 struct vfio_pci_region;
 struct p2pdma_provider;
 struct dma_buf_attachment;
+struct vfio_cxl_state;
 
 struct vfio_pci_eventfd {
 	struct eventfd_ctx	*ctx;
@@ -109,6 +110,8 @@ struct vfio_pci_core_device {
 	struct vfio_device	vdev;
 	struct pci_dev		*pdev;
 	const struct vfio_pci_device_ops *pci_ops;
+	const struct vfio_cxl_ops *cxl_ops;
+	struct vfio_cxl_state	*cxl;
 	void __iomem		*barmap[PCI_STD_NUM_BARS];
 	bool			bar_mmap_supported[PCI_STD_NUM_BARS];
 	/* Flags modified at runtime - dedicated storage unit */
-- 
2.25.1

[PATCH v4 08/27] vfio/cxl: Add the vfio-cxl module skeleton

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Add the vfio-cxl module that vfio-pci loads on demand for CXL Type-2
devices. It registers a set of callbacks with vfio-pci-core at
module_init and drops them at exit; the callbacks are stubs here and
are filled in by later patches.

VFIO_CXL depends on CXL_BUS and builds as vfio-cxl.ko, which
request_module("vfio-cxl") resolves through MODULE_ALIAS.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 MAINTAINERS                          |  7 +++++
 drivers/vfio/pci/Kconfig             |  2 ++
 drivers/vfio/pci/Makefile            |  2 ++
 drivers/vfio/pci/cxl/Kconfig         | 11 ++++++++
 drivers/vfio/pci/cxl/Makefile        |  3 ++
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 41 ++++++++++++++++++++++++++++
 6 files changed, 66 insertions(+)
 create mode 100644 drivers/vfio/pci/cxl/Kconfig
 create mode 100644 drivers/vfio/pci/cxl/Makefile
 create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_core.c

diff --git a/MAINTAINERS b/MAINTAINERS
index c665bf8f1a28..aa8cecbc4cf2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -28312,6 +28312,13 @@ L:	kvm@vger.kernel.org
 S:	Maintained
 F:	drivers/vfio/cdx/*
 
+VFIO CXL DRIVER
+M:	Manish Honap <mhonap@nvidia.com>
+L:	kvm@vger.kernel.org
+L:	linux-cxl@vger.kernel.org
+S:	Supported
+F:	drivers/vfio/pci/cxl/
+
 VFIO DRIVER
 M:	Alex Williamson <alex@shazbot.org>
 L:	kvm@vger.kernel.org
diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig
index 296bf01e185e..c3f54fc8ccc0 100644
--- a/drivers/vfio/pci/Kconfig
+++ b/drivers/vfio/pci/Kconfig
@@ -70,6 +70,8 @@ source "drivers/vfio/pci/virtio/Kconfig"
 
 source "drivers/vfio/pci/nvgrace-gpu/Kconfig"
 
+source "drivers/vfio/pci/cxl/Kconfig"
+
 source "drivers/vfio/pci/qat/Kconfig"
 
 source "drivers/vfio/pci/xe/Kconfig"
diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
index 6138f1bf241d..c3031070a1f3 100644
--- a/drivers/vfio/pci/Makefile
+++ b/drivers/vfio/pci/Makefile
@@ -21,6 +21,8 @@ obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/
 
 obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu/
 
+obj-$(CONFIG_VFIO_CXL) += cxl/
+
 obj-$(CONFIG_QAT_VFIO_PCI) += qat/
 
 obj-$(CONFIG_XE_VFIO_PCI) += xe/
diff --git a/drivers/vfio/pci/cxl/Kconfig b/drivers/vfio/pci/cxl/Kconfig
new file mode 100644
index 000000000000..1392fa4ed15d
--- /dev/null
+++ b/drivers/vfio/pci/cxl/Kconfig
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config VFIO_CXL
+	tristate "VFIO support for CXL Type-2 devices"
+	depends on CXL_BUS && CXL_MEM
+	select VFIO_PCI_CORE
+	help
+	  Add CXL HDM handling to vfio-pci so a CXL Type-2 device, such as a
+	  CXL-attached accelerator, can be assigned to a virtual machine.
+	  vfio-pci loads this module on demand when it binds a CXL device.
+
+	  If you don't know what to do here, say N.
diff --git a/drivers/vfio/pci/cxl/Makefile b/drivers/vfio/pci/cxl/Makefile
new file mode 100644
index 000000000000..8501bf626108
--- /dev/null
+++ b/drivers/vfio/pci/cxl/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_VFIO_CXL) += vfio-cxl.o
+vfio-cxl-y := vfio_cxl_core.o
diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
new file mode 100644
index 000000000000..cbec7319452c
--- /dev/null
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -0,0 +1,41 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * VFIO support for CXL Type-2 devices.
+ *
+ * Copyright (c) 2026 NVIDIA Corporation & Affiliates
+ */
+
+#include <linux/module.h>
+#include <linux/vfio_pci_core.h>
+
+static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
+{
+	return 0;
+}
+
+static void vfio_cxl_release_device(struct vfio_pci_core_device *vdev)
+{
+}
+
+static const struct vfio_cxl_ops vfio_cxl_ops = {
+	.init_device	= vfio_cxl_init_device,
+	.release_device	= vfio_cxl_release_device,
+	.owner		= THIS_MODULE,
+};
+
+static int __init vfio_cxl_init(void)
+{
+	return vfio_pci_core_register_cxl_ops(&vfio_cxl_ops);
+}
+
+static void __exit vfio_cxl_exit(void)
+{
+	vfio_pci_core_unregister_cxl_ops(&vfio_cxl_ops);
+}
+
+module_init(vfio_cxl_init);
+module_exit(vfio_cxl_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("VFIO support for CXL Type-2 devices");
+MODULE_ALIAS("vfio-cxl");
-- 
2.25.1

[PATCH v4 09/27] vfio/cxl: Create the CXL memory device at bind

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Read the HDM region size from decoder 0, cached in pdev->hdm at
enumeration, and use it to size the accelerator device state. A Type-2
accelerator has no mailbox, so cxl_set_capacity() establishes the
capacity that devm_cxl_probe_mem() then uses to join the device to the
CXL topology and resolve the host physical range.

Everything here is devm-scoped to the PCI device, so it is unwound when
vfio-pci unbinds. If pdev->hdm is not populated yet, defer the bind.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 53 ++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index cbec7319452c..125e11354a46 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -6,15 +6,67 @@
  */
 
 #include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/range.h>
 #include <linux/vfio_pci_core.h>
+#include <cxl/cxl.h>
+
+/**
+ * struct vfio_cxl_state - per-device state for a vfio-cxl device
+ * @cxlds: CXL device state; kept first for devm_cxl_dev_state_create()
+ * @cxlmd: memory device joined to the CXL topology at bind
+ * @hpa_range: host physical range of the HDM region
+ */
+struct vfio_cxl_state {
+	struct cxl_dev_state cxlds;
+	struct cxl_memdev *cxlmd;
+	struct range hpa_range;
+};
 
 static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
 {
+	struct pci_dev *pdev = vdev->pdev;
+	struct vfio_cxl_state *cxl;
+	struct cxl_memdev *cxlmd;
+	u64 hdm_size, serial;
+	u16 dvsec;
+	int ret;
+
+	/* pdev->hdm is populated at PCI enumeration; defer until it is. */
+	if (!pdev->hdm)
+		return -EPROBE_DEFER;
+
+	hdm_size = range_len(&pdev->hdm->settings[0].hpa_range);
+	if (!hdm_size)
+		return -ENXIO;
+
+	dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
+					  PCI_DVSEC_CXL_DEVICE);
+	serial = pci_get_dsn(pdev);
+
+	cxl = devm_cxl_dev_state_create(&pdev->dev, CXL_DEVTYPE_DEVMEM, serial,
+					dvsec, struct vfio_cxl_state, cxlds,
+					false);
+	if (!cxl)
+		return -ENOMEM;
+
+	ret = cxl_set_capacity(&cxl->cxlds, hdm_size);
+	if (ret)
+		return ret;
+
+	cxlmd = devm_cxl_probe_mem(&cxl->cxlds, &cxl->hpa_range);
+	if (IS_ERR(cxlmd))
+		return PTR_ERR(cxlmd);
+
+	cxl->cxlmd = cxlmd;
+	vdev->cxl = cxl;
+
 	return 0;
 }
 
 static void vfio_cxl_release_device(struct vfio_pci_core_device *vdev)
 {
+	vdev->cxl = NULL;
 }
 
 static const struct vfio_cxl_ops vfio_cxl_ops = {
@@ -39,3 +91,4 @@ module_exit(vfio_cxl_exit);
 MODULE_LICENSE("GPL");
 MODULE_DESCRIPTION("VFIO support for CXL Type-2 devices");
 MODULE_ALIAS("vfio-cxl");
+MODULE_IMPORT_NS("CXL");
-- 
2.25.1

[PATCH v4 10/27] vfio/cxl: Reject unsupported decoder topologies at bind

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

The guest drives a single virtual decoder whose memory maps 1:1, so a
device with more than one HDM decoder, or an interleaved one, cannot be
represented. Turn that assumption into an explicit refusal at bind rather
than a silent misprogramming later.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 125e11354a46..7edc53b25576 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -36,10 +36,30 @@ static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
 	if (!pdev->hdm)
 		return -EPROBE_DEFER;
 
+	/* The guest drives one virtual decoder; multiple are unsupported. */
+	if (pdev->hdm->decoder_count != 1)
+		return -EOPNOTSUPP;
+
 	hdm_size = range_len(&pdev->hdm->settings[0].hpa_range);
 	if (!hdm_size)
 		return -ENXIO;
 
+	/* Interleaved decoders are unsupported. */
+	if (pdev->hdm->settings[0].interleave_ways != 1)
+		return -EOPNOTSUPP;
+
+	/*
+	 * The guest drives resets through the CXL Device DVSEC and polls the
+	 * shadow for completion. If the host cannot service a function-scoped
+	 * CXL reset (no reset DVSEC, a multifunction device, or no HDM reset
+	 * support), that guest request could never complete, so refuse the
+	 * device rather than advertise a reset the guest would poll on forever.
+	 */
+	if (!cxl_reset_capable(pdev)) {
+		pci_err(pdev, "vfio-cxl: Unsupported device: host cannot service a CXL reset request\n");
+		return -EOPNOTSUPP;
+	}
+
 	dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
 					  PCI_DVSEC_CXL_DEVICE);
 	serial = pci_get_dsn(pdev);
-- 
2.25.1

[PATCH v4 11/27] vfio/cxl: Own the whole component register BAR

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

devm_cxl_probe_mem() maps the component registers through the CXL core.
By default the core claims each HDM/RAS sub-block, which then collides
with the full-BAR request that vfio-pci makes when the guest opens the
device.

Discover the component registers into the device reg_map and pass
bar_owned to cxl_pci_setup_regs(), so the core maps the sub-blocks
without claiming them and vfio-pci owns the whole BAR.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 7edc53b25576..966b7c22a6ac 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -10,6 +10,7 @@
 #include <linux/range.h>
 #include <linux/vfio_pci_core.h>
 #include <cxl/cxl.h>
+#include <cxl/pci.h>
 
 /**
  * struct vfio_cxl_state - per-device state for a vfio-cxl device
@@ -70,6 +71,16 @@ static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
 	if (!cxl)
 		return -ENOMEM;
 
+	/*
+	 * vfio-pci requests the whole component BAR when the guest opens the
+	 * device. Declare the BAR owned so the CXL core maps the HDM/RAS
+	 * sub-blocks without claiming them and that request does not collide.
+	 */
+	ret = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_COMPONENT,
+				 &cxl->cxlds.reg_map, true);
+	if (ret)
+		return ret;
+
 	ret = cxl_set_capacity(&cxl->cxlds, hdm_size);
 	if (ret)
 		return ret;
-- 
2.25.1

[PATCH v4 12/27] vfio/pci: Let a provider exclude a BAR sub-range from mmap

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Some devices expose registers in a BAR that must be reached only through
a trap, not a direct guest mapping. A CXL Type-2 device's HDM decoder
block is one: mapping it would let userspace reprogram the physical
decoder that governs host memory decode. Give a provider a way to mark a
BAR sub-range off-limits to mmap; it is advertised as a sparse-mmap
region and refused in the mmap path, while the provider's own region
still serves it.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci_core.c   | 72 ++++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_dmabuf.c | 13 ++++++
 drivers/vfio/pci/vfio_pci_priv.h   | 13 ++++++
 include/linux/vfio_pci_core.h      |  6 +++
 4 files changed, 104 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 0f9b5dfeea66..49dfbdaf3f05 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1009,6 +1009,67 @@ static int msix_mmappable_cap(struct vfio_pci_core_device *vdev,
 	return vfio_info_add_capability(caps, &header, sizeof(header));
 }
 
+/*
+ * A provider can keep a BAR sub-range off mmap (for example a CXL device's
+ * trapped HDM decoder block). Callers hold the resource so /dev/mem is already
+ * blocked; this only governs the vfio mmap path.
+ */
+void vfio_pci_core_set_mmap_exclude(struct vfio_pci_core_device *vdev, int bar,
+				    u64 start, u64 len)
+{
+	vdev->mmap_exclude_bar = bar;
+	vdev->mmap_exclude_start = start;
+	vdev->mmap_exclude_len = len;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_set_mmap_exclude);
+
+/* Advertise the BAR as mmappable minus the excluded sub-range. */
+static int vfio_pci_mmap_exclude_cap(struct vfio_pci_core_device *vdev,
+				     int index, struct vfio_info_cap *caps)
+{
+	u64 bar_len = pci_resource_len(vdev->pdev, index);
+	u64 excl_start = ALIGN_DOWN(vdev->mmap_exclude_start, PAGE_SIZE);
+	u64 excl_end = ALIGN(vdev->mmap_exclude_start + vdev->mmap_exclude_len,
+			     PAGE_SIZE);
+	struct vfio_region_info_cap_sparse_mmap *sparse;
+	int nr_areas = 0, i = 0, ret;
+	size_t size;
+
+	/*
+	 * mmap is page granular, so the mmappable areas must stop at the page
+	 * boundaries enclosing the excluded sub-range. The byte-granular
+	 * exclusion still governs the fault and read/write paths; only the
+	 * advertised mmap areas round out to whole pages.
+	 */
+	if (excl_start > 0)
+		nr_areas++;
+	if (excl_end < bar_len)
+		nr_areas++;
+
+	size = struct_size(sparse, areas, nr_areas);
+	sparse = kzalloc(size, GFP_KERNEL);
+	if (!sparse)
+		return -ENOMEM;
+
+	sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP;
+	sparse->header.version = 1;
+	sparse->nr_areas = nr_areas;
+
+	if (excl_start > 0) {
+		sparse->areas[i].offset = 0;
+		sparse->areas[i].size = excl_start;
+		i++;
+	}
+	if (excl_end < bar_len) {
+		sparse->areas[i].offset = excl_end;
+		sparse->areas[i].size = bar_len - excl_end;
+	}
+
+	ret = vfio_info_add_capability(caps, &sparse->header, size);
+	kfree(sparse);
+	return ret;
+}
+
 int vfio_pci_core_register_dev_region(struct vfio_pci_core_device *vdev,
 				      unsigned int type, unsigned int subtype,
 				      const struct vfio_pci_regops *ops,
@@ -1157,6 +1218,13 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
 				if (ret)
 					return ret;
 			}
+			if (vdev->mmap_exclude_len &&
+			    info->index == vdev->mmap_exclude_bar) {
+				ret = vfio_pci_mmap_exclude_cap(vdev, info->index,
+								caps);
+				if (ret)
+					return ret;
+			}
 		}
 
 		break;
@@ -1851,6 +1919,10 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma
 	if (req_start + req_len > phys_len)
 		return -EINVAL;
 
+	/* An excluded sub-range is reachable only through its trap, not mmap. */
+	if (vfio_pci_bar_is_excluded(vdev, index, req_start, req_len))
+		return -EINVAL;
+
 	/*
 	 * Ensure the BAR resource region is reserved for use.
 	 */
diff --git a/drivers/vfio/pci/vfio_pci_dmabuf.c b/drivers/vfio/pci/vfio_pci_dmabuf.c
index c16f460c01d6..51983105d38b 100644
--- a/drivers/vfio/pci/vfio_pci_dmabuf.c
+++ b/drivers/vfio/pci/vfio_pci_dmabuf.c
@@ -177,11 +177,24 @@ int vfio_pci_core_get_dmabuf_phys(struct vfio_pci_core_device *vdev,
 				  size_t nr_ranges)
 {
 	struct pci_dev *pdev = vdev->pdev;
+	unsigned int i;
 
 	*provider = pcim_p2pdma_provider(pdev, region_index);
 	if (!*provider)
 		return -EINVAL;
 
+	/*
+	 * A provider (e.g. vfio-cxl) can exclude a BAR sub-range that must be
+	 * reached only through its trap. The mmap and read/write paths already
+	 * refuse it; reject a DMA-BUF export overlapping it too, so a device fd
+	 * holder cannot map the excluded registers to a peer and bypass the trap.
+	 */
+	for (i = 0; i < nr_ranges; i++)
+		if (vfio_pci_bar_is_excluded(vdev, region_index,
+					     dma_ranges[i].offset,
+					     dma_ranges[i].length))
+			return -EINVAL;
+
 	return vfio_pci_core_fill_phys_vec(
 		phys_vec, dma_ranges, nr_ranges,
 		pci_resource_start(pdev, region_index),
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index fca9d0dfac90..902d17815ab6 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -44,6 +44,19 @@ ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev,
 ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 			size_t count, loff_t *ppos, bool iswrite);
 
+/*
+ * A provider (e.g. vfio-cxl) can carve a sub-range out of a BAR that must be
+ * reached only through its trap, never the direct BAR.  Returns true when
+ * [start, start + len) on this BAR overlaps that excluded range.
+ */
+static inline bool vfio_pci_bar_is_excluded(struct vfio_pci_core_device *vdev,
+					    int bar, u64 start, u64 len)
+{
+	return vdev->mmap_exclude_len && bar == vdev->mmap_exclude_bar &&
+	       start < vdev->mmap_exclude_start + vdev->mmap_exclude_len &&
+	       start + len > vdev->mmap_exclude_start;
+}
+
 #ifdef CONFIG_VFIO_PCI_VGA
 ssize_t vfio_pci_vga_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 			size_t count, loff_t *ppos, bool iswrite);
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 117cd67995d8..43755b91880f 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -162,6 +162,10 @@ struct vfio_pci_core_device {
 	struct notifier_block	nb;
 	struct rw_semaphore	memory_lock;
 	struct list_head	dmabufs;
+	/* BAR sub-range a provider keeps off mmap, reached only through a trap */
+	int			mmap_exclude_bar;
+	u64			mmap_exclude_start;
+	u64			mmap_exclude_len;
 };
 
 enum vfio_pci_io_width {
@@ -176,6 +180,8 @@ int vfio_pci_core_register_dev_region(struct vfio_pci_core_device *vdev,
 				      unsigned int type, unsigned int subtype,
 				      const struct vfio_pci_regops *ops,
 				      size_t size, u32 flags, void *data);
+void vfio_pci_core_set_mmap_exclude(struct vfio_pci_core_device *vdev, int bar,
+				    u64 start, u64 len);
 void vfio_pci_core_close_device(struct vfio_device *core_vdev);
 int vfio_pci_core_init_dev(struct vfio_device *core_vdev);
 void vfio_pci_core_release_dev(struct vfio_device *core_vdev);
-- 
2.25.1

[PATCH v4 13/27] vfio/pci: Refuse read/write to an excluded BAR sub-range

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Excluding a BAR sub-range from mmap is not enough on its own: the region
read/write path still reaches the underlying registers. Refuse an access
that overlaps the excluded range there too, so the only way in is the
provider's trap.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci_rdwr.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_rdwr.c b/drivers/vfio/pci/vfio_pci_rdwr.c
index 7f14dd46de17..2ce18e75a5ae 100644
--- a/drivers/vfio/pci/vfio_pci_rdwr.c
+++ b/drivers/vfio/pci/vfio_pci_rdwr.c
@@ -223,6 +223,10 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 
 	count = min(count, (size_t)(end - pos));
 
+	/* An excluded sub-range is reached only through its trap. */
+	if (vfio_pci_bar_is_excluded(vdev, bar, pos, count))
+		return -EINVAL;
+
 	if (bar == PCI_ROM_RESOURCE) {
 		/*
 		 * The ROM can fill less space than the BAR, so we start the
@@ -423,6 +427,10 @@ int vfio_pci_ioeventfd(struct vfio_pci_core_device *vdev, loff_t offset,
 	      pos >= vdev->msix_offset + vdev->msix_size))
 		return -EINVAL;
 
+	/* An excluded sub-range is reached only through its trap. */
+	if (vfio_pci_bar_is_excluded(vdev, bar, pos, count))
+		return -EINVAL;
+
 	if (count == 8)
 		return -EINVAL;
 
-- 
2.25.1

[PATCH v4 14/27] vfio: Add CXL region type for the HDM region

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A CXL Type-2 device exposes its HDM memory to the guest as a dedicated
VFIO region rather than a PCI BAR. Add a CXL region type and a memory
sub-type for it; vfio-cxl registers the window under these so userspace
can discover and map it.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 include/uapi/linux/vfio.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 5de618a3a5ee..540e62312671 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -345,6 +345,7 @@ struct vfio_region_info_cap_type {
 #define VFIO_REGION_TYPE_GFX                    (1)
 #define VFIO_REGION_TYPE_CCW			(2)
 #define VFIO_REGION_TYPE_MIGRATION_DEPRECATED   (3)
+#define VFIO_REGION_TYPE_CXL			(4)
 
 /* sub-types for VFIO_REGION_TYPE_PCI_* */
 
@@ -373,6 +374,10 @@ struct vfio_region_info_cap_type {
 /* sub-types for VFIO_REGION_TYPE_GFX */
 #define VFIO_REGION_SUBTYPE_GFX_EDID            (1)
 
+/* sub-types for VFIO_REGION_TYPE_CXL */
+/* CXL.mem HDM region of a Type-2 device, mmap-able */
+#define VFIO_REGION_SUBTYPE_CXL_MEM		(1)
+
 /**
  * struct vfio_region_gfx_edid - EDID region layout.
  *
-- 
2.25.1

[PATCH v4 15/27] vfio/pci: Call CXL open and close hooks around device use

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Volatile CXL register state must be sampled from live hardware when the
guest opens the device, not at bind, because a low-power transition
between bind and open can leave bind-time values stale.

Add open_device and close_device to the CXL ops and call them from the
common enable and close paths. A failed open unwinds the enable like any
other error. The callbacks are stubs here and are filled in next.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 11 +++++++
 drivers/vfio/pci/vfio_pci_core.c     | 44 ++++++++++++++++++++++------
 include/linux/vfio_pci_core.h        |  2 ++
 3 files changed, 48 insertions(+), 9 deletions(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 966b7c22a6ac..d19fd638f538 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -100,9 +100,20 @@ static void vfio_cxl_release_device(struct vfio_pci_core_device *vdev)
 	vdev->cxl = NULL;
 }
 
+static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
+{
+	return 0;
+}
+
+static void vfio_cxl_close_device(struct vfio_pci_core_device *vdev)
+{
+}
+
 static const struct vfio_cxl_ops vfio_cxl_ops = {
 	.init_device	= vfio_cxl_init_device,
 	.release_device	= vfio_cxl_release_device,
+	.open_device	= vfio_cxl_open_device,
+	.close_device	= vfio_cxl_close_device,
 	.owner		= THIS_MODULE,
 };
 
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 49dfbdaf3f05..470730cdc88b 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -584,6 +584,21 @@ static const struct dev_pm_ops vfio_pci_core_pm_ops = {
 			   NULL)
 };
 
+static void vfio_pci_core_unmap_bars(struct vfio_pci_core_device *vdev)
+{
+	struct pci_dev *pdev = vdev->pdev;
+	int i, bar;
+
+	for (i = 0; i < PCI_STD_NUM_BARS; i++) {
+		bar = i + PCI_STD_RESOURCES;
+		if (IS_ERR_OR_NULL(vdev->barmap[bar]))
+			continue;
+		pci_iounmap(pdev, vdev->barmap[bar]);
+		pci_release_selected_regions(pdev, 1 << bar);
+		vdev->barmap[bar] = NULL;
+	}
+}
+
 int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 {
 	struct pci_dev *pdev = vdev->pdev;
@@ -660,8 +675,23 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 
 	vfio_pci_core_map_bars(vdev);
 
+	if (vdev->cxl_ops) {
+		ret = vdev->cxl_ops->open_device(vdev);
+		if (ret)
+			goto out_free_config;
+	}
+
 	return 0;
 
+out_free_config:
+	/*
+	 * open_device() runs after vfio_config_init() and map_bars() have
+	 * succeeded, but a failed first open never reaches vfio_pci_core_disable().
+	 * Unwind the common vconfig and BAR state here so the allocations and BAR
+	 * requests are not leaked for a later open to overwrite.
+	 */
+	vfio_config_free(vdev);
+	vfio_pci_core_unmap_bars(vdev);
 out_free_zdev:
 	vfio_pci_zdev_close_device(vdev);
 out_free_state:
@@ -682,7 +712,7 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
 	struct pci_dev *pdev = vdev->pdev;
 	struct vfio_pci_dummy_resource *dummy_res, *tmp;
 	struct vfio_pci_ioeventfd *ioeventfd, *ioeventfd_tmp;
-	int i, bar;
+	int i;
 
 	/* For needs_reset */
 	lockdep_assert_held(&vdev->vdev.dev_set->lock);
@@ -737,14 +767,7 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
 
 	vfio_config_free(vdev);
 
-	for (i = 0; i < PCI_STD_NUM_BARS; i++) {
-		bar = i + PCI_STD_RESOURCES;
-		if (IS_ERR_OR_NULL(vdev->barmap[bar]))
-			continue;
-		pci_iounmap(pdev, vdev->barmap[bar]);
-		pci_release_selected_regions(pdev, 1 << bar);
-		vdev->barmap[bar] = NULL;
-	}
+	vfio_pci_core_unmap_bars(vdev);
 
 	list_for_each_entry_safe(dummy_res, tmp,
 				 &vdev->dummy_resources_list, res_next) {
@@ -827,6 +850,9 @@ void vfio_pci_core_close_device(struct vfio_device *core_vdev)
 #endif
 	vfio_pci_dma_buf_cleanup(vdev);
 
+	if (vdev->cxl_ops)
+		vdev->cxl_ops->close_device(vdev);
+
 	vfio_pci_core_disable(vdev);
 
 	mutex_lock(&vdev->igate);
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 43755b91880f..7354dae1dd85 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -70,6 +70,8 @@ struct vfio_pci_device_ops {
 struct vfio_cxl_ops {
 	int	(*init_device)(struct vfio_pci_core_device *vdev);
 	void	(*release_device)(struct vfio_pci_core_device *vdev);
+	int     (*open_device)(struct vfio_pci_core_device *vdev);
+	void    (*close_device)(struct vfio_pci_core_device *vdev);
 	/* Pinned per bound CXL device so vfio-cxl cannot unload under usage */
 	struct module *owner;
 };
-- 
2.25.1

[PATCH v4 16/27] vfio/cxl: Shadow the CXL DVSEC body at open

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Sample the CXL DVSEC body into a per-open shadow when the guest opens the
device, and free it at close. Reading it here rather than at bind picks
up any change from a low-power transition, and gives the DVSEC access
handler added next a per-tenant copy to serve from.

Annotate the shadow with __counted_by_ptr(dvsec_dwords) so its accesses
are bounds-checked against the recorded dword count.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 39 ++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index d19fd638f538..2e516a0929c6 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -8,6 +8,7 @@
 #include <linux/module.h>
 #include <linux/pci.h>
 #include <linux/range.h>
+#include <linux/slab.h>
 #include <linux/vfio_pci_core.h>
 #include <cxl/cxl.h>
 #include <cxl/pci.h>
@@ -17,11 +18,19 @@
  * @cxlds: CXL device state; kept first for devm_cxl_dev_state_create()
  * @cxlmd: memory device joined to the CXL topology at bind
  * @hpa_range: host physical range of the HDM region
+ * @dvsec: CXL device DVSEC config-space offset
+ * @dvsec_len: length of the DVSEC body
+ * @dvsec_dwords: dword count of @dvsec_shadow
+ * @dvsec_shadow: guest view of the CXL DVSEC body, sampled at open
  */
 struct vfio_cxl_state {
 	struct cxl_dev_state cxlds;
 	struct cxl_memdev *cxlmd;
 	struct range hpa_range;
+	u16 dvsec;
+	u32 dvsec_len;
+	u32 dvsec_dwords;
+	u32 *dvsec_shadow __counted_by_ptr(dvsec_dwords);
 };
 
 static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
@@ -71,6 +80,8 @@ static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
 	if (!cxl)
 		return -ENOMEM;
 
+	cxl->dvsec = dvsec;
+
 	/*
 	 * vfio-pci requests the whole component BAR when the guest opens the
 	 * device. Declare the BAR owned so the CXL core maps the HDM/RAS
@@ -102,11 +113,39 @@ static void vfio_cxl_release_device(struct vfio_pci_core_device *vdev)
 
 static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
 {
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	struct pci_dev *pdev = vdev->pdev;
+	u32 hdr, *shadow;
+	int i, dwords;
+
+	/*
+	 * Sample the DVSEC body now rather than at bind: a low-power
+	 * transition could have changed it since the device was bound.
+	 */
+	pci_read_config_dword(pdev, cxl->dvsec + PCI_DVSEC_HEADER1, &hdr);
+	cxl->dvsec_len = PCI_DVSEC_HEADER1_LEN(hdr);
+	dwords = cxl->dvsec_len / sizeof(u32);
+
+	shadow = kcalloc(dwords, sizeof(u32), GFP_KERNEL);
+	if (!shadow)
+		return -ENOMEM;
+
+	for (i = 0; i < dwords; i++)
+		pci_read_config_dword(pdev, cxl->dvsec + i * sizeof(u32),
+				      &shadow[i]);
+
+	cxl->dvsec_dwords = dwords;
+	cxl->dvsec_shadow = shadow;
+
 	return 0;
 }
 
 static void vfio_cxl_close_device(struct vfio_pci_core_device *vdev)
 {
+	struct vfio_cxl_state *cxl = vdev->cxl;
+
+	kfree(cxl->dvsec_shadow);
+	cxl->dvsec_shadow = NULL;
 }
 
 static const struct vfio_cxl_ops vfio_cxl_ops = {
-- 
2.25.1

[PATCH v4 17/27] vfio/cxl: Virtualize the CXL DVSEC

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Serve reads of the CXL DVSEC body from the per-open shadow and keep guest
writes in the shadow rather than letting them reach the hardware, so a
guest cannot reprogram the device through the DVSEC. Accesses outside the
CXL DVSEC return -ENODEV and take the default DVSEC handling, so a device
that also exposes a vendor DVSEC is unaffected.

Route each shadow write through the CXL r4.0 field class rather than
storing it verbatim: Control stays programmable, Status is
write-1-to-clear, and Capability, Lock and the Range registers keep their
firmware snapshot. The guest can no longer set Config Lock or scribble the
capability and range fields.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 88 ++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_config.c   | 36 +++++++++++-
 include/linux/vfio_pci_core.h        |  5 ++
 include/uapi/linux/pci_regs.h        |  1 +
 4 files changed, 129 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 2e516a0929c6..9fed909cb9d3 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -148,11 +148,99 @@ static void vfio_cxl_close_device(struct vfio_pci_core_device *vdev)
 	cxl->dvsec_shadow = NULL;
 }
 
+/* Read a 16-bit DVSEC field from the shadow; @off is DVSEC-relative. */
+static u16 vfio_cxl_dvsec16(struct vfio_cxl_state *cxl, u32 off)
+{
+	u32 dw = cxl->dvsec_shadow[off / sizeof(u32)];
+
+	return (dw >> (8 * (off % sizeof(u32)))) & 0xffff;
+}
+
+/*
+ * Apply the CXL r4.0 8.1.3 write class for the 16-bit DVSEC register at @off.
+ * Control is programmable, Status is write-1-to-clear, and Capability, Lock and
+ * the Range registers stay fixed at their firmware snapshot.
+ */
+static u16 vfio_cxl_dvsec_field(u32 off, u16 old, u16 wval, u16 wmask)
+{
+	switch (off) {
+	case PCI_DVSEC_CXL_CTRL:
+		/*
+		 * CXL.mem stays enabled for as long as the guest owns the device.
+		 * The HDM decoder maps the guest window to device memory, so a
+		 * store to it while CXL.mem is disabled completes on the device as
+		 * an error that the host fabric reports as an SError, which is
+		 * fatal. The spec does not pin down accesses to a decoder whose
+		 * CXL.mem is off and many hosts SError, so ignore a guest request
+		 * to clear the enable and keep the bit set.
+		 */
+		return ((old & ~wmask) | (wval & wmask)) | PCI_DVSEC_CXL_MEM_ENABLE;
+	case PCI_DVSEC_CXL_CTRL2:
+		return (old & ~wmask) | (wval & wmask);
+	case PCI_DVSEC_CXL_STATUS:
+	case PCI_DVSEC_CXL_STATUS2:
+		return old & ~(wval & wmask);
+	default:
+		return old;
+	}
+}
+
+/* Config accesses never cross a dword, so a single shadow entry covers them. */
+static int vfio_cxl_config_read(struct vfio_pci_core_device *vdev, int pos,
+				int count, __le32 *val)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	int boff = (pos - cxl->dvsec) % sizeof(u32);
+	__le32 dword;
+
+	if (pos < cxl->dvsec || pos >= cxl->dvsec + cxl->dvsec_len)
+		return -ENODEV;
+
+	dword = cpu_to_le32(cxl->dvsec_shadow[(pos - cxl->dvsec) / sizeof(u32)]);
+	memcpy(val, (u8 *)&dword + boff, count);
+
+	return count;
+}
+
+static int vfio_cxl_config_write(struct vfio_pci_core_device *vdev, int pos,
+				 int count, __le32 val)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	int idx = (pos - cxl->dvsec) / sizeof(u32);
+	int boff = (pos - cxl->dvsec) % sizeof(u32);
+	u32 off = idx * sizeof(u32);
+	__le32 le_wval = 0, le_wmask = 0;
+	u32 old, wval, wmask;
+	u16 lo, hi;
+
+	if (pos < cxl->dvsec || pos >= cxl->dvsec + cxl->dvsec_len)
+		return -ENODEV;
+
+	/*
+	 * Place the guest bytes and a matching byte mask at the write offset,
+	 * then let the per-field class decide what actually lands in the shadow.
+	 * The hardware is never touched.
+	 */
+	memcpy((u8 *)&le_wval + boff, &val, count);
+	memset((u8 *)&le_wmask + boff, 0xff, count);
+	old = cxl->dvsec_shadow[idx];
+	wval = le32_to_cpu(le_wval);
+	wmask = le32_to_cpu(le_wmask);
+
+	lo = vfio_cxl_dvsec_field(off, old, wval, wmask);
+	hi = vfio_cxl_dvsec_field(off + 2, old >> 16, wval >> 16, wmask >> 16);
+	cxl->dvsec_shadow[idx] = lo | ((u32)hi << 16);
+
+	return count;
+}
+
 static const struct vfio_cxl_ops vfio_cxl_ops = {
 	.init_device	= vfio_cxl_init_device,
 	.release_device	= vfio_cxl_release_device,
 	.open_device	= vfio_cxl_open_device,
 	.close_device	= vfio_cxl_close_device,
+	.config_read	= vfio_cxl_config_read,
+	.config_write	= vfio_cxl_config_write,
 	.owner		= THIS_MODULE,
 };
 
diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
index a10ed733f0e3..f088e4ce5e07 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -1102,6 +1102,39 @@ void vfio_pci_uninit_perm_bits(void)
 	free_perm_bits(&ecap_perms[PCI_EXT_CAP_ID_PWR]);
 }
 
+/*
+ * DVSEC accesses are dispatched to vfio-cxl for a CXL device. It handles the
+ * CXL DVSEC and returns -ENODEV for any other DVSEC, for which the default
+ * handling then applies.
+ */
+static int vfio_pci_dvsec_config_read(struct vfio_pci_core_device *vdev, int pos,
+				      int count, struct perm_bits *perm,
+				      int offset, __le32 *val)
+{
+	if (vdev->cxl_ops && vdev->cxl_ops->config_read) {
+		int ret = vdev->cxl_ops->config_read(vdev, pos, count, val);
+
+		if (ret != -ENODEV)
+			return ret;
+	}
+
+	return vfio_direct_config_read(vdev, pos, count, perm, offset, val);
+}
+
+static int vfio_pci_dvsec_config_write(struct vfio_pci_core_device *vdev,
+				       int pos, int count, struct perm_bits *perm,
+				       int offset, __le32 val)
+{
+	if (vdev->cxl_ops && vdev->cxl_ops->config_write) {
+		int ret = vdev->cxl_ops->config_write(vdev, pos, count, val);
+
+		if (ret != -ENODEV)
+			return ret;
+	}
+
+	return vfio_raw_config_write(vdev, pos, count, perm, offset, val);
+}
+
 int __init vfio_pci_init_perm_bits(void)
 {
 	int ret;
@@ -1121,7 +1154,8 @@ int __init vfio_pci_init_perm_bits(void)
 	ret |= init_pci_ext_cap_err_perm(&ecap_perms[PCI_EXT_CAP_ID_ERR]);
 	ret |= init_pci_ext_cap_pwr_perm(&ecap_perms[PCI_EXT_CAP_ID_PWR]);
 	ecap_perms[PCI_EXT_CAP_ID_VNDR].writefn = vfio_raw_config_write;
-	ecap_perms[PCI_EXT_CAP_ID_DVSEC].writefn = vfio_raw_config_write;
+	ecap_perms[PCI_EXT_CAP_ID_DVSEC].readfn = vfio_pci_dvsec_config_read;
+	ecap_perms[PCI_EXT_CAP_ID_DVSEC].writefn = vfio_pci_dvsec_config_write;
 
 	if (ret)
 		vfio_pci_uninit_perm_bits();
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 7354dae1dd85..b9202cb29d96 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -72,6 +72,11 @@ struct vfio_cxl_ops {
 	void	(*release_device)(struct vfio_pci_core_device *vdev);
 	int     (*open_device)(struct vfio_pci_core_device *vdev);
 	void    (*close_device)(struct vfio_pci_core_device *vdev);
+	int     (*config_read)(struct vfio_pci_core_device *vdev, int pos,
+			       int count, __le32 *val);
+	int     (*config_write)(struct vfio_pci_core_device *vdev, int pos,
+				int count, __le32 val);
+
 	/* Pinned per bound CXL device so vfio-cxl cannot unload under usage */
 	struct module *owner;
 };
diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
index 4f5942ebd29b..00ca1c777b44 100644
--- a/include/uapi/linux/pci_regs.h
+++ b/include/uapi/linux/pci_regs.h
@@ -1359,6 +1359,7 @@
 #define  PCI_DVSEC_CXL_CTRL				0xC
 #define   PCI_DVSEC_CXL_CACHE_ENABLE			_BITUL(0)
 #define   PCI_DVSEC_CXL_MEM_ENABLE			_BITUL(2)
+#define  PCI_DVSEC_CXL_STATUS				0xE
 #define  PCI_DVSEC_CXL_CTRL2				0x10
 #define   PCI_DVSEC_CXL_DISABLE_CACHING			_BITUL(0)
 #define   PCI_DVSEC_CXL_INIT_CACHE_WBI			_BITUL(1)
-- 
2.25.1

[PATCH v4 18/27] vfio/cxl: Expose the HDM memory and trap the decoder registers

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Register the HDM region as a CXL VFIO region so the guest can map
the device memory.

CXL.mem is coherent memory, so the guest and KVM need a write-back
mapping. Claim the host physical range exclusively at bind so a
firmware-owned cacheable alias cannot fault the host once the range is
mapped write-back.

The region is advertised mmap-able, so a fd holder can fault its
struct-page-less device memory in from the host CPU. A memory error on
that range would otherwise escalate to a host SError. Register the range
with memory_failure() via register_pfn_address_space() so such an error
is contained to unmapping the range and a SIGBUS to the fd holder.
register_pfn_address_space() returns -EOPNOTSUPP when
CONFIG_MEMORY_FAILURE is off, in which case the region is exposed
without containment.

A CXL Type-2 guest also reads the HDM decoder registers to learn the
HDM region it was handed. Those registers live in the component BAR that
vfio-pci owns, so map the decoder block at bind and copy it into a
per-open shadow, then expose a read-only region backed by that shadow.
The block location comes from the pdev->hdm enumeration cache rather
than the CXL core register map.

The guest programs a GPA into the decoder while the host resolves the
HPA, so it must never reach the physical registers. Serving reads from
the shadow keeps the two apart; write emulation follows in a later
change.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 347 ++++++++++++++++++++++++++-
 drivers/vfio/pci/vfio_pci_core.c     |  17 ++
 include/linux/vfio_pci_core.h        |   1 +
 include/uapi/linux/vfio.h            |   3 +
 4 files changed, 367 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 9fed909cb9d3..07047646edde 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -5,10 +5,14 @@
  * Copyright (c) 2026 NVIDIA Corporation & Affiliates
  */
 
+#include <linux/cleanup.h>
+#include <linux/memory-failure.h>
+#include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/pci.h>
 #include <linux/range.h>
 #include <linux/slab.h>
+#include <linux/uaccess.h>
 #include <linux/vfio_pci_core.h>
 #include <cxl/cxl.h>
 #include <cxl/pci.h>
@@ -18,21 +22,250 @@
  * @cxlds: CXL device state; kept first for devm_cxl_dev_state_create()
  * @cxlmd: memory device joined to the CXL topology at bind
  * @hpa_range: host physical range of the HDM region
+ * @dpa_pfn_space: HDM-region pfn range registered with memory_failure()
  * @dvsec: CXL device DVSEC config-space offset
  * @dvsec_len: length of the DVSEC body
  * @dvsec_dwords: dword count of @dvsec_shadow
  * @dvsec_shadow: guest view of the CXL DVSEC body, sampled at open
+ * @hdm_regs: mapped HDM decoder registers, source for the open-time snapshot
+ * @hdm_len: length of the HDM decoder register block
+ * @hdm_shadow: guest view of the HDM decoder registers, sampled at open
+ * @hdm_region_idx: vdev->region[] index of the HDM region
+ * @hdm_valid: true when the decoder is in a known-good restored state and host
+ *	       CPU access to the HDM range is safe; gated under memory_lock
  */
 struct vfio_cxl_state {
 	struct cxl_dev_state cxlds;
 	struct cxl_memdev *cxlmd;
 	struct range hpa_range;
+	struct pfn_address_space dpa_pfn_space;
 	u16 dvsec;
 	u32 dvsec_len;
 	u32 dvsec_dwords;
+
 	u32 *dvsec_shadow __counted_by_ptr(dvsec_dwords);
+	void __iomem *hdm_regs;
+	u32 hdm_len;
+	u32 hdm_dwords;
+
+	__le32 *hdm_shadow __counted_by_ptr(hdm_dwords);
+	int hdm_region_idx;
+	bool hdm_valid;
+};
+
+static unsigned long vfio_cxl_mem_pgoff(struct vm_area_struct *vma,
+					unsigned long addr)
+{
+	unsigned long mask = (1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1;
+
+	return (vma->vm_pgoff & mask) + ((addr - vma->vm_start) >> PAGE_SHIFT);
+}
+
+static vm_fault_t vfio_cxl_mem_huge_fault(struct vm_fault *vmf,
+					  unsigned int order)
+{
+	struct vm_area_struct *vma = vmf->vma;
+	struct vfio_pci_core_device *vdev = vma->vm_private_data;
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	unsigned long addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
+	unsigned long pfn = PHYS_PFN(cxl->hpa_range.start) +
+			    vfio_cxl_mem_pgoff(vma, addr);
+	vm_fault_t ret = VM_FAULT_FALLBACK;
+
+	if (is_aligned_for_order(vma, addr, pfn, order)) {
+		scoped_guard(rwsem_read, &vdev->memory_lock) {
+			/*
+			 * A reset or D3 transition takes memory_lock for write,
+			 * revokes this mapping and clears the decoder. Do not
+			 * insert a PFN for a decoder that is not in a known-good
+			 * state, or the host CPU could reach a disabled decoder.
+			 * vfio_pci_vmf_insert_pfn() adds the Memory-Space gate:
+			 * an HDM access while the device has Memory-Space disabled
+			 * aborts on the fabric as a fatal host SError, so it must
+			 * not be faulted in until the guest re-enables it.
+			 */
+			if (cxl->hdm_valid)
+				ret = vfio_pci_vmf_insert_pfn(vdev, vmf, pfn,
+							      order);
+			else
+				ret = VM_FAULT_SIGBUS;
+		}
+	}
+
+	return ret;
+}
+
+static vm_fault_t vfio_cxl_mem_fault(struct vm_fault *vmf)
+{
+	return vfio_cxl_mem_huge_fault(vmf, 0);
+}
+
+static const struct vm_operations_struct vfio_cxl_mem_vm_ops = {
+	.fault = vfio_cxl_mem_fault,
+#ifdef CONFIG_ARCH_SUPPORTS_HUGE_PFNMAP
+	.huge_fault = vfio_cxl_mem_huge_fault,
+#endif
+};
+
+static int vfio_cxl_mem_mmap(struct vfio_pci_core_device *vdev,
+			     struct vfio_pci_region *region,
+			     struct vm_area_struct *vma)
+{
+	unsigned long mask = (1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1;
+	u64 req_start = (vma->vm_pgoff & mask) << PAGE_SHIFT;
+	u64 req_len = vma->vm_end - vma->vm_start;
+
+	if (req_start + req_len > region->size)
+		return -EINVAL;
+
+	/*
+	 * CXL.mem is coherent memory, so leave the mapping write-back cacheable;
+	 * a device or non-cached mapping would break the coherence the guest and
+	 * KVM depend on. The host physical range is claimed exclusively at bind,
+	 * so no conflicting cacheable alias remains.
+	 */
+	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
+	vma->vm_ops = &vfio_cxl_mem_vm_ops;
+	vma->vm_private_data = vdev;
+
+	return 0;
+}
+
+static ssize_t vfio_cxl_mem_rw(struct vfio_pci_core_device *vdev,
+			       char __user *buf, size_t count, loff_t *ppos,
+			       bool iswrite)
+{
+	/*
+	 * The HDM region advertises READ and WRITE so a VMM can derive an
+	 * accessible mmap protection for it, but fd read/write is not supported.
+	 * The only host-side way to reach the range for a copy is a kernel
+	 * mapping of the CXL.mem host physical address (memremap, which reuses
+	 * the linear map for this RAM-backed range), and a CPU access through
+	 * that mapping aborts on the fabric as a fatal host SError, unlike the
+	 * guest-facing mmap fault path which maps the pfn directly. Reject the
+	 * transfer rather than fault the host; a consumer mmaps the region and
+	 * accesses it that way.
+	 */
+	return -EIO;
+}
+
+/*
+ * The CXL regions carry no per-region state (region->data is the shared,
+ * devm-managed vfio_cxl_state), so releasing a region is a no-op. The hook is
+ * still required: vfio_pci_core_disable() calls region->ops->release() for
+ * every region without a NULL check.
+ */
+static void vfio_cxl_region_release(struct vfio_pci_core_device *vdev,
+				    struct vfio_pci_region *region)
+{
+}
+
+static const struct vfio_pci_regops vfio_cxl_mem_regops = {
+	.rw = vfio_cxl_mem_rw,
+	.mmap = vfio_cxl_mem_mmap,
+	.release = vfio_cxl_region_release,
+};
+
+/*
+ * Map a poisoned HDM-region pfn back to the file offset of each user mapping so
+ * memory_failure() can unmap it and signal the fd holder. The region is a
+ * single linear range at hpa_range.start; recover the per-vma file offset the
+ * same way the fault handler derived the pfn.
+ */
+static int vfio_cxl_pfn_to_vma_pgoff(struct vm_area_struct *vma,
+				     unsigned long pfn, pgoff_t *pgoff)
+{
+	struct vfio_pci_core_device *vdev;
+	struct vfio_cxl_state *cxl;
+	pgoff_t vma_off, pfn_off;
+	unsigned long start_pfn;
+
+	if (vma->vm_ops != &vfio_cxl_mem_vm_ops)
+		return -ENOENT;
+
+	vdev = vma->vm_private_data;
+	cxl = vdev->cxl;
+
+	start_pfn = PHYS_PFN(cxl->hpa_range.start);
+	if (pfn < start_pfn ||
+	    pfn >= start_pfn + (range_len(&cxl->hpa_range) >> PAGE_SHIFT))
+		return -EFAULT;
+
+	pfn_off = pfn - start_pfn;
+	vma_off = vma->vm_pgoff &
+		  ((1UL << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
+	/* Skip VMAs that do not map the pfn, e.g. a partial mmap of the region. */
+	if (pfn_off < vma_off || pfn_off - vma_off >= vma_pages(vma))
+		return -EFAULT;
+
+	*pgoff = vma->vm_pgoff + (pfn_off - vma_off);
+	return 0;
+}
+
+/*
+ * The HDM region is struct-page-less device memory, so a memory error on it
+ * cannot be routed through the normal page path. Register the range with
+ * memory_failure() so such an error is contained to unmapping the range and a
+ * SIGBUS to the fd holder instead of escalating to a host SError.
+ */
+static int vfio_cxl_register_pfn_space(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	unsigned long start_pfn = PHYS_PFN(cxl->hpa_range.start);
+
+	cxl->dpa_pfn_space.node.start = start_pfn;
+	cxl->dpa_pfn_space.node.last =
+		start_pfn + (range_len(&cxl->hpa_range) >> PAGE_SHIFT) - 1;
+	cxl->dpa_pfn_space.mapping = vdev->vdev.inode->i_mapping;
+	cxl->dpa_pfn_space.pfn_to_vma_pgoff = vfio_cxl_pfn_to_vma_pgoff;
+
+	return register_pfn_address_space(&cxl->dpa_pfn_space);
+}
+
+static ssize_t vfio_cxl_comp_rw(struct vfio_pci_core_device *vdev,
+				char __user *buf, size_t count, loff_t *ppos,
+				bool iswrite)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
+
+	/*
+	 * The guest programs a GPA into this decoder and the host resolves the
+	 * HPA, so the guest never drives the physical decoder. Reads come from
+	 * the open-time snapshot; write emulation lands in a later change.
+	 */
+	if (iswrite)
+		return -EINVAL;
+
+	if (pos >= cxl->hdm_len)
+		return -EINVAL;
+
+	count = min_t(size_t, count, cxl->hdm_len - pos);
+	/*
+	 * The shadow mirrors the physical decoder, so BASE_LOW/HIGH carry the
+	 * host HPA. That is visible only to the trusted VMM holding the fd; the
+	 * VMM virtualizes the base so the guest sees its own GPA and never the
+	 * host address.
+	 */
+	if (copy_to_user(buf, (u8 *)cxl->hdm_shadow + pos, count))
+		return -EFAULT;
+
+	*ppos += count;
+	return count;
+}
+
+static const struct vfio_pci_regops vfio_cxl_comp_regops = {
+	.rw = vfio_cxl_comp_rw,
+	.release = vfio_cxl_region_release,
 };
 
+static void vfio_cxl_release_hpa(void *data)
+{
+	struct vfio_cxl_state *cxl = data;
+
+	release_mem_region(cxl->hpa_range.start, range_len(&cxl->hpa_range));
+}
+
 static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
 {
 	struct pci_dev *pdev = vdev->pdev;
@@ -92,6 +325,20 @@ static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
 	if (ret)
 		return ret;
 
+	/*
+	 * Map the HDM decoder registers to sample their programming at open.
+	 * The block location comes from the enumeration cache in pdev->hdm, so
+	 * this does not reach into the CXL core register map. vfio-pci owns the
+	 * BAR, so map without claiming the sub-block.
+	 */
+	cxl->hdm_regs = devm_ioremap(&pdev->dev,
+				     pci_resource_start(pdev, pdev->hdm->hdm_bar) +
+				     pdev->hdm->hdm_offset, pdev->hdm->hdm_size);
+	if (!cxl->hdm_regs)
+		return -ENOMEM;
+
+	cxl->hdm_len = pdev->hdm->hdm_size;
+
 	ret = cxl_set_capacity(&cxl->cxlds, hdm_size);
 	if (ret)
 		return ret;
@@ -100,9 +347,30 @@ static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
 	if (IS_ERR(cxlmd))
 		return PTR_ERR(cxlmd);
 
+	/*
+	 * Own the resolved host physical range outright, and exclusively: mark
+	 * it IORESOURCE_EXCLUSIVE so /dev/mem cannot map a conflicting alias even
+	 * on an IO_STRICT_DEVMEM=n kernel. Firmware that left it as System RAM
+	 * would otherwise keep a cacheable alias that faults the host once the
+	 * guest maps the range write-back. There is no devm form of the exclusive
+	 * request, so pair it with a devm release action.
+	 */
+	if (!request_mem_region_exclusive(cxl->hpa_range.start,
+					  range_len(&cxl->hpa_range),
+					  "vfio-cxl-hdm"))
+		return -EBUSY;
+	ret = devm_add_action_or_reset(&pdev->dev, vfio_cxl_release_hpa, cxl);
+	if (ret)
+		return ret;
+
 	cxl->cxlmd = cxlmd;
 	vdev->cxl = cxl;
 
+	/*
+	 * The VFIO regions and the poison-containment pfn space are set up in
+	 * open_device(): vfio_pci_core_disable() tears down all dynamic regions on
+	 * close, so they must be created per open rather than once at bind.
+	 */
 	return 0;
 }
 
@@ -114,9 +382,11 @@ static void vfio_cxl_release_device(struct vfio_pci_core_device *vdev)
 static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
 {
 	struct vfio_cxl_state *cxl = vdev->cxl;
+	void __iomem *hdm = cxl->hdm_regs;
 	struct pci_dev *pdev = vdev->pdev;
+	__le32 *hdm_shadow;
 	u32 hdr, *shadow;
-	int i, dwords;
+	int i, dwords, ret;
 
 	/*
 	 * Sample the DVSEC body now rather than at bind: a low-power
@@ -137,13 +407,88 @@ static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
 	cxl->dvsec_dwords = dwords;
 	cxl->dvsec_shadow = shadow;
 
+	dwords = cxl->hdm_len / sizeof(u32);
+	hdm_shadow = kcalloc(dwords, sizeof(__le32), GFP_KERNEL);
+	if (!hdm_shadow) {
+		kfree(shadow);
+		cxl->dvsec_shadow = NULL;
+		return -ENOMEM;
+	}
+
+	for (i = 0; i < dwords; i++)
+		hdm_shadow[i] = cpu_to_le32(readl(hdm + i * sizeof(u32)));
+
+	cxl->hdm_dwords = dwords;
+	cxl->hdm_shadow = hdm_shadow;
+
+	/*
+	 * vfio_pci_core_disable() frees all dynamic regions on close, so register
+	 * them here (per open) rather than at bind. A failed first-open never
+	 * reaches close_device(), so unwind on error.
+	 *
+	 * Advertise READ and WRITE alongside MMAP: a VMM derives the mmap
+	 * protection from these flags, so without them the HDM memory is mapped
+	 * PROT_NONE and a guest access faults (KVM cannot back the mapping). The
+	 * flags describe the mmap protection only; fd read/write returns -EIO,
+	 * because a host CPU read through a kernel mapping of the coherent
+	 * CXL.mem range aborts on the fabric (see vfio_cxl_mem_rw()).
+	 */
+	ret = vfio_pci_core_register_dev_region(vdev, VFIO_REGION_TYPE_CXL,
+						VFIO_REGION_SUBTYPE_CXL_MEM,
+						&vfio_cxl_mem_regops,
+						range_len(&cxl->hpa_range),
+						VFIO_REGION_INFO_FLAG_READ |
+						VFIO_REGION_INFO_FLAG_WRITE |
+						VFIO_REGION_INFO_FLAG_MMAP, cxl);
+	if (ret)
+		goto err_free_shadows;
+
+	ret = vfio_pci_core_register_dev_region(vdev, VFIO_REGION_TYPE_CXL,
+						VFIO_REGION_SUBTYPE_CXL_COMP_REGS,
+						&vfio_cxl_comp_regops, cxl->hdm_len,
+						VFIO_REGION_INFO_FLAG_READ, cxl);
+	if (ret)
+		goto err_unregister_hdm;
+
+	/*
+	 * The HDM region is advertised mmap-able, so a fd holder can fault its
+	 * struct-page-less device memory in from the host CPU. Register it with
+	 * memory_failure() to contain a memory error. -EOPNOTSUPP means
+	 * CONFIG_MEMORY_FAILURE is off, so run without containment.
+	 */
+	ret = vfio_cxl_register_pfn_space(vdev);
+	if (ret && ret != -EOPNOTSUPP)
+		goto err_unregister_comp;
+
+	/*
+	 * The decoder is firmware-committed and the shadow now mirrors it, so
+	 * host access to the HDM range is safe. Open the access gate; reset and
+	 * power transitions clear it until the decoder is restored.
+	 */
+	cxl->hdm_valid = true;
+
 	return 0;
+
+err_unregister_comp:
+	vfio_pci_core_unregister_dev_region(vdev);
+err_unregister_hdm:
+	vfio_pci_core_unregister_dev_region(vdev);
+err_free_shadows:
+	kfree(cxl->hdm_shadow);
+	cxl->hdm_shadow = NULL;
+	kfree(cxl->dvsec_shadow);
+	cxl->dvsec_shadow = NULL;
+	return ret;
 }
 
 static void vfio_cxl_close_device(struct vfio_pci_core_device *vdev)
 {
 	struct vfio_cxl_state *cxl = vdev->cxl;
 
+	cxl->hdm_valid = false;
+	unregister_pfn_address_space(&cxl->dpa_pfn_space);
+	kfree(cxl->hdm_shadow);
+	cxl->hdm_shadow = NULL;
 	kfree(cxl->dvsec_shadow);
 	cxl->dvsec_shadow = NULL;
 }
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 470730cdc88b..6f6ebcce86a1 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1123,6 +1123,23 @@ int vfio_pci_core_register_dev_region(struct vfio_pci_core_device *vdev,
 }
 EXPORT_SYMBOL_GPL(vfio_pci_core_register_dev_region);
 
+/*
+ * Unregister the most recently registered dynamic region. Used to unwind a
+ * partially built region set on an open-time error; regions are otherwise
+ * released together in vfio_pci_core_disable().
+ */
+void vfio_pci_core_unregister_dev_region(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_pci_region *region;
+
+	if (WARN_ON(!vdev->num_regions))
+		return;
+
+	region = &vdev->region[--vdev->num_regions];
+	region->ops->release(vdev, region);
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_unregister_dev_region);
+
 static int vfio_pci_info_atomic_cap(struct vfio_pci_core_device *vdev,
 				    struct vfio_info_cap *caps)
 {
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index b9202cb29d96..294e95b5e881 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -187,6 +187,7 @@ int vfio_pci_core_register_dev_region(struct vfio_pci_core_device *vdev,
 				      unsigned int type, unsigned int subtype,
 				      const struct vfio_pci_regops *ops,
 				      size_t size, u32 flags, void *data);
+void vfio_pci_core_unregister_dev_region(struct vfio_pci_core_device *vdev);
 void vfio_pci_core_set_mmap_exclude(struct vfio_pci_core_device *vdev, int bar,
 				    u64 start, u64 len);
 void vfio_pci_core_close_device(struct vfio_device *core_vdev);
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 540e62312671..d5b8106625bf 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -377,6 +377,9 @@ struct vfio_region_info_cap_type {
 /* sub-types for VFIO_REGION_TYPE_CXL */
 /* CXL.mem HDM region of a Type-2 device, mmap-able */
 #define VFIO_REGION_SUBTYPE_CXL_MEM		(1)
+/* CXL HDM decoder registers, trapped so the guest programs a GPA it never owns */
+#define VFIO_REGION_SUBTYPE_CXL_COMP_REGS	(2)
+/* Subtypes 3+ are reserved for future trapped CXL surfaces, e.g. RAS. */
 
 /**
  * struct vfio_region_gfx_edid - EDID region layout.
-- 
2.25.1

[PATCH v4 19/27] vfio/cxl: Keep the HDM decoder block off the direct BAR mapping

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

vfio-cxl owns the whole component BAR and serves the HDM decoder block
through a trapped region that runs the commit FSM against a shadow. Left
directly mmappable, the same block would let userspace reach the physical
decoder and move the host physical window. Exclude it from the BAR mmap so
the trap is the only path to it.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 07047646edde..f91f6eb8b8fb 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -339,6 +339,14 @@ static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev)
 
 	cxl->hdm_len = pdev->hdm->hdm_size;
 
+	/*
+	 * The HDM decoder block is served from the trapped component-register
+	 * region. Keep it out of the guest's direct BAR mapping so the physical
+	 * decoder, which governs host memory decode, cannot be reprogrammed.
+	 */
+	vfio_pci_core_set_mmap_exclude(vdev, pdev->hdm->hdm_bar,
+				       pdev->hdm->hdm_offset, cxl->hdm_len);
+
 	ret = cxl_set_capacity(&cxl->cxlds, hdm_size);
 	if (ret)
 		return ret;
-- 
2.25.1

[PATCH v4 20/27] vfio/cxl: Emulate the HDM decoder commit handshake

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Let the guest drive its virtual decoder now that the register window is
trapped. Writes stay in the per-open shadow so the guest never touches
the physical decoder; the host has already resolved the HPA.

The control register carries the commit handshake, so reflect a commit
request straight to committed and drop the error bit. A decoder that
committed with lock set stays frozen until the device is reset, when the
shadow is sampled afresh.

Apply per-field write semantics to the rest of the block rather than
storing every write verbatim: gate base and size on the committed state,
so they change only across a decommit, and drop writes to the read-only
capability register.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 156 ++++++++++++++++++++++++---
 1 file changed, 142 insertions(+), 14 deletions(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index f91f6eb8b8fb..ec938813bd91 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -15,6 +15,7 @@
 #include <linux/uaccess.h>
 #include <linux/vfio_pci_core.h>
 #include <cxl/cxl.h>
+#include <cxl/cxl_regs.h>
 #include <cxl/pci.h>
 
 /**
@@ -222,33 +223,159 @@ static int vfio_cxl_register_pfn_space(struct vfio_pci_core_device *vdev)
 	return register_pfn_address_space(&cxl->dpa_pfn_space);
 }
 
+/*
+ * Only an endpoint decoder's control register carries the commit handshake.
+ * A single non-interleaved decoder is assumed; switch topologies would widen
+ * which offsets qualify.
+ */
+static bool vfio_cxl_ctrl_offset(loff_t pos)
+{
+	unsigned int stride = CXL_HDM_DECODER0_CTRL_OFFSET(1) -
+			      CXL_HDM_DECODER0_CTRL_OFFSET(0);
+	loff_t off = pos - CXL_HDM_DECODER0_CTRL_OFFSET(0);
+
+	return pos >= CXL_HDM_DECODER0_CTRL_OFFSET(0) && off % stride == 0;
+}
+
+static void vfio_cxl_ctrl_write(struct vfio_cxl_state *cxl, u32 idx, u32 val)
+{
+	u32 old = le32_to_cpu(cxl->hdm_shadow[idx]);
+	u32 wmask = CXL_HDM_DECODER0_CTRL_IG_MASK |
+		    CXL_HDM_DECODER0_CTRL_IW_MASK |
+		    CXL_HDM_DECODER0_CTRL_LOCK |
+		    CXL_HDM_DECODER0_CTRL_COMMIT |
+		    CXL_HDM_DECODER0_CTRL_HOSTONLY;
+
+	/* A committed decoder that asked to lock stays put until reset. */
+	if ((old & CXL_HDM_DECODER0_CTRL_COMMITTED) &&
+	    (old & CXL_HDM_DECODER0_CTRL_LOCK))
+		return;
+
+	/*
+	 * Take only the guest-writable fields and preserve the reserved bits and
+	 * the emulation-owned status bits (COMMITTED/COMMIT_ERROR) from the
+	 * shadow, so the VMM never reads back guest-authored reserved state.
+	 */
+	val = (old & ~wmask) | (val & wmask);
+
+	/*
+	 * The host resolved the HPA before the guest ever saw the device, so a
+	 * commit request always lands and clearing it tears the guest view down.
+	 */
+	if (val & CXL_HDM_DECODER0_CTRL_COMMIT)
+		val = (val | CXL_HDM_DECODER0_CTRL_COMMITTED) &
+		      ~CXL_HDM_DECODER0_CTRL_COMMIT_ERROR;
+	else
+		val &= ~CXL_HDM_DECODER0_CTRL_COMMITTED;
+
+	cxl->hdm_shadow[idx] = cpu_to_le32(val);
+}
+
+/*
+ * Base, size, and the Target List / Skip registers are all RWL: they lock on
+ * commit, so every one of them is filtered through the committed guard.
+ */
+static bool vfio_cxl_base_size_offset(loff_t pos)
+{
+	return pos == CXL_HDM_DECODER0_BASE_LOW_OFFSET(0) ||
+	       pos == CXL_HDM_DECODER0_BASE_HIGH_OFFSET(0) ||
+	       pos == CXL_HDM_DECODER0_SIZE_LOW_OFFSET(0) ||
+	       pos == CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(0) ||
+	       pos == CXL_HDM_DECODER0_SKIP_LOW(0) ||
+	       pos == CXL_HDM_DECODER0_SKIP_HIGH(0);
+}
+
+/*
+ * Reserved dwords in the single-decoder HDM block: 0x08 and 0x0c between the
+ * global control register and decoder 0, and 0x2c after decoder 0's registers.
+ * Keep them read-only so the VMM never reads back guest-authored reserved state.
+ */
+static bool vfio_cxl_reserved_offset(loff_t pos)
+{
+	return pos == 0x08 || pos == 0x0c || pos == 0x2c;
+}
+
+/*
+ * BASE_LOW and SIZE_LOW expose only the 256MB-aligned upper nibble [31:28];
+ * bits [27:0] are RsvdP. Preserve the reserved low bits so the VMM never reads
+ * back an unaligned base or size.
+ */
+#define CXL_HDM_DECODER_LOW_ADDR_MASK 0xf0000000U
+
+static void vfio_cxl_base_size_write(struct vfio_cxl_state *cxl, u32 idx,
+				     __le32 val)
+{
+	u32 ctrl = le32_to_cpu(cxl->hdm_shadow[CXL_HDM_DECODER0_CTRL_OFFSET(0) /
+					       sizeof(u32)]);
+	loff_t off = (loff_t)idx * sizeof(u32);
+	u32 new = le32_to_cpu(val);
+
+	/* A committed decoder holds its position fields until it decommits. */
+	if (ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED)
+		return;
+
+	if (off == CXL_HDM_DECODER0_BASE_LOW_OFFSET(0) ||
+	    off == CXL_HDM_DECODER0_SIZE_LOW_OFFSET(0)) {
+		u32 old = le32_to_cpu(cxl->hdm_shadow[idx]);
+
+		new = (old & ~CXL_HDM_DECODER_LOW_ADDR_MASK) |
+		      (new & CXL_HDM_DECODER_LOW_ADDR_MASK);
+	}
+
+	cxl->hdm_shadow[idx] = cpu_to_le32(new);
+}
+
 static ssize_t vfio_cxl_comp_rw(struct vfio_pci_core_device *vdev,
 				char __user *buf, size_t count, loff_t *ppos,
 				bool iswrite)
 {
 	struct vfio_cxl_state *cxl = vdev->cxl;
 	loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
+	size_t o;
 
-	/*
-	 * The guest programs a GPA into this decoder and the host resolves the
-	 * HPA, so the guest never drives the physical decoder. Reads come from
-	 * the open-time snapshot; write emulation lands in a later change.
-	 */
-	if (iswrite)
+	if (pos >= cxl->hdm_len)
 		return -EINVAL;
 
-	if (pos >= cxl->hdm_len)
+	/* The decoder registers only take aligned dword accesses. */
+	if (pos % sizeof(u32) || count % sizeof(u32))
 		return -EINVAL;
 
 	count = min_t(size_t, count, cxl->hdm_len - pos);
+
+	if (!iswrite) {
+		/*
+		 * The shadow mirrors the physical decoder, so BASE_LOW/HIGH
+		 * carry the host HPA. That is visible only to the trusted VMM
+		 * holding the fd; the VMM virtualizes the base so the guest sees
+		 * its own GPA and never the host address.
+		 */
+		if (copy_to_user(buf, (u8 *)cxl->hdm_shadow + pos, count))
+			return -EFAULT;
+		*ppos += count;
+		return count;
+	}
+
 	/*
-	 * The shadow mirrors the physical decoder, so BASE_LOW/HIGH carry the
-	 * host HPA. That is visible only to the trusted VMM holding the fd; the
-	 * VMM virtualizes the base so the guest sees its own GPA and never the
-	 * host address.
+	 * The guest programs a GPA into this decoder while the host resolves
+	 * the HPA, so writes stay in the shadow. Each register follows its own
+	 * class: control runs the commit handshake, base and size are locked
+	 * once committed, and the capability header is fixed.
 	 */
-	if (copy_to_user(buf, (u8 *)cxl->hdm_shadow + pos, count))
-		return -EFAULT;
+	for (o = 0; o < count; o += sizeof(u32)) {
+		u32 idx = (pos + o) / sizeof(u32);
+		__le32 val;
+
+		if (copy_from_user(&val, buf + o, sizeof(val)))
+			return -EFAULT;
+
+		if (vfio_cxl_ctrl_offset(pos + o))
+			vfio_cxl_ctrl_write(cxl, idx, le32_to_cpu(val));
+		else if (vfio_cxl_base_size_offset(pos + o))
+			vfio_cxl_base_size_write(cxl, idx, val);
+		else if (pos + o >= sizeof(u32) &&
+			 !vfio_cxl_reserved_offset(pos + o))
+			cxl->hdm_shadow[idx] = val;
+	}
 
 	*ppos += count;
 	return count;
@@ -454,7 +581,8 @@ static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
 	ret = vfio_pci_core_register_dev_region(vdev, VFIO_REGION_TYPE_CXL,
 						VFIO_REGION_SUBTYPE_CXL_COMP_REGS,
 						&vfio_cxl_comp_regops, cxl->hdm_len,
-						VFIO_REGION_INFO_FLAG_READ, cxl);
+						VFIO_REGION_INFO_FLAG_READ |
+						VFIO_REGION_INFO_FLAG_WRITE, cxl);
 	if (ret)
 		goto err_unregister_hdm;
 
-- 
2.25.1

[PATCH v4 21/27] vfio/cxl: Describe the CXL device and decoder geometry to userspace

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Advertise the assignment with a device-info flag so a VMM can tell a
CXL Type-2 device apart from a plain vfio-pci one. On the trapped
component region, add a capability carrying the BAR and offset of the
HDM decoder registers so the VMM can place the window where the guest
expects to find it.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 16 ++++++++++++++++
 drivers/vfio/pci/vfio_pci_core.c     |  3 +++
 include/uapi/linux/vfio.h            | 18 ++++++++++++++++++
 3 files changed, 37 insertions(+)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index ec938813bd91..0fb5ed5d86b7 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -381,9 +381,25 @@ static ssize_t vfio_cxl_comp_rw(struct vfio_pci_core_device *vdev,
 	return count;
 }
 
+static int vfio_cxl_comp_add_capability(struct vfio_pci_core_device *vdev,
+					struct vfio_pci_region *region,
+					struct vfio_info_cap *caps)
+{
+	struct cxl_hdm_info *hdm = vdev->pdev->hdm;
+	struct vfio_region_info_cap_cxl_comp_regs cap = {
+		.header.id = VFIO_REGION_INFO_CAP_CXL_COMP_REGS,
+		.header.version = 1,
+		.bar = hdm->hdm_bar,
+		.offset = hdm->hdm_offset,
+	};
+
+	return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
+}
+
 static const struct vfio_pci_regops vfio_cxl_comp_regops = {
 	.rw = vfio_cxl_comp_rw,
 	.release = vfio_cxl_region_release,
+	.add_capability = vfio_cxl_comp_add_capability,
 };
 
 static void vfio_cxl_release_hpa(void *data)
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 6f6ebcce86a1..77f8f39dd670 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1192,6 +1192,9 @@ static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev,
 	if (vdev->reset_works)
 		info.flags |= VFIO_DEVICE_FLAGS_RESET;
 
+	if (vdev->cxl_ops)
+		info.flags |= VFIO_DEVICE_FLAGS_CXL;
+
 	info.num_regions = VFIO_PCI_NUM_REGIONS + vdev->num_regions;
 	info.num_irqs = VFIO_PCI_NUM_IRQS;
 
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index d5b8106625bf..df88535ede5f 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -215,6 +215,7 @@ struct vfio_device_info {
 #define VFIO_DEVICE_FLAGS_FSL_MC (1 << 6)	/* vfio-fsl-mc device */
 #define VFIO_DEVICE_FLAGS_CAPS	(1 << 7)	/* Info supports caps */
 #define VFIO_DEVICE_FLAGS_CDX	(1 << 8)	/* vfio-cdx device */
+#define VFIO_DEVICE_FLAGS_CXL	(1 << 9)	/* vfio-cxl device */
 	__u32	num_regions;	/* Max region index + 1 */
 	__u32	num_irqs;	/* Max IRQ index + 1 */
 	__u32   cap_offset;	/* Offset within info struct of first cap */
@@ -505,6 +506,23 @@ struct vfio_region_info_cap_nvlink2_lnkspd {
 	__u32 __pad;
 };
 
+/*
+ * Geometry of a CXL Type-2 device's HDM decoder registers, so a VMM can place
+ * the trapped component register window where the guest expects it. The trapped
+ * region spans the whole HDM decoder block (every decoder), not just decoder 0:
+ * a VMM reads the decoder count and each decoder's committed base from the block
+ * itself. Additional trapped component capabilities, such as CXL RAS, are
+ * exposed as their own region subtypes rather than by extending this cap.
+ */
+#define VFIO_REGION_INFO_CAP_CXL_COMP_REGS	6
+
+struct vfio_region_info_cap_cxl_comp_regs {
+	struct vfio_info_cap_header header;
+	__u32 bar;
+	__u32 __resv;	/* reserved, must be zero; versioning anchor for flags */
+	__aligned_u64 offset;
+};
+
 /**
  * VFIO_DEVICE_GET_IRQ_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 9,
  *				    struct vfio_irq_info)
-- 
2.25.1

[PATCH v4 22/27] vfio/cxl: Revoke the HDM mapping on reset and power transitions

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

The HDM region is a device region, not a BAR, so vfio_pci_zap_bars()
leaves its PTEs in place. A runtime-PM entry, a D3 transition, or a reset
would then leave the guest with live mappings into a quiesced device.

Add a zap hook, called alongside the BAR zap under memory_lock, that
unmaps the window. The fault path already refuses to re-insert PFNs while
the device is suspended or its Memory Space is disabled.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 29 ++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_core.c     |  8 ++++++++
 include/linux/vfio_pci_core.h        |  2 ++
 3 files changed, 39 insertions(+)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index 0fb5ed5d86b7..f1c6bf06c408 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -530,6 +530,31 @@ static void vfio_cxl_release_device(struct vfio_pci_core_device *vdev)
 	vdev->cxl = NULL;
 }
 
+static void vfio_cxl_zap(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	if (!cxl)
+		return;
+
+	/*
+	 * Revoke the mapping so a later access re-faults. Do not touch hdm_valid
+	 * here: zap also runs on a plain PCI Memory-Space disable, across which
+	 * the committed HDM decoder stays valid (CXL.mem is not gated by PCI
+	 * Memory-Space). hdm_valid tracks decoder validity and is cleared only by
+	 * the paths that can leave the decoder unrestored (a failed reset or PM
+	 * restore). A reset or D3 transition holds memory_lock for write while it
+	 * runs, so no fault races the revoke, and a runtime-suspended device is
+	 * caught by the pm_runtime_engaged check on the insert path.
+	 */
+	unmap_mapping_range(vdev->vdev.inode->i_mapping,
+			    VFIO_PCI_INDEX_TO_OFFSET(VFIO_PCI_NUM_REGIONS +
+						     cxl->hdm_region_idx),
+			    range_len(&cxl->hpa_range), true);
+}
+
 static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
 {
 	struct vfio_cxl_state *cxl = vdev->cxl;
@@ -594,6 +619,9 @@ static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
 	if (ret)
 		goto err_free_shadows;
 
+	/* Remember where the HDM region landed so it can be zapped by index. */
+	cxl->hdm_region_idx = vdev->num_regions - 1;
+
 	ret = vfio_pci_core_register_dev_region(vdev, VFIO_REGION_TYPE_CXL,
 						VFIO_REGION_SUBTYPE_CXL_COMP_REGS,
 						&vfio_cxl_comp_regops, cxl->hdm_len,
@@ -738,6 +766,7 @@ static const struct vfio_cxl_ops vfio_cxl_ops = {
 	.close_device	= vfio_cxl_close_device,
 	.config_read	= vfio_cxl_config_read,
 	.config_write	= vfio_cxl_config_write,
+	.zap		= vfio_cxl_zap,
 	.owner		= THIS_MODULE,
 };
 
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 77f8f39dd670..1a54f15d1c2c 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -1832,6 +1832,14 @@ void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev)
 {
 	down_write(&vdev->memory_lock);
 	vfio_pci_zap_bars(vdev);
+	/*
+	 * The HDM region lives in the device-region offset range that
+	 * vfio_pci_zap_bars() does not cover, so revoke it here too. Otherwise
+	 * a runtime-PM entry, D3 transition, or reset would leave the guest
+	 * with live mappings into a quiesced device.
+	 */
+	if (vdev->cxl_ops && vdev->cxl_ops->zap)
+		vdev->cxl_ops->zap(vdev);
 }
 
 u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev)
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 294e95b5e881..8b93949d4484 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -76,6 +76,8 @@ struct vfio_cxl_ops {
 			       int count, __le32 *val);
 	int     (*config_write)(struct vfio_pci_core_device *vdev, int pos,
 				int count, __le32 val);
+	/* Revoke the HDM mapping; paired with the BAR zap */
+	void    (*zap)(struct vfio_pci_core_device *vdev);
 
 	/* Pinned per bound CXL device so vfio-cxl cannot unload under usage */
 	struct module *owner;
-- 
2.25.1

[PATCH v4 23/27] vfio/cxl: Refresh the decoder snapshot after a device reset

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

A reset clears the HDM decoder registers, so the guest snapshot has to be
resampled once the reset settles, on every path that can reset the
function: the reset ioctl, an FLR driven through config space, and a bus
hot reset. Re-enable Memory Space first, since a config restore can leave
it off and the component-BAR read would then take an Unsupported Request.

The bus hot reset zaps BARs directly rather than through
vfio_pci_zap_and_down_write_memory_lock(), so it also needs the HDM
window zapped by hand; route both zap sites through a common helper.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 82 ++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_config.c   |  2 +
 drivers/vfio/pci/vfio_pci_core.c     | 25 ++++++++-
 drivers/vfio/pci/vfio_pci_priv.h     | 20 +++++++
 include/linux/vfio_pci_core.h        |  4 ++
 5 files changed, 131 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index f1c6bf06c408..f45eaa60bad2 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -555,6 +555,86 @@ static void vfio_cxl_zap(struct vfio_pci_core_device *vdev)
 			    range_len(&cxl->hpa_range), true);
 }
 
+static void vfio_cxl_post_reset(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	struct pci_dev *pdev = vdev->pdev;
+	bool re_enabled = false;
+	int i, dwords;
+	u16 cmd;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	if (!cxl || !cxl->hdm_shadow)
+		return;
+
+	/*
+	 * The decoder registers are read through the component BAR. A config
+	 * restore can leave Memory Space disabled, and the read would then
+	 * return an Unsupported Request, so re-enable it before sampling.
+	 */
+	pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+	if (!(cmd & PCI_COMMAND_MEMORY)) {
+		pci_write_config_word(pdev, PCI_COMMAND,
+				      cmd | PCI_COMMAND_MEMORY);
+		re_enabled = true;
+	}
+
+	dwords = cxl->hdm_len / sizeof(u32);
+	for (i = 0; i < dwords; i++)
+		cxl->hdm_shadow[i] = cpu_to_le32(readl(cxl->hdm_regs +
+						       i * sizeof(u32)));
+	/*
+	 * Leave Memory Space as it was found. The guest owns Memory Space
+	 * through vconfig, so a physical enable done only to sample must not
+	 * outlive the sampling or the function would decode while vconfig
+	 * reports it off.
+	 */
+	if (re_enabled)
+		pci_write_config_word(pdev, PCI_COMMAND, cmd);
+}
+
+static int vfio_cxl_pm_restore(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	struct pci_dev *pdev = vdev->pdev;
+	int rc;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	if (!cxl || !cxl->hdm_shadow) {
+		pci_dbg(pdev, "vfio-cxl: pm_restore: no shadow (device not open), skipping\n");
+		return 0;
+	}
+
+	/*
+	 * A D3hot->D0 transition can soft-reset the function and clear the HDM
+	 * decoder. Restore the physical decoder before the fault gate re-inserts
+	 * the mapping. The restore needs the device lock, taken here after
+	 * memory_lock to match the reset path ordering. On failure the decoder is
+	 * left unrestored, so close the access gate (zap no longer clears it) and
+	 * return the error so the caller keeps the HDM range inaccessible.
+	 */
+	if (!pci_dev_trylock(pdev)) {
+		pci_warn(pdev, "vfio-cxl: pm_restore: could not lock device, HDM not restored\n");
+		cxl->hdm_valid = false;
+		return -EBUSY;
+	}
+
+	rc = cxl_restore_hdm_after_pci_reset(pdev);
+	pci_dev_unlock(pdev);
+	if (rc) {
+		pci_err(pdev, "vfio-cxl: pm_restore: HDM restore failed: %d\n", rc);
+		cxl->hdm_valid = false;
+		return rc;
+	}
+
+	vfio_cxl_post_reset(vdev);
+	/* The decoder is restored and re-sampled, so reopen the access gate. */
+	cxl->hdm_valid = true;
+	return 0;
+}
+
 static int vfio_cxl_open_device(struct vfio_pci_core_device *vdev)
 {
 	struct vfio_cxl_state *cxl = vdev->cxl;
@@ -767,6 +847,8 @@ static const struct vfio_cxl_ops vfio_cxl_ops = {
 	.config_read	= vfio_cxl_config_read,
 	.config_write	= vfio_cxl_config_write,
 	.zap		= vfio_cxl_zap,
+	.post_reset	= vfio_cxl_post_reset,
+	.pm_restore	= vfio_cxl_pm_restore,
 	.owner		= THIS_MODULE,
 };
 
diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
index f088e4ce5e07..01d808546a4c 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -911,6 +911,7 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,
 			vfio_pci_zap_and_down_write_memory_lock(vdev);
 			vfio_pci_dma_buf_move(vdev, true);
 			pci_try_reset_function(vdev->pdev);
+			vfio_pci_cxl_post_reset(vdev);
 			if (__vfio_pci_memory_enabled(vdev))
 				vfio_pci_dma_buf_move(vdev, false);
 			up_write(&vdev->memory_lock);
@@ -996,6 +997,7 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,
 			vfio_pci_zap_and_down_write_memory_lock(vdev);
 			vfio_pci_dma_buf_move(vdev, true);
 			pci_try_reset_function(vdev->pdev);
+			vfio_pci_cxl_post_reset(vdev);
 			if (__vfio_pci_memory_enabled(vdev))
 				vfio_pci_dma_buf_move(vdev, false);
 			up_write(&vdev->memory_lock);
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 1a54f15d1c2c..fc8235c8b4fc 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -362,6 +362,14 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat
 		} else if (needs_restore) {
 			pci_load_and_free_saved_state(pdev, &vdev->pm_save);
 			pci_restore_state(pdev);
+			/*
+			 * A NoSoftRst- device soft-resets on D3hot->D0, which can
+			 * clear a CXL HDM decoder. Restore it before the fault
+			 * gate re-inserts the HDM mapping. memory_lock is held on
+			 * this path (the PM config write and runtime PM entry both
+			 * take it before the D0 transition).
+			 */
+			vfio_pci_cxl_pm_restore(vdev);
 		}
 	}
 
@@ -529,6 +537,13 @@ static int vfio_pci_core_runtime_resume(struct device *dev)
 		eventfd_signal(vdev->pm_wake_eventfd_ctx);
 		__vfio_pci_runtime_pm_exit(vdev);
 	}
+	/*
+	 * A NoSoftRst- function can soft-reset on the runtime D3hot->D0
+	 * transition and clear a CXL HDM decoder. Restore it while memory_lock
+	 * is held, before the fault gate can re-insert the HDM mapping. PCI
+	 * config restore alone does not restore the component decoder registers.
+	 */
+	vfio_pci_cxl_pm_restore(vdev);
 	up_write(&vdev->memory_lock);
 
 	if (vdev->pm_intx_masked)
@@ -1449,6 +1464,7 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
 
 	vfio_pci_dma_buf_move(vdev, true);
 	ret = pci_try_reset_function(vdev->pdev);
+	vfio_pci_cxl_post_reset(vdev);
 	if (__vfio_pci_memory_enabled(vdev))
 		vfio_pci_dma_buf_move(vdev, false);
 	up_write(&vdev->memory_lock);
@@ -1838,8 +1854,7 @@ void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev)
 	 * a runtime-PM entry, D3 transition, or reset would leave the guest
 	 * with live mappings into a quiesced device.
 	 */
-	if (vdev->cxl_ops && vdev->cxl_ops->zap)
-		vdev->cxl_ops->zap(vdev);
+	vfio_pci_cxl_zap(vdev);
 }
 
 u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev)
@@ -2780,6 +2795,8 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 
 		vfio_pci_dma_buf_move(vdev, true);
 		vfio_pci_zap_bars(vdev);
+		/* zap_bars misses the HDM window; bus reset needs it too */
+		vfio_pci_cxl_zap(vdev);
 	}
 
 	if (!list_entry_is_head(vdev,
@@ -2802,6 +2819,10 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 
 	ret = pci_reset_bus(pdev);
 
+	/* Re-sample decoder state for any CXL device the bus reset touched. */
+	list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list)
+		vfio_pci_cxl_post_reset(vdev);
+
 	vdev = list_last_entry(&dev_set->device_list,
 			       struct vfio_pci_core_device, vdev.dev_set_list);
 
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index 902d17815ab6..46e67573d264 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -82,6 +82,26 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev,
 			     pci_power_t state);
 
 void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev);
+
+static inline void vfio_pci_cxl_zap(struct vfio_pci_core_device *vdev)
+{
+	if (vdev->cxl_ops && vdev->cxl_ops->zap)
+		vdev->cxl_ops->zap(vdev);
+}
+
+static inline void vfio_pci_cxl_post_reset(struct vfio_pci_core_device *vdev)
+{
+	if (vdev->cxl_ops && vdev->cxl_ops->post_reset)
+		vdev->cxl_ops->post_reset(vdev);
+}
+
+static inline int vfio_pci_cxl_pm_restore(struct vfio_pci_core_device *vdev)
+{
+	if (vdev->cxl_ops && vdev->cxl_ops->pm_restore)
+		return vdev->cxl_ops->pm_restore(vdev);
+	return 0;
+}
+
 u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev);
 void vfio_pci_memory_unlock_and_restore(struct vfio_pci_core_device *vdev,
 					u16 cmd);
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 8b93949d4484..c438d968dc59 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -78,6 +78,10 @@ struct vfio_cxl_ops {
 				int count, __le32 val);
 	/* Revoke the HDM mapping; paired with the BAR zap */
 	void    (*zap)(struct vfio_pci_core_device *vdev);
+	/* Re-sample the decoder state once a reset has settled */
+	void    (*post_reset)(struct vfio_pci_core_device *vdev);
+	/* Restore the HDM decoder after a D3hot->D0 soft reset */
+	int     (*pm_restore)(struct vfio_pci_core_device *vdev);
 
 	/* Pinned per bound CXL device so vfio-cxl cannot unload under usage */
 	struct module *owner;
-- 
2.25.1

[PATCH v4 24/27] vfio/cxl: Service a guest-triggered CXL reset

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

When the guest sets Initiate_CXL_Reset in the CXL DVSEC, run the reset on
its behalf. The bit is not forwarded to hardware; the CXL core drives the
state machine through cxl_reset_dvsec_sequence(). The memory_lock, mapping
revoke and dma-buf quiesce are core-internal, so the core exposes a
wrapper that runs them around the CXL-specific work.

A CXL reset can clear config like an FLR, so the sequence is bracketed by
pci_save_state() and pci_restore_state(), with Bus Master dropped for the
window and kept dropped if a step failed, so the function cannot DMA over
decoders that were not restored. The guest owns Mem_Clr_Enable, read from
the shadow, and the outcome is stamped into STATUS2 for it to poll.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/cxl/vfio_cxl_core.c | 167 ++++++++++++++++++++-
 drivers/vfio/pci/vfio_pci_config.c   |   4 +-
 drivers/vfio/pci/vfio_pci_core.c     | 215 +++++++++++++++++++++++++--
 drivers/vfio/pci/vfio_pci_priv.h     |   2 +
 include/linux/vfio_pci_core.h        |   3 +
 5 files changed, 374 insertions(+), 17 deletions(-)

diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c
index f45eaa60bad2..b3eaefa03479 100644
--- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
+++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
@@ -7,6 +7,7 @@
 
 #include <linux/cleanup.h>
 #include <linux/memory-failure.h>
+#include <linux/memregion.h>
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/pci.h>
@@ -761,6 +762,16 @@ static u16 vfio_cxl_dvsec16(struct vfio_cxl_state *cxl, u32 off)
 	return (dw >> (8 * (off % sizeof(u32)))) & 0xffff;
 }
 
+/* Write a 16-bit DVSEC field into the shadow; the field must not straddle a dword. */
+static void vfio_cxl_dvsec_write16(struct vfio_cxl_state *cxl, u32 off, u16 val)
+{
+	u32 shift = 8 * (off % sizeof(u32));
+	u32 idx = off / sizeof(u32);
+
+	cxl->dvsec_shadow[idx] &= ~(0xffffU << shift);
+	cxl->dvsec_shadow[idx] |= (u32)val << shift;
+}
+
 /*
  * Apply the CXL r4.0 8.1.3 write class for the 16-bit DVSEC register at @off.
  * Control is programmable, Status is write-1-to-clear, and Capability, Lock and
@@ -815,12 +826,14 @@ static int vfio_cxl_config_write(struct vfio_pci_core_device *vdev, int pos,
 	int boff = (pos - cxl->dvsec) % sizeof(u32);
 	u32 off = idx * sizeof(u32);
 	__le32 le_wval = 0, le_wmask = 0;
+	u16 before, after, lo, hi;
 	u32 old, wval, wmask;
-	u16 lo, hi;
 
 	if (pos < cxl->dvsec || pos >= cxl->dvsec + cxl->dvsec_len)
 		return -ENODEV;
 
+	before = vfio_cxl_dvsec16(cxl, PCI_DVSEC_CXL_CTRL2);
+
 	/*
 	 * Place the guest bytes and a matching byte mask at the write offset,
 	 * then let the per-field class decide what actually lands in the shadow.
@@ -836,9 +849,159 @@ static int vfio_cxl_config_write(struct vfio_pci_core_device *vdev, int pos,
 	hi = vfio_cxl_dvsec_field(off + 2, old >> 16, wval >> 16, wmask >> 16);
 	cxl->dvsec_shadow[idx] = lo | ((u32)hi << 16);
 
+	/*
+	 * A 0->1 write of Initiate_CXL_Reset asks for a CXL reset. It is not
+	 * forwarded to hardware; cxl_reset_dvsec_sequence() drives the state
+	 * machine, and the outcome comes back through STATUS2.
+	 */
+	after = vfio_cxl_dvsec16(cxl, PCI_DVSEC_CXL_CTRL2);
+	if (!(before & PCI_DVSEC_CXL_INIT_CXL_RST) &&
+	    (after & PCI_DVSEC_CXL_INIT_CXL_RST) &&
+	    cxl_reset_capable(vdev->pdev))
+		vfio_pci_core_cxl_reset(vdev);
+
 	return count;
 }
 
+static int vfio_cxl_reset(struct vfio_pci_core_device *vdev)
+{
+	struct vfio_cxl_state *cxl = vdev->cxl;
+	struct pci_dev *pdev = vdev->pdev;
+	struct pci_saved_state *saved_state;
+	bool mem_clr;
+	u16 ctrl2, status2, cmd;
+	int ret;
+
+	lockdep_assert_held_write(&vdev->memory_lock);
+
+	/*
+	 * The host cxl_reset PCI method cannot run for a vfio-owned device: it
+	 * requests the HDM range that this driver already holds exclusively, so
+	 * it always fails busy. Drive the DVSEC reset directly here. Report
+	 * not-capable so the core reset path can fall back to a standard PCI
+	 * reset for a device with no CXL reset, such as a multifunction device.
+	 */
+	if (!cxl_reset_capable(pdev))
+		return -ENOTTY;
+
+	/*
+	 * Mem_Clr_En comes from the guest shadow CTRL2 while the device is open.
+	 * The core reset path also drives this reset at VM power on and off with
+	 * no shadow present, so read it from the live DVSEC then.
+	 */
+	if (cxl->dvsec_shadow)
+		ctrl2 = vfio_cxl_dvsec16(cxl, PCI_DVSEC_CXL_CTRL2);
+	else
+		pci_read_config_word(pdev, cxl->dvsec + PCI_DVSEC_CXL_CTRL2,
+				     &ctrl2);
+	mem_clr = ctrl2 & PCI_DVSEC_CXL_RST_MEM_CLR_EN;
+
+	/*
+	 * Mem_Clr is guest-controlled (Mem_Clr_En in the DVSEC CTRL2), so honor it
+	 * and pass it to cxl_reset_dvsec_sequence(). It zeroes the device memory,
+	 * but that sequence does not write back host CPU caches over the HDM range
+	 * the way the host cxl_reset method does. The range is mapped write-back,
+	 * so a dirty host line could survive the clear and overwrite it; invalidate
+	 * it first when the platform can.
+	 *
+	 * When cpu_cache_has_invalidate_memregion() is false proceed rather
+	 * than abort. Unlike native CXL region invalidation, which the host
+	 * manages and must fail safe, the host CPU never writes passthrough
+	 * HDM range: the guest owns it through its stage-2 mapping, so there
+	 * are no dirty host lines for the clear to lose.
+	 * Warn once so the case is visible rather than silent.
+	 */
+	if (mem_clr) {
+		if (cpu_cache_has_invalidate_memregion()) {
+			ret = cpu_cache_invalidate_memregion(cxl->hpa_range.start,
+							     range_len(&cxl->hpa_range));
+			if (ret) {
+				pci_err(pdev, "vfio-cxl: reset: CPU cache invalidate failed (%d), aborting reset\n",
+					ret);
+				return ret;
+			}
+		} else {
+			pci_warn_once(pdev, "vfio-cxl: reset: no CPU cache invalidation available; proceeding with Mem_Clr (host does not cache the HDM range)\n");
+		}
+	}
+
+	/*
+	 * A CXL reset can clear config like an FLR, so save state and drop Bus
+	 * Master for the reset window; the function masters the bus again only
+	 * once the decoders are restored.
+	 *
+	 * Sample the guest's Bus Master intent from live config before the reset
+	 * perturbs it, so it can be reapplied after the decoder is known good with
+	 * no window in which pci_restore_state() leaves Bus Master enabled.
+	 */
+	pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+
+	saved_state = pci_store_saved_state(pdev);
+	if (!saved_state && pdev->state_saved) {
+		pci_err(pdev, "vfio-cxl: reset: saved-state stash failed (-ENOMEM), aborting reset\n");
+		return -ENOMEM;
+	}
+	if (saved_state)
+		pci_save_state(pdev);
+	pci_clear_master(pdev);
+	ret = cxl_reset_dvsec_sequence(pdev, mem_clr);
+	pci_restore_state(pdev);
+	pci_clear_master(pdev);
+	/*
+	 * pci_restore_state() reinstated the guest's PCI_COMMAND, which may have
+	 * re-enabled Bus Master while the reset outcome, and thus decoder
+	 * validity, is not yet known. The pci_clear_master() above is its
+	 * immediate next statement, so there is no window in which a failed reset
+	 * could DMA over decoders that were not restored.
+	 */
+	if (saved_state)
+		pci_load_and_free_saved_state(pdev, &saved_state);
+	/*
+	 * Re-enable Bus Master only for a clean reset or -EBUSY (the reset never
+	 * ran, so the firmware-committed decoder is intact), and only if the guest
+	 * had it enabled (sampled before the reset).
+	 */
+	if ((!ret || ret == -EBUSY) && (cmd & PCI_COMMAND_MASTER))
+		pci_set_master(pdev);
+
+	vfio_cxl_post_reset(vdev);
+
+	/*
+	 * A clean reset restored the decoder, and -EBUSY means the reset never
+	 * ran so the firmware-committed decoder is intact: both are known-good.
+	 * Any other error may have left the decoder unrestored, so close the gate
+	 * until the next open or restore. Faults cannot race this: the caller
+	 * holds memory_lock for write across the whole reset.
+	 */
+	if (cxl->hdm_shadow) {
+		if (!ret || ret == -EBUSY)
+			cxl->hdm_valid = true;
+		else
+			cxl->hdm_valid = false;
+	}
+
+	/*
+	 * The guest-facing DVSEC bookkeeping only applies while the device is
+	 * open. Initiate_CXL_Reset self-clears in hardware; mirror that and
+	 * stamp the outcome onto a fresh hardware STATUS2 read for the polling
+	 * guest. A contended -EBUSY reports as an error so the guest can reissue
+	 * rather than poll a result that never comes.
+	 */
+	if (cxl->dvsec_shadow) {
+		vfio_cxl_dvsec_write16(cxl, PCI_DVSEC_CXL_CTRL2,
+				       vfio_cxl_dvsec16(cxl, PCI_DVSEC_CXL_CTRL2) &
+				       ~PCI_DVSEC_CXL_INIT_CXL_RST);
+
+		pci_read_config_word(pdev, cxl->dvsec + PCI_DVSEC_CXL_STATUS2,
+				     &status2);
+		status2 &= ~(PCI_DVSEC_CXL_RST_DONE | PCI_DVSEC_CXL_RST_ERR);
+		status2 |= ret ? PCI_DVSEC_CXL_RST_ERR : PCI_DVSEC_CXL_RST_DONE;
+		vfio_cxl_dvsec_write16(cxl, PCI_DVSEC_CXL_STATUS2, status2);
+	}
+
+	return ret;
+}
+
 static const struct vfio_cxl_ops vfio_cxl_ops = {
 	.init_device	= vfio_cxl_init_device,
 	.release_device	= vfio_cxl_release_device,
@@ -849,6 +1012,7 @@ static const struct vfio_cxl_ops vfio_cxl_ops = {
 	.zap		= vfio_cxl_zap,
 	.post_reset	= vfio_cxl_post_reset,
 	.pm_restore	= vfio_cxl_pm_restore,
+	.reset		= vfio_cxl_reset,
 	.owner		= THIS_MODULE,
 };
 
@@ -869,3 +1033,4 @@ MODULE_LICENSE("GPL");
 MODULE_DESCRIPTION("VFIO support for CXL Type-2 devices");
 MODULE_ALIAS("vfio-cxl");
 MODULE_IMPORT_NS("CXL");
+MODULE_IMPORT_NS("DEVMEM");
diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
index 01d808546a4c..f6509824988c 100644
--- a/drivers/vfio/pci/vfio_pci_config.c
+++ b/drivers/vfio/pci/vfio_pci_config.c
@@ -910,7 +910,7 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,
 		if (!ret && (cap & PCI_EXP_DEVCAP_FLR)) {
 			vfio_pci_zap_and_down_write_memory_lock(vdev);
 			vfio_pci_dma_buf_move(vdev, true);
-			pci_try_reset_function(vdev->pdev);
+			vfio_pci_reset_function(vdev);
 			vfio_pci_cxl_post_reset(vdev);
 			if (__vfio_pci_memory_enabled(vdev))
 				vfio_pci_dma_buf_move(vdev, false);
@@ -996,7 +996,7 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,
 		if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP)) {
 			vfio_pci_zap_and_down_write_memory_lock(vdev);
 			vfio_pci_dma_buf_move(vdev, true);
-			pci_try_reset_function(vdev->pdev);
+			vfio_pci_reset_function(vdev);
 			vfio_pci_cxl_post_reset(vdev);
 			if (__vfio_pci_memory_enabled(vdev))
 				vfio_pci_dma_buf_move(vdev, false);
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index fc8235c8b4fc..0fed8e00bc1d 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -635,8 +635,26 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 		goto out_power;
 
 	/* If reset fails because of the device lock, fail this path entirely */
-	ret = pci_try_reset_function(pdev);
-	if (ret == -EAGAIN)
+	if (vdev->cxl_ops && vdev->cxl_ops->reset) {
+		/*
+		 * VM power-on resets a CXL Type-2 device through its DVSEC
+		 * sequence. vconfig is not built yet here, so take memory_lock
+		 * and call the op directly rather than the wrapper.
+		 */
+		down_write(&vdev->memory_lock);
+		ret = vdev->cxl_ops->reset(vdev);
+		up_write(&vdev->memory_lock);
+	} else {
+		ret = pci_try_reset_function(pdev);
+	}
+	/*
+	 * -EAGAIN means the reset could not run. For a CXL device any reset
+	 * error must also fail the open: a failed DVSEC reset can leave the HDM
+	 * decoder cleared or unrestored, and continuing would expose the HDM
+	 * region for host access through a decoder in an unknown state.
+	 */
+	if (ret == -EAGAIN ||
+	    (vdev->cxl_ops && vdev->cxl_ops->reset && ret))
 		goto out_disable_device;
 
 	vdev->reset_works = !ret;
@@ -824,16 +842,30 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
 	 * overwrite the previously restored configuration information.
 	 */
 	if (vdev->reset_works) {
-		bridge = pci_upstream_bridge(pdev);
-		if (bridge && !pci_dev_trylock(bridge))
-			goto out_restore_state;
-		if (pci_dev_trylock(pdev)) {
-			if (!__pci_reset_function_locked(pdev))
+		if (vdev->cxl_ops && vdev->cxl_ops->reset) {
+			/*
+			 * VM power-off resets a CXL Type-2 device through its
+			 * DVSEC sequence. The sequence takes its own device lock,
+			 * so run it outside the lock below.
+			 * vconfig is already freed here, so call the op directly
+			 * under memory_lock rather than the wrapper.
+			 */
+			down_write(&vdev->memory_lock);
+			if (!vdev->cxl_ops->reset(vdev))
 				vdev->needs_reset = false;
-			pci_dev_unlock(pdev);
+			up_write(&vdev->memory_lock);
+		} else {
+			bridge = pci_upstream_bridge(pdev);
+			if (bridge && !pci_dev_trylock(bridge))
+				goto out_restore_state;
+			if (pci_dev_trylock(pdev)) {
+				if (!__pci_reset_function_locked(pdev))
+					vdev->needs_reset = false;
+				pci_dev_unlock(pdev);
+			}
+			if (bridge)
+				pci_dev_unlock(bridge);
 		}
-		if (bridge)
-			pci_dev_unlock(bridge);
 	}
 
 out_restore_state:
@@ -1441,6 +1473,20 @@ static int vfio_pci_ioctl_set_irqs(struct vfio_pci_core_device *vdev,
 	return ret;
 }
 
+/*
+ * Reset a function the way a guest asked for. A CXL Type-2 device resets
+ * through its DVSEC sequence: the host cxl_reset method would collide
+ * with the exclusive HDM range this driver holds and fail busy. Everything else
+ * takes a standard PCI function reset. The caller holds memory_lock, which the
+ * DVSEC sequence requires.
+ */
+int vfio_pci_reset_function(struct vfio_pci_core_device *vdev)
+{
+	if (vdev->cxl_ops && vdev->cxl_ops->reset)
+		return vdev->cxl_ops->reset(vdev);
+	return pci_try_reset_function(vdev->pdev);
+}
+
 static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
 				void __user *arg)
 {
@@ -1463,7 +1509,7 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
 	vfio_pci_set_power_state(vdev, PCI_D0);
 
 	vfio_pci_dma_buf_move(vdev, true);
-	ret = pci_try_reset_function(vdev->pdev);
+	ret = vfio_pci_reset_function(vdev);
 	vfio_pci_cxl_post_reset(vdev);
 	if (__vfio_pci_memory_enabled(vdev))
 		vfio_pci_dma_buf_move(vdev, false);
@@ -2736,6 +2782,21 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 		goto err_unlock;
 	}
 
+	/*
+	 * A multifunction CXL Type-2 device cannot be bus reset: its DVSEC
+	 * reset acts per function, so a shared secondary bus reset would reset
+	 * sibling functions out from under their own state. Reject it, matching
+	 * the cxl_reset bus method. A single-function device is quiesced through
+	 * its DVSEC sequence just before the reset below.
+	 */
+	list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) {
+		if (vdev->cxl_ops && vdev->cxl_ops->reset &&
+		    vdev->pdev->multifunction) {
+			ret = -ENOTTY;
+			goto err_unlock;
+		}
+	}
+
 	/*
 	 * Some of the devices in the dev_set can be in the runtime suspended
 	 * state. Increment the usage count for all the devices in the dev_set
@@ -2817,11 +2878,57 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 	list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list)
 		vfio_pci_set_power_state(vdev, PCI_D0);
 
+	/*
+	 * Quiesce each CXL Type-2 device through its DVSEC sequence before the
+	 * secondary bus reset: the bus reset alone does not write back the
+	 * device cache or tear down the HDM decoders. memory_lock is held. If a
+	 * quiesce fails, abort before the bus reset: resetting an unquiesced CXL
+	 * device risks data loss or a fabric error. Every device is locked here,
+	 * so unwind from the last one.
+	 */
+	list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) {
+		if (vdev->cxl_ops && vdev->cxl_ops->reset) {
+			ret = vdev->cxl_ops->reset(vdev);
+			if (ret) {
+				pci_warn(vdev->pdev, "vfio-cxl: hot reset: DVSEC quiesce failed (%d), aborting before bus reset\n",
+					 ret);
+				vdev = list_last_entry(&dev_set->device_list,
+						       struct vfio_pci_core_device,
+						       vdev.dev_set_list);
+				goto err_undo;
+			}
+		}
+	}
+
 	ret = pci_reset_bus(pdev);
 
-	/* Re-sample decoder state for any CXL device the bus reset touched. */
-	list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list)
-		vfio_pci_cxl_post_reset(vdev);
+	/*
+	 * pci_reset_bus() restored each device's PCI_COMMAND, which can re-enable
+	 * Bus Master, but the secondary bus reset cleared the physical HDM decoder
+	 * that the CXL quiesce above restored. Until it is restored a CXL device
+	 * could DMA over cleared decode, so for each CXL device drop Bus Master,
+	 * restore and re-sample the decoder, then re-enable Bus Master to the
+	 * guest's intent only once the decoder is known good. Re-sampling alone
+	 * would leave hdm_valid true over a cleared decoder. Keep the first restore
+	 * error so a failed restore is reported instead of the bus reset's success.
+	 */
+	list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) {
+		u16 cmd;
+		int rret;
+
+		if (!(vdev->cxl_ops && vdev->cxl_ops->reset))
+			continue;
+
+		pci_read_config_word(vdev->pdev, PCI_COMMAND, &cmd);
+		pci_clear_master(vdev->pdev);
+		rret = vfio_pci_cxl_pm_restore(vdev);
+		if (rret) {
+			if (!ret)
+				ret = rret;
+		} else if (cmd & PCI_COMMAND_MASTER) {
+			pci_set_master(vdev->pdev);
+		}
+	}
 
 	vdev = list_last_entry(&dev_set->device_list,
 			       struct vfio_pci_core_device, vdev.dev_set_list);
@@ -2876,6 +2983,16 @@ static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set)
 	if (!pdev)
 		return;
 
+	/*
+	 * A multifunction CXL Type-2 device cannot be bus reset (its DVSEC
+	 * reset is per function), so skip the automatic reset rather than reset
+	 * sibling functions out from under their state.
+	 */
+	list_for_each_entry(cur, &dev_set->device_list, vdev.dev_set_list)
+		if (cur->cxl_ops && cur->cxl_ops->reset &&
+		    cur->pdev->multifunction)
+			return;
+
 	/*
 	 * Some of the devices in the bus can be in the runtime suspended
 	 * state. Increment the usage count for all the devices in the dev_set
@@ -2884,9 +3001,56 @@ static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set)
 	if (vfio_pci_dev_set_pm_runtime_get(dev_set))
 		return;
 
+	/*
+	 * Quiesce each CXL Type-2 device through its DVSEC sequence before the
+	 * bus reset, which alone does not write back the device cache or tear
+	 * down the HDM decoders. Take memory_lock and zap the HDM window as the
+	 * explicit hot reset does. On lock contention or a failed quiesce, skip
+	 * the bus reset and leave needs_reset set for a later retry.
+	 */
+	list_for_each_entry(cur, &dev_set->device_list, vdev.dev_set_list) {
+		if (!(cur->cxl_ops && cur->cxl_ops->reset))
+			continue;
+		if (!down_write_trylock(&cur->memory_lock))
+			goto unwind;
+		vfio_pci_cxl_zap(cur);
+		if (cur->cxl_ops->reset(cur)) {
+			up_write(&cur->memory_lock);
+			goto unwind;
+		}
+	}
+
 	if (!pci_reset_bus(pdev))
 		reset_done = true;
 
+	list_for_each_entry(cur, &dev_set->device_list, vdev.dev_set_list) {
+		u16 cmd;
+
+		if (!(cur->cxl_ops && cur->cxl_ops->reset))
+			continue;
+		/*
+		 * pci_reset_bus() restored PCI_COMMAND, which can re-enable Bus
+		 * Master, but the bus reset cleared the physical HDM decoder. Drop
+		 * Bus Master, restore and re-sample the decoder, then re-enable Bus
+		 * Master to the guest's intent once the decoder is known good.
+		 * Re-sampling alone would leave hdm_valid true over a cleared
+		 * decoder.
+		 */
+		pci_read_config_word(cur->pdev, PCI_COMMAND, &cmd);
+		pci_clear_master(cur->pdev);
+		if (!vfio_pci_cxl_pm_restore(cur) && (cmd & PCI_COMMAND_MASTER))
+			pci_set_master(cur->pdev);
+		up_write(&cur->memory_lock);
+	}
+	goto out;
+
+unwind:
+	list_for_each_entry_continue_reverse(cur, &dev_set->device_list,
+					     vdev.dev_set_list)
+		if (cur->cxl_ops && cur->cxl_ops->reset)
+			up_write(&cur->memory_lock);
+
+out:
 	list_for_each_entry(cur, &dev_set->device_list, vdev.dev_set_list) {
 		if (reset_done)
 			cur->needs_reset = false;
@@ -2919,6 +3083,29 @@ void vfio_pci_core_unregister_cxl_ops(const struct vfio_cxl_ops *ops)
 }
 EXPORT_SYMBOL_GPL(vfio_pci_core_unregister_cxl_ops);
 
+/*
+ * Drive a guest-requested CXL reset. The memory_lock, mapping revoke and
+ * dma-buf quiesce are core-internal, so vfio-cxl calls in here to run them
+ * around its DVSEC reset sequence.
+ */
+int vfio_pci_core_cxl_reset(struct vfio_pci_core_device *vdev)
+{
+	int ret;
+
+	if (!vdev->cxl_ops || !vdev->cxl_ops->reset)
+		return -ENOTTY;
+
+	vfio_pci_zap_and_down_write_memory_lock(vdev);
+	vfio_pci_dma_buf_move(vdev, true);
+	ret = vdev->cxl_ops->reset(vdev);
+	if (__vfio_pci_memory_enabled(vdev))
+		vfio_pci_dma_buf_move(vdev, false);
+	up_write(&vdev->memory_lock);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_core_cxl_reset);
+
 static void vfio_pci_core_cleanup(void)
 {
 	vfio_pci_uninit_perm_bits();
diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h
index 46e67573d264..8055099cab03 100644
--- a/drivers/vfio/pci/vfio_pci_priv.h
+++ b/drivers/vfio/pci/vfio_pci_priv.h
@@ -102,6 +102,8 @@ static inline int vfio_pci_cxl_pm_restore(struct vfio_pci_core_device *vdev)
 	return 0;
 }
 
+int vfio_pci_reset_function(struct vfio_pci_core_device *vdev);
+
 u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev);
 void vfio_pci_memory_unlock_and_restore(struct vfio_pci_core_device *vdev,
 					u16 cmd);
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index c438d968dc59..18e206a35d8c 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -82,6 +82,8 @@ struct vfio_cxl_ops {
 	void    (*post_reset)(struct vfio_pci_core_device *vdev);
 	/* Restore the HDM decoder after a D3hot->D0 soft reset */
 	int     (*pm_restore)(struct vfio_pci_core_device *vdev);
+	/* Run the CXL reset sequence; the core holds memory_lock across it */
+	int     (*reset)(struct vfio_pci_core_device *vdev);
 
 	/* Pinned per bound CXL device so vfio-cxl cannot unload under usage */
 	struct module *owner;
@@ -89,6 +91,7 @@ struct vfio_cxl_ops {
 
 int vfio_pci_core_register_cxl_ops(const struct vfio_cxl_ops *ops);
 void vfio_pci_core_unregister_cxl_ops(const struct vfio_cxl_ops *ops);
+int vfio_pci_core_cxl_reset(struct vfio_pci_core_device *vdev);
 
 #if IS_ENABLED(CONFIG_VFIO_PCI_DMABUF)
 int vfio_pci_core_fill_phys_vec(struct phys_vec *phys_vec,
-- 
2.25.1

[PATCH v4 25/27] vfio/pci: Provide an opt-out for the CXL Type-2 extensions

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Add an opt-out so users can keep vfio-pci's CXL extensions out of the
path for individual devices or for an entire vfio-pci instance. The
runtime gates are:

  - Module parameter vfio_pci.disable_cxl (bool, 0444). Setting
    disable_cxl=1 at modprobe time makes vfio_pci_probe() set
    vdev->disable_cxl on every device it binds.

  - Variant drivers (nvgrace, mlx5, and others) may set vdev->disable_cxl
    in their own probe for per-device control without the module
    parameter. The bit lives on struct vfio_pci_core_device so it is
    reachable from any variant.

vfio_pci_core_init_dev() consults vdev->disable_cxl before it probes for
a CXL device, so a device that opts out is driven as plain vfio-pci:
vfio-cxl is not loaded, init_device() never runs, and the device gets no
VFIO_DEVICE_FLAGS_CXL, no HDM or component-register regions, and no DVSEC
virtualization.

The module parameter is built only when CONFIG_VFIO_CXL is enabled; the
disable_cxl bit itself is unconditional so a variant driver can set it
regardless. This mirrors the long-standing disable_denylist opt-out.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 drivers/vfio/pci/vfio_pci.c      | 9 +++++++++
 drivers/vfio/pci/vfio_pci_core.c | 8 +++++---
 include/linux/vfio_pci_core.h    | 1 +
 3 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index 830369ff878d..0ad041fffe48 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -60,6 +60,12 @@ static bool disable_denylist;
 module_param(disable_denylist, bool, 0444);
 MODULE_PARM_DESC(disable_denylist, "Disable use of device denylist. Disabling the denylist allows binding to devices with known errata that may lead to exploitable stability or security issues when accessed by untrusted users.");
 
+#if IS_ENABLED(CONFIG_VFIO_CXL)
+static bool disable_cxl;
+module_param(disable_cxl, bool, 0444);
+MODULE_PARM_DESC(disable_cxl, "Disable CXL Type-2 extensions for all devices bound to vfio-pci. A variant driver may instead set vdev->disable_cxl in its own .init callback.");
+#endif
+
 static bool vfio_pci_dev_in_denylist(struct pci_dev *pdev)
 {
 	switch (pdev->vendor) {
@@ -142,6 +148,9 @@ static int vfio_pci_init_dev(struct vfio_device *core_vdev)
 #ifdef CONFIG_VFIO_PCI_VGA
 	vdev->disable_vga = disable_vga;
 #endif
+#if IS_ENABLED(CONFIG_VFIO_CXL)
+	vdev->disable_cxl = disable_cxl;
+#endif
 
 	return vfio_pci_core_init_dev(core_vdev);
 }
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 0fed8e00bc1d..4b51a0f1e847 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -2423,10 +2423,12 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
 	xa_init(&vdev->ctx);
 
 	/*
-	 * Load vfio-cxl on demand for a CXL device. If it is absent, drive the
-	 * device as plain vfio-pci rather than failing the bind.
+	 * Load vfio-cxl on demand for a CXL device unless the user opted out.
+	 * If it is opted out or absent, drive the device as plain vfio-pci
+	 * rather than failing the bind.
 	 */
-	if (pcie_is_cxl(vdev->pdev) && vfio_pci_is_cxl_type2(vdev->pdev)) {
+	if (!vdev->disable_cxl && pcie_is_cxl(vdev->pdev) &&
+	    vfio_pci_is_cxl_type2(vdev->pdev)) {
 		const struct vfio_cxl_ops *ops;
 
 		request_module("vfio-cxl");
diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h
index 18e206a35d8c..8e3723a55c17 100644
--- a/include/linux/vfio_pci_core.h
+++ b/include/linux/vfio_pci_core.h
@@ -158,6 +158,7 @@ struct vfio_pci_core_device {
 	bool			disable_idle_d3:1;
 	bool			nointxmask:1;
 	bool			disable_vga:1;
+	bool			disable_cxl:1;
 	/* Flags modified at runtime - dedicated storage unit */
 	bool			needs_reset;
 	bool			pm_intx_masked;
-- 
2.25.1

[PATCH v4 26/27] Documentation: vfio-pci: Document CXL Type-2 device passthrough

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Describe the vfio-cxl provider module: the address model, the two regions,
the guest decoder commit handshake, DVSEC virtualization, and the reset
contract, so the supported topology and the kernel/VMM split are on record.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 Documentation/driver-api/index.rst        |   1 +
 Documentation/driver-api/vfio-pci-cxl.rst | 141 ++++++++++++++++++++++
 MAINTAINERS                               |   1 +
 3 files changed, 143 insertions(+)
 create mode 100644 Documentation/driver-api/vfio-pci-cxl.rst

diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst
index eaf7161ff957..7578f5528c28 100644
--- a/Documentation/driver-api/index.rst
+++ b/Documentation/driver-api/index.rst
@@ -46,6 +46,7 @@ of interest to most developers working on device drivers.
    uio-howto
    vfio-mediated-device
    vfio
+   vfio-pci-cxl
    vfio-pci-device-specific-driver-acceptance
 
 Bus-level documentation
diff --git a/Documentation/driver-api/vfio-pci-cxl.rst b/Documentation/driver-api/vfio-pci-cxl.rst
new file mode 100644
index 000000000000..84b41a8761a0
--- /dev/null
+++ b/Documentation/driver-api/vfio-pci-cxl.rst
@@ -0,0 +1,141 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=======================================
+VFIO-PCI: CXL Type-2 device passthrough
+=======================================
+
+Overview
+========
+
+A CXL Type-2 device is an accelerator (for example a GPU) that exposes
+host-managed device memory through an HDM decoder. vfio-pci alone does
+not expose the HDM decoder registers or the CXL Device DVSEC, and it
+does not place device memory at a guest-chosen address.
+
+The optional ``vfio-cxl`` module provides that. It is a provider for
+vfio-pci-core. vfio-pci-core does not implement CXL registers; it loads
+``vfio-cxl`` when it binds a CXL device.
+
+Address model
+=============
+
+Device memory uses three address spaces: DPA (device), HPA (host
+physical), and GPA (guest physical). The host kernel assigns the device
+memory a host physical range before the guest sees the device. The guest
+only chooses where that memory appears in its own physical address
+space, by programming a virtual endpoint HDM decoder. The guest does not
+reprogram the physical decoder.
+
+The kernel holds the HPA and does not see the GPA. The guest programs a
+GPA and does not see the HPA. The VMM holds the device fd, reads the
+committed base from the decoder-register shadow (the trapped component
+region described below), and maps the HPA-backed region at the GPA the
+guest committed. The base the guest reads back is the GPA, not the HPA.
+
+Driver model
+============
+
+There is no separate PCI driver. vfio-pci binds the device. During bind,
+vfio-pci-core detects a CXL device (``pcie_is_cxl()``), loads
+``vfio-cxl`` with ``request_module()``, and calls the registered
+``struct vfio_cxl_ops``. The module reference is pinned for the bind
+lifetime so ``vfio-cxl`` cannot unload while a device is bound.
+
+A non-CXL device, or a CXL device whose CXL setup fails, uses the
+ordinary vfio-pci paths.
+
+Regions
+=======
+
+``vfio-cxl`` adds two regions under ``VFIO_REGION_TYPE_CXL``:
+
+``VFIO_REGION_SUBTYPE_CXL_MEM``
+    The HDM region, backed by the fixed host physical range. It can be
+    mapped with mmap. The fault handler inserts the host PFNs, including
+    2 MB PMDs when the mapping is aligned. The VMM maps this region into
+    guest memory at the committed GPA.
+
+``VFIO_REGION_SUBTYPE_CXL_COMP_REGS``
+    The trapped HDM decoder registers. Access is read/write only (no
+    mmap) and must be dword aligned. A misaligned or out-of-range access
+    returns ``-EINVAL``. The kernel serves the registers from a per-open
+    shadow and runs the decoder state machine on writes. The region
+    includes a ``VFIO_REGION_INFO_CAP_CXL_COMP_REGS`` capability that
+    reports the component BAR and the offset of the decoder block within
+    it, so the VMM can place the trapped window at the address the guest
+    expects.
+
+The rest of the component BAR is a normal vfio-pci BAR.
+
+Guest decoder and commit
+========================
+
+The guest programs its endpoint decoder through the trapped region: it
+writes a base (a GPA), a size, and then the COMMIT bit. The host has
+already resolved the host physical placement, so a commit always reaches
+COMMITTED in the shadow. The physical decoder is not written. A decoder
+committed with LOCK_ON_COMMIT stays frozen until the device is reset.
+The shadow is sampled from hardware at each open, so a reset clears the
+frozen state on the next open.
+
+The VMM observes the commit, reads the committed base, and maps the HDM
+region at that GPA.
+
+CXL Device DVSEC
+================
+
+The kernel virtualizes the CXL Device DVSEC body through the config-space
+permission hooks. Reads and writes inside the DVSEC body use a per-open
+shadow. A guest write stays in the shadow and does not reach hardware.
+Accesses outside the DVSEC body go to the device as usual.
+
+Reset
+=====
+
+A guest triggers a CXL reset by writing Initiate_CXL_Reset in the CXL
+Device DVSEC. The kernel revokes the HDM mapping, saves and restores
+config around the reset, runs the CXL reset, and writes the result into
+DVSEC STATUS2 for the polling guest.
+
+Host-side resets (the reset ioctl, an FLR through config space, and a
+bus hot reset) revoke the mapping the same way and restore and re-sample
+the decoder shadow afterward. Because the kernel re-samples the
+firmware-committed decoder, the shadow returns to the committed state
+without a new guest commit.
+
+A guest that had decommitted an unlocked decoder therefore issues no new
+commit. If the VMM dropped its mapping, it must rescan the decoder after
+the DVSEC reset, the D3hot->D0 transition, and an FLR. No new commit
+will arrive. A committed, locked decoder cannot be decommitted, so this
+only applies to the unlocked case.
+
+UAPI
+====
+
+``VFIO_DEVICE_FLAGS_CXL``
+    Set in ``VFIO_DEVICE_GET_INFO`` flags for a CXL Type-2 device.
+
+``VFIO_REGION_TYPE_CXL`` with ``VFIO_REGION_SUBTYPE_CXL_MEM`` /
+``VFIO_REGION_SUBTYPE_CXL_COMP_REGS``
+    Reported through the region-info ``VFIO_REGION_INFO_CAP_TYPE``
+    capability. Userspace finds each region by scanning for the type and
+    subtype.
+
+``VFIO_REGION_INFO_CAP_CXL_COMP_REGS``
+    On the component-register region, reports the component BAR index
+    and the decoder-block offset within it.
+
+The HDM decoder register layout is available to a VMM without a private
+kernel header via ``uapi/cxl/cxl_regs.h``.
+
+Scope
+=====
+
+This support covers a single, non-interleaved endpoint decoder on a
+directly attached device. Multi-decoder devices, interleave, and
+switch-attached topologies are not supported. The interfaces are
+structured so those cases can be added later without changing the UAPI
+described here.
+
+A selftest, ``tools/testing/selftests/vfio/vfio_cxl_type2_test.c``,
+exercises the interfaces above on a bound device.
diff --git a/MAINTAINERS b/MAINTAINERS
index aa8cecbc4cf2..b9361a8d618e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -28317,6 +28317,7 @@ M:	Manish Honap <mhonap@nvidia.com>
 L:	kvm@vger.kernel.org
 L:	linux-cxl@vger.kernel.org
 S:	Supported
+F:	Documentation/driver-api/vfio-pci-cxl.rst
 F:	drivers/vfio/pci/cxl/
 
 VFIO DRIVER
-- 
2.25.1

[PATCH v4 27/27] selftests/vfio: Add CXL Type-2 passthrough corner-case tests

[mhonap]

From: Manish Honap <mhonap@nvidia.com>

Exercise the vfio-cxl contract on a bound CXL Type-2 device: the two VFIO
regions and the geometry capability, the HDM memory mmap (including a 2 MB
huge fault), the dword-aligned trapped decoder block, and the
lock-on-commit FSM. The decoder writes land in the per-open shadow and
each test reopens the device, so the FSM tests repeat cleanly.

Cover the HDM memory two ways: a host-CPU load/store of the mmap, and the
path a VMM actually uses, mmap plus a stage-2 IOAS map for the device's
ATS access. The mmap flag is required for the IOAS path, so assert it is
advertised rather than skipping when it is absent.

Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
 MAINTAINERS                                   |   1 +
 tools/testing/selftests/vfio/Makefile         |   1 +
 .../selftests/vfio/lib/vfio_pci_device.c      |  57 +-
 .../selftests/vfio/vfio_cxl_type2_test.c      | 799 ++++++++++++++++++
 4 files changed, 855 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/vfio/vfio_cxl_type2_test.c

diff --git a/MAINTAINERS b/MAINTAINERS
index b9361a8d618e..192b1681b3bd 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -28319,6 +28319,7 @@ L:	linux-cxl@vger.kernel.org
 S:	Supported
 F:	Documentation/driver-api/vfio-pci-cxl.rst
 F:	drivers/vfio/pci/cxl/
+F:	tools/testing/selftests/vfio/vfio_cxl_type2_test.c
 
 VFIO DRIVER
 M:	Alex Williamson <alex@shazbot.org>
diff --git a/tools/testing/selftests/vfio/Makefile b/tools/testing/selftests/vfio/Makefile
index 2c32c48db509..08f88e88cb4d 100644
--- a/tools/testing/selftests/vfio/Makefile
+++ b/tools/testing/selftests/vfio/Makefile
@@ -13,6 +13,7 @@ TEST_GEN_PROGS += vfio_pci_device_test
 TEST_GEN_PROGS += vfio_pci_device_init_perf_test
 TEST_GEN_PROGS += vfio_pci_driver_test
 TEST_GEN_PROGS += vfio_pci_sriov_uapi_test
+TEST_GEN_PROGS += vfio_cxl_type2_test
 
 TEST_FILES += scripts/cleanup.sh
 TEST_FILES += scripts/lib.sh
diff --git a/tools/testing/selftests/vfio/lib/vfio_pci_device.c b/tools/testing/selftests/vfio/lib/vfio_pci_device.c
index 94dc5fcecbeb..ab49b41653c4 100644
--- a/tools/testing/selftests/vfio/lib/vfio_pci_device.c
+++ b/tools/testing/selftests/vfio/lib/vfio_pci_device.c
@@ -160,9 +160,31 @@ static void vfio_pci_region_get(struct vfio_pci_device *device, int index,
 	ioctl_assert(device->fd, VFIO_DEVICE_GET_REGION_INFO, info);
 }
 
+/* Return the sparse-mmap capability in @info, or NULL if the region has none. */
+static struct vfio_region_info_cap_sparse_mmap *
+vfio_pci_sparse_mmap_cap(struct vfio_region_info *info)
+{
+	struct vfio_info_cap_header *hdr;
+	u32 offset;
+
+	if (!(info->flags & VFIO_REGION_INFO_FLAG_CAPS))
+		return NULL;
+
+	for (offset = info->cap_offset; offset; offset = hdr->next) {
+		hdr = (void *)info + offset;
+		if (hdr->id == VFIO_REGION_INFO_CAP_SPARSE_MMAP)
+			return (struct vfio_region_info_cap_sparse_mmap *)hdr;
+	}
+
+	return NULL;
+}
+
 static void vfio_pci_bar_map(struct vfio_pci_device *device, int index)
 {
 	struct vfio_pci_bar *bar = &device->bars[index];
+	struct vfio_region_info_cap_sparse_mmap *sparse;
+	u8 infobuf[1024] = {};
+	struct vfio_region_info *info = (void *)infobuf;
 	size_t align, size;
 	int prot = 0;
 	void *vaddr;
@@ -190,9 +212,38 @@ static void vfio_pci_bar_map(struct vfio_pci_device *device, int index)
 	align = min_t(size_t, size, SZ_1G);
 
 	vaddr = mmap_reserve(size, align, 0);
-	bar->vaddr = mmap(vaddr, size, prot, MAP_SHARED | MAP_FIXED,
-			  device->fd, bar->info.offset);
-	VFIO_ASSERT_NE(bar->vaddr, MAP_FAILED);
+
+	/*
+	 * A BAR that is only partially mmappable, such as a CXL Type-2 component
+	 * BAR with the HDM decoder block trapped, advertises the mmappable
+	 * ranges through a sparse-mmap capability. Map each area within the
+	 * reservation and leave the excluded ranges unmapped; mapping the whole
+	 * BAR would be rejected.
+	 */
+	info->argsz = sizeof(infobuf);
+	info->index = index;
+	ioctl_assert(device->fd, VFIO_DEVICE_GET_REGION_INFO, info);
+	sparse = vfio_pci_sparse_mmap_cap(info);
+	if (sparse) {
+		u32 i;
+
+		bar->vaddr = vaddr;
+		for (i = 0; i < sparse->nr_areas; i++) {
+			void *p;
+
+			if (!sparse->areas[i].size)
+				continue;
+			p = mmap(vaddr + sparse->areas[i].offset,
+				 sparse->areas[i].size, prot,
+				 MAP_SHARED | MAP_FIXED, device->fd,
+				 bar->info.offset + sparse->areas[i].offset);
+			VFIO_ASSERT_NE(p, MAP_FAILED);
+		}
+	} else {
+		bar->vaddr = mmap(vaddr, size, prot, MAP_SHARED | MAP_FIXED,
+				  device->fd, bar->info.offset);
+		VFIO_ASSERT_NE(bar->vaddr, MAP_FAILED);
+	}
 
 	madvise(bar->vaddr, size, MADV_HUGEPAGE);
 }
diff --git a/tools/testing/selftests/vfio/vfio_cxl_type2_test.c b/tools/testing/selftests/vfio/vfio_cxl_type2_test.c
new file mode 100644
index 000000000000..8c23ddd014ca
--- /dev/null
+++ b/tools/testing/selftests/vfio/vfio_cxl_type2_test.c
@@ -0,0 +1,799 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * vfio_cxl_type2_test - corner-case tests for the vfio-cxl kernel contract.
+ *
+ * Exercises the user-visible surface the vfio-cxl module adds to a CXL Type-2
+ * device: the two VFIO regions (HDM memory and the trapped HDM decoder block),
+ * the component-register geometry capability, and the lock-on-commit decoder
+ * FSM the kernel runs on the trapped block.
+ *
+ * Unlike a plain vfio-pci device the guest programs its own endpoint decoder,
+ * so the trapped block enforces the commit handshake and freezes a locked
+ * decoder. These tests drive that FSM directly. Writes to the decoder block
+ * land in the per-open kernel shadow only, never on the physical decoder, and
+ * each test reopens the device (fresh shadow), so the FSM tests are safe to
+ * repeat and do not leak state between tests.
+ *
+ * Usage: ./vfio_cxl_type2_test <BDF>  (or export VFIO_SELFTESTS_BDF=<BDF>).
+ * The device must be bound to vfio-pci with the vfio-cxl module available.
+ *
+ * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.
+ */
+
+#include <fcntl.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+
+#include <linux/pci_regs.h>
+#include <linux/sizes.h>
+#include <linux/vfio.h>
+
+#include <cxl/cxl_regs.h>
+
+#include <libvfio.h>
+
+#include "kselftest_harness.h"
+
+#define PCI_DVSEC_VENDOR_ID_CXL		0x1e98
+#define PCI_DVSEC_ID_CXL_DEVICE		0x0000
+
+/* CXL r3.1 8.1.9.1: Register Block Identifier for the component registers. */
+#define CXL_REGLOC_RBI_COMPONENT	1
+
+/*
+ * Register Locator DVSEC block-1 field masks. The uapi pci_regs.h names expand
+ * to __GENMASK(), which is not a macro in this userspace include path, so use
+ * explicit values.
+ */
+#define REG_LOCATOR_BIR_MASK		0x00000007
+#define REG_LOCATOR_BLOCK_ID_MASK	0x0000ff00
+#define REG_LOCATOR_BLOCK_OFF_LOW_MASK	0xffff0000
+
+/*
+ * vfio-pci's region-offset packing is kernel-internal (vfio_pci_core.h), not
+ * UAPI. Define it locally; the guards let a future kernel hoist it to UAPI.
+ */
+#ifndef VFIO_PCI_OFFSET_SHIFT
+#define VFIO_PCI_OFFSET_SHIFT		40
+#endif
+#ifndef VFIO_PCI_INDEX_TO_OFFSET
+#define VFIO_PCI_INDEX_TO_OFFSET(i)	((uint64_t)(i) << VFIO_PCI_OFFSET_SHIFT)
+#endif
+
+static const char *device_bdf;
+
+/* Locate a region-info capability by id inside a GET_REGION_INFO buffer. */
+static const struct vfio_info_cap_header *
+find_region_cap(const void *buf, size_t bufsz, uint16_t id)
+{
+	const struct vfio_region_info *ri = buf;
+	const struct vfio_info_cap_header *cap;
+	size_t off = ri->cap_offset;
+
+	while (off && off + sizeof(*cap) <= bufsz) {
+		cap = (const void *)((const char *)buf + off);
+		if (cap->id == id)
+			return cap;
+		off = cap->next;
+	}
+	return NULL;
+}
+
+/*
+ * Find a CXL region by scanning every region's VFIO_REGION_INFO_CAP_TYPE for
+ * the CXL type and the requested subtype. Returns the region index or -1.
+ * @buf is a caller scratch buffer left holding the matched region's info
+ * (with caps).
+ */
+static int find_cxl_region(int fd, uint32_t nregions, uint32_t subtype,
+			   void *buf, size_t bufsz)
+{
+	uint32_t i;
+
+	for (i = 0; i < nregions; i++) {
+		struct vfio_region_info *ri = buf;
+		const struct vfio_region_info_cap_type *t;
+		const struct vfio_info_cap_header *hdr;
+
+		memset(buf, 0, bufsz);
+		ri->argsz = bufsz;
+		ri->index = i;
+		if (ioctl(fd, VFIO_DEVICE_GET_REGION_INFO, ri))
+			continue;
+		if (!(ri->flags & VFIO_REGION_INFO_FLAG_CAPS))
+			continue;
+
+		hdr = find_region_cap(buf, bufsz, VFIO_REGION_INFO_CAP_TYPE);
+		if (!hdr)
+			continue;
+		t = (const void *)hdr;
+		if (t->type == VFIO_REGION_TYPE_CXL && t->subtype == subtype)
+			return i;
+	}
+	return -1;
+}
+
+/* Walk the PCI extended capability list for the CXL Device DVSEC. */
+static uint16_t find_cxl_dvsec(struct vfio_pci_device *dev)
+{
+	uint16_t pos = PCI_CFG_SPACE_SIZE;
+	int iter = 0;
+
+	while (pos && iter++ < 64) {
+		uint32_t hdr = vfio_pci_config_readl(dev, pos);
+		uint16_t cap_id = hdr & 0xffff;
+		uint16_t next = (hdr >> 20) & 0xffc;
+		uint32_t h1, h2;
+
+		if (cap_id == PCI_EXT_CAP_ID_DVSEC) {
+			h1 = vfio_pci_config_readl(dev, pos + 4);
+			h2 = vfio_pci_config_readl(dev, pos + 8);
+			if ((h1 & 0xffff) == PCI_DVSEC_VENDOR_ID_CXL &&
+			    (h2 & 0xffff) == PCI_DVSEC_ID_CXL_DEVICE)
+				return pos;
+		}
+		pos = next;
+	}
+	return 0;
+}
+
+FIXTURE(vfio_cxl) {
+	struct iommu *iommu;
+	struct vfio_pci_device *dev;
+
+	int mem_idx;
+	uint64_t mem_size;
+	uint32_t mem_flags;
+	int comp_idx;
+	uint64_t comp_size;
+	uint32_t comp_bar;
+	uint64_t comp_offset;	/* HDM block offset within comp_bar */
+	uint64_t comp_off;	/* mmap/rw base offset of the comp region */
+	uint16_t dvsec;
+};
+
+FIXTURE_SETUP(vfio_cxl)
+{
+	uint8_t infobuf[512] = {};
+	struct vfio_device_info *info = (void *)infobuf;
+	const struct vfio_region_info_cap_cxl_comp_regs *geo;
+	const struct vfio_info_cap_header *hdr;
+	uint8_t rbuf[1024];
+
+	self->iommu = iommu_init(default_iommu_mode);
+	self->dev = vfio_pci_device_init(device_bdf, self->iommu);
+
+	info->argsz = sizeof(infobuf);
+	ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_INFO, info));
+
+	if (!(info->flags & VFIO_DEVICE_FLAGS_CXL))
+		SKIP(return, "not a CXL Type-2 device");
+
+	self->mem_idx = find_cxl_region(self->dev->fd, info->num_regions,
+					VFIO_REGION_SUBTYPE_CXL_MEM,
+					rbuf, sizeof(rbuf));
+	ASSERT_GE(self->mem_idx, 0);
+	self->mem_size = ((struct vfio_region_info *)rbuf)->size;
+	self->mem_flags = ((struct vfio_region_info *)rbuf)->flags;
+
+	self->comp_idx = find_cxl_region(self->dev->fd, info->num_regions,
+					 VFIO_REGION_SUBTYPE_CXL_COMP_REGS,
+					 rbuf, sizeof(rbuf));
+	ASSERT_GE(self->comp_idx, 0);
+	self->comp_size = ((struct vfio_region_info *)rbuf)->size;
+
+	/* The geometry cap rides on the component-register region. */
+	hdr = find_region_cap(rbuf, sizeof(rbuf),
+			      VFIO_REGION_INFO_CAP_CXL_COMP_REGS);
+	ASSERT_NE(NULL, hdr);
+	geo = (const void *)hdr;
+	self->comp_bar = geo->bar;
+	self->comp_offset = geo->offset;
+
+	self->comp_off = VFIO_PCI_INDEX_TO_OFFSET(self->comp_idx);
+	self->dvsec = find_cxl_dvsec(self->dev);
+}
+
+FIXTURE_TEARDOWN(vfio_cxl)
+{
+	vfio_pci_device_cleanup(self->dev);
+	iommu_cleanup(self->iommu);
+}
+
+/* GET_INFO advertises the flag and both CXL regions with a sane geometry cap. */
+TEST_F(vfio_cxl, device_is_cxl)
+{
+	ASSERT_NE(self->mem_idx, self->comp_idx);
+	ASSERT_GT(self->mem_size, 0);
+	ASSERT_GT(self->comp_size, 0);
+	ASSERT_LT(self->comp_bar, PCI_STD_NUM_BARS);
+	/* The HDM memory must advertise mmap; a VMM needs it for stage-2. */
+	ASSERT_NE(0, self->mem_flags & VFIO_REGION_INFO_FLAG_MMAP);
+}
+
+/*
+ * The component BAR carries the physical HDM decoder block, which vfio traps
+ * and excludes from mmap so the guest cannot reprogram it directly. Mapping the
+ * whole BAR must fail; mapping the ranges around the excluded block, as the
+ * sparse-mmap capability advertises, must succeed.
+ */
+TEST_F(vfio_cxl, comp_bar_sparse_mmap)
+{
+	size_t page_size = getpagesize();
+	uint8_t rbuf[1024] = {};
+	struct vfio_region_info *ri = (void *)rbuf;
+	const struct vfio_region_info_cap_sparse_mmap *sm;
+	const struct vfio_info_cap_header *hdr;
+	uint64_t bar_off, decoder_page;
+	void *map;
+	uint32_t i;
+
+	/* Region info for the component BAR, with capabilities. */
+	ri->argsz = sizeof(rbuf);
+	ri->index = self->comp_bar;
+	ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_REGION_INFO, ri));
+	ASSERT_NE(0, ri->flags & VFIO_REGION_INFO_FLAG_MMAP);
+	bar_off = ri->offset;
+
+	/* The trapped decoder block splits the BAR, so it must be sparse. */
+	hdr = find_region_cap(rbuf, sizeof(rbuf),
+			      VFIO_REGION_INFO_CAP_SPARSE_MMAP);
+	ASSERT_NE(NULL, hdr);
+	sm = (const void *)hdr;
+	ASSERT_GT(sm->nr_areas, 0);
+
+	/* Mapping the whole BAR must fail: it covers the excluded block. */
+	map = mmap(NULL, ri->size, PROT_READ | PROT_WRITE, MAP_SHARED,
+		   self->dev->fd, bar_off);
+	ASSERT_EQ(MAP_FAILED, map);
+
+	/* Every advertised area is page aligned and must map. */
+	for (i = 0; i < sm->nr_areas; i++) {
+		uint64_t ao = sm->areas[i].offset;
+		uint64_t as = sm->areas[i].size;
+
+		if (!as)
+			continue;
+		ASSERT_EQ(0, ao & (page_size - 1));
+		ASSERT_EQ(0, as & (page_size - 1));
+
+		map = mmap(NULL, as, PROT_READ | PROT_WRITE, MAP_SHARED,
+			   self->dev->fd, bar_off + ao);
+		ASSERT_NE(MAP_FAILED, map);
+		ASSERT_EQ(0, munmap(map, as));
+	}
+
+	/* The page holding the decoder block must never be mmappable. */
+	decoder_page = self->comp_offset & ~(uint64_t)(page_size - 1);
+	map = mmap(NULL, page_size, PROT_READ | PROT_WRITE, MAP_SHARED,
+		   self->dev->fd, bar_off + decoder_page);
+	ASSERT_EQ(MAP_FAILED, map);
+}
+
+/* mmap one page of the HDM memory, write a pattern, read it back. */
+TEST_F(vfio_cxl, hdm_mem_mmap_rw)
+{
+	uint64_t off = VFIO_PCI_INDEX_TO_OFFSET(self->mem_idx);
+	uint32_t pattern = 0xdeadbeefU, readback = 0;
+	void *map;
+
+	if (self->mem_size < SZ_4K)
+		SKIP(return, "HDM memory < 4K");
+
+	map = mmap(NULL, SZ_4K, PROT_READ | PROT_WRITE, MAP_SHARED,
+		   self->dev->fd, off);
+	ASSERT_NE(MAP_FAILED, map);
+
+	memcpy(map, &pattern, sizeof(pattern));
+	memcpy(&readback, map, sizeof(readback));
+	ASSERT_EQ(pattern, readback);
+
+	ASSERT_EQ(0, munmap(map, SZ_4K));
+}
+
+/*
+ * A 2 MB-aligned window should map as a huge (PMD) fault. The kernel falls back
+ * to base pages when it cannot, so only correctness (write/read) is asserted.
+ */
+TEST_F(vfio_cxl, hdm_mem_huge_mmap)
+{
+	uint64_t off = VFIO_PCI_INDEX_TO_OFFSET(self->mem_idx);
+	uint32_t pattern = 0x5a5a5a5aU, readback = 0;
+	void *map, *last;
+
+	if (self->mem_size < SZ_2M)
+		SKIP(return, "HDM memory < 2M");
+
+	map = mmap(NULL, SZ_2M, PROT_READ | PROT_WRITE, MAP_SHARED,
+		   self->dev->fd, off);
+	ASSERT_NE(MAP_FAILED, map);
+
+	/* Touch the last dword so a 2 MB PMD fault covers the whole window. */
+	last = (char *)map + SZ_2M - sizeof(pattern);
+	memcpy(last, &pattern, sizeof(pattern));
+	memcpy(&readback, last, sizeof(readback));
+	ASSERT_EQ(pattern, readback);
+
+	ASSERT_EQ(0, munmap(map, SZ_2M));
+}
+
+/*
+ * A guest driver disables and re-enables PCI Memory-Space during init and
+ * reset. The committed HDM decoder stays valid across that toggle, so once
+ * Memory-Space is re-enabled the coherent HDM memory must be reachable again
+ * without a reset. This is the regression test for the hdm_valid access gate
+ * being cleared by a Memory-Space disable and never restored, which left a
+ * later valid mmap fault wrongly SIGBUS-ing.
+ *
+ * The region is exercised only through the mmap path (as a VMM does) and only
+ * while Memory-Space is enabled. An access with Memory-Space disabled aborts
+ * on the fabric as a fatal host error, so the test never attempts one: the
+ * toggle in between is pure config-space writes.
+ */
+TEST_F(vfio_cxl, hdm_mem_survives_mem_space_toggle)
+{
+	uint64_t off = VFIO_PCI_INDEX_TO_OFFSET(self->mem_idx);
+	uint32_t pattern = 0x12345678U, readback = 0;
+	uint16_t cmd;
+	void *map;
+
+	if (self->mem_size < SZ_4K)
+		SKIP(return, "HDM memory < 4K");
+
+	/* Seed a known pattern through the mmap path with Memory-Space on. */
+	cmd = vfio_pci_config_readw(self->dev, PCI_COMMAND);
+	vfio_pci_config_writew(self->dev, PCI_COMMAND,
+			       cmd | PCI_COMMAND_MEMORY);
+	map = mmap(NULL, SZ_4K, PROT_READ | PROT_WRITE, MAP_SHARED,
+		   self->dev->fd, off);
+	ASSERT_NE(MAP_FAILED, map);
+	memcpy(map, &pattern, sizeof(pattern));
+	ASSERT_EQ(0, munmap(map, SZ_4K));
+
+	/*
+	 * Toggle Memory-Space off and back on with no HDM access in between,
+	 * as a guest driver does during init/reset.
+	 */
+	vfio_pci_config_writew(self->dev, PCI_COMMAND,
+			       cmd & ~PCI_COMMAND_MEMORY);
+	vfio_pci_config_writew(self->dev, PCI_COMMAND,
+			       cmd | PCI_COMMAND_MEMORY);
+
+	/*
+	 * The committed decoder stayed valid across the toggle, so a fresh mmap
+	 * fault succeeds and the seeded pattern reads back, without a reset.
+	 * Before the fix the gate was cleared by the disable and never restored,
+	 * so the fault wrongly SIGBUS-ed.
+	 */
+	map = mmap(NULL, SZ_4K, PROT_READ | PROT_WRITE, MAP_SHARED,
+		   self->dev->fd, off);
+	ASSERT_NE(MAP_FAILED, map);
+	memcpy(&readback, map, sizeof(readback));
+	ASSERT_EQ(pattern, readback);
+	ASSERT_EQ(0, munmap(map, SZ_4K));
+
+	/* Restore PCI_COMMAND. */
+	vfio_pci_config_writew(self->dev, PCI_COMMAND, cmd);
+}
+
+/*
+ * Mirror how a VMM uses the region: mmap the HDM memory and map it into the
+ * IOAS (stage-2) so the device can reach it over ATS. The mmap flag is required
+ * for that path, so its absence is a failure, not a skip. The host CPU does not
+ * dereference the mapping; the guest reaches it through stage-2.
+ */
+TEST_F(vfio_cxl, hdm_mem_ioas_map)
+{
+	uint64_t off = VFIO_PCI_INDEX_TO_OFFSET(self->mem_idx);
+	struct iova_allocator *iova_alloc;
+	struct dma_region region;
+	void *map;
+
+	ASSERT_NE(0, self->mem_flags & VFIO_REGION_INFO_FLAG_MMAP);
+
+	/* iova_allocator_alloc() requires a power-of-2 size. */
+	if (self->mem_size < SZ_2M)
+		SKIP(return, "HDM memory < 2M");
+
+	map = mmap(NULL, SZ_2M, PROT_READ | PROT_WRITE, MAP_SHARED,
+		   self->dev->fd, off);
+	ASSERT_NE(MAP_FAILED, map);
+
+	iova_alloc = iova_allocator_init(self->iommu);
+	region.vaddr = map;
+	region.size = SZ_2M;
+	region.iova = iova_allocator_alloc(iova_alloc, SZ_2M);
+
+	iommu_map(self->iommu, &region);
+	iommu_unmap(self->iommu, &region);
+
+	iova_allocator_cleanup(iova_alloc);
+	ASSERT_EQ(0, munmap(map, SZ_2M));
+}
+
+/* The trapped block starts at the HDM decoder registers; CTRL 0 reads back. */
+TEST_F(vfio_cxl, comp_regs_hdm_read)
+{
+	uint64_t ctrl = self->comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0);
+	uint32_t val = 0;
+
+	ASSERT_GE(self->comp_size, CXL_HDM_DECODER0_CTRL_OFFSET(0) + 4);
+	ASSERT_EQ((ssize_t)sizeof(val),
+		  pread(self->dev->fd, &val, sizeof(val), ctrl));
+}
+
+/* The decoder registers only take aligned dword accesses. */
+TEST_F(vfio_cxl, comp_regs_reject_unaligned)
+{
+	uint32_t val = 0;
+	uint16_t half = 0;
+
+	/* Unaligned offset. */
+	ASSERT_EQ(-1, pread(self->dev->fd, &val, sizeof(val),
+			    self->comp_off + 1));
+	/* Non-dword size. */
+	ASSERT_EQ(-1, pread(self->dev->fd, &half, sizeof(half),
+			    self->comp_off));
+}
+
+/* Accesses past the region end are rejected. */
+TEST_F(vfio_cxl, comp_regs_reject_out_of_range)
+{
+	uint32_t val = 0;
+
+	ASSERT_EQ(-1, pread(self->dev->fd, &val, sizeof(val),
+			    self->comp_off + self->comp_size));
+}
+
+/*
+ * Commit handshake. This series supports only a firmware committed+locked
+ * decoder, which is the state the device boots in: the host resolved the HPA
+ * before the guest saw the device, so the guest view is frozen until a reset
+ * and a decommit request is ignored. If a decoder is ever seen uncommitted, the
+ * shadow FSM instead lets a commit request reach COMMITTED and a clear tear it
+ * back down; assert whichever contract applies to the decoder's actual state.
+ */
+TEST_F(vfio_cxl, hdm_commit_fsm)
+{
+	uint64_t ctrl = self->comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0);
+	uint32_t v, orig;
+
+	ASSERT_EQ((ssize_t)sizeof(orig),
+		  pread(self->dev->fd, &orig, sizeof(orig), ctrl));
+
+	if ((orig & CXL_HDM_DECODER0_CTRL_COMMITTED) &&
+	    (orig & CXL_HDM_DECODER0_CTRL_LOCK)) {
+		/* Committed+locked: a decommit request must be ignored. */
+		v = orig & ~CXL_HDM_DECODER0_CTRL_COMMIT;
+		ASSERT_EQ((ssize_t)sizeof(v),
+			  pwrite(self->dev->fd, &v, sizeof(v), ctrl));
+		ASSERT_EQ((ssize_t)sizeof(v),
+			  pread(self->dev->fd, &v, sizeof(v), ctrl));
+		ASSERT_TRUE(v & CXL_HDM_DECODER0_CTRL_COMMITTED);
+		return;
+	}
+
+	/* Uncommitted: the commit handshake round-trips through the shadow. */
+	v = orig | CXL_HDM_DECODER0_CTRL_COMMIT;
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pwrite(self->dev->fd, &v, sizeof(v), ctrl));
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pread(self->dev->fd, &v, sizeof(v), ctrl));
+	ASSERT_TRUE(v & CXL_HDM_DECODER0_CTRL_COMMITTED);
+
+	v &= ~CXL_HDM_DECODER0_CTRL_COMMIT;
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pwrite(self->dev->fd, &v, sizeof(v), ctrl));
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pread(self->dev->fd, &v, sizeof(v), ctrl));
+	ASSERT_FALSE(v & CXL_HDM_DECODER0_CTRL_COMMITTED);
+}
+
+/*
+ * Lock on commit: a decoder committed with LOCK set is frozen until reset, so
+ * a later attempt to clear COMMIT is ignored. The freeze lives in the per-open
+ * shadow, so the next test's reopen starts clean.
+ */
+TEST_F(vfio_cxl, hdm_lock_on_commit)
+{
+	uint64_t ctrl = self->comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0);
+	uint32_t v;
+
+	v = CXL_HDM_DECODER0_CTRL_COMMIT | CXL_HDM_DECODER0_CTRL_LOCK;
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pwrite(self->dev->fd, &v, sizeof(v), ctrl));
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pread(self->dev->fd, &v, sizeof(v), ctrl));
+	ASSERT_TRUE(v & CXL_HDM_DECODER0_CTRL_COMMITTED);
+	ASSERT_TRUE(v & CXL_HDM_DECODER0_CTRL_LOCK);
+
+	/* Attempt to decommit the locked decoder; it must stay committed. */
+	v = 0;
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pwrite(self->dev->fd, &v, sizeof(v), ctrl));
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pread(self->dev->fd, &v, sizeof(v), ctrl));
+	ASSERT_TRUE(v & CXL_HDM_DECODER0_CTRL_COMMITTED);
+	ASSERT_TRUE(v & CXL_HDM_DECODER0_CTRL_LOCK);
+}
+
+/*
+ * A base written to the trapped block round-trips through the shadow only while
+ * the decoder is uncommitted. In the supported production state the decoder is
+ * already committed, so its base is read-only and the write is ignored; assert
+ * whichever contract applies to the decoder's actual state.
+ */
+TEST_F(vfio_cxl, hdm_base_shadow_roundtrip)
+{
+	uint64_t lo_off = self->comp_off + CXL_HDM_DECODER0_BASE_LOW_OFFSET(0);
+	uint64_t ctrl_off = self->comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0);
+	uint32_t v = 0x30000000U;	/* 256 MB-aligned low base bits */
+	uint32_t rb = 0, orig = 0, ctrl = 0;
+
+	ASSERT_GE(self->comp_size, CXL_HDM_DECODER0_BASE_LOW_OFFSET(0) + 4);
+	ASSERT_EQ((ssize_t)sizeof(ctrl),
+		  pread(self->dev->fd, &ctrl, sizeof(ctrl), ctrl_off));
+	ASSERT_EQ((ssize_t)sizeof(orig),
+		  pread(self->dev->fd, &orig, sizeof(orig), lo_off));
+
+	ASSERT_EQ((ssize_t)sizeof(v),
+		  pwrite(self->dev->fd, &v, sizeof(v), lo_off));
+	ASSERT_EQ((ssize_t)sizeof(rb),
+		  pread(self->dev->fd, &rb, sizeof(rb), lo_off));
+
+	if (ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED) {
+		/* Committed decoder holds its base read-only; write ignored. */
+		ASSERT_EQ(orig, rb);
+	} else {
+		/* Uncommitted decoder accepts the base into the shadow. */
+		ASSERT_EQ(v, rb);
+	}
+}
+
+/*
+ * The CXL Device DVSEC body is virtualized by the kernel; a config read of it
+ * is served from the shadow (no SIGBUS / -EIO). The value itself is
+ * firmware-dependent, so only success is asserted.
+ */
+TEST_F(vfio_cxl, dvsec_body_read)
+{
+	uint32_t v;
+
+	if (!self->dvsec)
+		SKIP(return, "CXL Device DVSEC not found");
+
+	v = vfio_pci_config_readl(self->dev, self->dvsec + PCI_DVSEC_HEADER1);
+	ASSERT_NE(0xffffffffU, v);
+}
+
+/*
+ * Guest-initiated CXL reset: a 0->1 write of Initiate_CXL_Reset in the DVSEC
+ * asks the kernel to run the reset sequence. The bit self-clears and STATUS2
+ * reports the outcome. This is the path a guest drives through QEMU, so the
+ * kernel must complete it and report RESET_COMPLETE, not RESET_ERROR.
+ */
+TEST_F(vfio_cxl, guest_cxl_reset)
+{
+	uint16_t cap, ctrl2, status2;
+
+	if (!self->dvsec)
+		SKIP(return, "CXL Device DVSEC not found");
+
+	cap = vfio_pci_config_readw(self->dev, self->dvsec + PCI_DVSEC_CXL_CAP);
+	if (!(cap & PCI_DVSEC_CXL_RST_CAPABLE))
+		SKIP(return, "device does not support CXL reset");
+
+	/* Request the reset through the control register. */
+	ctrl2 = vfio_pci_config_readw(self->dev,
+				      self->dvsec + PCI_DVSEC_CXL_CTRL2);
+	vfio_pci_config_writew(self->dev, self->dvsec + PCI_DVSEC_CXL_CTRL2,
+			       ctrl2 | PCI_DVSEC_CXL_INIT_CXL_RST);
+
+	/* Initiate_CXL_Reset self-clears once the sequence has run. */
+	ctrl2 = vfio_pci_config_readw(self->dev,
+				      self->dvsec + PCI_DVSEC_CXL_CTRL2);
+	ASSERT_FALSE(ctrl2 & PCI_DVSEC_CXL_INIT_CXL_RST);
+
+	/* STATUS2 reports the outcome; a completed reset sets RESET_COMPLETE. */
+	status2 = vfio_pci_config_readw(self->dev,
+					self->dvsec + PCI_DVSEC_CXL_STATUS2);
+	ASSERT_TRUE(status2 & PCI_DVSEC_CXL_RST_DONE);
+	ASSERT_FALSE(status2 & PCI_DVSEC_CXL_RST_ERR);
+}
+
+/*
+ * The component BAR is reachable by fd read everywhere except the trapped
+ * decoder block, which is served only through the comp-regs region. A VMM
+ * relies on this split when it forwards accesses that land in the excluded
+ * mmap page: non-decoder bytes go to the BAR, the decoder goes to the trap.
+ */
+TEST_F(vfio_cxl, comp_bar_rdwr_split)
+{
+	uint64_t bar_off = VFIO_PCI_INDEX_TO_OFFSET(self->comp_bar);
+	uint32_t val;
+
+	/* A non-decoder dword of the BAR reads back through the fd. */
+	ASSERT_EQ((ssize_t)sizeof(val),
+		  pread(self->dev->fd, &val, sizeof(val), bar_off));
+
+	/* The trapped decoder block is off-limits to raw BAR fd access. */
+	ASSERT_EQ(-1, pread(self->dev->fd, &val, sizeof(val),
+			    bar_off + self->comp_offset));
+}
+
+/*
+ * Mirror the guest's HDM discovery: a guest does NOT use VFIO's geometry cap.
+ * It finds the CXL Register Locator DVSEC in config space, takes the component
+ * BAR and offset from it, walks the component-register capability array over
+ * the BAR to locate the HDM decoder, then reads decoder 0's base/size/ctrl and
+ * derives the memory range from a committed decoder.
+ */
+TEST_F(vfio_cxl, guest_hdm_discovery)
+{
+	uint64_t bar_off = VFIO_PCI_INDEX_TO_OFFSET(self->comp_bar);
+	uint32_t reg_lo, reg_hi, cap_array, cap_count, hdr;
+	uint32_t bl, bh, sl, sh, ctrl;
+	uint64_t block_off, cm, hdm_off = 0, base, size;
+	uint16_t pos = PCI_CFG_SPACE_SIZE, regloc = 0, block1;
+	int iter = 0, bar, i;
+
+	/* 1. Find the CXL Register Locator DVSEC in config space. */
+	while (pos && iter++ < 64) {
+		uint32_t h = vfio_pci_config_readl(self->dev, pos);
+
+		if ((h & 0xffff) == PCI_EXT_CAP_ID_DVSEC) {
+			uint32_t h1 = vfio_pci_config_readl(self->dev, pos + 4);
+			uint32_t h2 = vfio_pci_config_readl(self->dev, pos + 8);
+
+			if ((h1 & 0xffff) == PCI_DVSEC_VENDOR_ID_CXL &&
+			    (h2 & 0xffff) == PCI_DVSEC_CXL_REG_LOCATOR) {
+				regloc = pos;
+				break;
+			}
+		}
+		pos = (h >> 20) & 0xffc;
+	}
+	ASSERT_NE(0, regloc);
+
+	/* 2. Take the component register block BAR and offset from block 1. */
+	block1 = regloc + PCI_DVSEC_CXL_REG_LOCATOR_BLOCK1;
+	reg_lo = vfio_pci_config_readl(self->dev, block1);
+	reg_hi = vfio_pci_config_readl(self->dev, block1 + 4);
+
+	ASSERT_EQ(CXL_REGLOC_RBI_COMPONENT,
+		  (reg_lo & REG_LOCATOR_BLOCK_ID_MASK) >> 8);
+	bar = reg_lo & REG_LOCATOR_BIR_MASK;
+	block_off = ((uint64_t)reg_hi << 32) |
+		    (reg_lo & REG_LOCATOR_BLOCK_OFF_LOW_MASK);
+
+	/* The DVSEC must name the same BAR the geometry cap reported. */
+	ASSERT_EQ(self->comp_bar, bar);
+
+	/* 3. Walk the CM capability array over the BAR to find the HDM cap. */
+	cm = block_off + CXL_CM_OFFSET;
+	ASSERT_EQ((ssize_t)sizeof(cap_array),
+		  pread(self->dev->fd, &cap_array, sizeof(cap_array),
+			bar_off + cm + CXL_CM_CAP_HDR_OFFSET));
+	ASSERT_EQ(CM_CAP_HDR_CAP_ID, cap_array & CXL_CM_CAP_HDR_ID_MASK);
+
+	cap_count = (cap_array & CXL_CM_CAP_HDR_ARRAY_SIZE_MASK) >> 24;
+	for (i = 1; i <= (int)cap_count; i++) {
+		ASSERT_EQ((ssize_t)sizeof(hdr),
+			  pread(self->dev->fd, &hdr, sizeof(hdr),
+				bar_off + cm + i * 4));
+		if ((hdr & CXL_CM_CAP_HDR_ID_MASK) == CXL_CM_CAP_CAP_ID_HDM) {
+			hdm_off = cm + ((hdr & CXL_CM_CAP_PTR_MASK) >> 20);
+			break;
+		}
+	}
+	ASSERT_NE(0, hdm_off);
+
+	/* The guest's manual walk must land on the decoder the kernel traps. */
+	ASSERT_EQ(self->comp_offset, hdm_off);
+
+	/* 4. Read decoder 0 through the trapped region and derive base/size. */
+	ASSERT_EQ((ssize_t)sizeof(bl),
+		  pread(self->dev->fd, &bl, sizeof(bl),
+			self->comp_off + CXL_HDM_DECODER0_BASE_LOW_OFFSET(0)));
+	ASSERT_EQ((ssize_t)sizeof(bh),
+		  pread(self->dev->fd, &bh, sizeof(bh),
+			self->comp_off + CXL_HDM_DECODER0_BASE_HIGH_OFFSET(0)));
+	ASSERT_EQ((ssize_t)sizeof(sl),
+		  pread(self->dev->fd, &sl, sizeof(sl),
+			self->comp_off + CXL_HDM_DECODER0_SIZE_LOW_OFFSET(0)));
+	ASSERT_EQ((ssize_t)sizeof(sh),
+		  pread(self->dev->fd, &sh, sizeof(sh),
+			self->comp_off + CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(0)));
+	ASSERT_EQ((ssize_t)sizeof(ctrl),
+		  pread(self->dev->fd, &ctrl, sizeof(ctrl),
+			self->comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0)));
+
+	base = ((uint64_t)bh << 32) | bl;
+	size = ((uint64_t)sh << 32) | sl;
+
+	/*
+	 * A guest only accepts a committed decoder. When firmware left decoder 0
+	 * committed the derived range must be non-empty; the base is read to
+	 * mirror the driver even though its value is firmware-defined.
+	 */
+	if (!(ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED))
+		SKIP(return, "HDM decoder 0 not committed by firmware");
+
+	ASSERT_GT(size, 0);
+	ASSERT_LT(base, base + size);
+}
+
+/*
+ * Tie the committed decoder's advertised geometry to the HDM memory region a
+ * VMM hands the guest. Read decoder 0's base and size through the trapped
+ * region, confirm the mmap-able region covers exactly that range, then map the
+ * advertised base and touch it. A region that advertises the decoder but maps
+ * PROT_NONE (missing READ/WRITE flags) or a size that disagrees with the
+ * decoder passes discovery yet faults the guest on first access; catch that
+ * here instead of on hardware.
+ */
+TEST_F(vfio_cxl, hdm_mem_touch_committed_base)
+{
+	uint64_t mem_off = VFIO_PCI_INDEX_TO_OFFSET(self->mem_idx);
+	uint32_t pattern = 0xc0ffee11U, readback = 0;
+	uint32_t bl, bh, sl, sh, ctrl;
+	uint64_t base, size;
+	void *map;
+
+	ASSERT_EQ((ssize_t)sizeof(ctrl),
+		  pread(self->dev->fd, &ctrl, sizeof(ctrl),
+			self->comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0)));
+	if (!(ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED))
+		SKIP(return, "HDM decoder 0 not committed by firmware");
+
+	ASSERT_EQ((ssize_t)sizeof(bl),
+		  pread(self->dev->fd, &bl, sizeof(bl),
+			self->comp_off + CXL_HDM_DECODER0_BASE_LOW_OFFSET(0)));
+	ASSERT_EQ((ssize_t)sizeof(bh),
+		  pread(self->dev->fd, &bh, sizeof(bh),
+			self->comp_off + CXL_HDM_DECODER0_BASE_HIGH_OFFSET(0)));
+	ASSERT_EQ((ssize_t)sizeof(sl),
+		  pread(self->dev->fd, &sl, sizeof(sl),
+			self->comp_off + CXL_HDM_DECODER0_SIZE_LOW_OFFSET(0)));
+	ASSERT_EQ((ssize_t)sizeof(sh),
+		  pread(self->dev->fd, &sh, sizeof(sh),
+			self->comp_off + CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(0)));
+
+	base = ((uint64_t)bh << 32) | bl;
+	size = ((uint64_t)sh << 32) | sl;
+
+	ASSERT_GT(size, 0);
+	ASSERT_LT(base, base + size);
+	/* The mmap-able HDM region must cover exactly the committed decoder. */
+	ASSERT_EQ(size, self->mem_size);
+
+	if (self->mem_size < SZ_4K)
+		SKIP(return, "HDM memory < 4K");
+
+	/*
+	 * Region offset 0 is the decoder's advertised base. Map it read/write and
+	 * touch it: a PROT_NONE mapping (region missing READ/WRITE flags) faults
+	 * here rather than round-tripping the pattern.
+	 */
+	map = mmap(NULL, SZ_4K, PROT_READ | PROT_WRITE, MAP_SHARED,
+		   self->dev->fd, mem_off);
+	ASSERT_NE(MAP_FAILED, map);
+
+	memcpy(map, &pattern, sizeof(pattern));
+	memcpy(&readback, map, sizeof(readback));
+	ASSERT_EQ(pattern, readback);
+
+	ASSERT_EQ(0, munmap(map, SZ_4K));
+}
+
+int main(int argc, char *argv[])
+{
+	device_bdf = vfio_selftests_get_bdf(&argc, argv);
+	return test_harness_run(argc, argv);
+}
-- 
2.25.1