* [PATCH 01/10] linux-headers: Update vfio.h for CXL Type-2 passthrough
2026-08-13 13:06 [PATCH 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci mhonap
@ 2026-08-13 13:06 ` mhonap
2026-08-13 13:06 ` [PATCH 02/10] hw/vfio/region: Add vfio_region_setup_with_ops() mhonap
` (9 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: mhonap @ 2026-08-13 13:06 UTC (permalink / raw)
To: alex, ankita, jic23, dave.jiang, alejandro.lucero-palau,
smadhavan, pierrick.bouvier, mst, imammedo, anisinha, pbonzini,
eric.auger, peter.maydell, richard.henderson, clg, cohuck
Cc: kjaju, vsethi, zhiw, mhonap, qemu-devel, qemu-arm
From: Manish Honap <mhonap@nvidia.com>
Sync the vfio.h additions from the kernel vfio-cxl series: the
VFIO_DEVICE_FLAGS_CXL device flag, the VFIO_REGION_TYPE_CXL region type
with the HDM memory and component-register sub-types, and the
component-register geometry capability that reports the BAR and offset of
the trapped HDM decoder block.
The capability ID is provisional until the kernel series lands, at which
point this becomes a scripted linux-headers update.
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
| 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
--git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h
index f3282b8e86..f3e6c94b00 100644
--- a/linux-headers/linux/vfio.h
+++ b/linux-headers/linux/vfio.h
@@ -215,6 +215,7 @@ struct vfio_device_info {
#define VFIO_DEVICE_FLAGS_FSL_MC (1 << 6) /* vfio-fsl-mc device */
#define VFIO_DEVICE_FLAGS_CAPS (1 << 7) /* Info supports caps */
#define VFIO_DEVICE_FLAGS_CDX (1 << 8) /* vfio-cdx device */
+#define VFIO_DEVICE_FLAGS_CXL (1 << 9) /* vfio-cxl device */
__u32 num_regions; /* Max region index + 1 */
__u32 num_irqs; /* Max IRQ index + 1 */
__u32 cap_offset; /* Offset within info struct of first cap */
@@ -345,6 +346,7 @@ struct vfio_region_info_cap_type {
#define VFIO_REGION_TYPE_GFX (1)
#define VFIO_REGION_TYPE_CCW (2)
#define VFIO_REGION_TYPE_MIGRATION_DEPRECATED (3)
+#define VFIO_REGION_TYPE_CXL (4)
/* sub-types for VFIO_REGION_TYPE_PCI_* */
@@ -373,6 +375,12 @@ struct vfio_region_info_cap_type {
/* sub-types for VFIO_REGION_TYPE_GFX */
#define VFIO_REGION_SUBTYPE_GFX_EDID (1)
+/* sub-types for VFIO_REGION_TYPE_CXL */
+/* CXL.mem HDM memory window of a Type-2 device, mmap-able */
+#define VFIO_REGION_SUBTYPE_CXL_MEM (1)
+/* CXL HDM decoder registers, trapped so the guest programs a GPA it never owns */
+#define VFIO_REGION_SUBTYPE_CXL_COMP_REGS (2)
+
/**
* struct vfio_region_gfx_edid - EDID region layout.
*
@@ -497,6 +505,20 @@ struct vfio_region_info_cap_nvlink2_lnkspd {
__u32 __pad;
};
+/*
+ * Geometry of a CXL Type-2 device's HDM decoder registers, so a VMM can place
+ * the trapped component register window where the guest expects it. The cap ID
+ * is provisional pending an upstream allocation.
+ */
+#define VFIO_REGION_INFO_CAP_CXL_COMP_REGS 6
+
+struct vfio_region_info_cap_cxl_comp_regs {
+ struct vfio_info_cap_header header;
+ __u32 bar;
+ __u32 __resv;
+ __aligned_u64 offset;
+};
+
/**
* VFIO_DEVICE_GET_IRQ_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 9,
* struct vfio_irq_info)
--
2.25.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 02/10] hw/vfio/region: Add vfio_region_setup_with_ops()
2026-08-13 13:06 [PATCH 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci mhonap
2026-08-13 13:06 ` [PATCH 01/10] linux-headers: Update vfio.h for CXL Type-2 passthrough mhonap
@ 2026-08-13 13:06 ` mhonap
2026-08-13 13:06 ` [PATCH 03/10] hw/vfio/pci: Detect a CXL Type-2 device and read its geometry mhonap
` (8 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: mhonap @ 2026-08-13 13:06 UTC (permalink / raw)
To: alex, ankita, jic23, dave.jiang, alejandro.lucero-palau,
smadhavan, pierrick.bouvier, mst, imammedo, anisinha, pbonzini,
eric.auger, peter.maydell, richard.henderson, clg, cohuck
Cc: kjaju, vsethi, zhiw, mhonap, qemu-devel, qemu-arm
From: Manish Honap <mhonap@nvidia.com>
A CXL Type-2 device traps its HDM decoder block, so that region needs
custom MemoryRegionOps rather than the pass-through default. Split the
setup body out and let a caller supply the ops; NULL keeps the existing
behaviour.
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
hw/vfio/region.c | 27 ++++++++++++++++++++++++---
hw/vfio/vfio-region.h | 3 +++
2 files changed, 27 insertions(+), 3 deletions(-)
diff --git a/hw/vfio/region.c b/hw/vfio/region.c
index 54ad11a6c8..21b54978b5 100644
--- a/hw/vfio/region.c
+++ b/hw/vfio/region.c
@@ -228,8 +228,9 @@ static int vfio_setup_region_sparse_mmaps(VFIORegion *region,
return 0;
}
-int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region,
- int index, const char *name, Error **errp)
+static int vfio_region_do_setup(Object *obj, VFIODevice *vbasedev,
+ VFIORegion *region, int index, const char *name,
+ const MemoryRegionOps *ops, Error **errp)
{
struct vfio_region_info *info = NULL;
int ret;
@@ -249,7 +250,7 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region,
if (region->size) {
region->mem = g_new0(MemoryRegion, 1);
- memory_region_init_io(region->mem, obj, &vfio_region_ops,
+ memory_region_init_io(region->mem, obj, ops,
region, name, region->size);
if (!vbasedev->no_mmap &&
@@ -273,6 +274,26 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region,
return 0;
}
+int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region,
+ int index, const char *name, Error **errp)
+{
+ return vfio_region_do_setup(obj, vbasedev, region, index, name,
+ &vfio_region_ops, errp);
+}
+
+/*
+ * Like vfio_region_setup() but traps the region through @ops instead of the
+ * default pass-through, so a caller can intercept accesses (the CXL HDM
+ * decoder block). A NULL @ops keeps the default.
+ */
+int vfio_region_setup_with_ops(Object *obj, VFIODevice *vbasedev,
+ VFIORegion *region, int index, const char *name,
+ const MemoryRegionOps *ops, Error **errp)
+{
+ return vfio_region_do_setup(obj, vbasedev, region, index, name,
+ ops ? ops : &vfio_region_ops, errp);
+}
+
static void vfio_subregion_unmap(VFIORegion *region, int index)
{
trace_vfio_region_unmap(memory_region_name(®ion->mmaps[index].mem),
diff --git a/hw/vfio/vfio-region.h b/hw/vfio/vfio-region.h
index 58b236f113..8d5699013a 100644
--- a/hw/vfio/vfio-region.h
+++ b/hw/vfio/vfio-region.h
@@ -39,6 +39,9 @@ uint64_t vfio_region_read(void *opaque,
hwaddr addr, unsigned size);
int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region,
int index, const char *name, Error **errp);
+int vfio_region_setup_with_ops(Object *obj, VFIODevice *vbasedev,
+ VFIORegion *region, int index, const char *name,
+ const MemoryRegionOps *ops, Error **errp);
int vfio_region_mmap(VFIORegion *region);
void vfio_region_mmaps_set_enabled(VFIORegion *region, bool enabled);
void vfio_region_exit(VFIORegion *region);
--
2.25.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 03/10] hw/vfio/pci: Detect a CXL Type-2 device and read its geometry
2026-08-13 13:06 [PATCH 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci mhonap
2026-08-13 13:06 ` [PATCH 01/10] linux-headers: Update vfio.h for CXL Type-2 passthrough mhonap
2026-08-13 13:06 ` [PATCH 02/10] hw/vfio/region: Add vfio_region_setup_with_ops() mhonap
@ 2026-08-13 13:06 ` mhonap
2026-08-13 13:06 ` [PATCH 04/10] hw/vfio/pci: Enforce the passthrough topology for a CXL device mhonap
` (7 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: mhonap @ 2026-08-13 13:06 UTC (permalink / raw)
To: alex, ankita, jic23, dave.jiang, alejandro.lucero-palau,
smadhavan, pierrick.bouvier, mst, imammedo, anisinha, pbonzini,
eric.auger, peter.maydell, richard.henderson, clg, cohuck
Cc: kjaju, vsethi, zhiw, mhonap, qemu-devel, qemu-arm
From: Manish Honap <mhonap@nvidia.com>
The kernel marks a passed-through CXL Type-2 device with a device flag and
exposes two regions: the HDM memory (host physical) and the trapped HDM
decoder register block. Read the flag, locate both regions by type, and
take the component BAR and block offset from the geometry capability.
Realize only records this; later patches build the guest mapping on top.
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
hw/vfio/pci.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++
hw/vfio/pci.h | 15 ++++++++++++
2 files changed, 83 insertions(+)
diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 428ab2f069..157786df72 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -3570,6 +3570,60 @@ bool vfio_pci_interrupt_setup(VFIOPCIDevice *vdev, Error **errp)
return true;
}
+/*
+ * Learn the CXL geometry the kernel reports: the HPA-backed HDM memory region
+ * and the trapped HDM decoder block (which BAR carries it and at what offset).
+ * A non-CXL device leaves cxl.enabled false and takes no CXL paths.
+ */
+static bool vfio_cxl_setup(VFIOPCIDevice *vdev, Error **errp)
+{
+ VFIODevice *vbasedev = &vdev->vbasedev;
+ VFIOCXL *cxl = &vdev->cxl;
+ struct vfio_region_info *mem_info = NULL, *comp_info = NULL;
+ struct vfio_region_info_cap_cxl_comp_regs *cap;
+ struct vfio_info_cap_header *hdr;
+
+ if (!(vbasedev->flags & VFIO_DEVICE_FLAGS_CXL)) {
+ return true;
+ }
+
+ if (vfio_device_get_region_info_type(vbasedev, VFIO_REGION_TYPE_CXL,
+ VFIO_REGION_SUBTYPE_CXL_MEM,
+ &mem_info)) {
+ error_setg(errp, "vfio-cxl: %s: CXL memory region not found",
+ vbasedev->name);
+ return false;
+ }
+
+ if (vfio_device_get_region_info_type(vbasedev, VFIO_REGION_TYPE_CXL,
+ VFIO_REGION_SUBTYPE_CXL_COMP_REGS,
+ &comp_info)) {
+ error_setg(errp,
+ "vfio-cxl: %s: CXL component-register region not found",
+ vbasedev->name);
+ return false;
+ }
+
+ hdr = vfio_get_region_info_cap(comp_info,
+ VFIO_REGION_INFO_CAP_CXL_COMP_REGS);
+ if (!hdr) {
+ error_setg(errp,
+ "vfio-cxl: %s: component-register geometry not reported",
+ vbasedev->name);
+ return false;
+ }
+ cap = container_of(hdr, struct vfio_region_info_cap_cxl_comp_regs, header);
+
+ cxl->mem_region_index = mem_info->index;
+ cxl->comp_regs_region_index = comp_info->index;
+ cxl->dpa_size = mem_info->size;
+ cxl->comp_bar = cap->bar;
+ cxl->hdm_offset = cap->offset;
+ cxl->enabled = true;
+
+ return true;
+}
+
static void vfio_pci_realize(PCIDevice *pdev, Error **errp)
{
ERRP_GUARD();
@@ -3700,6 +3754,20 @@ static void vfio_pci_realize(PCIDevice *pdev, Error **errp)
}
}
+ if (!vfio_cxl_setup(vdev, errp)) {
+ /*
+ * vfio_migration_realize() above installed a migration blocker in auto
+ * mode (generic vfio-pci exposes no migration ops). out_deregister does
+ * not remove it, so a rejected CXL setup would leave VM migration
+ * blocked until QEMU restarts. Drop it here, under the same
+ * failover-pair condition used to install it.
+ */
+ if (!pdev->failover_pair_id) {
+ vfio_migration_exit(vbasedev);
+ }
+ goto out_deregister;
+ }
+
vfio_pci_register_err_notifier(vdev);
vfio_pci_register_req_notifier(vdev);
vfio_setup_resetfn_quirk(vdev);
diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h
index c9ab949870..7fdd695704 100644
--- a/hw/vfio/pci.h
+++ b/hw/vfio/pci.h
@@ -122,10 +122,25 @@ typedef struct VFIOMSIXInfo {
OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI_DEVICE)
+/*
+ * State for a CXL Type-2 device. The kernel owns the host physical placement
+ * of the device memory; QEMU only maps it at the guest physical address the
+ * guest commits into its endpoint HDM decoder.
+ */
+typedef struct VFIOCXL {
+ bool enabled;
+ uint32_t mem_region_index; /* HPA-backed HDM memory VFIO region */
+ uint32_t comp_regs_region_index; /* trapped HDM decoder register block */
+ uint32_t comp_bar; /* component BAR carrying that block */
+ uint64_t hdm_offset; /* block offset within the component BAR */
+ uint64_t dpa_size; /* size of the HDM memory region */
+} VFIOCXL;
+
struct VFIOPCIDevice {
PCIDevice parent_obj;
VFIODevice vbasedev;
+ VFIOCXL cxl;
VFIOINTx intx;
unsigned int config_size;
uint8_t *emulated_config_bits; /* QEMU emulated bits, little-endian */
--
2.25.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 04/10] hw/vfio/pci: Enforce the passthrough topology for a CXL device
2026-08-13 13:06 [PATCH 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci mhonap
` (2 preceding siblings ...)
2026-08-13 13:06 ` [PATCH 03/10] hw/vfio/pci: Detect a CXL Type-2 device and read its geometry mhonap
@ 2026-08-13 13:06 ` mhonap
2026-08-13 13:06 ` [PATCH 05/10] hw/vfio/pci: Back the CXL memory with a RAM-device region mhonap
` (6 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: mhonap @ 2026-08-13 13:06 UTC (permalink / raw)
To: alex, ankita, jic23, dave.jiang, alejandro.lucero-palau,
smadhavan, pierrick.bouvier, mst, imammedo, anisinha, pbonzini,
eric.auger, peter.maydell, richard.henderson, clg, cohuck
Cc: kjaju, vsethi, zhiw, mhonap, qemu-devel, qemu-arm
From: Manish Honap <mhonap@nvidia.com>
Only the guest's endpoint HDM decoder is programmed by the guest; the
decoders above it are covered only while the pxb-cxl host bridge stays in
passthrough mode. Reject a switch in the path, or a host bridge taken out
of passthrough, at realize before any mapping exists. The topology walk
lives in one helper so switch support later relaxes it in one place.
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
hw/vfio/pci.c | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 84 insertions(+)
diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 157786df72..1f21da9513 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -27,6 +27,7 @@
#include "hw/pci/msi.h"
#include "hw/pci/msix.h"
#include "hw/pci/pci_bridge.h"
+#include "hw/cxl/cxl.h"
#include "hw/core/qdev-properties.h"
#include "hw/core/qdev-properties-system.h"
#include "hw/vfio/vfio-cpr.h"
@@ -3570,6 +3571,84 @@ bool vfio_pci_interrupt_setup(VFIOPCIDevice *vdev, Error **errp)
return true;
}
+/*
+ * Walk the endpoint's ancestor bridges up to its pxb-cxl host bridge. A CXL
+ * switch in the path is rejected: the guest would then have to program switch
+ * decoders, which the passthrough model does not cover. This is the single
+ * place the supported-topology assumption lives, so switch support later
+ * relaxes it here.
+ */
+static PXBCXLDev *vfio_cxl_find_pxb(VFIOPCIDevice *vdev, Error **errp)
+{
+ PCIBus *bus = pci_get_bus(&vdev->parent_obj);
+
+ /*
+ * The endpoint must sit directly on a CXL root port's downstream bus. A
+ * cxl-rp carries QEMU_PCIE_CAP_CXL, so it builds a TYPE_CXL_BUS downstream
+ * bus; a generic pcie-root-port builds a TYPE_PCIE_BUS yet still counts as
+ * the single downstream port checked at bind, so without this the guest
+ * could realize a Type-2 device with no CXL root port above it: bound in
+ * appearance but unusable. The bus type is the right test here, not
+ * pci_bus_is_cxl(): the PCI_BUS_CXL flag is set only on the pxb-cxl root
+ * bus (where the root ports sit), not on the root port's downstream bus.
+ */
+ if (!object_dynamic_cast(OBJECT(bus), TYPE_CXL_BUS)) {
+ error_setg(errp,
+ "vfio-cxl: %s: endpoint is not on a CXL root port (cxl-rp)",
+ vdev->vbasedev.name);
+ return NULL;
+ }
+
+ for (; bus; ) {
+ PCIDevice *bridge = bus->parent_dev;
+
+ if (!bridge) {
+ break;
+ }
+ if (object_dynamic_cast(OBJECT(bridge), TYPE_CXL_USP) ||
+ object_dynamic_cast(OBJECT(bridge), TYPE_CXL_DSP)) {
+ error_setg(errp,
+ "vfio-cxl: %s: switch-attached topology not supported",
+ vdev->vbasedev.name);
+ return NULL;
+ }
+ if (object_dynamic_cast(OBJECT(bridge), TYPE_PXB_CXL_DEV)) {
+ return PXB_CXL_DEV(bridge);
+ }
+ if (pci_bus_is_root(bus)) {
+ break;
+ }
+ bus = pci_get_bus(bridge);
+ }
+
+ error_setg(errp, "vfio-cxl: %s: not attached below a pxb-cxl host bridge",
+ vdev->vbasedev.name);
+ return NULL;
+}
+
+/*
+ * Option I enforcement: only the guest's endpoint decoder is programmed, so
+ * the host bridge must stay in HDM passthrough mode (decoders above the
+ * endpoint are then trivially correct). Reject anything else at realize, before
+ * a mapping is built. cxl_get_hb_passthrough() reflects reset state and is not
+ * settled yet here, so gate on the static hdm_for_passthrough property.
+ */
+static bool vfio_cxl_check_topology(VFIOPCIDevice *vdev, Error **errp)
+{
+ PXBCXLDev *pxb = vfio_cxl_find_pxb(vdev, errp);
+
+ if (!pxb) {
+ return false;
+ }
+ if (pxb->hdm_for_passthrough) {
+ error_setg(errp,
+ "vfio-cxl: %s: the pxb-cxl must be in HDM passthrough mode "
+ "(do not set hdm_for_passthrough)", vdev->vbasedev.name);
+ return false;
+ }
+ return true;
+}
+
/*
* Learn the CXL geometry the kernel reports: the HPA-backed HDM memory region
* and the trapped HDM decoder block (which BAR carries it and at what offset).
@@ -3619,6 +3698,11 @@ static bool vfio_cxl_setup(VFIOPCIDevice *vdev, Error **errp)
cxl->dpa_size = mem_info->size;
cxl->comp_bar = cap->bar;
cxl->hdm_offset = cap->offset;
+
+ if (!vfio_cxl_check_topology(vdev, errp)) {
+ return false;
+ }
+
cxl->enabled = true;
return true;
--
2.25.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 05/10] hw/vfio/pci: Back the CXL memory with a RAM-device region
2026-08-13 13:06 [PATCH 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci mhonap
` (3 preceding siblings ...)
2026-08-13 13:06 ` [PATCH 04/10] hw/vfio/pci: Enforce the passthrough topology for a CXL device mhonap
@ 2026-08-13 13:06 ` mhonap
2026-08-13 13:06 ` [PATCH 06/10] hw/vfio/pci: Bind a CXL device to its fixed memory window mhonap
` (5 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: mhonap @ 2026-08-13 13:06 UTC (permalink / raw)
To: alex, ankita, jic23, dave.jiang, alejandro.lucero-palau,
smadhavan, pierrick.bouvier, mst, imammedo, anisinha, pbonzini,
eric.auger, peter.maydell, richard.henderson, clg, cohuck
Cc: kjaju, vsethi, zhiw, mhonap, qemu-devel, qemu-arm
From: Manish Honap <mhonap@nvidia.com>
mmap the kernel's HDM memory region so its host physical pages back a
RAM-device MemoryRegion. It stays out of the guest address space until
the guest commits its decoder and QEMU learns the GPA to place it at.
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
hw/vfio/pci.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
hw/vfio/pci.h | 1 +
2 files changed, 46 insertions(+)
diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 1f21da9513..066333d52d 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -3221,8 +3221,11 @@ bool vfio_pci_populate_device(VFIOPCIDevice *vdev, Error **errp)
return true;
}
+static void vfio_cxl_teardown(VFIOPCIDevice *vdev);
+
void vfio_pci_put_device(VFIOPCIDevice *vdev)
{
+ vfio_cxl_teardown(vdev);
vfio_display_finalize(vdev);
vfio_bars_finalize(vdev);
vfio_cpr_pci_unregister_device(vdev);
@@ -3666,6 +3669,19 @@ static bool vfio_cxl_setup(VFIOPCIDevice *vdev, Error **errp)
return true;
}
+ /*
+ * The HDM memory is exposed only as an mmap-backed RAM-device region; the
+ * kernel rejects fd read/write on it. With x-no-mmap the region would fall
+ * back to that always-failing fd path, so every guest access to the HDM
+ * range would fault. Reject it here rather than start an unusable VM.
+ */
+ if (vbasedev->no_mmap) {
+ error_setg(errp, "vfio-cxl: %s: x-no-mmap is not supported for a CXL "
+ "device; its HDM memory has no fd read/write fallback",
+ vbasedev->name);
+ return false;
+ }
+
if (vfio_device_get_region_info_type(vbasedev, VFIO_REGION_TYPE_CXL,
VFIO_REGION_SUBTYPE_CXL_MEM,
&mem_info)) {
@@ -3703,11 +3719,40 @@ static bool vfio_cxl_setup(VFIOPCIDevice *vdev, Error **errp)
return false;
}
+ /*
+ * The HDM memory is host physical. mmap it now; it is added to the guest
+ * address space only once the guest commits its endpoint decoder.
+ */
+ if (vfio_region_setup(OBJECT(vdev), vbasedev, &cxl->mem_region,
+ cxl->mem_region_index, "cxl-mem", errp)) {
+ return false;
+ }
+ if (vfio_region_mmap(&cxl->mem_region)) {
+ error_setg(errp, "vfio-cxl: %s: failed to mmap the HDM memory region",
+ vbasedev->name);
+ vfio_region_exit(&cxl->mem_region);
+ vfio_region_finalize(&cxl->mem_region);
+ return false;
+ }
+
cxl->enabled = true;
return true;
}
+static void vfio_cxl_teardown(VFIOPCIDevice *vdev)
+{
+ VFIOCXL *cxl = &vdev->cxl;
+
+ if (!cxl->enabled) {
+ return;
+ }
+ if (cxl->mem_region.mem) {
+ vfio_region_exit(&cxl->mem_region);
+ vfio_region_finalize(&cxl->mem_region);
+ }
+}
+
static void vfio_pci_realize(PCIDevice *pdev, Error **errp)
{
ERRP_GUARD();
diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h
index 7fdd695704..06d15e807d 100644
--- a/hw/vfio/pci.h
+++ b/hw/vfio/pci.h
@@ -134,6 +134,7 @@ typedef struct VFIOCXL {
uint32_t comp_bar; /* component BAR carrying that block */
uint64_t hdm_offset; /* block offset within the component BAR */
uint64_t dpa_size; /* size of the HDM memory region */
+ VFIORegion mem_region; /* HDM memory, mapped at committed GPA */
} VFIOCXL;
struct VFIOPCIDevice {
--
2.25.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 06/10] hw/vfio/pci: Bind a CXL device to its fixed memory window
2026-08-13 13:06 [PATCH 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci mhonap
` (4 preceding siblings ...)
2026-08-13 13:06 ` [PATCH 05/10] hw/vfio/pci: Back the CXL memory with a RAM-device region mhonap
@ 2026-08-13 13:06 ` mhonap
2026-08-13 13:06 ` [PATCH 07/10] hw/vfio/pci: Map the CXL memory on the guest decoder commit mhonap
` (4 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: mhonap @ 2026-08-13 13:06 UTC (permalink / raw)
To: alex, ankita, jic23, dave.jiang, alejandro.lucero-palau,
smadhavan, pierrick.bouvier, mst, imammedo, anisinha, pbonzini,
eric.auger, peter.maydell, richard.henderson, clg, cohuck
Cc: kjaju, vsethi, zhiw, mhonap, qemu-devel, qemu-arm
From: Manish Honap <mhonap@nvidia.com>
The guest programs a GPA inside a CXL fixed memory window and QEMU maps
the HDM memory there, so the device must sit under exactly one
single-target CFMWS. Reject any ambiguous, interleaved, unsized, or
missing window rather than guess. Match windows by target name, since the
resolved target pointers are only filled in by a machine-done notifier
that may run after this one.
The CFMWS windows are placed at machine-init-done, so a cold-plugged
device can only be validated from that notifier, where a bad topology is
fatal to startup. Factor the check into a helper that reports through errp
so a later change can also validate a hotplugged device from realize.
The endpoint's HDM size is device information QEMU already reads from the
kernel region, not a value the caller should have to guess. Sizing the
window from the device is not yet possible because it is placed in the
guest PA map before this device is realized, so treat the device
geometry as the source of truth for the check: reject a window too small
to hold the endpoint and name the exact size to set, and warn on an
oversized window, since the padding becomes guest CEDT that migration
has to preserve.
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
hw/pci-bridge/pci_expander_bridge_stubs.c | 6 +
hw/vfio/pci.c | 190 ++++++++++++++++++++++
hw/vfio/pci.h | 3 +
3 files changed, 199 insertions(+)
diff --git a/hw/pci-bridge/pci_expander_bridge_stubs.c b/hw/pci-bridge/pci_expander_bridge_stubs.c
index b35180311f..c44ad7fab6 100644
--- a/hw/pci-bridge/pci_expander_bridge_stubs.c
+++ b/hw/pci-bridge/pci_expander_bridge_stubs.c
@@ -10,5 +10,11 @@
#include "hw/pci/pci_bus.h"
#include "hw/pci-bridge/pci_expander_bridge.h"
#include "hw/cxl/cxl.h"
+#include "hw/cxl/cxl_component.h"
void pxb_cxl_hook_up_registers(CXLState *state, PCIBus *bus, Error **errp) {};
+
+bool cxl_get_hb_passthrough(PCIHostState *hb)
+{
+ return false;
+}
diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 066333d52d..bf6f36d514 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -27,7 +27,12 @@
#include "hw/pci/msi.h"
#include "hw/pci/msix.h"
#include "hw/pci/pci_bridge.h"
+#include "hw/pci/pci_host.h"
+#include "hw/pci/pcie_port.h"
#include "hw/cxl/cxl.h"
+#include "hw/cxl/cxl_host.h"
+#include "hw/cxl/cxl_component.h"
+#include "system/system.h"
#include "hw/core/qdev-properties.h"
#include "hw/core/qdev-properties-system.h"
#include "hw/vfio/vfio-cpr.h"
@@ -3652,6 +3657,187 @@ static bool vfio_cxl_check_topology(VFIOPCIDevice *vdev, Error **errp)
return true;
}
+/*
+ * Count the CXL fixed memory windows that target this device's pxb-cxl and
+ * report the matched window's base, size and target count. Match on the target
+ * names: cxl_fmws_link_targets() resolves target_hbs[] only at machine_done,
+ * which may run after this, so the resolved pointers can still be NULL here.
+ */
+static int vfio_cxl_match_fmws(PXBCXLDev *pxb, hwaddr *base, uint64_t *size,
+ int *nwindows, int *ntargets)
+{
+ GSList *list = cxl_fmws_get_all_sorted();
+ GSList *iter;
+ int matches = 0;
+
+ *nwindows = g_slist_length(list);
+ *base = 0;
+ *size = 0;
+ *ntargets = 0;
+
+ for (iter = list; iter; iter = iter->next) {
+ CXLFixedWindow *fw = CXL_FMW(iter->data);
+ int i;
+
+ for (i = 0; i < fw->num_targets; i++) {
+ bool ambiguous = false;
+ Object *t = object_resolve_path_type(fw->targets[i],
+ TYPE_PXB_CXL_DEV, &ambiguous);
+
+ if (t && !ambiguous && PXB_CXL_DEV(t) == pxb) {
+ *base = fw->base;
+ *size = fw->size;
+ *ntargets = fw->num_targets;
+ matches++;
+ break;
+ }
+ }
+ }
+ g_slist_free(list);
+ return matches;
+}
+
+/*
+ * Fix the bounds of the device's memory window once the topology has settled.
+ * Exactly one single-target CFMWS, with an assigned base and room for the HDM
+ * memory, is required; anything else is a misconfiguration the guest cannot
+ * recover from, so reject it rather than guess a window. QEMU presents this
+ * window base as the decoder base to the guest, so the guest works purely in
+ * GPA; the host physical placement stays with the kernel.
+ *
+ * Returns false and sets errp on a bad topology so the caller can either fail a
+ * hotplug device_add or abort a cold boot.
+ */
+static bool vfio_cxl_do_bind_fmws(VFIOPCIDevice *vdev, Error **errp)
+{
+ VFIOCXL *cxl = &vdev->cxl;
+ const char *name = vdev->vbasedev.name;
+ int nwindows = 0, ntargets = 0, matches;
+ hwaddr base = 0;
+ uint64_t size = 0, need;
+ PXBCXLDev *pxb;
+
+ pxb = vfio_cxl_find_pxb(vdev, errp);
+ if (!pxb) {
+ return false;
+ }
+ /*
+ * The pxb-cxl enters HDM passthrough only with a single root port and no
+ * emulated decoders, but cxl->passthrough is set by pxb_cxl_dev_reset(),
+ * which runs at the system reset after machine init done, so it is not
+ * readable from a machine-init-done notifier yet. Check the same topology
+ * the reset keys on; hdm_for_passthrough was already rejected at realize by
+ * vfio_cxl_check_topology().
+ */
+ if (pcie_count_ds_ports(PCI_HOST_BRIDGE(pxb->cxl_host_bridge)->bus) != 1) {
+ error_setg(errp,
+ "vfio-cxl: %s: pxb-cxl not in HDM passthrough mode "
+ "(use a single cxl-rp)", name);
+ return false;
+ }
+
+ /*
+ * A PCIe root-port link is point to point, but QEMU only warns when a
+ * second endpoint is cold-plugged at a nonzero slot below the port. Both
+ * endpoints would match the same single-target CFMWS and map their HDM
+ * memory at the same base, aliasing one over the other. Require exactly one
+ * endpoint below the root port.
+ */
+ {
+ PCIBus *ep_bus = pci_get_bus(&vdev->parent_obj);
+ int slot, fn, nendpoints = 0;
+
+ for (slot = 0; slot < PCI_SLOT_MAX; slot++) {
+ for (fn = 0; fn < PCI_FUNC_MAX; fn++) {
+ if (ep_bus->devices[PCI_DEVFN(slot, fn)]) {
+ nendpoints++;
+ break;
+ }
+ }
+ }
+ if (nendpoints != 1) {
+ error_setg(errp,
+ "vfio-cxl: %s: %d devices below the cxl-rp; a passed "
+ "through CXL endpoint must be alone below its root port",
+ name, nendpoints);
+ return false;
+ }
+ }
+
+ matches = vfio_cxl_match_fmws(pxb, &base, &size, &nwindows, &ntargets);
+ if (matches == 1 && ntargets == 1) {
+ if (!base) {
+ error_setg(errp,
+ "vfio-cxl: %s: matched CFMWS has no base; reduce its "
+ "size or grow the guest PA space", name);
+ return false;
+ }
+ /*
+ * QEMU reads the endpoint's HDM size from the kernel region, so the
+ * window size is really device information, not a value to make the
+ * caller guess. Sizing the CFMWS from the device the way firmware does
+ * is not possible here: the window is placed in the guest PA map by
+ * cxl_fmws_set_memmap() before this device is realized, so its size is
+ * fixed before dpa_size is known. Until a core change can size the
+ * window from the device, take the device geometry as the source of
+ * truth: reject a window too small to hold the endpoint, and warn when
+ * it is larger than needed, since the padding becomes guest CEDT that
+ * migration has to preserve.
+ */
+ need = ROUND_UP(cxl->dpa_size, 256 * MiB);
+ if (size < need) {
+ error_setg(errp,
+ "vfio-cxl: %s: CFMWS size 0x%" PRIx64 " cannot hold the "
+ "endpoint; set the cxl-fmw size to 0x%" PRIx64,
+ name, size, need);
+ return false;
+ }
+ if (size > need) {
+ warn_report("vfio-cxl: %s: CFMWS size 0x%" PRIx64 " exceeds the "
+ "endpoint's 0x%" PRIx64 "; set the cxl-fmw size to 0x%"
+ PRIx64 " to keep the guest memory map stable across "
+ "migration", name, size, cxl->dpa_size, need);
+ }
+ cxl->fmws_base = base;
+ cxl->fmws_size = size;
+ return true;
+ }
+
+ if (matches == 1) {
+ error_setg(errp,
+ "vfio-cxl: %s: its CFMWS interleaves %d targets; use a "
+ "single-target window", name, ntargets);
+ } else if (matches > 1) {
+ error_setg(errp,
+ "vfio-cxl: %s: pxb-cxl is targeted by %d CFMWS; use one",
+ name, matches);
+ } else {
+ error_setg(errp,
+ "vfio-cxl: %s: no CFMWS targets this device (%d present)",
+ name, nwindows);
+ }
+ return false;
+}
+
+/*
+ * Cold-plug path: the CFMWS windows are placed at machine init done, so the
+ * binding can only be validated from this notifier. The VM has not run yet, so
+ * a configuration error is fatal to startup. A hotplugged device is validated
+ * in realize instead (see vfio_cxl_setup), where the failure fails device_add
+ * without taking down the running VM.
+ */
+static void vfio_cxl_bind_fmws(Notifier *n, void *data)
+{
+ VFIOCXL *cxl = container_of(n, VFIOCXL, machine_done);
+ VFIOPCIDevice *vdev = container_of(cxl, VFIOPCIDevice, cxl);
+ Error *err = NULL;
+
+ if (!vfio_cxl_do_bind_fmws(vdev, &err)) {
+ error_report_err(err);
+ exit(1);
+ }
+}
+
/*
* Learn the CXL geometry the kernel reports: the HPA-backed HDM memory region
* and the trapped HDM decoder block (which BAR carries it and at what offset).
@@ -3735,6 +3921,9 @@ static bool vfio_cxl_setup(VFIOPCIDevice *vdev, Error **errp)
return false;
}
+ cxl->machine_done.notify = vfio_cxl_bind_fmws;
+ qemu_add_machine_init_done_notifier(&cxl->machine_done);
+
cxl->enabled = true;
return true;
@@ -3747,6 +3936,7 @@ static void vfio_cxl_teardown(VFIOPCIDevice *vdev)
if (!cxl->enabled) {
return;
}
+ qemu_remove_machine_init_done_notifier(&cxl->machine_done);
if (cxl->mem_region.mem) {
vfio_region_exit(&cxl->mem_region);
vfio_region_finalize(&cxl->mem_region);
diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h
index 06d15e807d..6f184520bf 100644
--- a/hw/vfio/pci.h
+++ b/hw/vfio/pci.h
@@ -135,6 +135,9 @@ typedef struct VFIOCXL {
uint64_t hdm_offset; /* block offset within the component BAR */
uint64_t dpa_size; /* size of the HDM memory region */
VFIORegion mem_region; /* HDM memory, mapped at committed GPA */
+ Notifier machine_done; /* CFMWS validated at machine_done */
+ hwaddr fmws_base; /* base of the memory window */
+ uint64_t fmws_size; /* size of that window */
} VFIOCXL;
struct VFIOPCIDevice {
--
2.25.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 07/10] hw/vfio/pci: Map the CXL memory on the guest decoder commit
2026-08-13 13:06 [PATCH 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci mhonap
` (5 preceding siblings ...)
2026-08-13 13:06 ` [PATCH 06/10] hw/vfio/pci: Bind a CXL device to its fixed memory window mhonap
@ 2026-08-13 13:06 ` mhonap
2026-08-13 13:06 ` [PATCH 08/10] docs/cxl: Document CXL Type-2 device passthrough mhonap
` (3 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: mhonap @ 2026-08-13 13:06 UTC (permalink / raw)
To: alex, ankita, jic23, dave.jiang, alejandro.lucero-palau,
smadhavan, pierrick.bouvier, mst, imammedo, anisinha, pbonzini,
eric.auger, peter.maydell, richard.henderson, clg, cohuck
Cc: kjaju, vsethi, zhiw, mhonap, qemu-devel, qemu-arm
From: Manish Honap <mhonap@nvidia.com>
Overlay the trapped HDM decoder block over the component BAR and forward
its accesses to the kernel, which runs the lock-on-commit FSM. This
hardware presents a firmware-fixed decoder, so QEMU does not honor the
base the guest programmed: when a decoder is committed, QEMU maps the HDM
memory at the base of the device's CFMWS window and presents that same
base back to the guest through the trapped decoder read, so the guest
works purely in GPA and never sees the host physical base. A decommit
removes the mapping.
Overlap the CFMWS window MemoryRegion the CXL host bridge maps at that
base, at a higher priority, so precedence is defined by priority rather
than subregion insertion order.
The trapped component region and its err: cleanup exist here, so this is
also where a hotplugged device is bound: validate the CFMWS from realize
and report through errp, so a bad device_add fails cleanly instead of
aborting the running VM. A cold-plugged device is still validated from
the machine-init-done notifier.
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
hw/vfio/pci.c | 371 ++++++++++++++++++++++++++++++++++++++++++++++++--
hw/vfio/pci.h | 5 +
2 files changed, 368 insertions(+), 8 deletions(-)
diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index bf6f36d514..dac3011739 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -1453,6 +1453,39 @@ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len)
return val;
}
+static void vfio_cxl_decoder_changed(VFIOPCIDevice *vdev);
+static void vfio_cxl_unmap_mem(VFIOPCIDevice *vdev);
+
+/*
+ * Return true if this config write sets a Function Level Reset bit the kernel
+ * acts on: the PCIe Device Control FLR bit or the Advanced Features FLR bit.
+ * FLR bits are write-1-to-trigger and self-clearing, so inspect the written
+ * value rather than the post-write config.
+ */
+static bool vfio_cxl_flr_write(VFIOPCIDevice *vdev, uint32_t addr,
+ uint32_t val, int len)
+{
+ PCIDevice *pdev = &vdev->parent_obj;
+ uint32_t off;
+
+ if (pdev->exp.exp_cap) {
+ /* PCI_EXP_DEVCTL_BCR_FLR (bit 15) is the high byte of Device Ctrl. */
+ off = pdev->exp.exp_cap + PCI_EXP_DEVCTL + 1;
+ if (off >= addr && off < addr + len &&
+ ((val >> (8 * (off - addr))) & (PCI_EXP_DEVCTL_BCR_FLR >> 8))) {
+ return true;
+ }
+ }
+ if (vdev->cxl.af_offset) {
+ off = vdev->cxl.af_offset + PCI_AF_CTRL;
+ if (off >= addr && off < addr + len &&
+ ((val >> (8 * (off - addr))) & PCI_AF_CTRL_FLR)) {
+ return true;
+ }
+ }
+ return false;
+}
+
void vfio_pci_write_config(PCIDevice *pdev,
uint32_t addr, uint32_t val, int len)
{
@@ -1522,9 +1555,74 @@ void vfio_pci_write_config(PCIDevice *pdev,
vfio_sub_page_bar_update_mapping(pdev, bar);
}
}
+
+ /*
+ * A firmware-committed CXL decoder is already committed at boot, so no
+ * guest control write triggers the HDM mapping. Map it once the guest
+ * enables memory decoding, so the region enters the guest address space
+ * and the IOAS while the device is live. On a clear of memory decoding,
+ * withdraw the overlay: the kernel revokes the HDM PTEs on the same
+ * write, so a retained memslot would fault a guest access during the
+ * disabled interval onto a zapped VMA and stop the VM; the enable path
+ * re-maps it.
+ */
+ if (vdev->cxl.enabled &&
+ range_covers_byte(addr, len, PCI_COMMAND)) {
+ if (pci_get_word(pdev->config + PCI_COMMAND) & PCI_COMMAND_MEMORY) {
+ vfio_cxl_decoder_changed(vdev);
+ } else {
+ vfio_cxl_unmap_mem(vdev);
+ }
+ }
} else {
/* Write everything to QEMU to keep emulated bits correct */
pci_default_write_config(pdev, addr, val, len);
+
+ /*
+ * A guest CXL reset is a write to the CXL Device DVSEC ctrl2 register,
+ * forwarded to the kernel above, which re-commits this firmware-fixed
+ * decoder. If the guest decommitted the decoder before the reset, which
+ * unmapped the HDM memory, that re-commit is not otherwise visible to
+ * QEMU, so rescan the decoder here to restore the mapping. Gated on
+ * memory decoding still being enabled, like the enable path above.
+ */
+ if (vdev->cxl.enabled && vdev->cxl.dvsec_offset &&
+ ranges_overlap(addr, len,
+ vdev->cxl.dvsec_offset +
+ offsetof(CXLDVSECDevice, ctrl2),
+ sizeof_field(CXLDVSECDevice, ctrl2)) &&
+ (pci_get_word(pdev->config + PCI_COMMAND) & PCI_COMMAND_MEMORY)) {
+ vfio_cxl_decoder_changed(vdev);
+ }
+
+ /*
+ * A NoSoftRst- device the guest cycles D3hot->D0 has its physical
+ * decoder restored by the kernel on resume, which re-commits this
+ * firmware-fixed decoder just like a reset. That re-commit is not
+ * otherwise visible to QEMU, so on the transition back to D0 rescan
+ * the decoder to restore a mapping the guest dropped before it
+ * suspended. Gated on memory decoding, like the paths above.
+ */
+ if (vdev->cxl.enabled && pdev->pm_cap &&
+ range_covers_byte(addr, len, pdev->pm_cap + PCI_PM_CTRL) &&
+ (pci_get_word(pdev->config + pdev->pm_cap + PCI_PM_CTRL) &
+ PCI_PM_CTRL_STATE_MASK) == 0 &&
+ (pci_get_word(pdev->config + PCI_COMMAND) & PCI_COMMAND_MEMORY)) {
+ vfio_cxl_decoder_changed(vdev);
+ }
+
+ /*
+ * A Function Level Reset the guest triggers through the PCIe Device
+ * Control or Advanced Features FLR bit is forwarded to the kernel,
+ * which re-commits this firmware-fixed decoder just like a CXL reset.
+ * That re-commit is not otherwise visible to QEMU, so rescan the
+ * decoder to restore a mapping the guest dropped before the FLR. Gated
+ * on memory decoding, like the paths above.
+ */
+ if (vdev->cxl.enabled && vfio_cxl_flr_write(vdev, addr, val, len) &&
+ (pci_get_word(pdev->config + PCI_COMMAND) & PCI_COMMAND_MEMORY)) {
+ vfio_cxl_decoder_changed(vdev);
+ }
}
}
@@ -3838,6 +3936,219 @@ static void vfio_cxl_bind_fmws(Notifier *n, void *data)
}
}
+/*
+ * HDM decoder registers, relative to the trapped decoder block. The block holds
+ * the HDM Decoder Capability register followed by one register set per decoder,
+ * each VFIO_CXL_HDM_DECODER_STRIDE apart. Index the sets by decoder so
+ * multi-decoder support is a loop bound rather than a rewrite; this generation
+ * commits a single decoder.
+ */
+#define VFIO_CXL_HDM_CAP 0x00
+#define VFIO_CXL_HDM_DECODER_STRIDE 0x20
+#define VFIO_CXL_HDM_DECODER_BASE_LOW(n) \
+ (0x10 + (n) * VFIO_CXL_HDM_DECODER_STRIDE)
+#define VFIO_CXL_HDM_DECODER_BASE_HIGH(n) \
+ (0x14 + (n) * VFIO_CXL_HDM_DECODER_STRIDE)
+#define VFIO_CXL_HDM_DECODER_CTRL(n) \
+ (0x20 + (n) * VFIO_CXL_HDM_DECODER_STRIDE)
+#define VFIO_CXL_HDM_CTRL_COMMITTED (1 << 10)
+#define VFIO_CXL_HDM_BASE_LOW_MASK 0xf0000000U
+
+static void vfio_cxl_unmap_mem(VFIOPCIDevice *vdev)
+{
+ VFIOCXL *cxl = &vdev->cxl;
+
+ if (!cxl->dpa_mapped) {
+ return;
+ }
+ memory_region_del_subregion(get_system_memory(), cxl->mem_region.mem);
+ cxl->dpa_mapped = false;
+ cxl->mapped_base = 0;
+}
+
+/*
+ * The HDM Decoder Capability register encodes the decoder count in its low
+ * nibble (0 -> 1, 1 -> 2, 2 -> 4, 3 -> 6, 4 -> 8 ...). Read it from the trapped
+ * block so the commit handler walks every decoder rather than assuming decoder
+ * 0. An unreadable or unknown encoding is treated as a single decoder.
+ */
+static unsigned vfio_cxl_decoder_count(VFIOPCIDevice *vdev)
+{
+ off_t off = vdev->cxl.comp_regs_region.fd_offset;
+ uint32_t cap = 0;
+
+ if (pread(vdev->vbasedev.fd, &cap, 4, off + VFIO_CXL_HDM_CAP) != 4) {
+ return 1;
+ }
+ switch (le32_to_cpu(cap) & 0xf) {
+ case 0: return 1;
+ case 1: return 2;
+ case 2: return 4;
+ case 3: return 6;
+ case 4: return 8;
+ default: return 1;
+ }
+}
+
+/*
+ * The guest committed or tore down an endpoint decoder. Walk each decoder in
+ * the trapped HDM block, not just decoder 0. QEMU does not honor the base the
+ * guest programmed: the host decoder is firmware-fixed, so QEMU maps the HDM
+ * memory at the base of the device's CFMWS window and presents that same base
+ * back to the guest (see vfio_cxl_comp_regs_read). The guest works purely in
+ * GPA and never sees the host physical base the kernel shadow holds; the shadow
+ * base and size are not consulted here.
+ *
+ * This generation commits a single, non-interleaved decoder (the kernel
+ * refuses to bind otherwise) and exposes one HDM memory region, so the walk
+ * stops at the first committed decoder. Keeping it indexed makes multi-decoder
+ * support a kernel-gate relaxation rather than a QEMU rewrite.
+ */
+static void vfio_cxl_decoder_changed(VFIOPCIDevice *vdev)
+{
+ VFIOCXL *cxl = &vdev->cxl;
+ VFIODevice *vbasedev = &vdev->vbasedev;
+ off_t off = cxl->comp_regs_region.fd_offset;
+ unsigned n, count = vfio_cxl_decoder_count(vdev);
+
+ for (n = 0; n < count; n++) {
+ uint32_t ctrl = 0;
+
+ if (pread(vbasedev->fd, &ctrl, 4,
+ off + VFIO_CXL_HDM_DECODER_CTRL(n)) != 4) {
+ return;
+ }
+ if (!(le32_to_cpu(ctrl) & VFIO_CXL_HDM_CTRL_COMMITTED)) {
+ continue;
+ }
+
+ /*
+ * Map the HDM memory at the guest physical base of the device's CFMWS
+ * window, which is also the base QEMU presents to the guest for this
+ * decoder (see vfio_cxl_comp_regs_read). The guest never sees the host
+ * physical base the kernel shadow holds.
+ */
+ if (cxl->dpa_mapped && cxl->mapped_base == cxl->fmws_base) {
+ return;
+ }
+ /*
+ * Overlap the CFMWS window MemoryRegion the CXL host bridge already
+ * maps at this base, at a higher priority, so precedence is defined by
+ * priority rather than subregion insertion order.
+ */
+ memory_region_transaction_begin();
+ vfio_cxl_unmap_mem(vdev);
+ memory_region_add_subregion_overlap(get_system_memory(),
+ cxl->fmws_base,
+ cxl->mem_region.mem, 1);
+ memory_region_transaction_commit();
+ cxl->mapped_base = cxl->fmws_base;
+ cxl->dpa_mapped = true;
+ return;
+ }
+
+ /* No committed decoder in the block: tear down any existing mapping. */
+ vfio_cxl_unmap_mem(vdev);
+}
+
+static uint64_t vfio_cxl_comp_regs_read(void *opaque, hwaddr addr,
+ unsigned size)
+{
+ VFIORegion *region = opaque;
+ VFIODevice *vbasedev = region->vbasedev;
+ VFIOPCIDevice *vdev =
+ container_of(region, VFIOPCIDevice, cxl.comp_regs_region);
+ VFIOCXL *cxl = &vdev->cxl;
+ uint32_t val = 0xffffffff;
+
+ if (pread(vbasedev->fd, &val, size, region->fd_offset + addr) != size) {
+ error_report("vfio-cxl: %s: HDM decoder read at 0x%" HWADDR_PRIx
+ " failed", vbasedev->name, addr);
+ }
+ val = le32_to_cpu(val);
+
+ /*
+ * The kernel shadow holds the host physical base for a firmware-committed
+ * decoder. Never expose that to the guest: present the guest physical base
+ * QEMU maps the HDM memory at (the device's CFMWS window). Any decoder's
+ * base registers are virtualized; ctrl, size and the capability header pass
+ * through. The base registers repeat every VFIO_CXL_HDM_DECODER_STRIDE.
+ */
+ if (addr >= VFIO_CXL_HDM_DECODER_BASE_LOW(0) &&
+ (addr - VFIO_CXL_HDM_DECODER_BASE_LOW(0)) %
+ VFIO_CXL_HDM_DECODER_STRIDE == 0) {
+ val = (val & ~VFIO_CXL_HDM_BASE_LOW_MASK) |
+ ((uint32_t)cxl->fmws_base & VFIO_CXL_HDM_BASE_LOW_MASK);
+ } else if (addr >= VFIO_CXL_HDM_DECODER_BASE_HIGH(0) &&
+ (addr - VFIO_CXL_HDM_DECODER_BASE_HIGH(0)) %
+ VFIO_CXL_HDM_DECODER_STRIDE == 0) {
+ val = (uint32_t)(cxl->fmws_base >> 32);
+ }
+
+ return val;
+}
+
+static void vfio_cxl_comp_regs_write(void *opaque, hwaddr addr, uint64_t data,
+ unsigned size)
+{
+ VFIORegion *region = opaque;
+ VFIODevice *vbasedev = region->vbasedev;
+ VFIOPCIDevice *vdev =
+ container_of(region, VFIOPCIDevice, cxl.comp_regs_region);
+ uint32_t val = cpu_to_le32((uint32_t)data);
+
+ if (pwrite(vbasedev->fd, &val, size, region->fd_offset + addr) != size) {
+ error_report("vfio-cxl: %s: HDM decoder write at 0x%" HWADDR_PRIx
+ " failed", vbasedev->name, addr);
+ return;
+ }
+
+ /*
+ * The kernel runs the lock-on-commit FSM in the write above, so the
+ * committed state is settled by now; a control write on any decoder can
+ * change the mapping.
+ */
+ if (addr >= VFIO_CXL_HDM_DECODER_CTRL(0) &&
+ (addr - VFIO_CXL_HDM_DECODER_CTRL(0)) %
+ VFIO_CXL_HDM_DECODER_STRIDE == 0) {
+ vfio_cxl_decoder_changed(vdev);
+ }
+}
+
+static const MemoryRegionOps vfio_cxl_comp_regs_ops = {
+ .read = vfio_cxl_comp_regs_read,
+ .write = vfio_cxl_comp_regs_write,
+ .endianness = DEVICE_LITTLE_ENDIAN,
+ .valid = { .min_access_size = 4, .max_access_size = 4 },
+ .impl = { .min_access_size = 4, .max_access_size = 4 },
+};
+
+/*
+ * Locate the CXL Device DVSEC (CXL r3.1 8.1.3) in config space. The guest
+ * triggers a CXL reset by writing its ctrl2 register; QEMU rescans the decoder
+ * after that write so the HDM mapping is restored (see vfio_pci_write_config).
+ * Returns the DVSEC config offset, or 0 if the device does not expose it.
+ */
+static uint16_t vfio_cxl_find_device_dvsec(PCIDevice *pdev)
+{
+ uint16_t offset;
+
+ for (offset = PCI_CONFIG_SPACE_SIZE; offset;
+ offset = PCI_EXT_CAP_NEXT(pci_get_long(pdev->config + offset))) {
+ uint32_t hdr = pci_get_long(pdev->config + offset);
+
+ if (PCI_EXT_CAP_ID(hdr) == PCI_EXT_CAP_ID_DVSEC &&
+ (pci_get_long(pdev->config + offset + PCI_DVSEC_HEADER1) & 0xffff)
+ == CXL_VENDOR_ID &&
+ pci_get_word(pdev->config + offset + PCI_DVSEC_HEADER2)
+ == PCIE_CXL_DEVICE_DVSEC) {
+ return offset;
+ }
+ }
+
+ return 0;
+}
+
/*
* Learn the CXL geometry the kernel reports: the HPA-backed HDM memory region
* and the trapped HDM decoder block (which BAR carries it and at what offset).
@@ -3900,6 +4211,8 @@ static bool vfio_cxl_setup(VFIOPCIDevice *vdev, Error **errp)
cxl->dpa_size = mem_info->size;
cxl->comp_bar = cap->bar;
cxl->hdm_offset = cap->offset;
+ cxl->dvsec_offset = vfio_cxl_find_device_dvsec(&vdev->parent_obj);
+ cxl->af_offset = pci_find_capability(&vdev->parent_obj, PCI_CAP_ID_AF);
if (!vfio_cxl_check_topology(vdev, errp)) {
return false;
@@ -3916,31 +4229,73 @@ static bool vfio_cxl_setup(VFIOPCIDevice *vdev, Error **errp)
if (vfio_region_mmap(&cxl->mem_region)) {
error_setg(errp, "vfio-cxl: %s: failed to mmap the HDM memory region",
vbasedev->name);
- vfio_region_exit(&cxl->mem_region);
- vfio_region_finalize(&cxl->mem_region);
- return false;
+ goto err;
}
- cxl->machine_done.notify = vfio_cxl_bind_fmws;
- qemu_add_machine_init_done_notifier(&cxl->machine_done);
+ /*
+ * Trap the HDM decoder block: overlay it, priority 1, over the directly
+ * mapped component BAR, so guest decoder accesses reach the kernel FSM and
+ * the commit becomes visible to QEMU.
+ */
+ if (!vdev->bars[cxl->comp_bar].mr) {
+ error_setg(errp, "vfio-cxl: %s: component BAR %u is not present",
+ vbasedev->name, cxl->comp_bar);
+ goto err;
+ }
+ if (vfio_region_setup_with_ops(OBJECT(vdev), vbasedev,
+ &cxl->comp_regs_region,
+ cxl->comp_regs_region_index, "cxl-comp-regs",
+ &vfio_cxl_comp_regs_ops, errp)) {
+ goto err;
+ }
+ memory_region_add_subregion_overlap(vdev->bars[cxl->comp_bar].mr,
+ cxl->hdm_offset,
+ cxl->comp_regs_region.mem, 1);
+
+ if (DEVICE(vdev)->hotplugged) {
+ /*
+ * The machine is already up, so the CFMWS windows are placed and the
+ * binding can be validated now. Report a failure through errp so a bad
+ * device_add fails cleanly instead of aborting the running VM.
+ */
+ if (!vfio_cxl_do_bind_fmws(vdev, errp)) {
+ goto err;
+ }
+ } else {
+ cxl->machine_done.notify = vfio_cxl_bind_fmws;
+ qemu_add_machine_init_done_notifier(&cxl->machine_done);
+ }
cxl->enabled = true;
return true;
+
+err:
+ vfio_cxl_teardown(vdev);
+ return false;
}
static void vfio_cxl_teardown(VFIOPCIDevice *vdev)
{
VFIOCXL *cxl = &vdev->cxl;
- if (!cxl->enabled) {
- return;
+ if (cxl->comp_regs_region.mem) {
+ if (vdev->bars[cxl->comp_bar].mr) {
+ memory_region_del_subregion(vdev->bars[cxl->comp_bar].mr,
+ cxl->comp_regs_region.mem);
+ }
+ vfio_region_exit(&cxl->comp_regs_region);
+ vfio_region_finalize(&cxl->comp_regs_region);
}
- qemu_remove_machine_init_done_notifier(&cxl->machine_done);
+ vfio_cxl_unmap_mem(vdev);
if (cxl->mem_region.mem) {
vfio_region_exit(&cxl->mem_region);
vfio_region_finalize(&cxl->mem_region);
}
+ if (cxl->machine_done.notify) {
+ qemu_remove_machine_init_done_notifier(&cxl->machine_done);
+ cxl->machine_done.notify = NULL;
+ }
}
static void vfio_pci_realize(PCIDevice *pdev, Error **errp)
diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h
index 6f184520bf..c6c07a2185 100644
--- a/hw/vfio/pci.h
+++ b/hw/vfio/pci.h
@@ -133,11 +133,16 @@ typedef struct VFIOCXL {
uint32_t comp_regs_region_index; /* trapped HDM decoder register block */
uint32_t comp_bar; /* component BAR carrying that block */
uint64_t hdm_offset; /* block offset within the component BAR */
+ uint16_t dvsec_offset; /* CXL Device DVSEC offset, 0 if none */
+ uint16_t af_offset; /* PCI Advanced Features cap, 0 if none */
uint64_t dpa_size; /* size of the HDM memory region */
VFIORegion mem_region; /* HDM memory, mapped at committed GPA */
Notifier machine_done; /* CFMWS validated at machine_done */
hwaddr fmws_base; /* base of the memory window */
uint64_t fmws_size; /* size of that window */
+ VFIORegion comp_regs_region; /* trapped HDM decoder block */
+ hwaddr mapped_base; /* GPA the HDM memory is mapped at */
+ bool dpa_mapped; /* HDM memory currently in system memory */
} VFIOCXL;
struct VFIOPCIDevice {
--
2.25.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 08/10] docs/cxl: Document CXL Type-2 device passthrough
2026-08-13 13:06 [PATCH 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci mhonap
` (6 preceding siblings ...)
2026-08-13 13:06 ` [PATCH 07/10] hw/vfio/pci: Map the CXL memory on the guest decoder commit mhonap
@ 2026-08-13 13:06 ` mhonap
2026-08-13 13:06 ` [PATCH 09/10] hw/arm/smmu-common: Allow pxb-cxl as an SMMUv3 primary bus mhonap
` (2 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: mhonap @ 2026-08-13 13:06 UTC (permalink / raw)
To: alex, ankita, jic23, dave.jiang, alejandro.lucero-palau,
smadhavan, pierrick.bouvier, mst, imammedo, anisinha, pbonzini,
eric.auger, peter.maydell, richard.henderson, clg, cohuck
Cc: kjaju, vsethi, zhiw, mhonap, qemu-devel, qemu-arm
From: Manish Honap <mhonap@nvidia.com>
Describe the supported single-endpoint passthrough topology, the command
line, and the kernel dependency, and note that a guest CXL reset is
handled by the host kernel rather than QEMU.
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
docs/system/devices/cxl.rst | 46 +++++++++++++++++++++++++++++++++++++
1 file changed, 46 insertions(+)
diff --git a/docs/system/devices/cxl.rst b/docs/system/devices/cxl.rst
index 9d0771cdfd..33a9fb169b 100644
--- a/docs/system/devices/cxl.rst
+++ b/docs/system/devices/cxl.rst
@@ -418,6 +418,52 @@ Volatile Memory device::
-device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \
-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G
+Type 2 device passthrough
+-------------------------
+
+A CXL Type 2 device (an accelerator with host-managed device memory) can be
+assigned to a guest with vfio-pci, so the guest reaches the device memory
+through its own CXL stack. This pairs with the kernel vfio-cxl series: the
+host kernel fixes the device memory at a host physical range before the guest
+sees the device, and QEMU maps that range at the guest physical address the
+guest programs into its endpoint HDM decoder. The guest chooses only the GPA;
+the host physical placement is never reprogrammed by the guest.
+
+The device memory reaches the guest as a CXL fixed memory window (``cxl-fmw``),
+advertised through CEDT, exactly like a Type 3 window; there is no separate
+device memory-map slot. Only the endpoint decoder is programmed, by the guest,
+so the host bridge must stay in HDM passthrough mode: a single ``cxl-rp`` under
+the ``pxb-cxl`` and no ``hdm_for_passthrough``. Switch-attached and interleaved
+topologies are rejected.
+
+Example command line::
+
+ -machine q35,cxl=on
+ -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1
+ -device cxl-rp,port=0,bus=cxl.1,id=rp0,chassis=0,slot=2
+ -device vfio-pci,host=<BDF>,bus=rp0,id=cxl-ep0
+ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=<device-mem-size>
+
+The window must be a single-target ``cxl-fmw`` that targets the device's
+``pxb-cxl`` and is at least the size of the device memory. The guest triggers a
+CXL reset by writing the CXL Device DVSEC; the host kernel runs that sequence,
+so QEMU has no reset handling of its own.
+
+On arm64, pass ``accel=on`` to the ``arm-smmuv3`` when passing a Type 2 device
+through. The accelerated SMMUv3 describes the device MSI doorbell to the guest
+through an IORT Reserved Memory Range (RMR) node, which reserves a fixed guest
+IOVA, so OSPM must preserve the firmware PCI resource assignments rather than
+re-enumerate them. QEMU requests that through PCI Firmware ``_DSM`` function 5
+(preserve firmware PCI configuration), which on arm64 is emitted for the CXL
+host bridge only when the machine requests preserved configuration, that is,
+the accelerated SMMUv3 path. This ``_DSM`` is not about the CXL decoder:
+vfio-pci keeps the host BAR fixed, and QEMU's trapped component-register block
+is a subregion of the guest BAR MemoryRegion, so it follows any guest-visible
+BAR relocation on its own. x86 does not use the accelerated SMMU, so it does
+not advertise or implement preserve-configuration function 5; the CXL host
+bridge still emits the ``_DSM`` method, but its function 0 returns an empty
+support mask.
+
Deprecations
------------
--
2.25.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 09/10] hw/arm/smmu-common: Allow pxb-cxl as an SMMUv3 primary bus
2026-08-13 13:06 [PATCH 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci mhonap
` (7 preceding siblings ...)
2026-08-13 13:06 ` [PATCH 08/10] docs/cxl: Document CXL Type-2 device passthrough mhonap
@ 2026-08-13 13:06 ` mhonap
2026-08-13 13:06 ` [PATCH 10/10] hw/pci-host: Emit a _DSM on pxb-cxl to preserve firmware PCI config mhonap
2026-08-13 17:42 ` [PATCH 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci Cédric Le Goater
10 siblings, 0 replies; 12+ messages in thread
From: mhonap @ 2026-08-13 13:06 UTC (permalink / raw)
To: alex, ankita, jic23, dave.jiang, alejandro.lucero-palau,
smadhavan, pierrick.bouvier, mst, imammedo, anisinha, pbonzini,
eric.auger, peter.maydell, richard.henderson, clg, cohuck
Cc: kjaju, vsethi, zhiw, mhonap, qemu-devel, qemu-arm
From: Manish Honap <mhonap@nvidia.com>
The SMMUv3 primary-bus check only accepted pxb-pcie as a valid extra
root complex, so a CXL device behind a pxb-cxl could not reach the
IOMMU: attaching an arm-smmuv3 to a pxb-cxl bus failed at realize with
"SMMU should be attached to a default PCIe root complex (pcie.0) or a
pxb-pcie based root complex".
A pxb-cxl uses the same PCIe-compatible bus as a pxb-pcie, so accept it
too. A CXL Type-2 device passed through with vfio-pci sits behind a
pxb-cxl and needs SMMU translation for its DMA and ATS just like a
pxb-pcie device.
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
hw/arm/smmu-common.c | 19 ++++++++++---------
1 file changed, 10 insertions(+), 9 deletions(-)
diff --git a/hw/arm/smmu-common.c b/hw/arm/smmu-common.c
index 8e40ba603d..42fb098ab2 100644
--- a/hw/arm/smmu-common.c
+++ b/hw/arm/smmu-common.c
@@ -963,19 +963,20 @@ static void smmu_base_realize(DeviceState *dev, Error **errp)
s->iommu_ops = &smmu_ops;
}
/*
- * We only allow default PCIe Root Complex(pcie.0) or pxb-pcie based extra
- * root complexes to be associated with SMMU.
+ * We only allow the default PCIe root complex (pcie.0) or pxb-pcie /
+ * pxb-cxl based extra root complexes to be associated with SMMU.
*/
if (pci_bus_is_express(pci_bus) && pci_bus_is_root(pci_bus) &&
object_dynamic_cast(OBJECT(pci_bus)->parent, TYPE_PCI_HOST_BRIDGE)) {
/*
- * This condition matches either the default pcie.0, pxb-pcie, or
- * pxb-cxl. For both pxb-pcie and pxb-cxl, parent_dev will be set.
- * Currently, we don't allow pxb-cxl as it requires further
- * verification. Therefore, make sure this is indeed pxb-pcie.
+ * pcie.0 has no parent_dev; pxb-pcie and pxb-cxl do. Accept both bus
+ * types explicitly so other root complexes are still rejected. A
+ * pxb-cxl uses the same PCIe-compatible bus, so a CXL device behind it
+ * needs the SMMU just like a pxb-pcie one.
*/
if (pci_bus->parent_dev) {
- if (!object_dynamic_cast(OBJECT(pci_bus), TYPE_PXB_PCIE_BUS)) {
+ if (!object_dynamic_cast(OBJECT(pci_bus), TYPE_PXB_PCIE_BUS) &&
+ !object_dynamic_cast(OBJECT(pci_bus), TYPE_PXB_CXL_BUS)) {
goto out_err;
}
}
@@ -990,8 +991,8 @@ static void smmu_base_realize(DeviceState *dev, Error **errp)
return;
}
out_err:
- error_setg(errp, "SMMU should be attached to a default PCIe root complex"
- "(pcie.0) or a pxb-pcie based root complex");
+ error_setg(errp, "SMMU should be attached to a default PCIe root complex "
+ "(pcie.0), a pxb-pcie, or a pxb-cxl based root complex");
}
/*
--
2.25.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 10/10] hw/pci-host: Emit a _DSM on pxb-cxl to preserve firmware PCI config
2026-08-13 13:06 [PATCH 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci mhonap
` (8 preceding siblings ...)
2026-08-13 13:06 ` [PATCH 09/10] hw/arm/smmu-common: Allow pxb-cxl as an SMMUv3 primary bus mhonap
@ 2026-08-13 13:06 ` mhonap
2026-08-13 17:42 ` [PATCH 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci Cédric Le Goater
10 siblings, 0 replies; 12+ messages in thread
From: mhonap @ 2026-08-13 13:06 UTC (permalink / raw)
To: alex, ankita, jic23, dave.jiang, alejandro.lucero-palau,
smadhavan, pierrick.bouvier, mst, imammedo, anisinha, pbonzini,
eric.auger, peter.maydell, richard.henderson, clg, cohuck
Cc: kjaju, vsethi, zhiw, mhonap, qemu-devel, qemu-arm
From: Manish Honap <mhonap@nvidia.com>
A pxb-cxl host bridge had no PCI host-bridge _DSM method, so it could not
emit function 5 (preserve firmware PCI configuration) even when the
machine asks OSPM to keep the firmware resource assignments.
That preservation is required by the accelerated SMMUv3 (accel=on), which
describes MSI 1:1 mappings through IORT RMR nodes. An RMR reserves a fixed
IOVA range, so OSPM must not re-enumerate and reassign the PCI resources
underneath it. virt sets pci_preserve_config for that case and expects
every host bridge to emit the _DSM; the pxb-cxl bridge was the one that
did not, so a CXL topology under an accelerated SMMU lost the guarantee.
This is not about the component register BAR moving. vfio-pci marks every
BAR dword ALL_VIRT, so a guest BAR write updates the virtual config only
and cannot move the host resource, and QEMU's trapped comp-regs region is
a subregion of the BAR MemoryRegion, so it follows any guest-visible
relocation while the kernel's physical decoder mapping stays fixed. The
CXL.mem window is delivered through CEDT/CFMWS and is likewise unaffected
by PCI resource assignment.
Wire preserve_config through GPEXConfig into the CXL host bridge OSC path
so pxb-cxl bridges emit the _DSM function 5. Rename build_cxl_osc_method()
to acpi_dsdt_add_cxl_host_bridge_methods() to match the pxb-pcie analogue
acpi_dsdt_add_host_bridge_methods(), since it now appends both the _OSC
and the _DSM. x86 passes false as it does not use the accelerated SMMU.
Signed-off-by: Manish Honap <mhonap@nvidia.com>
---
hw/acpi/Kconfig | 1 +
hw/acpi/cxl-stub.c | 2 +-
hw/acpi/cxl.c | 4 +++-
hw/acpi/pci.c | 40 +++++++++++++++++++++++++++++++++++++++
hw/i386/acpi-build.c | 2 +-
hw/pci-host/gpex-acpi.c | 42 ++---------------------------------------
include/hw/acpi/cxl.h | 2 +-
include/hw/acpi/pci.h | 1 +
8 files changed, 50 insertions(+), 44 deletions(-)
diff --git a/hw/acpi/Kconfig b/hw/acpi/Kconfig
index daabbe6cd1..9490b75c94 100644
--- a/hw/acpi/Kconfig
+++ b/hw/acpi/Kconfig
@@ -88,3 +88,4 @@ config ACPI_ERST
config ACPI_CXL
bool
depends on ACPI
+ select ACPI_PCI
diff --git a/hw/acpi/cxl-stub.c b/hw/acpi/cxl-stub.c
index 15bc21076b..d7c6731975 100644
--- a/hw/acpi/cxl-stub.c
+++ b/hw/acpi/cxl-stub.c
@@ -6,7 +6,7 @@
#include "hw/acpi/aml-build.h"
#include "hw/acpi/cxl.h"
-void build_cxl_osc_method(Aml *dev)
+void acpi_dsdt_add_cxl_host_bridge_methods(Aml *dev, bool preserve_config)
{
g_assert_not_reached();
}
diff --git a/hw/acpi/cxl.c b/hw/acpi/cxl.c
index f92f7fa3d5..7607ba500f 100644
--- a/hw/acpi/cxl.c
+++ b/hw/acpi/cxl.c
@@ -23,6 +23,7 @@
#include "hw/pci/pci_host.h"
#include "hw/cxl/cxl.h"
#include "hw/cxl/cxl_host.h"
+#include "hw/acpi/pci.h"
#include "hw/mem/memory-device.h"
#include "hw/acpi/acpi.h"
#include "hw/acpi/aml-build.h"
@@ -320,11 +321,12 @@ static Aml *__build_cxl_osc_method(void)
return method;
}
-void build_cxl_osc_method(Aml *dev)
+void acpi_dsdt_add_cxl_host_bridge_methods(Aml *dev, bool preserve_config)
{
aml_append(dev, aml_name_decl("SUPP", aml_int(0)));
aml_append(dev, aml_name_decl("CTRL", aml_int(0)));
aml_append(dev, aml_name_decl("SUPC", aml_int(0)));
aml_append(dev, aml_name_decl("CTRC", aml_int(0)));
aml_append(dev, __build_cxl_osc_method());
+ aml_append(dev, build_pci_host_bridge_dsm_method(preserve_config));
}
diff --git a/hw/acpi/pci.c b/hw/acpi/pci.c
index 8c7ed10479..1a20f8469b 100644
--- a/hw/acpi/pci.c
+++ b/hw/acpi/pci.c
@@ -351,3 +351,43 @@ Aml *build_pci_host_bridge_osc_method(bool enable_native_pcie_hotplug)
aml_append(method, aml_return(aml_arg(3)));
return method;
}
+
+Aml *build_pci_host_bridge_dsm_method(bool preserve_config)
+{
+ Aml *method = aml_method("_DSM", 4, AML_NOTSERIALIZED);
+ Aml *UUID, *ifctx, *ifctx1, *buf;
+ uint8_t byte_list[1] = {0};
+
+ /*
+ * PCI Firmware Specification 3.0
+ * 4.6.1. _DSM for PCI Express Slot Information
+ * The UUID in _DSM in this context is
+ * {E5C937D0-3553-4D7A-9117-EA4D19C3434D}
+ */
+ UUID = aml_touuid("E5C937D0-3553-4D7A-9117-EA4D19C3434D");
+ ifctx = aml_if(aml_equal(aml_arg(0), UUID));
+ ifctx1 = aml_if(aml_equal(aml_arg(2), aml_int(0)));
+ if (preserve_config) {
+ /* support functions other than 0, specifically function 5 */
+ byte_list[0] = 0x21;
+ }
+ buf = aml_buffer(1, byte_list);
+ aml_append(ifctx1, aml_return(buf));
+ aml_append(ifctx, ifctx1);
+ if (preserve_config) {
+ Aml *ifctx2 = aml_if(aml_equal(aml_arg(2), aml_int(5)));
+ /*
+ * 0 - The operating system must not ignore the PCI configuration that
+ * firmware has done at boot time.
+ */
+ aml_append(ifctx2, aml_return(aml_int(0)));
+ aml_append(ifctx, ifctx2);
+ }
+
+ aml_append(method, ifctx);
+
+ byte_list[0] = 0;
+ buf = aml_buffer(1, byte_list);
+ aml_append(method, aml_return(buf));
+ return method;
+}
diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 8837b69687..a24bff8c88 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -1019,7 +1019,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
aml_append(aml_pkg, aml_eisaid("PNP0A08"));
aml_append(aml_pkg, aml_eisaid("PNP0A03"));
aml_append(dev, aml_name_decl("_CID", aml_pkg));
- build_cxl_osc_method(dev);
+ acpi_dsdt_add_cxl_host_bridge_methods(dev, false);
} else if (pci_bus_is_express(bus)) {
aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A08")));
aml_append(dev, aml_name_decl("_CID", aml_eisaid("PNP0A03")));
diff --git a/hw/pci-host/gpex-acpi.c b/hw/pci-host/gpex-acpi.c
index d9820f9b41..014f647c7f 100644
--- a/hw/pci-host/gpex-acpi.c
+++ b/hw/pci-host/gpex-acpi.c
@@ -51,45 +51,6 @@ static void acpi_dsdt_add_pci_route_table(Aml *dev, uint32_t irq,
}
}
-static Aml *build_pci_host_bridge_dsm_method(bool preserve_config)
-{
- Aml *method = aml_method("_DSM", 4, AML_NOTSERIALIZED);
- Aml *UUID, *ifctx, *ifctx1, *buf;
- uint8_t byte_list[1] = {0};
-
- /* PCI Firmware Specification 3.0
- * 4.6.1. _DSM for PCI Express Slot Information
- * The UUID in _DSM in this context is
- * {E5C937D0-3553-4D7A-9117-EA4D19C3434D}
- */
- UUID = aml_touuid("E5C937D0-3553-4D7A-9117-EA4D19C3434D");
- ifctx = aml_if(aml_equal(aml_arg(0), UUID));
- ifctx1 = aml_if(aml_equal(aml_arg(2), aml_int(0)));
- if (preserve_config) {
- /* support functions other than 0, specifically function 5 */
- byte_list[0] = 0x21;
- }
- buf = aml_buffer(1, byte_list);
- aml_append(ifctx1, aml_return(buf));
- aml_append(ifctx, ifctx1);
- if (preserve_config) {
- Aml *ifctx2 = aml_if(aml_equal(aml_arg(2), aml_int(5)));
- /*
- * 0 - The operating system must not ignore the PCI configuration that
- * firmware has done at boot time.
- */
- aml_append(ifctx2, aml_return(aml_int(0)));
- aml_append(ifctx, ifctx2);
- }
-
- aml_append(method, ifctx);
-
- byte_list[0] = 0;
- buf = aml_buffer(1, byte_list);
- aml_append(method, aml_return(buf));
- return method;
-}
-
static void acpi_dsdt_add_host_bridge_methods(Aml *dev,
bool enable_native_pcie_hotplug,
bool preserve_config)
@@ -164,7 +125,8 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)
aml_append(dev, aml_name_decl("_CRS", crs));
if (is_cxl) {
- build_cxl_osc_method(dev);
+ acpi_dsdt_add_cxl_host_bridge_methods(dev,
+ cfg->preserve_config);
} else {
/* pxb bridges do not have ACPI PCI Hot-plug enabled */
acpi_dsdt_add_host_bridge_methods(dev, true,
diff --git a/include/hw/acpi/cxl.h b/include/hw/acpi/cxl.h
index 8f22c71530..6fe6c9c58d 100644
--- a/include/hw/acpi/cxl.h
+++ b/include/hw/acpi/cxl.h
@@ -24,7 +24,7 @@
void cxl_build_cedt(GArray *table_offsets, GArray *table_data,
BIOSLinker *linker, const char *oem_id,
const char *oem_table_id, CXLState *cxl_state);
-void build_cxl_osc_method(Aml *dev);
+void acpi_dsdt_add_cxl_host_bridge_methods(Aml *dev, bool preserve_config);
void build_cxl_dsm_method(Aml *dev);
#endif
diff --git a/include/hw/acpi/pci.h b/include/hw/acpi/pci.h
index 20b672575f..c7de33f8cf 100644
--- a/include/hw/acpi/pci.h
+++ b/include/hw/acpi/pci.h
@@ -42,6 +42,7 @@ void build_pci_bridge_aml(AcpiDevAmlIf *adev, Aml *scope);
void build_srat_generic_affinity_structures(GArray *table_data);
Aml *build_pci_host_bridge_osc_method(bool enable_native_pcie_hotplug);
+Aml *build_pci_host_bridge_dsm_method(bool preserve_config);
Aml *build_pci_bridge_edsm(void);
#endif
--
2.25.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci
2026-08-13 13:06 [PATCH 00/10] QEMU: CXL Type-2 device passthrough via vfio-pci mhonap
` (9 preceding siblings ...)
2026-08-13 13:06 ` [PATCH 10/10] hw/pci-host: Emit a _DSM on pxb-cxl to preserve firmware PCI config mhonap
@ 2026-08-13 17:42 ` Cédric Le Goater
10 siblings, 0 replies; 12+ messages in thread
From: Cédric Le Goater @ 2026-08-13 17:42 UTC (permalink / raw)
To: mhonap, alex, ankita, jic23, dave.jiang, alejandro.lucero-palau,
smadhavan, pierrick.bouvier, mst, imammedo, anisinha, pbonzini,
eric.auger, peter.maydell, richard.henderson, cohuck
Cc: kjaju, vsethi, zhiw, qemu-devel, qemu-arm
Hello Manish,
On 8/13/26 15:06, mhonap@nvidia.com wrote:
> From: Manish Honap <mhonap@nvidia.com>
>
> This series adds QEMU support for passing a CXL Type-2 device (an
> accelerator with host-managed device memory, e.g. a GPU) to a guest via
> vfio-pci. The guest drives its own virtual endpoint HDM decoder and QEMU
> maps the device memory at the guest physical address the guest commits,
> while the host owns the host physical placement.
>
> It is a full rewrite of the RFC v1 series [1] on clean upstream master,
> reworked to address Jonathan's review.
>
> Base: master (post pull-9p-20260725), commit 6333226c2a.
>
>
> Kernel dependency
> -----------------
>
> Pairs with the kernel vfio-cxl series "vfio/cxl: CXL Type-2 device
> passthrough" [2].
> - The kernel exposes the device memory as an HPA-backed VFIO region, traps
> the HDM decoder block and runs its lock-on-commit FSM, handles the CXL
> DVSEC (including guest-triggered reset), and reports two things through
> VFIO:
> - A device flag (VFIO_DEVICE_FLAGS_CXL) and
> - The component-register geometry (VFIO_REGION_INFO_CAP_CXL_COMP_REGS).
>
>
> Sample supported topology
> -------------------------
>
> Guest disk, network, and system-RAM lines are omitted:
>
> -machine virt,accel=kvm,gic-version=3,hmat=on,cxl=on,ras=on, \
> highmem-mmio-size=4T
> -object iommufd,id=iommufd0
> -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1
> -device cxl-rp,port=1,bus=cxl.1,id=rport0.1,chassis=4, \
> pref64-reserve=2G,mem-reserve=1G
> -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=256G
> -device arm-smmuv3,primary-bus=cxl.1,id=smmuv3.0,accel=on,ats=on, \
> ril=on,ssidsize=8,oas=48
> -device vfio-pci-nohotplug,host=<BDF>,bus=rport0.1,id=dev0, \
> iommufd=iommufd0
> -object acpi-generic-initiator,id=gi0,pci-dev=dev0,node=2
> ... (one acpi-generic-initiator per guest NUMA node the HDM memory backs)
>
>
> Address model
> -------------
>
> The kernel fixes device memory to a host physical range before the guest
> sees the device, and hardware presents a firmware-committed, locked
> endpoint HDM decoder whose registers hold that host physical base.
>
> QEMU never exposes Host physical base. It virtualizes the decoder base registers
> in the trapped component-register read and returns the base of the device's
> CFMWS window, a guest physical address, so the guest only ever sees a GPA.
>
> QEMU maps the RAM-device region (backed by the fixed HPA) at that CFMWS base.
>
> The kernel never learns the GPA and the guest never learns the HPA.
>
> Because the decoder is already committed at boot, no guest commit write
> triggers the mapping. QEMU maps once the guest enables memory decoding
> (the Command register Memory-Space bit) and re-checks on any decoder
> control write, so the region enters the guest address space and the IOAS
> while the device is live.
>
> When the guest clears Memory-Space, QEMU withdraws the mapping, matching the
> kernel's revoke of the backing PTEs on the same write, so a guest access during
> the disabled interval cannot fault a zapped mapping and stop the VM; the enable
> path re-installs it.
>
> This also helps to keep the guest-visible base as the CFMWS base by
> construction, and the host physical placement stays with the kernel.
>
> The RFC added the pxb-cxl _DSM because OS may treat PCI configuration as
> reassignable and a BAR move would break the CXL.mem mapping. This change keeps
> it narrower than the RFC v1 _DSM: it applies only to the host bridge that
> carries the passed-through CXL device.
>
>
> Reset
> -----
>
> There is no QEMU reset patch. A guest CXL reset is a DVSEC write that
> lands in vfio config space and is handled entirely by the host kernel,
> which stamps the outcome into DVSEC STATUS2. The kernel re-commits this
> firmware-fixed decoder across the reset, and the guest reaches its memory
> through the mapping QEMU already installed.
>
>
> Reviewer feedback addressed
> ---------------------------
>
> The RFC [1] thread has the full discussion.
>
> Jonathan Cameron
> - The high-MMIO window and the cxl-fmws-base property are removed. The
> guest-visible base is the CFMWS base by construction, so it equals the
> decoder base and stays stable without a hack.
> - The guest programs its virtual (GPA) decoder and QEMU maps the memory at
> commit time.
> - The host owns the HPA, resolved before the guest sees the device.
> - The committed decoder is the fast path this series ships. The
> guest-programmed uncommitted case is delivered by cxl-core resolving the
> range at enumeration, not by a QEMU or vfio dynamic branch, so it is a
> later cxl-core item this series does not depend on.
> - FIRMWARE_COMMITTED is dropped; the cap is no longer exposed.
> - The one-endpoint, non-interleaved, no-switch topology is enforced at
> realize.
> - PCI/BAR configuration and CXL.mem stay independent.
>
>
> Validation
> ----------
>
> - Every patch passes scripts/checkpatch.pl --codespell --strict
> (patch 1 carries the expected imported-from-Linux warning)
> - The series applies cleanly on the stated base.
> - Ran couple of rounds of masoncl/review-prompts on this series before posting.
>
>
> Pending items
> -------------
>
> Future enhancements for the multi-decoder, interleaved devices and switched
> topologies.
>
> Trapped CXL RAS registers are planned as a new VFIO region subtype that the same
> region-by-subtype detection already handles, not as a change to the
> component-register cap.
>
> The bios-tables test refresh for the new _DSM is still to be added.
Before diving into CXL and doing a review of the QEMU patches, could you
provide a status update on the kernel vfio-cxl series [2] ?
The QEMU side is a consumer of the VFIO regions and capabilities the
kernel exposes, and patch 1 notes the capability ID is still provisional.
Knowing where the kernel series stands would help. That said, the UAPI
looks simple enough.
What has become of the CXL Type-2 device emulation effort from Zhi Wang?
https://lore.kernel.org/qemu-devel/20241212130422.69380-1-zhiw@nvidia.com/
That series introduced a bare-minimum emulated CXL Type-2 device in
QEMU, so kernel and QEMU developers could test the CXL Type-2 stack
without real hardware. I liked the idea quite a lot.
CXL is still new and complex, and hardware is scarce. Having an
emulated device and passthrough support gives us a full environment to
flush out design issues across the stack (Linux kernel, QEMU, libvirt)
and in associated subsystems like IOMMU and ACPI, without needing
physical devices. It is also useful for education and for running CI
tests. Do you see both efforts as complementary, and is the emulation
work still active?
I will take a closer look at the series after some time off.
Thanks,
C.
^ permalink raw reply [flat|nested] 12+ messages in thread