[PATCH RFC 0/7] QEMU/hw/cxl: Allow CXL Type 3 memory regions to be initialized at guest boot

[PATCH RFC 0/7] QEMU/hw/cxl: Allow CXL Type 3 memory regions to be initialized at guest boot

[Yukiya Takaki]

Hello everyone,

I’m a student intern, and this is my first time sending patches to an open source mailing list.

This patch series configures QEMU’s CXL Type 3 devices so that guest operating systems can automatically discover
and create CXL regions during boot, aligning QEMU’s behavior with real hardware.

Background and motivation
  Currently, during guest boot QEMU does not provide the information or register programming needed for the guest OS
  to initialize CXL Type 3 memory, so using it still requires an explicit action in the guest (for example, running the cxl create-region command).
  By contrast, on some physical systems the firmware automatically allocates CXL memory ranges and exposes them to the OS,
   allowing the Linux CXL driver to discover and create regions during boot.
  This patch series is a first, experimental attempt to reproduce that behavior in QEMU.

  Some parts of this implementation are based on Li, Zhijian’s work in:
  https://github.com/zhijianli88/qemu/tree/program-decoder-v2

  I would like to ask for feedback on whether this approach makes sense and whether I should continue to enhance it.

Approach
  This patch first adds a program_hdm_decoder option to cxl-type3.
  When program-hdm-decoder is enabled on cxl-type3, QEMU will:
    1) program endpoint/host-bridge HDM decoder state,
    2) expose Type 3 ranges as E820_SOFT_RESERVED on x86,
    3) create memory-only NUMA nodes for those ranges,
    4) emit SRAT memory-affinity entries mapping each Type 3 HPA range to its node ID

  On real hardware, HDM decoder programming and memory layout are typically handled by firmware, so they would ideally be managed by SeaBIOS.
  However, in the current QEMU/SeaBIOS model, E820 and SRAT are built on the QEMU side, and pushing HDM decoder programming
  into SeaBIOS would introduce additional complexity and risk inconsistencies.

Testing
  We ran tests in the following environment and confirmed that CXL regions were created at boot, NUMA nodes were set up,
  and E820_SOFT_RESERVED entries appeared in the E820 table:

  - BIOS: SeaBIOS c2a33ad9ad1452e23b41c4ac44a3bc6be8ebc4cf
  - CXL memory: 4 GB
  - CFMWS: 4 GB
  - Host bridge: 1

  The following is the startup script.

  sudo /path/to/qemu/build/qemu-system-x86_64 \
    -name guest=cxl \
    -nographic \
    -machine q35,accel=tcg,nvdimm=on,cxl=on \
    -cpu max -smp 4,sockets=4,cores=1,threads=1 \
    -m size=4G,slots=10,maxmem=20G \
    -drive file=/path/to/image.qcow2,format=qcow2,index=0,media=disk,id=hd \
    -bios /path/to/seabios/out/bios.bin \
    -nic bridge,br=virbr0,model=e1000e,mac=52:54:00:b5:4e:7c \
    -object memory-backend-ram,size=2G,id=m0 \
    -object memory-backend-ram,size=2G,id=m1 \
    -numa node,nodeid=0,cpus=0-1,memdev=m0 \
    -numa node,nodeid=1,cpus=2-3,memdev=m1 \
    -device pcie-root-port,id=pci-root,slot=4,bus=pcie.0,chassis=0 \
    -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1,hdm_for_passthrough=true \
    -device cxl-rp,port=0,id=root_port13,bus=cxl.1,chassis=0,slot=2 \
    -object memory-backend-ram,size=4G,share=on,id=vmem0 \
    -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0,program-hdm-decoder=true \
    -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k
  
  Furthermore, we conducted the experiment in which we added two CXL Type3.

  - CXL memory: 4 GB × 2
  - CFMWS: 4 GB × 2

  The following is the startup script.
    
  sudo /path/to/qemu/build/qemu-system-x86_64 \
    -name guest=cxl \
    -nographic \
    -machine q35,accel=tcg,nvdimm=on,cxl=on,hmat=on \
    -cpu max -smp 4,sockets=4,cores=1,threads=1 \
    -m size=4G,slots=10,maxmem=20G \
    -drive file=/path/to/image.qcow2,format=qcow2,index=0,media=disk,id=hd \
    -bios /path/to/seabios/out/bios.bin \
    -nic bridge,br=virbr0,model=e1000e,mac=52:54:00:b5:4e:7c \
    -object memory-backend-ram,size=2G,id=m0 \
    -object memory-backend-ram,size=2G,id=m1 \
    -numa node,nodeid=0,cpus=0-1,memdev=m0 \
    -numa node,nodeid=1,cpus=2-3,memdev=m1 \
    -device pcie-root-port,id=pci-root,slot=4,bus=pcie.0,chassis=0 \
    -device pcie-root-port,id=pci-root-net,chassis=5,slot=5,bus=pcie.0 \
    -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1,hdm_for_passthrough=true \
    -device cxl-rp,port=0,id=root_port13,bus=cxl.1,chassis=0,slot=2 \
    -device cxl-rp,port=3,id=root_port14,bus=cxl.1,chassis=2,slot=4 \
    -object memory-backend-ram,size=4G,share=on,id=vmem0 \
    -object memory-backend-ram,size=4G,share=on,id=vmem1 \
    -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0,program-hdm-decoder=true \
    -device cxl-type3,bus=root_port14,volatile-memdev=vmem1,id=cxl-vmem1,program-hdm-decoder=true \
    -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k,cxl-fmw.1.targets.0=cxl.1,cxl-fmw.1.size=4G,cxl-fmw.1.interleave-granularity=4k


How to use
  Specify the following parameters when starting QEMU:

  - bios <SeaBIOS binary that includes commit c2a33ad9ad1452e23b41c4ac44a3bc6be8ebc4cf> to ensure that E820_SOFT_RESERVED entries are exposed
  - Add hmat=on to the -machine option [1]
  - Add program-hdm-decoder=true to each -device cxl-type3 option
    [1] On Linux, the kernel first checks E820_SOFT_RESERVED ranges. When multiple CXL Type 3 devices share a contiguous HPA range,
        these ranges tend to be merged into a single region, so the CXL driver cannot reliably initialize per‑device regions based on E820 alone.
        SRAT entries for CXL memory are therefore needed, and the guest seems to discard SRAT when `hmat=on` is not specified,
        so HMAT must be enabled.

QEMU base commit: bddb785e791e415db38884848d068457e226f5e3

TODO
  With the current implementation of this patch, the following configurations are either not supported or have not been sufficiently tested.
  As a result, automatic creation of CXL regions in the guest OS may fail:

  - A CXL persistent memory device is included
  - Interleaved configurations
  - Cases where the memory size specified by CFMWS does not match the size of the CXL Type 3 backend memory
  - Configurations where CXL memory is attached via a switch
  - Multi-target CFMWS environments
  - ARM environments
  - HMAT does not yet include information about CXL memory devices

  The following behaviors are also provisional implementations and may need further refinement:
   - A new NUMA node ID is assigned to each CXL memory device. This should be improved either to reflect more accurate NUMA distances for CXL memory,
     or to allow the NUMA node IDs to be specified by the user.
   - Assignment of CXL Type 3 devices to CFMWS address ranges. On physical systems, platform firmware typically generates CFMWS entries
     based on the detected CXL topology and its address-space allocation. In QEMU, however, CFMWS entries are configured explicitly
     through machine options. The current implementation therefore has to infer which CFMWS range should be used for each Type 3 device.
     This selection logic may need further refinement. Possible approaches include improving the selection and validation logic,
     adding an explicit option to associate each Type 3 device with a CFMWS entry,
     or generating CFMWS entries automatically from the configured CXL topology and devices.
   - HDM decoder reset sequencing and state reconstruction.
     The current implementation prevents the host bridge HDM decoder configuration from being overwritten later in the same reset
     by moving the common initialization of the host bridge component registers from the reset path to the realization path.
     However, ideally, after the registers have been restored to their reset values, the host bridge HDM decoder configuration
     automatically programmed by QEMU should be reconstructed as part of reset processing.

Thanks,

------

Yukiya Takaki (7):
  QEMU/hw/cxl: add program-hdm-decoder option for CXL Type 3 devices
  QEMU/hw/cxl:add HPA offset tracking for CXL Type 3 devices
  QEMU: hw/cxl: add helper to get HPA range for CXL Type 3
  QEMU/hw/cxl: auto‑program HDM decoders for CXL Type 3
  QEMU/hw/cxl: expose CXL Type 3 ranges as E820_SOFT_RESERVED on x86
  QEMU/hw/cxl: Create NUMA nodes for CXL Type 3 memory
  add CXL Type 3 memory ranges to SRAT

 hw/acpi/cxl_type3.c                 |  30 +++++
 hw/acpi/meson.build                 |   1 +
 hw/arm/cxl.c                        |  10 ++
 hw/arm/meson.build                  |   2 +
 hw/cxl/cxl-host.c                   |  63 ++++++++++
 hw/i386/acpi-build.c                |   7 ++
 hw/i386/cxl.c                       |  11 ++
 hw/i386/meson.build                 |   1 +
 hw/i386/pc.c                        |   2 +
 hw/mem/cxl_type3.c                  | 177 ++++++++++++++++++++++++++++
 hw/pci-bridge/pci_expander_bridge.c |  13 +-
 include/hw/cxl/cxl.h                |   2 +
 include/hw/cxl/cxl_device.h         |   6 +
 include/hw/cxl/cxl_host.h           |   3 +
 include/hw/mem/cxl_type3.h          |   8 ++
 15 files changed, 334 insertions(+), 2 deletions(-)
 create mode 100644 hw/acpi/cxl_type3.c
 create mode 100644 hw/arm/cxl.c
 create mode 100644 hw/i386/cxl.c
 create mode 100644 include/hw/mem/cxl_type3.h

-- 
2.52.0

[PATCH RFC 1/7] QEMU/hw/cxl: add program-hdm-decoder option for CXL Type 3 devices

[Yukiya Takaki]

Introduce a `program-hdm-decoder` property for CXL Type 3 devices.

When this property is enabled, QEMU will program the HDM decoder so that
the guest can automatically create CXL regions for Type 3 memory at boot.

The property is a boolean and is exposed as `program-hdm-decoder` on the
`cxl-type3` device. It defaults to `false`, so existing configurations are
not affected unless the user explicitly enables it.

Co-developed-by: Li Zhijian <lizhijian@fujitsu.com>
Signed-off-by: Yukiya Takaki <fj2110jc@fujitsu.com>
---
 hw/mem/cxl_type3.c          | 2 ++
 include/hw/cxl/cxl_device.h | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index a04dc74ecc..b65ddd264d 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -1360,6 +1360,8 @@ static const Property ct3_props[] = {
     DEFINE_PROP_LINK("lsa", CXLType3Dev, lsa, TYPE_MEMORY_BACKEND,
                      HostMemoryBackend *),
     DEFINE_PROP_UINT64("sn", CXLType3Dev, sn, UI64_NULL),
+    DEFINE_PROP_BOOL("program-hdm-decoder", CXLType3Dev,
+                     program_hdm_decoder, false),
     DEFINE_PROP_STRING("cdat", CXLType3Dev, cxl_cstate.cdat.filename),
     DEFINE_PROP_UINT8("num-dc-regions", CXLType3Dev, dc.num_regions, 0),
     DEFINE_PROP_LINK("volatile-dc-memdev", CXLType3Dev, dc.host_dc,
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index ba551fa5f9..a2c830d181 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -777,6 +777,8 @@ struct CXLType3Dev {
     /* BI flows */
     bool hdmdb;
 
+    bool program_hdm_decoder;
+
     struct dynamic_capacity {
         HostMemoryBackend *host_dc;
         AddressSpace host_dc_as;
-- 
2.52.0

[PATCH RFC 2/7] QEMU/hw/cxl:add HPA offset tracking for CXL Type 3 devices

[Yukiya Takaki]

E820, SRAT and HDM decoder programming in this series all need per‑device
host physical address (HPA) information for CXL Type 3 memory. Today,
CXLType3Dev does not expose this in a convenient way.

This patch adds minimal state to CXLHost and CXLType3Dev, plus a helper
to walk from a Type 3 device back to its host, so that each CXL Type 3
device can be assigned a stable HPA offset within the CFMW window and an
index that later code can rely on when building firmware tables and
regions.

Co-developed-by: Li Zhijian <lizhijian@fujitsu.com>
Signed-off-by: Yukiya Takaki <fj2110jc@fujitsu.com>
---
 hw/mem/cxl_type3.c          | 37 +++++++++++++++++++++++++++++++++++++
 include/hw/cxl/cxl.h        |  2 ++
 include/hw/cxl/cxl_device.h |  2 ++
 3 files changed, 41 insertions(+)

diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index b65ddd264d..d08a4255c9 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -732,9 +732,39 @@ static void cxl_destroy_dc_regions(CXLType3Dev *ct3d)
     }
 }
 
+
+static CXLHost *ct3d_to_cxl_pxb(DeviceState *dev)
+{
+    BusState *bus = dev->parent_bus;
+
+    DeviceState *cxl_rp_dev = NULL;
+    if (bus && bus->parent) {
+        cxl_rp_dev = bus->parent;
+    }
+    assert(cxl_rp_dev);
+
+    DeviceState *cxl_host_dev = NULL;
+    if (cxl_rp_dev) {
+        BusState *rp_bus = cxl_rp_dev->parent_bus;
+        if (rp_bus && rp_bus->parent) {
+            cxl_host_dev = rp_bus->parent;
+        }
+    }
+    assert(cxl_host_dev);
+
+    if (cxl_host_dev &&
+        object_dynamic_cast(OBJECT(cxl_host_dev), TYPE_PXB_CXL_HOST)) {
+        CXLHost *host = PXB_CXL_HOST(cxl_host_dev);
+        return host;
+    }
+
+    return NULL;
+}
+
 static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
 {
     DeviceState *ds = DEVICE(ct3d);
+    CXLHost *host = ct3d_to_cxl_pxb(ds);
 
     if (!ct3d->hostmem && !ct3d->hostvmem && !ct3d->hostpmem
         && !ct3d->dc.num_regions) {
@@ -816,6 +846,13 @@ static bool cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
         g_free(p_name);
     }
 
+    if (ct3d->program_hdm_decoder) {
+        ct3d->hpa_offset = host->unused_hpa_offset;
+        ct3d->idx = host->unused_dev_num;
+        host->unused_hpa_offset += ct3d->cxl_dstate.static_mem_size;
+        host->unused_dev_num++;
+    }
+
     ct3d->dc.total_capacity = 0;
     if (ct3d->dc.num_regions > 0) {
         MemoryRegion *dc_mr;
diff --git a/include/hw/cxl/cxl.h b/include/hw/cxl/cxl.h
index 7cf3dc622c..b8b3071f81 100644
--- a/include/hw/cxl/cxl.h
+++ b/include/hw/cxl/cxl.h
@@ -67,6 +67,8 @@ struct CXLHost {
 
     CXLComponentState cxl_cstate;
     bool passthrough;
+    hwaddr unused_hpa_offset;
+    int unused_dev_num;
 };
 
 #define TYPE_PXB_CXL_HOST "pxb-cxl-host"
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index a2c830d181..c3a729f8c5 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -778,6 +778,8 @@ struct CXLType3Dev {
     bool hdmdb;
 
     bool program_hdm_decoder;
+    hwaddr hpa_offset;
+    int idx;
 
     struct dynamic_capacity {
         HostMemoryBackend *host_dc;
-- 
2.52.0

[PATCH RFC 3/7] QEMU: hw/cxl: add helper to get HPA range for CXL Type 3

[Yukiya Takaki]

The previously added `hpa_offset` on CXLType3Dev records the offset of a
device within its CFMW window, but there is no simple helper to obtain the
full host physical address (HPA) range for a given Type 3 device.

This patch adds `ct3_type3_get_hpm_range()`, which returns the HPA base and
size for a CXLType3Dev by combining the CFMW base, the per‑device offset
and the device size. The helper is declared in the header so that other
code can use it when building tables and regions.

Co-developed-by: Li Zhijian <lizhijian@fujitsu.com>
Signed-off-by: Yukiya Takaki <fj2110jc@fujitsu.com>
---
 hw/mem/cxl_type3.c          | 40 +++++++++++++++++++++++++++++++++++++
 include/hw/cxl/cxl_device.h |  1 +
 2 files changed, 41 insertions(+)

diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index d08a4255c9..811f8e247d 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -1358,6 +1358,46 @@ MemTxResult cxl_type3_write(PCIDevice *d, hwaddr host_addr, uint64_t data,
     return address_space_write(as, dpa_offset, attrs, &data, size);
 }
 
+static uint64_t c3t_addr_size(DeviceState *dev)
+{
+    CXLType3Dev *ct3d = CXL_TYPE3(dev);
+    MemoryRegion *vmr = host_memory_backend_get_memory(ct3d->hostvmem);
+
+    return memory_region_size(vmr);
+}
+
+static uint64_t c3t_hpa_base(DeviceState *dev)
+{
+    GSList *cfmws_list, *iter;
+
+    cfmws_list = cxl_fmws_get_all_sorted();
+    for (iter = cfmws_list; iter; iter = iter->next) {
+        struct CXLFixedWindow *fw;
+
+        if (!object_dynamic_cast(iter->data, TYPE_CXL_FMW)) {
+            continue;
+        }
+        fw = CXL_FMW(iter->data);
+
+        PXBCXLDev *pxb = fw->target_hbs[0];
+        if (ct3d_to_cxl_pxb(dev) != pxb->cxl_host_bridge) {
+            continue;
+        }
+
+        g_slist_free(cfmws_list);
+        return fw->base;
+    }
+    g_slist_free(cfmws_list);
+
+    return 0;
+}
+
+void ct3_type3_get_hpm_range(CXLType3Dev *dev, uint64_t *base, uint64_t *size)
+{
+    *base = c3t_hpa_base(DEVICE(dev)) + dev->hpa_offset;
+    *size = c3t_addr_size(DEVICE(dev));
+}
+
 static void ct3d_reset(DeviceState *dev)
 {
     CXLType3Dev *ct3d = CXL_TYPE3(dev);
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index c3a729f8c5..07654f4b62 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -878,6 +878,7 @@ void ct3_clear_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa,
                                    uint64_t len);
 bool ct3_test_region_block_backed(CXLType3Dev *ct3d, uint64_t dpa,
                                   uint64_t len);
+void ct3_type3_get_hpm_range(CXLType3Dev *dev, uint64_t *base, uint64_t *size);
 void cxl_assign_event_header(CXLEventRecordHdr *hdr,
                              const QemuUUID *uuid, uint32_t flags,
                              uint8_t length, uint64_t timestamp,
-- 
2.52.0

[PATCH RFC 4/7] QEMU/hw/cxl: auto‑program HDM decoders for CXL Type 3

[Yukiya Takaki]

On some real systems, firmware programs the HDM decoders on both the
endpoint and the CXL host bridge so that the OS can discover CXL memory
ranges at boot and the CXL driver can automatically create regions.

This patch adds helpers to program the HDM decoders in QEMU using the
per‑device HPA range from `ct3_type3_get_hpm_range()`, and wires them
into the CXL Type 3 reset path. When `program_hdm_decoder` is enabled,
both the endpoint and host‑bridge HDM decoders are configured so that
the guest sees the same kind of automatically assigned CXL regions as
on real hardware.

The reset hold phase processes child devices before their parents.
Consequently, the Type 3 reset handler programs the host bridge decoder
before the pxb-cxl reset handler runs. Calling
cxl_component_register_init_common() from the pxb-cxl reset handler
would then clear the component register block and erase the decoder
configuration that was just programmed.

When the affected HPA ranges are advertised through E820 and SRAT,
which will be added in the following patches, Linux attempts to use them
as system memory. Without a valid host bridge decoder, accesses to
these ranges cannot be routed to the Type 3 devices. This corrupts
kernel memory-management data and eventually causes a kernel panic.

Initialize the common host bridge component registers when the PXB is
realized instead. Keep the reset handler limited to the
topology-dependent HDM capability setup so that the automatically
programmed decoder settings are not overwritten.

Co-developed-by: Li Zhijian <lizhijian@fujitsu.com>
Signed-off-by: Yukiya Takaki <fj2110jc@fujitsu.com>
---
 hw/mem/cxl_type3.c                  | 98 +++++++++++++++++++++++++++++
 hw/pci-bridge/pci_expander_bridge.c | 13 +++-
 2 files changed, 109 insertions(+), 2 deletions(-)

diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index 811f8e247d..60e4c0e20f 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -19,6 +19,9 @@
 #include "hw/pci/pci.h"
 #include "hw/core/qdev-properties.h"
 #include "hw/core/qdev-properties-system.h"
+#include "hw/pci/pci_bridge.h"
+#include "hw/pci/pcie_port.h"
+#include "hw/cxl/cxl_host.h"
 #include "qapi/error.h"
 #include "qemu/log.h"
 #include "qemu/module.h"
@@ -1358,6 +1361,59 @@ MemTxResult cxl_type3_write(PCIDevice *d, hwaddr host_addr, uint64_t data,
     return address_space_write(as, dpa_offset, attrs, &data, size);
 }
 
+static void __update_hdm_decoder_n(uint32_t *cache_mem, int which,
+                                   uint32_t ctrl, uint64_t base,
+                                   uint64_t size)
+{
+    int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO;
+
+    stl_le_p(cache_mem + R_CXL_HDM_DECODER0_BASE_LO + which * hdm_inc,
+             base & 0xffffffff);
+    stl_le_p(cache_mem + R_CXL_HDM_DECODER0_BASE_HI + which * hdm_inc,
+             base >> 32);
+    stl_le_p(cache_mem + R_CXL_HDM_DECODER0_SIZE_LO + which * hdm_inc,
+             size & 0xffffffff);
+    stl_le_p(cache_mem + R_CXL_HDM_DECODER0_SIZE_HI + which * hdm_inc,
+             size >> 32);
+    stl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL + which * hdm_inc, ctrl);
+}
+
+static void update_hdm_decoder_endpoint(uint32_t *cache_mem, int which,
+                                        uint32_t ctrl, uint64_t base,
+                                        uint64_t size)
+{
+    return __update_hdm_decoder_n(cache_mem, which, ctrl, base, size);
+}
+
+static void update_hdm_decoder_pxb(uint32_t *cache_mem, int which,
+                                   uint64_t tl, uint32_t ctrl,
+                                   uint64_t base, uint64_t size)
+{
+    int hdm_inc = R_CXL_HDM_DECODER1_BASE_LO - R_CXL_HDM_DECODER0_BASE_LO;
+
+    stl_le_p(cache_mem +
+             R_CXL_HDM_DECODER0_TARGET_LIST_LO + which * hdm_inc,
+             tl & 0xffffffff);
+    stl_le_p(cache_mem +
+             R_CXL_HDM_DECODER0_TARGET_LIST_HI + which * hdm_inc,
+             tl >> 32);
+
+    __update_hdm_decoder_n(cache_mem, which, ctrl, base, size);
+}
+
+static uint8_t ct3d_root_port_number(DeviceState *dev)
+{
+    BusState *bus = dev->parent_bus;
+    DeviceState *rp_dev;
+
+    assert(bus);
+    rp_dev = bus->parent;
+    assert(rp_dev);
+    assert(object_dynamic_cast(OBJECT(rp_dev), TYPE_PCIE_ROOT_PORT));
+
+    return PCIE_PORT(rp_dev)->port;
+}
+
 static uint64_t c3t_addr_size(DeviceState *dev)
 {
     CXLType3Dev *ct3d = CXL_TYPE3(dev);
@@ -1398,6 +1454,42 @@ void ct3_type3_get_hpm_range(CXLType3Dev *dev, uint64_t *base, uint64_t *size)
     *size = c3t_addr_size(DEVICE(dev));
 }
 
+static void program_pxb_hdm_decoder(DeviceState *dev)
+{
+    CXLHost *cxl = ct3d_to_cxl_pxb(dev);
+    CXLType3Dev *ct3d = CXL_TYPE3(dev);
+    int hdm_num = ct3d->idx;
+    int order = ct3d_root_port_number(dev);
+    assert(cxl);
+    uint64_t base, size;
+    ct3_type3_get_hpm_range(ct3d, &base, &size);
+
+    CXLComponentState *cxl_cstate = &cxl->cxl_cstate;
+    uint32_t *cache_mem = cxl_cstate->crb.cache_mem_registers;
+
+    uint32_t ctrl = R_CXL_HDM_DECODER0_CTRL_COMMIT_MASK |
+                    R_CXL_HDM_DECODER0_CTRL_COMMITTED_MASK |
+                    R_CXL_HDM_DECODER0_CTRL_TYPE_MASK;
+
+    update_hdm_decoder_pxb(cache_mem, hdm_num, order, ctrl, base, size);
+}
+
+static void program_hdm_decoder(DeviceState *dev)
+{
+    CXLType3Dev *ct3d = CXL_TYPE3(dev);
+    uint64_t base, size;
+    ComponentRegisters *cregs = &ct3d->cxl_cstate.crb;
+    uint32_t *cache_mem = cregs->cache_mem_registers;
+
+    ct3_type3_get_hpm_range(ct3d, &base, &size);
+
+    uint32_t ctrl = R_CXL_HDM_DECODER0_CTRL_COMMIT_MASK |
+                    R_CXL_HDM_DECODER0_CTRL_COMMITTED_MASK |
+                    R_CXL_HDM_DECODER0_CTRL_TYPE_MASK;
+
+    update_hdm_decoder_endpoint(cache_mem, 0, ctrl, base, size);
+}
+
 static void ct3d_reset(DeviceState *dev)
 {
     CXLType3Dev *ct3d = CXL_TYPE3(dev);
@@ -1408,6 +1500,12 @@ static void ct3d_reset(DeviceState *dev)
                               ct3d->flitmode);
     cxl_component_register_init_common(reg_state, write_msk,
                                        CXL2_TYPE3_DEVICE, ct3d->hdmdb);
+
+    if (ct3d->program_hdm_decoder) {
+        program_hdm_decoder(dev);
+        program_pxb_hdm_decoder(dev);
+    }
+
     cxl_device_register_init_t3(ct3d, CXL_T3_MSIX_MBOX);
 
     /*
diff --git a/hw/pci-bridge/pci_expander_bridge.c b/hw/pci-bridge/pci_expander_bridge.c
index 40ffbc4e08..e8fd7343d1 100644
--- a/hw/pci-bridge/pci_expander_bridge.c
+++ b/hw/pci-bridge/pci_expander_bridge.c
@@ -297,16 +297,24 @@ static int pxb_map_irq_fn(PCIDevice *pci_dev, int pin)
     return pin - PCI_SLOT(pxb->devfn);
 }
 
+static void pxb_cxl_dev_init(DeviceState *dev)
+{
+    CXLHost *cxl = PXB_CXL_DEV(dev)->cxl_host_bridge;
+    CXLComponentState *cxl_cstate = &cxl->cxl_cstate;
+    uint32_t *reg_state = cxl_cstate->crb.cache_mem_registers;
+    uint32_t *write_msk = cxl_cstate->crb.cache_mem_regs_write_mask;
+
+    cxl_component_register_init_common(reg_state, write_msk, CXL2_RC, false);
+}
+
 static void pxb_cxl_dev_reset(DeviceState *dev)
 {
     CXLHost *cxl = PXB_CXL_DEV(dev)->cxl_host_bridge;
     CXLComponentState *cxl_cstate = &cxl->cxl_cstate;
     PCIHostState *hb = PCI_HOST_BRIDGE(cxl);
     uint32_t *reg_state = cxl_cstate->crb.cache_mem_registers;
-    uint32_t *write_msk = cxl_cstate->crb.cache_mem_regs_write_mask;
     int dsp_count = 0;
 
-    cxl_component_register_init_common(reg_state, write_msk, CXL2_RC, false);
     /*
      * The CXL specification allows for host bridges with no HDM decoders
      * if they only have a single root port.
@@ -507,6 +515,7 @@ static void pxb_cxl_dev_realize(PCIDevice *dev, Error **errp)
     if (!pxb_dev_realize_common(dev, CXL, errp)) {
         return;
     }
+    pxb_cxl_dev_init(DEVICE(dev));
     pxb_cxl_dev_reset(DEVICE(dev));
 }
 
-- 
2.52.0

[PATCH RFC 5/7] QEMU/hw/cxl: expose CXL Type 3 ranges as E820_SOFT_RESERVED on x86

[Yukiya Takaki]

On real systems, CXL Type 3 memory is exposed to the OS via the E820
table as E820_SOFT_RESERVED, so the kernel can treat it as a dedicated
reserved range rather than normal RAM.

This patch adds a small helper and a recursive walker so that, on x86,
the HPA ranges assigned to CXL Type 3 devices are added to the E820 table
as E820_SOFT_RESERVED when `program_hdm_decoder` is enabled. An ARM stub
is also added as a placeholder for future support.

Signed-off-by: Yukiya Takaki <fj2110jc@fujitsu.com>
---
 hw/arm/cxl.c              | 10 ++++++++++
 hw/arm/meson.build        |  2 ++
 hw/cxl/cxl-host.c         | 25 +++++++++++++++++++++++++
 hw/i386/cxl.c             | 11 +++++++++++
 hw/i386/meson.build       |  1 +
 hw/i386/pc.c              |  1 +
 include/hw/cxl/cxl_host.h |  2 ++
 7 files changed, 52 insertions(+)
 create mode 100644 hw/arm/cxl.c
 create mode 100644 hw/i386/cxl.c

diff --git a/hw/arm/cxl.c b/hw/arm/cxl.c
new file mode 100644
index 0000000000..1cbcf2d8e5
--- /dev/null
+++ b/hw/arm/cxl.c
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+#include "qemu/osdep.h"
+#include "exec/hwaddr.h"
+#include "hw/cxl/cxl_host.h"
+
+void cxl_c3_add_memory_map(hwaddr base, hwaddr size)
+{
+    /* TODO: Implement memory map addition for CXL Type 3 on ARM */
+}
diff --git a/hw/arm/meson.build b/hw/arm/meson.build
index bcc068de04..2a4f09fbef 100644
--- a/hw/arm/meson.build
+++ b/hw/arm/meson.build
@@ -116,6 +116,8 @@ arm_common_ss.add(when: ['CONFIG_AXIADO_EVK', 'TARGET_AARCH64'], if_true: files(
   'ax3000-boards.c',
   'ax3000-evk.c'))
 
+arm_common_ss.add(when: 'CONFIG_CXL', if_true: files('cxl.c'))
+
 arm_common_ss.add(files('boot.c'))
 
 hw_common_arch += {'arm': arm_common_ss}
diff --git a/hw/cxl/cxl-host.c b/hw/cxl/cxl-host.c
index e279ce5fe3..1967c9a9c3 100644
--- a/hw/cxl/cxl-host.c
+++ b/hw/cxl/cxl-host.c
@@ -689,12 +689,37 @@ static int cxl_fmws_mmio_map(Object *obj, void *opaque)
     return 0;
 }
 
+static int cxl_c3_mmio_map(Object *obj, void *opaque)
+{
+    uint64_t base, size;
+    CXLType3Dev *c3d;
+
+    if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) {
+        return 0;
+    }
+    c3d = CXL_TYPE3(obj);
+
+    if (!c3d->program_hdm_decoder) {
+        return 0;
+    }
+
+    ct3_type3_get_hpm_range(c3d, &base, &size);
+    cxl_c3_add_memory_map(base, size);
+
+    return 0;
+}
+
 void cxl_fmws_update_mmio(void)
 {
     /* Ordering is not required for this */
     object_child_foreach_recursive(object_get_root(), cxl_fmws_mmio_map, NULL);
 }
 
+void cxl_c3_update_mmio(void)
+{
+    object_child_foreach_recursive(object_get_root(), cxl_c3_mmio_map, NULL);
+}
+
 hwaddr cxl_fmws_set_memmap(hwaddr base, hwaddr max_addr)
 {
     GSList *cfmws_list, *iter;
diff --git a/hw/i386/cxl.c b/hw/i386/cxl.c
new file mode 100644
index 0000000000..02dbe4d3f0
--- /dev/null
+++ b/hw/i386/cxl.c
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+#include "qemu/osdep.h"
+#include "exec/hwaddr.h"
+#include "hw/cxl/cxl_host.h"
+#include "hw/i386/e820_memory_layout.h"
+
+void cxl_c3_add_memory_map(hwaddr base, hwaddr size)
+{
+    e820_add_entry(base, size, E820_SOFT_RESERVED);
+}
diff --git a/hw/i386/meson.build b/hw/i386/meson.build
index e16930c431..b212b3f6fa 100644
--- a/hw/i386/meson.build
+++ b/hw/i386/meson.build
@@ -24,6 +24,7 @@ i386_ss.add(when: 'CONFIG_VTD', if_true: files('intel_iommu.c'))
 i386_ss.add(when: 'CONFIG_VTD_ACCEL', if_true: files('intel_iommu_accel.c'))
 i386_ss.add(when: 'CONFIG_SGX', if_true: files('sgx-epc.c','sgx.c'),
                                 if_false: files('sgx-stub.c'))
+i386_ss.add(when: 'CONFIG_CXL', if_true: files('cxl.c'))
 if have_hmp
   stub_ss.add(files('sgx-hmp-stub.c'))
 endif
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 2d8c87b0e3..f843629543 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -534,6 +534,7 @@ void pc_machine_done(Notifier *notifier, void *data)
 
     if (pcms->cxl_devices_state.is_enabled) {
         cxl_fmws_link_targets(&error_fatal);
+        cxl_c3_update_mmio();
     }
 
     /* set the number of CPUs */
diff --git a/include/hw/cxl/cxl_host.h b/include/hw/cxl/cxl_host.h
index 21619bb748..6c142a5b89 100644
--- a/include/hw/cxl/cxl_host.h
+++ b/include/hw/cxl/cxl_host.h
@@ -18,6 +18,8 @@ void cxl_fmws_link_targets(Error **errp);
 void cxl_hook_up_pxb_registers(PCIBus *bus, CXLState *state, Error **errp);
 hwaddr cxl_fmws_set_memmap(hwaddr base, hwaddr max_addr);
 void cxl_fmws_update_mmio(void);
+void cxl_c3_update_mmio(void);
+void cxl_c3_add_memory_map(hwaddr base, hwaddr size);
 GSList *cxl_fmws_get_all_sorted(void);
 
 extern const MemoryRegionOps cfmws_ops;
-- 
2.52.0

[PATCH RFC 6/7] QEMU/hw/cxl: Create NUMA nodes for CXL Type 3 memory

[Yukiya Takaki]

On the actual hardware, CXL memory is exposed to the OS as a "Memory Only Node."
This allows the OS to recognize that accessing CXL memory takes time.
As a preliminary step, a function was created to automatically add a NUMA node for each instance of CXL memory.
This function generates one NUMA node per instance of CXL memory.
This function is executed when program_hdm_decoder is set.

Signed-off-by: Yukiya Takaki <fj2110jc@fujitsu.com>
---
 hw/cxl/cxl-host.c           | 38 +++++++++++++++++++++++++++++++++++++
 hw/i386/pc.c                |  1 +
 include/hw/cxl/cxl_device.h |  1 +
 include/hw/cxl/cxl_host.h   |  1 +
 4 files changed, 41 insertions(+)

diff --git a/hw/cxl/cxl-host.c b/hw/cxl/cxl-host.c
index 1967c9a9c3..ba321841dc 100644
--- a/hw/cxl/cxl-host.c
+++ b/hw/cxl/cxl-host.c
@@ -720,6 +720,44 @@ void cxl_c3_update_mmio(void)
     object_child_foreach_recursive(object_get_root(), cxl_c3_mmio_map, NULL);
 }
 
+static int setup_cxl_numa_node_inner(Object *obj, void *opaque)
+{
+    MachineState *machine = opaque;
+    DeviceState *dev;
+    CXLType3Dev *c3d;
+    int node_id;
+
+    if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) {
+        return 0;
+    }
+
+    dev = DEVICE(obj);
+    c3d = CXL_TYPE3(dev);
+    node_id = machine->numa_state->num_nodes;
+
+    if (!c3d->program_hdm_decoder) {
+        return 0;
+    }
+
+    NumaOptions option = {
+        .type = NUMA_OPTIONS_TYPE_NODE,
+        .u.node = {
+            .has_nodeid = true,
+            .nodeid = node_id,
+        },
+    };
+
+    set_numa_options(machine, &option, &error_fatal);
+    c3d->node = node_id;
+    return 0;
+}
+
+void setup_cxl_numa_node(MachineState *machine)
+{
+    object_child_foreach_recursive(object_get_root(),
+                                   setup_cxl_numa_node_inner, machine);
+}
+
 hwaddr cxl_fmws_set_memmap(hwaddr base, hwaddr max_addr)
 {
     GSList *cfmws_list, *iter;
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index f843629543..b1d1f03468 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -534,6 +534,7 @@ void pc_machine_done(Notifier *notifier, void *data)
 
     if (pcms->cxl_devices_state.is_enabled) {
         cxl_fmws_link_targets(&error_fatal);
+        setup_cxl_numa_node(MACHINE(pcms));
         cxl_c3_update_mmio();
     }
 
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index 07654f4b62..9001aacf0e 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -780,6 +780,7 @@ struct CXLType3Dev {
     bool program_hdm_decoder;
     hwaddr hpa_offset;
     int idx;
+    int node;
 
     struct dynamic_capacity {
         HostMemoryBackend *host_dc;
diff --git a/include/hw/cxl/cxl_host.h b/include/hw/cxl/cxl_host.h
index 6c142a5b89..70285328c4 100644
--- a/include/hw/cxl/cxl_host.h
+++ b/include/hw/cxl/cxl_host.h
@@ -19,6 +19,7 @@ void cxl_hook_up_pxb_registers(PCIBus *bus, CXLState *state, Error **errp);
 hwaddr cxl_fmws_set_memmap(hwaddr base, hwaddr max_addr);
 void cxl_fmws_update_mmio(void);
 void cxl_c3_update_mmio(void);
+void setup_cxl_numa_node(MachineState *machine);
 void cxl_c3_add_memory_map(hwaddr base, hwaddr size);
 GSList *cxl_fmws_get_all_sorted(void);
 
-- 
2.52.0