Re: [PATCH 4/4] hw/cxl: Map committed HDM decoder ranges as RAM for direct DMA

Re: [PATCH 4/4] hw/cxl: Map committed HDM decoder ranges as RAM for direct DMA

[Jonathan Cameron]

On Tue,  2 Jun 2026 15:41:27 +0800
Chen Pei <cp0613@linux.alibaba.com> wrote:

> The CXL Fixed Memory Window (CFMW) is registered with
> memory_region_init_io() and has no backing ram_block.  As a result
> address_space_map() on a guest physical address inside the CFMW
> takes the bounce-buffer path and is bounded by
> DEFAULT_MAX_BOUNCE_BUFFER_SIZE (4 KiB by default for the system
> AddressSpace).  Once a Type-3 device is brought online as system RAM
> (daxctl online-memory), the kernel happily allocates DMA buffers from
> the CFMW range and any virtio operation whose scatter list exceeds
> 4 KiB or that overlaps with another in-flight transfer fails with:
> 
>     qemu-system-riscv64: virtio: bogus descriptor or out of resources

This is half the bug if you are running without the stuff I talk about
below - you can't run instructions out of that memory either.

As to the bonce buffer that should be larger anyway - there has
been some discussion in the past on how to do that for the global one
(there is support for PCI specific cases) but we never got to a
standard solution.

> 
> The bug is not RISC-V specific: CFMW registration and the bounce
> buffer limit are both arch-agnostic, so any guest that onlines CXL
> memory and issues DMA larger than 4 KiB into it is affected.  It
> shows up first on RISC-V virt because that is where the rest of
> this series enables the daxctl + virtio path end-to-end.
> 
> Reproduce on RISC-V virt with cxl=on and a single Type-3 device:
> 
>     cxl create-region -m -t ram -d decoder0.0 -w 1 mem0 -s 4G
>     daxctl online-memory dax0.0
>     free -h    # triggers the error and stalls the guest
> 
> Fix it by overlaying a RAM alias of the device's memory backend
> (hostvmem / hostpmem) at the committed HDM decoder's HPA range, with
> higher priority than the CFMW I/O region.  flatview_translate() then
> hits the alias, address_space_map() returns a direct host pointer,
> and DMA proceeds without bouncing.  This mirrors the existing QEMU
> pattern of PCI BAR and IOMMU MR overlays.  The alias is torn down on
> hdm_decoder_uncommit() so subsequent region tear-down + re-creation
> works.
> 

+CC linux-cxl as that's where you are more likely to get feedback on 
CXL aspects.

For this one there is a more comprehensive solution that restricts this
to the cases that are valid (no interleaving going on).
https://lore.kernel.org/qemu-devel/20260318171918.146-1-alireza.sanaee@huawei.com/#t
It is rather more complex as a result.

For other cases we have always intentionally left it as not supporting
what you have here because we want the interleave to be correct and testable.

As far as I'm concerned that series of Alireza's has been ready to merge for
a while. I've been a bit busy with other things so haven't been poking
for that though.  Might be a week or two before I get back on top of things.

Anyhow, please give Ali's series a go and see if that works for you.
There may well be other corners we haven't thought of!

Jonathan



> Signed-off-by: Chen Pei <cp0613@linux.alibaba.com>



> ---
>  hw/mem/cxl_type3.c          | 81 +++++++++++++++++++++++++++++++++++++
>  include/hw/cxl/cxl_device.h |  4 ++
>  2 files changed, 85 insertions(+)
> 
> diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
> index 4739239da3..f962bce66a 100644
> --- a/hw/mem/cxl_type3.c
> +++ b/hw/mem/cxl_type3.c
> @@ -24,6 +24,7 @@
>  #include "qemu/module.h"
>  #include "qemu/pmem.h"
>  #include "qemu/range.h"
> +#include "system/address-spaces.h"
>  #include "qemu/rcu.h"
>  #include "qemu/guest-random.h"
>  #include "system/hostmem.h"
> @@ -420,6 +421,11 @@ static void hdm_decoder_commit(CXLType3Dev *ct3d, int which)
>      ComponentRegisters *cregs = &ct3d->cxl_cstate.crb;
>      uint32_t *cache_mem = cregs->cache_mem_registers;
>      uint32_t ctrl;
> +    uint32_t low, high;
> +    uint64_t decoder_base, decoder_size;
> +    MemoryRegion *mr = NULL;
> +    uint64_t dpa_offset = 0;
> +    char *alias_name;
>  
>      ctrl = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL + which * hdm_inc);
>      /* TODO: Sanity checks that the decoder is possible */
> @@ -427,6 +433,73 @@ static void hdm_decoder_commit(CXLType3Dev *ct3d, int which)
>      ctrl = FIELD_DP32(ctrl, CXL_HDM_DECODER0_CTRL, COMMITTED, 1);
>  
>      stl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL + which * hdm_inc, ctrl);
> +
> +    /*
> +     * Create a RAM alias in system memory for the committed decoder range.
> +     * This enables direct DMA mapping (address_space_map) for devices like
> +     * virtio that need to DMA to/from CXL memory.  Without this, the CFMW
> +     * I/O region would require bounce buffering which is limited to 4KB.
> +     */
> +    low = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_BASE_LO + which * hdm_inc);
> +    high = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_BASE_HI + which * hdm_inc);
> +    decoder_base = ((uint64_t)high << 32) | (low & 0xf0000000);
> +
> +    low = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_SIZE_LO + which * hdm_inc);
> +    high = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_SIZE_HI + which * hdm_inc);
> +    decoder_size = ((uint64_t)high << 32) | (low & 0xf0000000);
> +
> +    if (!decoder_base || !decoder_size) {
> +        return;
> +    }
> +
> +    /* Calculate DPA offset by summing sizes of preceding decoders */
> +    for (int i = 0; i < which; i++) {
> +        uint32_t prev_low, prev_high;
> +        uint64_t prev_size;
> +        uint32_t prev_ctrl;
> +
> +        prev_ctrl = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL +
> +                             i * hdm_inc);
> +        if (!FIELD_EX32(prev_ctrl, CXL_HDM_DECODER0_CTRL, COMMITTED)) {
> +            continue;
> +        }
> +        prev_low = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_SIZE_LO +
> +                            i * hdm_inc);
> +        prev_high = ldl_le_p(cache_mem + R_CXL_HDM_DECODER0_SIZE_HI +
> +                             i * hdm_inc);
> +        prev_size = ((uint64_t)prev_high << 32) | (prev_low & 0xf0000000);
> +        dpa_offset += prev_size;
> +    }
> +
> +    /* Determine which memory backend to alias */
> +    if (ct3d->hostvmem) {
> +        MemoryRegion *vmr = host_memory_backend_get_memory(ct3d->hostvmem);
> +        uint64_t vmr_size = memory_region_size(vmr);
> +
> +        if (dpa_offset < vmr_size) {
> +            mr = vmr;
> +        }
> +    }
> +    if (!mr && ct3d->hostpmem) {
> +        MemoryRegion *pmr = host_memory_backend_get_memory(ct3d->hostpmem);
> +        uint64_t vmr_size = ct3d->hostvmem ?
> +            memory_region_size(
> +                host_memory_backend_get_memory(ct3d->hostvmem)) : 0;
> +        mr = pmr;
> +        dpa_offset -= vmr_size;
> +    }
> +
> +    if (!mr) {
> +        return;
> +    }
> +
> +    alias_name = g_strdup_printf("cxl-hdm%d-ram-alias", which);
> +    memory_region_init_alias(&ct3d->hdm_ram_alias[which], OBJECT(ct3d),
> +                             alias_name, mr, dpa_offset, decoder_size);
> +    memory_region_add_subregion_overlap(get_system_memory(), decoder_base,
> +                                        &ct3d->hdm_ram_alias[which], 1);
> +    ct3d->hdm_ram_alias_valid[which] = true;
> +    g_free(alias_name);
>  }
>  
>  static void hdm_decoder_uncommit(CXLType3Dev *ct3d, int which)
> @@ -442,6 +515,14 @@ static void hdm_decoder_uncommit(CXLType3Dev *ct3d, int which)
>      ctrl = FIELD_DP32(ctrl, CXL_HDM_DECODER0_CTRL, COMMITTED, 0);
>  
>      stl_le_p(cache_mem + R_CXL_HDM_DECODER0_CTRL + which * hdm_inc, ctrl);
> +
> +    /* Remove the RAM alias if it was added during commit */
> +    if (ct3d->hdm_ram_alias_valid[which]) {
> +        memory_region_del_subregion(get_system_memory(),
> +                                    &ct3d->hdm_ram_alias[which]);
> +        object_unparent(OBJECT(&ct3d->hdm_ram_alias[which]));
> +        ct3d->hdm_ram_alias_valid[which] = false;
> +    }
>  }
>  
>  static int ct3d_qmp_uncor_err_to_cxl(CxlUncorErrorType qmp_err)
> diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
> index 393f312217..07deef2e2c 100644
> --- a/include/hw/cxl/cxl_device.h
> +++ b/include/hw/cxl/cxl_device.h
> @@ -714,6 +714,10 @@ struct CXLType3Dev {
>      /* State */
>      AddressSpace hostvmem_as;
>      AddressSpace hostpmem_as;
> +
> +    /* RAM aliases for HDM decoders - enables direct DMA mapping */
> +    MemoryRegion hdm_ram_alias[CXL_HDM_DECODER_COUNT];
> +    bool hdm_ram_alias_valid[CXL_HDM_DECODER_COUNT];
>      CXLComponentState cxl_cstate;
>      CXLDeviceState cxl_dstate;
>      CXLCCI cci; /* Primary PCI mailbox CCI */

Re: [PATCH 4/4] hw/cxl: Map committed HDM decoder ranges as RAM for direct DMA

[Chen Pei]

On Tue, 9 Jun 2026 13:56:11 +0100, Jonathan Cameron <jic23@kernel.org> wrote:

Hi Jonathan,

> +CC linux-cxl as that's where you are more likely to get feedback on 
> CXL aspects.
> 
> For this one there is a more comprehensive solution that restricts this
> to the cases that are valid (no interleaving going on).
> https://lore.kernel.org/qemu-devel/20260318171918.146-1-alireza.sanaee@huawei.com/#t
> It is rather more complex as a result.
> 
> For other cases we have always intentionally left it as not supporting
> what you have here because we want the interleave to be correct and testable.
> 
> As far as I'm concerned that series of Alireza's has been ready to merge for
> a while. I've been a bit busy with other things so haven't been poking
> for that though.  Might be a week or two before I get back on top of things.
> 
> Anyhow, please give Ali's series a go and see if that works for you.
> There may well be other corners we haven't thought of!

Thanks for pointing out Alireza's series.

I'll test Alireza's series against the RISC-V virt machine with the
same CXL + daxctl + virtio DMA workflow. If it resolves the issue,
I'll drop this patch from v2 and just rebase patches 1-3.

I'll report back once I've tested.

Best,
Pei

Re: [PATCH 2/4] hw/riscv/virt-acpi-build: Add _DEP to ACPI0017 for CXL host bridge dependency

[Jonathan Cameron]

On Tue,  2 Jun 2026 15:41:25 +0800
Chen Pei <cp0613@linux.alibaba.com> wrote:

> On RISC-V QEMU virt platform with CXL enabled, the probe ordering
> of acpi_pci_root (ACPI0016) and cxl_acpi (ACPI0017) is not
> guaranteed. If cxl_acpi probes before acpi_pci_root has attached
> the CXL host bridges, the CXL port topology will be incomplete
> because to_cxl_host_bridge() silently skips devices whose PCI root
> is not yet ready.
> 
> Add a _DEP object to the ACPI0017 device in the DSDT, declaring
> its dependency on the ACPI0016 CXL host bridge devices. This tells
> the OS to defer ACPI0017 enumeration until all ACPI0016 devices
> have been attached by acpi_pci_root.
> 
> This requires a corresponding kernel change to call
> acpi_dev_clear_dependencies() in acpi_pci_root_add().
> 
> Signed-off-by: Chen Pei <cp0613@linux.alibaba.com>

It may be a good idea to add a bios tables test as well.
 
We only have the x86 q35 one today because I argued at the time
the ARM64 one would be just duplication.  Now we have this new _DEP
stuff we should probably add something to cover it.

Please also include an iasl -d dump of the relevant additions to DSDT
in the patch description.  Much easier to review than the code that
generates it!

In general looks fine to me and great that you are clearing this up.
Seems it's luck that x86 and ARM64 worked without this (or strictly
speaking other things enforcing the ordering).  Ultimately we probably
want to add this to those two architectures a well.

One small thing inline.

Thanks,

Jonathan


> ---
>  hw/riscv/virt-acpi-build.c | 32 ++++++++++++++++++++++++++++++++
>  1 file changed, 32 insertions(+)
> 
> diff --git a/hw/riscv/virt-acpi-build.c b/hw/riscv/virt-acpi-build.c
> index 309d64b322..a5bafd1dcf 100644
> --- a/hw/riscv/virt-acpi-build.c
> +++ b/hw/riscv/virt-acpi-build.c
> @@ -510,6 +510,38 @@ static void build_dsdt(GArray *table_data,
>      if (s->cxl_devices_state.is_enabled) {
>          Aml *cxl_dev = aml_device("CXLM");
>          aml_append(cxl_dev, aml_name_decl("_HID", aml_string("ACPI0017")));
> +
> +        /*
> +         * Declare a _DEP on every ACPI0016 CXL host bridge so the OS
> +         * defers ACPI0017 enumeration until acpi_pci_root has attached
> +         * the CXL host bridges. Without this, cxl_acpi may probe before
> +         * to_cxl_host_bridge() can resolve the PCI root and the CXL
> +         * port topology comes up empty.
> +         */
> +        if (s->bus) {
> +            PCIBus *bus;
> +            uint32_t num_cxl_hbs = 0;
> +
> +            QLIST_FOREACH(bus, &s->bus->child, sibling) {
> +                if (pci_bus_is_root(bus) && pci_bus_is_cxl(bus)) {
> +                    num_cxl_hbs++;

I think you only care if there is at least one. Instead of counting, just set
a bool and break out early if you find one.

> +                }
> +            }
> +
> +            if (num_cxl_hbs > 0) {
> +                Aml *dep_pkg = aml_package(num_cxl_hbs);
> +
> +                QLIST_FOREACH(bus, &s->bus->child, sibling) {
> +                    if (pci_bus_is_root(bus) && pci_bus_is_cxl(bus)) {
> +                        aml_append(dep_pkg,
> +                                   aml_name("\\_SB.PC%.02X",
> +                                            pci_bus_num(bus)));
> +                    }
> +                }
> +                aml_append(cxl_dev, aml_name_decl("_DEP", dep_pkg));
> +            }
> +        }
> +
>          Aml *method = aml_method("_STA", 0, AML_NOTSERIALIZED);
>          aml_append(method, aml_return(aml_int(0x0B)));
>          aml_append(cxl_dev, method);

Re: [PATCH 2/4] hw/riscv/virt-acpi-build: Add _DEP to ACPI0017 for CXL host bridge dependency

[Peter Maydell]

On Tue, 9 Jun 2026 at 13:48, Jonathan Cameron <jic23@kernel.org> wrote:
>
> On Tue,  2 Jun 2026 15:41:25 +0800
> Chen Pei <cp0613@linux.alibaba.com> wrote:
>
> > On RISC-V QEMU virt platform with CXL enabled, the probe ordering
> > of acpi_pci_root (ACPI0016) and cxl_acpi (ACPI0017) is not
> > guaranteed. If cxl_acpi probes before acpi_pci_root has attached
> > the CXL host bridges, the CXL port topology will be incomplete
> > because to_cxl_host_bridge() silently skips devices whose PCI root
> > is not yet ready.
> >
> > Add a _DEP object to the ACPI0017 device in the DSDT, declaring
> > its dependency on the ACPI0016 CXL host bridge devices. This tells
> > the OS to defer ACPI0017 enumeration until all ACPI0016 devices
> > have been attached by acpi_pci_root.
> >
> > This requires a corresponding kernel change to call
> > acpi_dev_clear_dependencies() in acpi_pci_root_add().
> >
> > Signed-off-by: Chen Pei <cp0613@linux.alibaba.com>
>
> It may be a good idea to add a bios tables test as well.
>
> We only have the x86 q35 one today because I argued at the time
> the ARM64 one would be just duplication.  Now we have this new _DEP
> stuff we should probably add something to cover it.
>
> Please also include an iasl -d dump of the relevant additions to DSDT
> in the patch description.  Much easier to review than the code that
> generates it!
>
> In general looks fine to me and great that you are clearing this up.
> Seems it's luck that x86 and ARM64 worked without this (or strictly
> speaking other things enforcing the ordering).  Ultimately we probably
> want to add this to those two architectures a well.
>
> One small thing inline.

> > +
> > +            QLIST_FOREACH(bus, &s->bus->child, sibling) {
> > +                if (pci_bus_is_root(bus) && pci_bus_is_cxl(bus)) {
> > +                    num_cxl_hbs++;
>
> I think you only care if there is at least one. Instead of counting, just set
> a bool and break out early if you find one.

> > +                Aml *dep_pkg = aml_package(num_cxl_hbs);

Looks like we need it so we can set the num_elements here.

(If we had a version of aml_package() that let you set the number
of elements after the fact we could do this all with a single loop,
but it doesn't look like we have any other places where that would
be useful. Almost all the callers pass a fixed number.)

-- PMM

Re: [PATCH 2/4] hw/riscv/virt-acpi-build: Add _DEP to ACPI0017 for CXL host bridge dependency

[Chen Pei]

On Tue, 9 Jun 2026 13:47:09 +0100, Jonathan Cameron <jic23@kernel.org> wrote:

Hi Jonathan,

Thanks for the detailed review!

> It may be a good idea to add a bios tables test as well.

Okay. I'll add a RISC-V ACPI bios tables test in v2 to cover
the _DEP additions and prevent regressions.

> Please also include an iasl -d dump of the relevant additions to DSDT
> in the patch description. Much easier to review than the code that
> generates it!

Will do. I'll include the iasl -d decompiled output of the ACPI0017
device (with _DEP) in the v2 commit message.

> In general looks fine to me and great that you are clearing this up.
> Seems it's luck that x86 and ARM64 worked without this (or strictly
> speaking other things enforcing the ordering). Ultimately we probably
> want to add this to those two architectures as well.

Agreed. I can send a follow-up series to add _DEP for x86/ARM64
once this lands on RISC-V.

> > +        if (s->bus) {
> > +            PCIBus *bus;
> > +            uint32_t num_cxl_hbs = 0;
> > +
> > +            QLIST_FOREACH(bus, &s->bus->child, sibling) {
> > +                if (pci_bus_is_root(bus) && pci_bus_is_cxl(bus)) {
> > +                    num_cxl_hbs++;
> 
> I think you only care if there is at least one. Instead of counting, just set
> a bool and break out early if you find one.

As Peter pointed out, the count is actually needed for
aml_package(num_cxl_hbs) to set the correct number of elements.
We'd need two loops regardless (one to count, one to populate), so
I'll keep the current structure.

Best,
Pei

Re: [PATCH 3/4] hw/riscv/virt, gpex: Provide 32-bit MMIO window for CXL host bridges

[Jonathan Cameron]

On Tue,  2 Jun 2026 15:41:26 +0800
Chen Pei <cp0613@linux.alibaba.com> wrote:

> CXL component register BAR (BAR0 on CXL Root Port and Type3 device)
> and the CXL device register BAR (BAR2 on Type3 device) are declared
> as 64-bit non-prefetchable memory.  A standard PCIe-to-PCI bridge
> exposes a 32-bit non-prefetchable memory window plus an (optional)
> 64-bit prefetchable memory window, but no 64-bit non-prefetchable
> window.

When you say 'standard' do you mean that is all the PCI spec allows
for? (Right now I'm not working for a SIG member so don't have a copy
to check)

>  Linux therefore places 64-bit non-prefetchable BARs in the
> 32-bit non-prefetchable bridge window, which requires the bridge to
> own enough address space below 4 GiB.
> 

This sounds a bit like the issue that Dave Jiang reported with recent
EDK2 on x86 (fedora upgraded).  We don't see it with the older EDK2
that ships with QEMU. I haven't yet figured out exactly why.
Arguably whatever they changed is a regression but we don't have good
enough testing in place to have detected it early enough.

I'd like some input from PCI / ACPI experts on this. +CC Michael and
Igor.

Like the previous patch we'd definitely want some testing around this
to make sure it doesn't accidentally get broken in future.


> On RISC-V virt the 32-bit PCIe MMIO range (1 GiB at 0x40000000) is
> currently consumed entirely by PCI0, so CXL host bridges (ACPI0016)
> have no non-prefetchable window and Linux fails to assign these BARs.
> Marking the BARs prefetchable would work around it, but the CXL
> component registers have read/write side effects and are not
> prefetchable per the PCIe specification.
> 
> Reserve the top 256 MiB of the 32-bit MMIO window exclusively for
> CXL host bridges:
> - Shrink PCI0's mmio32 window by 256 MiB in virt.c so that UEFI's
>   PciHostBridgeDxe and the ACPI _CRS for PCI0 never claim that range
> - Store the reserved range in a new gpex_cfg.cxl_mmio32 field
> - In gpex-acpi.c, emit the cxl_mmio32 range as the Memory resource
>   in the CXL host bridge _CRS instead of re-using build_crs() (which
>   returns an empty set when UEFI has not assigned resources yet)
> - Reduce the FDT 'ranges' for PCI0 by the same 256 MiB so that UEFI
>   firmware driven by device-tree also respects the reservation
> 
> Signed-off-by: Chen Pei <cp0613@linux.alibaba.com>
> ---
>  hw/pci-host/gpex-acpi.c    | 36 +++++++++++++++++++++--
>  hw/riscv/virt.c            | 58 +++++++++++++++++++++++++++++++-------
>  include/hw/pci-host/gpex.h |  1 +
>  3 files changed, 83 insertions(+), 12 deletions(-)
> 
> diff --git a/hw/pci-host/gpex-acpi.c b/hw/pci-host/gpex-acpi.c
> index d9820f9b41..d8b943b665 100644
> --- a/hw/pci-host/gpex-acpi.c
> +++ b/hw/pci-host/gpex-acpi.c
> @@ -158,9 +158,41 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)
>               * Resources defined for PXBs are composed of the following parts:
>               * 1. The resources the pci-bridge/pcie-root-port need.
>               * 2. The resources the devices behind pxb need.
> +             *
> +             * For CXL host bridges on platforms where UEFI (driven by
> +             * FDT 'ranges') does not assign PCI resources for the CXL
> +             * root bridge before ACPI table construction, build_crs()
> +             * would return an empty resource set.  When the platform
> +             * has reserved a dedicated MMIO window for CXL host bridges
> +             * (cfg->cxl_mmio32), emit that window as a static _CRS
> +             * instead.  The platform is responsible for shrinking PCI0's
> +             * mmio32 window so the two do not overlap.
>               */
> -            crs = build_crs(PCI_HOST_BRIDGE(BUS(bus)->parent), &crs_range_set,
> -                            cfg->pio.base, 0, 0, 0);
> +            if (is_cxl && cfg->cxl_mmio32.size) {
> +                uint64_t cxl_base = cfg->cxl_mmio32.base;
> +                uint64_t cxl_size = cfg->cxl_mmio32.size;
> +
> +                crs = aml_resource_template();
> +
> +                /* 32-bit MMIO range for CXL devices */
> +                aml_append(crs,
> +                    aml_dword_memory(AML_POS_DECODE, AML_MIN_FIXED,
> +                                     AML_MAX_FIXED, AML_NON_CACHEABLE,
> +                                     AML_READ_WRITE, 0,
> +                                     cxl_base,
> +                                     cxl_base + cxl_size - 1,
> +                                     0, cxl_size));
> +
> +                /* Bus number range */
> +                aml_append(crs,
> +                    aml_word_bus_number(AML_MIN_FIXED, AML_MAX_FIXED,
> +                                       AML_POS_DECODE, 0,
> +                                       bus_num, bus_num + 15,
> +                                       0, 16));
> +            } else {
> +                crs = build_crs(PCI_HOST_BRIDGE(BUS(bus)->parent),
> +                                &crs_range_set, cfg->pio.base, 0, 0, 0);
> +            }
>              aml_append(dev, aml_name_decl("_CRS", crs));
>  
>              if (is_cxl) {
> diff --git a/hw/riscv/virt.c b/hw/riscv/virt.c
> index 899f632de7..929c01fb26 100644
> --- a/hw/riscv/virt.c
> +++ b/hw/riscv/virt.c
> @@ -113,6 +113,9 @@ static const MemMapEntry virt_memmap[] = {
>  /* PCIe high mmio for RV64, size is fixed but base depends on top of RAM */
>  #define VIRT64_HIGH_PCIE_MMIO_SIZE  (16 * GiB)
>  
> +/* 32-bit MMIO range carved out of VIRT_PCIE_MMIO for CXL host bridges */
> +#define VIRT_CXL_MMIO32_SIZE        (256 * MiB)
> +
>  static MemMapEntry virt_high_pcie_memmap;
>  
>  #define VIRT_FLASH_SECTOR_SIZE (256 * KiB)
> @@ -890,15 +893,28 @@ static void create_fdt_pcie(RISCVVirtState *s,
>      }
>      qemu_fdt_setprop_sized_cells(ms->fdt, name, "reg", 2,
>          s->memmap[VIRT_PCIE_ECAM].base, 2, s->memmap[VIRT_PCIE_ECAM].size);
> -    qemu_fdt_setprop_sized_cells(ms->fdt, name, "ranges",
> -        1, FDT_PCI_RANGE_IOPORT, 2, 0,
> -        2, s->memmap[VIRT_PCIE_PIO].base, 2, s->memmap[VIRT_PCIE_PIO].size,
> -        1, FDT_PCI_RANGE_MMIO,
> -        2, s->memmap[VIRT_PCIE_MMIO].base,
> -        2, s->memmap[VIRT_PCIE_MMIO].base, 2, s->memmap[VIRT_PCIE_MMIO].size,
> -        1, FDT_PCI_RANGE_MMIO_64BIT,
> -        2, virt_high_pcie_memmap.base,
> -        2, virt_high_pcie_memmap.base, 2, virt_high_pcie_memmap.size);
> +    {
> +        /*
> +         * When CXL is enabled, reserve the last 256 MiB of the 32-bit
> +         * MMIO window for CXL host bridges and exclude it from the main
> +         * PCIe host bridge's FDT 'ranges' so UEFI's PciHostBridgeDxe
> +         * does not allocate that range to PCI0.  The CXL host bridge
> +         * _CRS declares this range independently.
> +         */
> +        hwaddr mmio32_size = s->memmap[VIRT_PCIE_MMIO].size;
> +        if (s->cxl_devices_state.is_enabled) {
> +            mmio32_size -= VIRT_CXL_MMIO32_SIZE;
> +        }
> +        qemu_fdt_setprop_sized_cells(ms->fdt, name, "ranges",
> +            1, FDT_PCI_RANGE_IOPORT, 2, 0,
> +            2, s->memmap[VIRT_PCIE_PIO].base, 2, s->memmap[VIRT_PCIE_PIO].size,
> +            1, FDT_PCI_RANGE_MMIO,
> +            2, s->memmap[VIRT_PCIE_MMIO].base,
> +            2, s->memmap[VIRT_PCIE_MMIO].base, 2, mmio32_size,
> +            1, FDT_PCI_RANGE_MMIO_64BIT,
> +            2, virt_high_pcie_memmap.base,
> +            2, virt_high_pcie_memmap.base, 2, virt_high_pcie_memmap.size);
> +    }
>  
>      if (virt_is_iommu_sys_enabled(s)) {
>          qemu_fdt_setprop_cells(ms->fdt, name, "iommu-map",
> @@ -1730,7 +1746,29 @@ static void virt_machine_init(MachineState *machine)
>              qdev_get_gpio_in(virtio_irqchip, VIRTIO_IRQ + i));
>      }
>  
> -    gpex_pcie_init(system_memory, pcie_irqchip, s);
> +    DeviceState *pcie_dev = gpex_pcie_init(system_memory, pcie_irqchip, s);
> +
> +    /*
> +     * If CXL is enabled, reserve the last 256 MiB of the 32-bit MMIO
> +     * window for CXL host bridges so the bridge non-prefetchable window
> +     * can hold CXL device BARs (component registers and similar 64-bit
> +     * non-prefetchable BARs that need a < 4 GiB address).
> +     *
> +     * - Shrink PCI0's mmio32 advertised in the ACPI _CRS by the same
> +     *   256 MiB so the two ranges do not overlap (the FDT 'ranges'
> +     *   shrink happens in create_fdt_pcie()).
> +     * - Store the reserved range in cxl_mmio32 so gpex-acpi.c can emit
> +     *   a correct _CRS for the CXL host bridge (ACPI0016).
> +     */
> +    if (s->cxl_devices_state.is_enabled) {
> +        GPEXHost *gpex = GPEX_HOST(pcie_dev);
> +        gpex->gpex_cfg.cxl_mmio32.size = VIRT_CXL_MMIO32_SIZE;
> +        gpex->gpex_cfg.cxl_mmio32.base =
> +            s->memmap[VIRT_PCIE_MMIO].base +
> +            s->memmap[VIRT_PCIE_MMIO].size - VIRT_CXL_MMIO32_SIZE;
> +        /* Shrink PCI0's advertised 32-bit MMIO window to exclude CXL range */
> +        gpex->gpex_cfg.mmio32.size -= VIRT_CXL_MMIO32_SIZE;
> +    }
>  
>      create_platform_bus(s, mmio_irqchip);
>  
> diff --git a/include/hw/pci-host/gpex.h b/include/hw/pci-host/gpex.h
> index 1da9c85bce..d38fbbacd6 100644
> --- a/include/hw/pci-host/gpex.h
> +++ b/include/hw/pci-host/gpex.h
> @@ -43,6 +43,7 @@ struct GPEXConfig {
>      MemMapEntry mmio32;
>      MemMapEntry mmio64;
>      MemMapEntry pio;
> +    MemMapEntry cxl_mmio32;
>      int         irq;
>      PCIBus      *bus;
>      bool        pci_native_hotplug;

Re: [PATCH 3/4] hw/riscv/virt, gpex: Provide 32-bit MMIO window for CXL host bridges

[Chen Pei]

On Tue, 9 Jun 2026 13:56:11 +0100, Jonathan Cameron <jic23@kernel.org> wrote:

Hi Jonathan,

> > CXL component register BAR (BAR0 on CXL Root Port and Type3 device)
> > and the CXL device register BAR (BAR2 on Type3 device) are declared
> > as 64-bit non-prefetchable memory.  A standard PCIe-to-PCI bridge
> > exposes a 32-bit non-prefetchable memory window plus an (optional)
> > 64-bit prefetchable memory window, but no 64-bit non-prefetchable
> > window.
> 
> When you say 'standard' do you mean that is all the PCI spec allows
> for?

Yes, I confirmed against the PCI-to-PCI Bridge Architecture
Specification, Revision 1.2 (PCI-SIG, 2003). Relevant sections:

  - Section 3.2.5.8 (Memory Base Register and Memory Limit Register):
      "The upper 12 bits of both the Memory Base and Memory Limit
       registers are read/write and correspond to the upper 12 address
       bits, AD[31::20], of 32-bit addresses."
    -> The non-prefetchable memory window is 32-bit only, and the
       Type 1 header defines no upper-32-bit extension for it.

  - Section 3.2.5.9 (Prefetchable Memory Base/Limit Register):
      The bottom 4 bits encode 64-bit support: 0h = 32-bit only,
      01h = 64-bit (extended via 3.2.5.10).

  - Section 3.2.5.10 (Prefetchable Base/Limit Upper 32 Bits):
      Optional registers that hold AD[63::32] for the 64-bit
      *prefetchable* range.

So the architecture only allows a 64-bit window when it is also
prefetchable; there is no 64-bit non-prefetchable form. PCIe
inherits this Type 1 header layout unchanged.

Public mirror of the spec text (Rev 1.1, identical wording for these
sections):
  https://0x04.net/~mwk/doc/pci/PCI-to-PCI%20Bridge%20Architecture%20Specification.pdf

Official Rev 1.2 page (PCI-SIG, member access):
  https://pcisig.com/specifications/conventional/pci_bridge_2_1

> >  Linux therefore places 64-bit non-prefetchable BARs in the
> > 32-bit non-prefetchable bridge window, which requires the bridge to
> > own enough address space below 4 GiB.
> > 
> 
> This sounds a bit like the issue that Dave Jiang reported with recent
> EDK2 on x86 (fedora upgraded).  We don't see it with the older EDK2
> that ships with QEMU. I haven't yet figured out exactly why.
> Arguably whatever they changed is a regression but we don't have good
> enough testing in place to have detected it early enough.
> 
> I'd like some input from PCI / ACPI experts on this. +CC Michael and
> Igor.
> 
> Like the previous patch we'd definitely want some testing around this
> to make sure it doesn't accidentally get broken in future.

The symptoms do look related -- in both cases UEFI's
PciHostBridgeDxe ends up not giving a 32-bit non-prefetchable
window to the CXL host bridge (ACPI0016), so build_crs() in
gpex-acpi.c returns an empty resource set. I haven't dug deep
enough into Dave's case to say whether the underlying trigger is
the same as on RISC-V virt though.

Would appreciate input from Michael / Igor / Dave on whether the
approach in this patch (carve out a dedicated cxl_mmio32 range and
emit a static _CRS for ACPI0016) is a reasonable shape for a
shared fix, or if there's a preferred direction on the PCI/ACPI
side.

Best,
Pei

[PATCH 0/4] hw/riscv/virt: Add CXL support to the RISC-V virt machine

[Chen Pei]

This series adds CXL support to the RISC-V virt machine, following the
approach used by the ARM virt machine: CXL host bridges (pxb-cxl) are
described as ACPI0016 devices, an ACPI0017 (CXLM) device is added to the
DSDT, and a CEDT table is built.  A bios-tables test is added to pin down
the generated ACPI tables.

Prerequisite
------------
The v2 posting depended on Alireza Sanaee's v8 series [1] (the
performant non-interleaved CFMW lookup path).  That series has since
been merged upstream, so this version has no outstanding prerequisite
and applies directly on current upstream master.

Changes since v2
----------------
  - hw/riscv/virt: The MMIO-window patch no longer touches the common
    gpex code (Igor).  The gpex_cfg.cxl_mmio32 field and the is_cxl
    static _CRS branch are dropped; the ACPI0016 _CRS now comes from the
    generic build_crs() path.  Since EDK2 does not enumerate the pxb-cxl
    expander bridge, riscv simulates the firmware PCI initialization
    (reserved window + depth-first bridge bus numbers) and re-applies it
    via a reset handler.
  - hw/riscv/virt: Drops the machine-global window/bus-range synthesis
    that did not scale past one pxb-cxl (Junjie).  The series targets a
    single CXL host bridge for now, documented in the commit message and
    a TODO.
  - hw/riscv/virt: CXL host register region and FMW setup folded into a
    single cxl_host_state_init() helper; redundant braces removed in
    create_fdt_pcie() (Daniel).
  - hw/riscv/virt-acpi-build: the _DEP commit message documents the
    kernel-compatibility behaviour (Alistair).
  - Carried review tags: Sunil V L's Reviewed-by on the _DEP patch and
    Alistair Francis's Acked-by on the test.

Changes since v1
---------------
  - hw/riscv/virt: PCIBus *bus renamed to PCIBus *pci_bus (Jonathan).
  - hw/riscv/virt: Dropped outer if (s->pci_bus) guard around
    cxl_hook_up_pxb_registers(); the function already handles a NULL
    bus internally (Jonathan).
  - hw/riscv/virt-acpi-build: All s->bus references updated to
    s->pci_bus; iasl -d decompiled DSDT fragment added to commit
    message.
  - hw/riscv/virt,gpex: Commit message expanded with PCI-to-PCI Bridge
    Spec §3.2.5.8/9/10 citations (Jonathan).
  - Original patch 4 ("Map committed HDM decoder ranges as RAM for
    direct DMA") dropped; superseded by Alireza Sanaee's v8 series [1],
    which is now merged upstream and thus no longer a prerequisite.
  - New patch 4: RISC-V ACPI bios-tables test for CXL, with golden AML
    files generated and included.

[1] https://lore.kernel.org/qemu-devel/20260318171918.146-1-alireza.sanaee@huawei.com/

Test
----
Built riscv64-softmmu and ran the new bios-tables test together with the
existing riscv64 ACPI tests; all pass.  Also booted an RVA22 kernel with
pxb-cxl + cxl-rp + cxl-type3 + a CFMW: the root port and Type3 device
enumerate (0000:0c:00.0 / 0000:0d:00.0) and 'cxl list' reports the
memdev (4 GiB) and the CFMWS root decoder.

QEMU invocation (CXL-relevant options shown):

    qemu-system-riscv64 \
        -M virt,aia=aplic-imsic,acpi=on,cxl=on \
        -cpu rv64 -smp 2 -m 4G,maxmem=8G,slots=8 \
        -object memory-backend-ram,id=vmem0,share=on,size=4G \
        -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
        -device cxl-rp,port=0,bus=cxl.1,id=rp0,chassis=0,slot=2 \
        -device cxl-type3,bus=rp0,volatile-memdev=vmem0,id=cxl-vmem0 \
        -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G \
        ...

Verification (total system memory grows by 4 GiB after onlining):

    # cxl list
    # cxl enable-memdev mem0
    # cxl create-region -m -t ram -d decoder0.0 -w 1 mem0 -s 4G
    # daxctl online-memory dax0.0
    # free -h

Chen Pei (4):
  hw/riscv/virt: Add CXL support to the RISC-V virt machine
  hw/riscv/virt-acpi-build: Add _DEP to ACPI0017 for CXL host bridge
    dependency
  hw/riscv/virt: Provide a 32-bit MMIO window for CXL host bridges
  tests/qtest: Add RISC-V ACPI bios tables test for CXL

 hw/riscv/Kconfig                      |   2 +
 hw/riscv/virt-acpi-build.c            |  52 +++++++
 hw/riscv/virt.c                       | 191 +++++++++++++++++++++++++-
 include/hw/riscv/virt.h               |   3 +
 tests/data/acpi/riscv64/virt/CEDT.cxl | Bin 0 -> 108 bytes
 tests/data/acpi/riscv64/virt/DSDT.cxl | Bin 0 -> 6212 bytes
 tests/qtest/bios-tables-test.c        |  54 ++++++++
 7 files changed, 301 insertions(+), 1 deletion(-)
 create mode 100644 tests/data/acpi/riscv64/virt/CEDT.cxl
 create mode 100644 tests/data/acpi/riscv64/virt/DSDT.cxl

-- 
2.50.1

[PATCH 2/4] hw/riscv/virt-acpi-build: Add _DEP to ACPI0017 for CXL host bridge dependency

[Chen Pei]

On RISC-V QEMU virt platform with CXL enabled, the probe ordering
of acpi_pci_root (ACPI0016) and cxl_acpi (ACPI0017) is not
guaranteed. If cxl_acpi probes before acpi_pci_root has attached
the CXL host bridges, the CXL port topology will be incomplete
because to_cxl_host_bridge() silently skips devices whose PCI root
is not yet ready.

Add a _DEP object to the ACPI0017 device in the DSDT, declaring
its dependency on the ACPI0016 CXL host bridge devices. This tells
the OS to defer ACPI0017 enumeration until all ACPI0016 devices
have been attached by acpi_pci_root.

This requires a corresponding kernel change to call
acpi_dev_clear_dependencies() in acpi_pci_root_add().

Note on kernel compatibility: this is a "new QEMU feature requires a
sufficiently new kernel" situation.  Kernels that process _DEP but
lack the acpi_dev_clear_dependencies() call will leave the CXLM device
deferred, so CXL memory is unusable there; kernels old enough to ignore
_DEP entirely behave exactly as before this change.  In neither case is
existing (non-CXL) PCIe affected.

The resulting DSDT fragment (iasl -d output) for a single CXL host
bridge at bus 0x01 looks like:

  Device (CXLM)
  {
      Name (_HID, "ACPI0017")
      Name (_DEP, Package (0x01)
      {
          \_SB.PC01
      })
      Method (_STA, 0, NotSerialized)
      {
          Return (0x0B)
      }
      ...
  }

Reviewed-by: Sunil V L <sunilvl@oss.qualcomm.com>
Signed-off-by: Chen Pei <cp0613@linux.alibaba.com>
---
 hw/riscv/virt-acpi-build.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/hw/riscv/virt-acpi-build.c b/hw/riscv/virt-acpi-build.c
index 3188cdc5bf..56292f04a7 100644
--- a/hw/riscv/virt-acpi-build.c
+++ b/hw/riscv/virt-acpi-build.c
@@ -521,6 +521,38 @@ static void build_dsdt(GArray *table_data,
     if (s->cxl_devices_state.is_enabled) {
         Aml *cxl_dev = aml_device("CXLM");
         aml_append(cxl_dev, aml_name_decl("_HID", aml_string("ACPI0017")));
+
+        /*
+         * Declare a _DEP on every ACPI0016 CXL host bridge so the OS
+         * defers ACPI0017 enumeration until acpi_pci_root has attached
+         * the CXL host bridges. Without this, cxl_acpi may probe before
+         * to_cxl_host_bridge() can resolve the PCI root and the CXL
+         * port topology comes up empty.
+         */
+        if (s->pci_bus) {
+            PCIBus *bus;
+            uint32_t num_cxl_hbs = 0;
+
+            QLIST_FOREACH(bus, &s->pci_bus->child, sibling) {
+                if (pci_bus_is_root(bus) && pci_bus_is_cxl(bus)) {
+                    num_cxl_hbs++;
+                }
+            }
+
+            if (num_cxl_hbs > 0) {
+                Aml *dep_pkg = aml_package(num_cxl_hbs);
+
+                QLIST_FOREACH(bus, &s->pci_bus->child, sibling) {
+                    if (pci_bus_is_root(bus) && pci_bus_is_cxl(bus)) {
+                        aml_append(dep_pkg,
+                                   aml_name("\\_SB.PC%.02X",
+                                            pci_bus_num(bus)));
+                    }
+                }
+                aml_append(cxl_dev, aml_name_decl("_DEP", dep_pkg));
+            }
+        }
+
         Aml *method = aml_method("_STA", 0, AML_NOTSERIALIZED);
         aml_append(method, aml_return(aml_int(0x0B)));
         aml_append(cxl_dev, method);
-- 
2.50.1