Re: CXL Namespaces of ACPI disappearing in Qemu demo

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Mon, 4 Sep 2023 18:27:10 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> 
Hi Yuquan

Given this question isn't just ARM specific included qemu-devel in the cc list as
that gets much wider reading than qemu-arm.

> Due to my poor experience and knowledge on cxl development history, I am sorry to continue
> to ask some very simple and fundamental questions  : (
> 
> In hw/arm/virt :
> [VIRT_CXL_HOST] =           { 0x0, 64 * KiB * 16 }, /* 16 UID */
> 
> It seems like the specific space for MMIO of cxl host bridges.  Why not just use the existing "VIRT_PCIE_MMIO" 
> space for them? 

At the system design level, MMIO space of Root complex register space via RCRB
does not map in a similar fashion to PCIE MMIO space (which is handled via
address decoding in the PCIE fabric). It is much more similar to MMIO for platform
devices - as such the implementation handles in like a platform device (well 16 of
them which seemed enough for any sane usecase).


Now the next bit I've only quickly explored so may have some errors!

For instance on a GPEX the main MMIO region is an alias to the sysbus mmio region 1.
https://elixir.bootlin.com/qemu/latest/source/hw/arm/virt.c#L1452
That region is then mapped to a generic pci_root_bus for use by bar
mappings etc.

https://elixir.bootlin.com/qemu/latest/source/hw/pci-host/gpex.c#L136

So in theory we could make some space for the CXL root bridge RCRB registers
but it would make various generic paths more complex.  In a real system
those registers are likely to be far from the PCI MMIO space anyway so the
way it's modeled is probably more realistic than pushing the RCRB into the
existing allocation.

I hope that clarifies our reasoning for handling this MMIO region separately.

Jonathan

> 
> I would be grateful if you have some free time to help check this issue : )
> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

Hi, Jonathan
On 2023-09-04 20:43,  jonathan.cameron wrote:
> 
> At the system design level, MMIO space of Root complex register space via RCRB
> does not map in a similar fashion to PCIE MMIO space (which is handled via
> address decoding in the PCIE fabric). It is much more similar to MMIO for platform
> devices - as such the implementation handles in like a platform device (well 16 of
> them which seemed enough for any sane usecase).
> 
> 

Oh,thanks! According to above, therefore, the core factor is the implementation of RCRB.

> 
> So in theory we could make some space for the CXL root bridge RCRB registers
> but it would make various generic paths more complex.  In a real system
> those registers are likely to be far from the PCI MMIO space anyway so the
> way it's modeled is probably more realistic than pushing the RCRB into the
> existing allocation.
> 

Here implies that all CXL root bridge will use RCRB registers.

From Table 8-17 and Figure 9-14 in CXL3.0 specification, I understood that only RCH DP &
RCD UP will use RCRBs, and CXL host bridges VH mode will use other way to realize
the CHBCR. I had tried to find more explanation in CXL spec, but I haven't found. Hence 
this is why I am confused.

Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 2493 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Tue, 5 Sep 2023 18:45:02 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> On 2023-09-04 20:43,  jonathan.cameron wrote:
> > 
> > At the system design level, MMIO space of Root complex register space via RCRB
> > does not map in a similar fashion to PCIE MMIO space (which is handled via
> > address decoding in the PCIE fabric). It is much more similar to MMIO for platform
> > devices - as such the implementation handles in like a platform device (well 16 of
> > them which seemed enough for any sane usecase).
> > 
> >   
> 
> Oh,thanks! According to above, therefore, the core factor is the implementation of RCRB.
> 
> > 
> > So in theory we could make some space for the CXL root bridge RCRB registers
> > but it would make various generic paths more complex.  In a real system
> > those registers are likely to be far from the PCI MMIO space anyway so the
> > way it's modeled is probably more realistic than pushing the RCRB into the
> > existing allocation.
> >   
> 
> Here implies that all CXL root bridge will use RCRB registers.
> 
> From Table 8-17 and Figure 9-14 in CXL3.0 specification, I understood that only RCH DP &
> RCD UP will use RCRBs, and CXL host bridges VH mode will use other way to realize
> the CHBCR. I had tried to find more explanation in CXL spec, but I haven't found. Hence 
> this is why I am confused.

Ah. That distinction is a bit messy.  Both an RCRB and CHBCR (CXL Host Bridge Component
Registers) are similar in the sense that they are mapped in host memory space. 
As I understand it the distinction is more about the format / contents of that memory
than how you access them. As an aside, they are described by a static ACPI table,
so they can't be in the MMIO space used for BARs etc.

Jonathan


> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

Hi, Jonathan
On 2023-09-05 22:34,  jonathan.cameron wrote:
> 
> As I understand it the distinction is more about the format / contents of that memory
> than how you access them. 

Yes, RCH DP RCRB includes registers from PCIe Type 1 Config Header and
PCIe capabilities and extended capabilities while CHBCR includes component registers 
with the same layout and discovery mechanism in other CXL components.

> As an aside, they are described by a static ACPI table,
> so they can't be in the MMIO space used for BARs etc.
> 

In CXL 3.0 Spec, the Figure 9-14 (CXL Link/Protocol Register Mapping in a CXL VH)
shows that CHBCR is mapped by "Host Proprietary Static Bar". According to your guidance,
it is not a standard PCIe BAR using PCIe MMIO Space, so I understand it is a special BAR for 
MMIO of a platform device?


Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 2501 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Wed, 6 Sep 2023 19:22:19 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> On 2023-09-05 22:34,  jonathan.cameron wrote:
> > 
> > As I understand it the distinction is more about the format / contents of that memory
> > than how you access them.   
> 
> Yes, RCH DP RCRB includes registers from PCIe Type 1 Config Header and
> PCIe capabilities and extended capabilities while CHBCR includes component registers 
> with the same layout and discovery mechanism in other CXL components.
> 
> > As an aside, they are described by a static ACPI table,
> > so they can't be in the MMIO space used for BARs etc.
> >   
> 
> In CXL 3.0 Spec, the Figure 9-14 (CXL Link/Protocol Register Mapping in a CXL VH)
> shows that CHBCR is mapped by "Host Proprietary Static Bar". According to your guidance,
> it is not a standard PCIe BAR using PCIe MMIO Space, so I understand it is a special BAR for 
> MMIO of a platform device?

Hmm. Host proprietary so basically you can in theory do anything you like.

In QEMU emulation at least it's not in the PCIe MMIO space.  I'd not rule out other
implementations putting it somewhere in that space.  For now I'm not seeing
a) Anything that says our choice is invalid.
b) Any advantage in making the handling of PCIe MMIO space more complex to shoe
   horn this in there.

Jonathan
> 
> 
> Many thanks
> Yuquan

A confusion about CXL in arm virt machine

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

Hi, Gregory

There is one confusion about CXL in QEMU I hope to consult. 
If you have some time to look at this email, I would have better understanding of CXL 
emulation in QEMU.

On docs/system/devices/cxl.rst ,  Gregory wrote:
A very simple setup with just one directly attached CXL Type 3 Volatile Memory device::
qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \
......

As the current master branch of QEMU has not yet complemented the CXL option/expansion 
in arm virt machine, how this example command lines worked? Or here used another branch 
rather than master?

Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 2769 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Fri, 11 Aug 2023 18:31:28 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, 
> On 2023-08-10 21:56,  jonathan.cameron wrote:
>  
> So took a look at your issue - be it on the cxl-2023-08-07 branch rebased on qemu/master
> from today (side effect of looking at the segfault that was stopping me getting to this).
>  
> For me at least the branch does create an ACPI0017 DSDT entry and an ACPI0016 one
> and all the CXL devices turn up in /sys/bus/cxl/devices as expected.
>  
> 
> Oh, thanks for your guidance. It works so now I can get  ACPI0017 & ACPI0016 information in DSDT.   : )
> 
> By the way,  I found that if we add a pcie root port which create the same bus number as we assigned to pxb-cxl, 
> the enumeration of cxl and pcie would be different from what we expected.  In this case,  we cannot find 
> CXL devices in /sys/bus/cxl/devices. 

So this seems to be a case of shooting ourselves in the foot, but not catching the nonsensical configuration
(as you observer later! :)
pxb-pcie complains if you try and add two at the same bus number, but that doesn't protect against overlapping
ranges because they aren't known until after enumeration (which is done by the bios - and I guess the bios
doesn't sanity check for this insanity).  Qemu could take another look when it builds the ACPI tables a
second time though.

Looking at edk2 logs I can see it is happily populating the root bus 1 on my arm64 setup and that it
observes there are no subordinate buses available for the main PCIe bus (0) that QEMU is creating by
default. The _CRS entries look correct but the kernel ignores them it seems. 

It is very much not a valid configuration so there is no reason the kernel should cope with it.

Maybe it's worth considering some hardening code?


> 
> According to my test, the error happened in 
> "devm_cxl_register_pci_bus()"  of  "add_host_bridge_uport" in  "cxl_acpi_probe".
> Actually,  in above case, the incorrect enumeration of pcie will also occur with pxb-pcie except for pxb-cxl, 
> hence I guess the kernel did not deal with such case and users just need to avoid it if they need a correct
> enumeration result.

Agreed - Protecting against ever corner case of impossible configuration is tricky to
do.

> 
> My qemu script (which will cause the incorrect enumeration):
> qemu-system-x86_64 \
> -M q35,nvdimm=on,cxl=on \
> -m 4G,maxmem=8G,slots=8 \
> -smp 1 \
> -object memory-backend-file,id=cxl-mem1,share=on,mem-path=./memfile/cxltest3.raw,size=256M \
> -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=./memfile/lsa3.raw,size=256M \
> -device ioh3420,bus=pcie.0,id=root_port1,chassis=0,slot=1,addr=04 \
> -device qemu-xhci,bus=root_port1 \
> -device pxb-cxl,bus_nr=1,bus=pcie.0,id=cxl.1 \
> -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
> -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
> -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G \
> ......
> 
> Many thanks
> Yuquan