CXL Namespaces of ACPI disappearing in Qemu demo

CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 1114 bytes --]

Hi, Jonathan

When I tested the CXL topology in qemu, I found the linux kernel could find 
and match the ACPI0016&ACPI0017 in relevant drivers, however, I could not 
find these information in dumped ACPI tables (like DSDT). 

Hence, I guess there is some problem in my method, maybe you have some 
suggestions if you have some time to look at this email.

My environment:
qemu: Jonathan_qemu/cxl-2023-08-07
Linux: v6.3.0
ACPI Disassembler: iasl/version 20230628

My qemu script is the same as the first example in qemu/docs/system/devices/cxl.rst, 
as below:
  qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \
  ...
  -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \
  -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \
  -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
  -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
  -device cxl-type3,bus=root_port13,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
  -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G


Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 2378 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Thu, 10 Aug 2023 17:30:43 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> 
> When I tested the CXL topology in qemu, I found the linux kernel could find 
> and match the ACPI0016&ACPI0017 in relevant drivers, however, I could not 
> find these information in dumped ACPI tables (like DSDT). 
> 
> Hence, I guess there is some problem in my method, maybe you have some 
> suggestions if you have some time to look at this email.
> 
> My environment:
> qemu: Jonathan_qemu/cxl-2023-08-07
> Linux: v6.3.0
> ACPI Disassembler: iasl/version 20230628
> 
> My qemu script is the same as the first example in qemu/docs/system/devices/cxl.rst, 
> as below:
>   qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \
>   ...
>   -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \
>   -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \
>   -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
>   -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
>   -device cxl-type3,bus=root_port13,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
>   -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G

Hi Yuquan,

Ah. I may have messed up dropping a patch set that was factoring out all the PXB stuff to SSDT
in my latest tree.

I dropped it because it was nasty to rebase and didn’t seem likely to go upstream particularly
quickly as we pushed the decision on it back behind a load of other PCI changes.

I tested the latest tree heavily on arm, but was lazy on the x86 front :(
as I forgot that that change might have broken things (and all the new stuff was architecture
agnostic).

Right now for x86 tests with my tree I'm getting a freeze on booting so some debug needed.
Even better, on upstream qemu with near vanilla kernel I get a segfault instead in pretty
much the same place.
This is going to be a 'fun' day I guess. 

Jonathan



> 
> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Thu, 10 Aug 2023 11:04:29 +0100
Jonathan Cameron <Jonathan.Cameron@Huawei.com> wrote:

> On Thu, 10 Aug 2023 17:30:43 +0800
> Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:
> 
> > Hi, Jonathan
> > 
> > When I tested the CXL topology in qemu, I found the linux kernel could find 
> > and match the ACPI0016&ACPI0017 in relevant drivers, however, I could not 
> > find these information in dumped ACPI tables (like DSDT). 
> > 
> > Hence, I guess there is some problem in my method, maybe you have some 
> > suggestions if you have some time to look at this email.
> > 
> > My environment:
> > qemu: Jonathan_qemu/cxl-2023-08-07
> > Linux: v6.3.0
> > ACPI Disassembler: iasl/version 20230628
> > 
> > My qemu script is the same as the first example innn qemu/docs/system/devices/cxl.rst, 
> > as below:
> >   qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \
> >   ...
> >   -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \
> >   -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \
> >   -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
> >   -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
> >   -device cxl-type3,bus=root_port13,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
> >   -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G  
> 
> Hi Yuquan,
> 
> Ah. I may have messed up dropping a patch set that was factoring out all the PXB stuff to SSDT
> in my latest tree.
> 
> I dropped it because it was nasty to rebase and didn’t seem likely to go upstream particularly
> quickly as we pushed the decision on it back behind a load of other PCI changes.
> 
> I tested the latest tree heavily on arm, but was lazy on the x86 front :(
> as I forgot that that change might have broken things (and all the new stuff was architecture
> agnostic).
> 
> Right now for x86 tests with my tree I'm getting a freeze on booting so some debug needed.
> Even better, on upstream qemu with near vanilla kernel I get a segfault instead in pretty
> much the same place.
> This is going to be a 'fun' day I guess. 

I haven't gotten very far with that other issue (beyond reporting where the segfault was)
but in the meantime I can boot machines as long as they only have one core.

So took a look at your issue - be it on the cxl-2023-08-07 branch rebased on qemu/master
from today (side effect of looking at the segfault that was stopping me getting to this).

For me at least the branch does create an ACPI0017 DSDT entry and an ACPI0016 one
and all the CXL devices turn up in /sys/bus/cxl/devices as expected.

So I'm afraid I have no idea what is going wrong for you :(  Maybe you are running
a BIOS that is overwriting them? - I'm just using the default from the same qemu build.

Jonathan


> 
> Jonathan
> 
> 
> 
> > 
> > 
> > Many thanks
> > Yuquan  
> 

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 1852 bytes --]

Hi, 
On 2023-08-10 21:56,  jonathan.cameron wrote:
 
So took a look at your issue - be it on the cxl-2023-08-07 branch rebased on qemu/master
from today (side effect of looking at the segfault that was stopping me getting to this).
 
For me at least the branch does create an ACPI0017 DSDT entry and an ACPI0016 one
and all the CXL devices turn up in /sys/bus/cxl/devices as expected.
 

Oh, thanks for your guidance. It works so now I can get  ACPI0017 & ACPI0016 information in DSDT.   : )

By the way,  I found that if we add a pcie root port which create the same bus number as we assigned to pxb-cxl, 
the enumeration of cxl and pcie would be different from what we expected.  In this case,  we cannot find 
CXL devices in /sys/bus/cxl/devices. 

According to my test, the error happened in 
"devm_cxl_register_pci_bus()"  of  "add_host_bridge_uport" in  "cxl_acpi_probe".
Actually,  in above case, the incorrect enumeration of pcie will also occur with pxb-pcie except for pxb-cxl, 
hence I guess the kernel did not deal with such case and users just need to avoid it if they need a correct
enumeration result.

My qemu script (which will cause the incorrect enumeration):
qemu-system-x86_64 \
-M q35,nvdimm=on,cxl=on \
-m 4G,maxmem=8G,slots=8 \
-smp 1 \
-object memory-backend-file,id=cxl-mem1,share=on,mem-path=./memfile/cxltest3.raw,size=256M \
-object memory-backend-file,id=cxl-lsa1,share=on,mem-path=./memfile/lsa3.raw,size=256M \
-device ioh3420,bus=pcie.0,id=root_port1,chassis=0,slot=1,addr=04 \
-device qemu-xhci,bus=root_port1 \
-device pxb-cxl,bus_nr=1,bus=pcie.0,id=cxl.1 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
-device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G \
......

Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 3704 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 823 bytes --]

Hi, Gregory
I am sorry to disturb you, but there was still an ignored problem about CXL, 
Although I had sent an email to jonathan (maybe he is busy recently so he forgot to reply),  
the Link is : https://lists.nongnu.org/archive/html/qemu-arm/2023-08/msg00278.html 

Maybe the core question is that how should we use CXL components with a standard PCIe system：
1) In Qemu, since the boards or machines in qemu can only use pxb-cxl (attached on pcie.0) to add
cxl host bridge, therefore, users should avoid to assign an occupied bus number by other devices
to pxb-cxl.

2) In real hardware, CXL components should use independent CXL root/tree (ACPI0017&ACPI0016)
to separate from the namespace of default pcie root/domain.

I would be grateful if you have some free time to help check this issue : )




[-- Attachment #2: Type: text/html, Size: 3050 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Fri, 11 Aug 2023 18:31:28 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, 
> On 2023-08-10 21:56,  jonathan.cameron wrote:
>  
> So took a look at your issue - be it on the cxl-2023-08-07 branch rebased on qemu/master
> from today (side effect of looking at the segfault that was stopping me getting to this).
>  
> For me at least the branch does create an ACPI0017 DSDT entry and an ACPI0016 one
> and all the CXL devices turn up in /sys/bus/cxl/devices as expected.
>  
> 
> Oh, thanks for your guidance. It works so now I can get  ACPI0017 & ACPI0016 information in DSDT.   : )
> 
> By the way,  I found that if we add a pcie root port which create the same bus number as we assigned to pxb-cxl, 
> the enumeration of cxl and pcie would be different from what we expected.  In this case,  we cannot find 
> CXL devices in /sys/bus/cxl/devices. 

So this seems to be a case of shooting ourselves in the foot, but not catching the nonsensical configuration
(as you observer later! :)
pxb-pcie complains if you try and add two at the same bus number, but that doesn't protect against overlapping
ranges because they aren't known until after enumeration (which is done by the bios - and I guess the bios
doesn't sanity check for this insanity).  Qemu could take another look when it builds the ACPI tables a
second time though.

Looking at edk2 logs I can see it is happily populating the root bus 1 on my arm64 setup and that it
observes there are no subordinate buses available for the main PCIe bus (0) that QEMU is creating by
default. The _CRS entries look correct but the kernel ignores them it seems. 

It is very much not a valid configuration so there is no reason the kernel should cope with it.

Maybe it's worth considering some hardening code?


> 
> According to my test, the error happened in 
> "devm_cxl_register_pci_bus()"  of  "add_host_bridge_uport" in  "cxl_acpi_probe".
> Actually,  in above case, the incorrect enumeration of pcie will also occur with pxb-pcie except for pxb-cxl, 
> hence I guess the kernel did not deal with such case and users just need to avoid it if they need a correct
> enumeration result.

Agreed - Protecting against ever corner case of impossible configuration is tricky to
do.

> 
> My qemu script (which will cause the incorrect enumeration):
> qemu-system-x86_64 \
> -M q35,nvdimm=on,cxl=on \
> -m 4G,maxmem=8G,slots=8 \
> -smp 1 \
> -object memory-backend-file,id=cxl-mem1,share=on,mem-path=./memfile/cxltest3.raw,size=256M \
> -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=./memfile/lsa3.raw,size=256M \
> -device ioh3420,bus=pcie.0,id=root_port1,chassis=0,slot=1,addr=04 \
> -device qemu-xhci,bus=root_port1 \
> -device pxb-cxl,bus_nr=1,bus=pcie.0,id=cxl.1 \
> -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
> -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
> -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G \
> ......
> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Fri, 11 Aug 2023 18:31:28 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, 
> On 2023-08-10 21:56,  jonathan.cameron wrote:
>  
> So took a look at your issue - be it on the cxl-2023-08-07 branch rebased on qemu/master
> from today (side effect of looking at the segfault that was stopping me getting to this).
>  
> For me at least the branch does create an ACPI0017 DSDT entry and an ACPI0016 one
> and all the CXL devices turn up in /sys/bus/cxl/devices as expected.
>  
> 
> Oh, thanks for your guidance. It works so now I can get  ACPI0017 & ACPI0016 information in DSDT.   : )
> 
> By the way,  I found that if we add a pcie root port which create the same bus number as we assigned to pxb-cxl, 
> the enumeration of cxl and pcie would be different from what we expected.  In this case,  we cannot find 
> CXL devices in /sys/bus/cxl/devices. 

So this seems to be a case of shooting ourselves in the foot, but not catching the nonsensical configuration
(as you observer later! :)
pxb-pcie complains if you try and add two at the same bus number, but that doesn't protect against overlapping
ranges because they aren't known until after enumeration (which is done by the bios - and I guess the bios
doesn't sanity check for this insanity).  Qemu could take another look when it builds the ACPI tables a
second time though.

Looking at edk2 logs I can see it is happily populating the root bus 1 on my arm64 setup and that it
observes there are no subordinate buses available for the main PCIe bus (0) that QEMU is creating by
default. The _CRS entries look correct but the kernel ignores them it seems. 

It is very much not a valid configuration so there is no reason the kernel should cope with it.

Maybe it's worth considering some hardening code?


> 
> According to my test, the error happened in 
> "devm_cxl_register_pci_bus()"  of  "add_host_bridge_uport" in  "cxl_acpi_probe".
> Actually,  in above case, the incorrect enumeration of pcie will also occur with pxb-pcie except for pxb-cxl, 
> hence I guess the kernel did not deal with such case and users just need to avoid it if they need a correct
> enumeration result.

Agreed - Protecting against ever corner case of impossible configuration is tricky to
do.

> 
> My qemu script (which will cause the incorrect enumeration):
> qemu-system-x86_64 \
> -M q35,nvdimm=on,cxl=on \
> -m 4G,maxmem=8G,slots=8 \
> -smp 1 \
> -object memory-backend-file,id=cxl-mem1,share=on,mem-path=./memfile/cxltest3.raw,size=256M \
> -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=./memfile/lsa3.raw,size=256M \
> -device ioh3420,bus=pcie.0,id=root_port1,chassis=0,slot=1,addr=04 \
> -device qemu-xhci,bus=root_port1 \
> -device pxb-cxl,bus_nr=1,bus=pcie.0,id=cxl.1 \
> -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
> -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
> -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G \
> ......
> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Gregory Price]

On Tue, Aug 22, 2023 at 03:22:02PM +0800, Yuquan Wang wrote:
> Hi, Gregory
> I am sorry to disturb you, but there was still an ignored problem about CXL, 
> Although I had sent an email to jonathan (maybe he is busy recently so he forgot to reply),  
> the Link is : https://lists.nongnu.org/archive/html/qemu-arm/2023-08/msg00278.html 
> 
> Maybe the core question is that how should we use CXL components with a standard PCIe system：
> 1) In Qemu, since the boards or machines in qemu can only use pxb-cxl (attached on pcie.0) to add
> cxl host bridge, therefore, users should avoid to assign an occupied bus number by other devices
> to pxb-cxl.
> 
> 2) In real hardware, CXL components should use independent CXL root/tree (ACPI0017&ACPI0016)
> to separate from the namespace of default pcie root/domain.
> 
> I would be grateful if you have some free time to help check this issue : )
> 

Generally speaking, ACPI questions are outside the scope of my
knowledge, but to the extent that i have debugged QEMU for CXL I can say
that #1 is correct - the host bridge adapter should not share PCI bus
id's with anything else or you will have issues.  This can be
frustrating if you don't know what the magic numbers are.

for my work, i have been using this when i need to attach two pxb-cxl
deivces

qemu-system-x86_64 \
-drive file=./cxl.qcow2,format=qcow2,index=0,media=disk,id=hd \
-m 4G,slots=4,maxmem=8G \
-smp 4 \
-machine type=q35,cxl=on,hmat=on \
-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=191 \


In regards to real hardware, I'm honestly not knowledgable enough about
ACPI or hardware to provide an intelligible answer unfortunately.

~Gregory

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 384 bytes --]

Hi, Gregory
On 2023-08-24 03:44,  Gregory wrote:

In regards to real hardware, I'm honestly not knowledgable enough about
ACPI or hardware to provide an intelligible answer unfortunately.

Never mind. 

Your precious guidance and replies along with Jonathan's are meaningful for me, because I was 
just touching CXL field in work as a fresh graduate.

Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 1205 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Wed, 23 Aug 2023 15:44:35 -0400
Gregory Price <gregory.price@memverge.com> wrote:

> On Tue, Aug 22, 2023 at 03:22:02PM +0800, Yuquan Wang wrote:
> > Hi, Gregory
> > I am sorry to disturb you, but there was still an ignored problem about CXL, 
> > Although I had sent an email to jonathan (maybe he is busy recently so he forgot to reply),  
> > the Link is : https://lists.nongnu.org/archive/html/qemu-arm/2023-08/msg00278.html 
> > 
> > Maybe the core question is that how should we use CXL components with a standard PCIe system：
> > 1) In Qemu, since the boards or machines in qemu can only use pxb-cxl (attached on pcie.0) to add
> > cxl host bridge, therefore, users should avoid to assign an occupied bus number by other devices
> > to pxb-cxl.
> > 
> > 2) In real hardware, CXL components should use independent CXL root/tree (ACPI0017&ACPI0016)
> > to separate from the namespace of default pcie root/domain.
> > 
> > I would be grateful if you have some free time to help check this issue : )
> >   
> 
> Generally speaking, ACPI questions are outside the scope of my
> knowledge, but to the extent that i have debugged QEMU for CXL I can say
> that #1 is correct - the host bridge adapter should not share PCI bus
> id's with anything else or you will have issues.  This can be
> frustrating if you don't know what the magic numbers are.
> 
> for my work, i have been using this when i need to attach two pxb-cxl
> deivces
> 
> qemu-system-x86_64 \
> -drive file=./cxl.qcow2,format=qcow2,index=0,media=disk,id=hd \
> -m 4G,slots=4,maxmem=8G \
> -smp 4 \
> -machine type=q35,cxl=on,hmat=on \
> -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \
> -device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=191 \
> 
> 
> In regards to real hardware, I'm honestly not knowledgable enough about
> ACPI or hardware to provide an intelligible answer unfortunately.
> 
> ~Gregory

One additional thing to note. Nothing stops you putting a PCI device on a CXL
root port, or downstream switch port.  Some of the reporting is probably wrong
at the moment, but I've messed around with NVME below one and that 'works' fine
as Linux doesn't read the relevant registers anyway.

Jonathan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

Hi, Jonathan

Due to my poor experience and knowledge on cxl development history, I am sorry to continue
to ask some very simple and fundamental questions  : (

In hw/arm/virt :
[VIRT_CXL_HOST] =           { 0x0, 64 * KiB * 16 }, /* 16 UID */

It seems like the specific space for MMIO of cxl host bridges.  Why not just use the existing "VIRT_PCIE_MMIO" 
space for them? 

I would be grateful if you have some free time to help check this issue : )

Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 1910 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Mon, 4 Sep 2023 18:27:10 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> 
Hi Yuquan

Given this question isn't just ARM specific included qemu-devel in the cc list as
that gets much wider reading than qemu-arm.

> Due to my poor experience and knowledge on cxl development history, I am sorry to continue
> to ask some very simple and fundamental questions  : (
> 
> In hw/arm/virt :
> [VIRT_CXL_HOST] =           { 0x0, 64 * KiB * 16 }, /* 16 UID */
> 
> It seems like the specific space for MMIO of cxl host bridges.  Why not just use the existing "VIRT_PCIE_MMIO" 
> space for them? 

At the system design level, MMIO space of Root complex register space via RCRB
does not map in a similar fashion to PCIE MMIO space (which is handled via
address decoding in the PCIE fabric). It is much more similar to MMIO for platform
devices - as such the implementation handles in like a platform device (well 16 of
them which seemed enough for any sane usecase).


Now the next bit I've only quickly explored so may have some errors!

For instance on a GPEX the main MMIO region is an alias to the sysbus mmio region 1.
https://elixir.bootlin.com/qemu/latest/source/hw/arm/virt.c#L1452
That region is then mapped to a generic pci_root_bus for use by bar
mappings etc.

https://elixir.bootlin.com/qemu/latest/source/hw/pci-host/gpex.c#L136

So in theory we could make some space for the CXL root bridge RCRB registers
but it would make various generic paths more complex.  In a real system
those registers are likely to be far from the PCI MMIO space anyway so the
way it's modeled is probably more realistic than pushing the RCRB into the
existing allocation.

I hope that clarifies our reasoning for handling this MMIO region separately.

Jonathan

> 
> I would be grateful if you have some free time to help check this issue : )
> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Mon, 4 Sep 2023 18:27:10 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> 
Hi Yuquan

Given this question isn't just ARM specific included qemu-devel in the cc list as
that gets much wider reading than qemu-arm.

> Due to my poor experience and knowledge on cxl development history, I am sorry to continue
> to ask some very simple and fundamental questions  : (
> 
> In hw/arm/virt :
> [VIRT_CXL_HOST] =           { 0x0, 64 * KiB * 16 }, /* 16 UID */
> 
> It seems like the specific space for MMIO of cxl host bridges.  Why not just use the existing "VIRT_PCIE_MMIO" 
> space for them? 

At the system design level, MMIO space of Root complex register space via RCRB
does not map in a similar fashion to PCIE MMIO space (which is handled via
address decoding in the PCIE fabric). It is much more similar to MMIO for platform
devices - as such the implementation handles in like a platform device (well 16 of
them which seemed enough for any sane usecase).


Now the next bit I've only quickly explored so may have some errors!

For instance on a GPEX the main MMIO region is an alias to the sysbus mmio region 1.
https://elixir.bootlin.com/qemu/latest/source/hw/arm/virt.c#L1452
That region is then mapped to a generic pci_root_bus for use by bar
mappings etc.

https://elixir.bootlin.com/qemu/latest/source/hw/pci-host/gpex.c#L136

So in theory we could make some space for the CXL root bridge RCRB registers
but it would make various generic paths more complex.  In a real system
those registers are likely to be far from the PCI MMIO space anyway so the
way it's modeled is probably more realistic than pushing the RCRB into the
existing allocation.

I hope that clarifies our reasoning for handling this MMIO region separately.

Jonathan

> 
> I would be grateful if you have some free time to help check this issue : )
> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

Hi, Jonathan
On 2023-09-04 20:43,  jonathan.cameron wrote:
> 
> At the system design level, MMIO space of Root complex register space via RCRB
> does not map in a similar fashion to PCIE MMIO space (which is handled via
> address decoding in the PCIE fabric). It is much more similar to MMIO for platform
> devices - as such the implementation handles in like a platform device (well 16 of
> them which seemed enough for any sane usecase).
> 
> 

Oh,thanks! According to above, therefore, the core factor is the implementation of RCRB.

> 
> So in theory we could make some space for the CXL root bridge RCRB registers
> but it would make various generic paths more complex.  In a real system
> those registers are likely to be far from the PCI MMIO space anyway so the
> way it's modeled is probably more realistic than pushing the RCRB into the
> existing allocation.
> 

Here implies that all CXL root bridge will use RCRB registers.

From Table 8-17 and Figure 9-14 in CXL3.0 specification, I understood that only RCH DP &
RCD UP will use RCRBs, and CXL host bridges VH mode will use other way to realize
the CHBCR. I had tried to find more explanation in CXL spec, but I haven't found. Hence 
this is why I am confused.

Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 2493 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Tue, 5 Sep 2023 18:45:02 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> On 2023-09-04 20:43,  jonathan.cameron wrote:
> > 
> > At the system design level, MMIO space of Root complex register space via RCRB
> > does not map in a similar fashion to PCIE MMIO space (which is handled via
> > address decoding in the PCIE fabric). It is much more similar to MMIO for platform
> > devices - as such the implementation handles in like a platform device (well 16 of
> > them which seemed enough for any sane usecase).
> > 
> >   
> 
> Oh,thanks! According to above, therefore, the core factor is the implementation of RCRB.
> 
> > 
> > So in theory we could make some space for the CXL root bridge RCRB registers
> > but it would make various generic paths more complex.  In a real system
> > those registers are likely to be far from the PCI MMIO space anyway so the
> > way it's modeled is probably more realistic than pushing the RCRB into the
> > existing allocation.
> >   
> 
> Here implies that all CXL root bridge will use RCRB registers.
> 
> From Table 8-17 and Figure 9-14 in CXL3.0 specification, I understood that only RCH DP &
> RCD UP will use RCRBs, and CXL host bridges VH mode will use other way to realize
> the CHBCR. I had tried to find more explanation in CXL spec, but I haven't found. Hence 
> this is why I am confused.

Ah. That distinction is a bit messy.  Both an RCRB and CHBCR (CXL Host Bridge Component
Registers) are similar in the sense that they are mapped in host memory space. 
As I understand it the distinction is more about the format / contents of that memory
than how you access them. As an aside, they are described by a static ACPI table,
so they can't be in the MMIO space used for BARs etc.

Jonathan


> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Tue, 5 Sep 2023 18:45:02 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> On 2023-09-04 20:43,  jonathan.cameron wrote:
> > 
> > At the system design level, MMIO space of Root complex register space via RCRB
> > does not map in a similar fashion to PCIE MMIO space (which is handled via
> > address decoding in the PCIE fabric). It is much more similar to MMIO for platform
> > devices - as such the implementation handles in like a platform device (well 16 of
> > them which seemed enough for any sane usecase).
> > 
> >   
> 
> Oh,thanks! According to above, therefore, the core factor is the implementation of RCRB.
> 
> > 
> > So in theory we could make some space for the CXL root bridge RCRB registers
> > but it would make various generic paths more complex.  In a real system
> > those registers are likely to be far from the PCI MMIO space anyway so the
> > way it's modeled is probably more realistic than pushing the RCRB into the
> > existing allocation.
> >   
> 
> Here implies that all CXL root bridge will use RCRB registers.
> 
> From Table 8-17 and Figure 9-14 in CXL3.0 specification, I understood that only RCH DP &
> RCD UP will use RCRBs, and CXL host bridges VH mode will use other way to realize
> the CHBCR. I had tried to find more explanation in CXL spec, but I haven't found. Hence 
> this is why I am confused.

Ah. That distinction is a bit messy.  Both an RCRB and CHBCR (CXL Host Bridge Component
Registers) are similar in the sense that they are mapped in host memory space. 
As I understand it the distinction is more about the format / contents of that memory
than how you access them. As an aside, they are described by a static ACPI table,
so they can't be in the MMIO space used for BARs etc.

Jonathan


> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

Hi, Jonathan
On 2023-09-05 22:34,  jonathan.cameron wrote:
> 
> As I understand it the distinction is more about the format / contents of that memory
> than how you access them. 

Yes, RCH DP RCRB includes registers from PCIe Type 1 Config Header and
PCIe capabilities and extended capabilities while CHBCR includes component registers 
with the same layout and discovery mechanism in other CXL components.

> As an aside, they are described by a static ACPI table,
> so they can't be in the MMIO space used for BARs etc.
> 

In CXL 3.0 Spec, the Figure 9-14 (CXL Link/Protocol Register Mapping in a CXL VH)
shows that CHBCR is mapped by "Host Proprietary Static Bar". According to your guidance,
it is not a standard PCIe BAR using PCIe MMIO Space, so I understand it is a special BAR for 
MMIO of a platform device?


Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 2501 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Wed, 6 Sep 2023 19:22:19 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> On 2023-09-05 22:34,  jonathan.cameron wrote:
> > 
> > As I understand it the distinction is more about the format / contents of that memory
> > than how you access them.   
> 
> Yes, RCH DP RCRB includes registers from PCIe Type 1 Config Header and
> PCIe capabilities and extended capabilities while CHBCR includes component registers 
> with the same layout and discovery mechanism in other CXL components.
> 
> > As an aside, they are described by a static ACPI table,
> > so they can't be in the MMIO space used for BARs etc.
> >   
> 
> In CXL 3.0 Spec, the Figure 9-14 (CXL Link/Protocol Register Mapping in a CXL VH)
> shows that CHBCR is mapped by "Host Proprietary Static Bar". According to your guidance,
> it is not a standard PCIe BAR using PCIe MMIO Space, so I understand it is a special BAR for 
> MMIO of a platform device?

Hmm. Host proprietary so basically you can in theory do anything you like.

In QEMU emulation at least it's not in the PCIe MMIO space.  I'd not rule out other
implementations putting it somewhere in that space.  For now I'm not seeing
a) Anything that says our choice is invalid.
b) Any advantage in making the handling of PCIe MMIO space more complex to shoe
   horn this in there.

Jonathan
> 
> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Wed, 6 Sep 2023 19:22:19 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> On 2023-09-05 22:34,  jonathan.cameron wrote:
> > 
> > As I understand it the distinction is more about the format / contents of that memory
> > than how you access them.   
> 
> Yes, RCH DP RCRB includes registers from PCIe Type 1 Config Header and
> PCIe capabilities and extended capabilities while CHBCR includes component registers 
> with the same layout and discovery mechanism in other CXL components.
> 
> > As an aside, they are described by a static ACPI table,
> > so they can't be in the MMIO space used for BARs etc.
> >   
> 
> In CXL 3.0 Spec, the Figure 9-14 (CXL Link/Protocol Register Mapping in a CXL VH)
> shows that CHBCR is mapped by "Host Proprietary Static Bar". According to your guidance,
> it is not a standard PCIe BAR using PCIe MMIO Space, so I understand it is a special BAR for 
> MMIO of a platform device?

Hmm. Host proprietary so basically you can in theory do anything you like.

In QEMU emulation at least it's not in the PCIe MMIO space.  I'd not rule out other
implementations putting it somewhere in that space.  For now I'm not seeing
a) Anything that says our choice is invalid.
b) Any advantage in making the handling of PCIe MMIO space more complex to shoe
   horn this in there.

Jonathan
> 
> 
> Many thanks
> Yuquan

Re: A confusion about cxl.mem in CXL drivers

[Jonathan Cameron]

On Wed, 20 Sep 2023 18:24:46 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> 
> There jumped a silly confusion when I am analyzing cxl drivers: 
> Since Host to access the cxl device memory and to access its mmio registers
> both use load/store instructions, how system software distinguish 
> with cxl.io and cxl.mem protocols?
> 
> Many thanks
> Yuquan

Hi Yuquan

I'm afraid I don't really understand the question.  So I'm guessing a bit whilst
trying to answer.

Is it about the kernel side of things, or more on what we are doing in QEMU
where we use iomem regions for both CXL.io and CXL.mem? For CXL.mem that
is done in QEMU to give us the ability to do fine grained address decoding
(below page level) but it doesn't in practice matter to the OS on top. It's
just an implementation detail in QEMU. There are knock on effects if you run with
KVM though as instructions can end up being read from the memory
(so that's not advised!).
However for both CXL.io and CXL.mem it is a host physical address range
and how the OS deals with it is dependent on how it is mapped.

So from driver side of things, the CXL.IO stuff is either in ECAM (for config
space) or mapped as PCIe BARs.  The CXL.mem stuff is mapped via the Host Physical
Addresses described in a CXL Fixed Memory Window.

So the right type of access is used based on the underlying hardware performing
the routing for the appropriate Host Physical Address range.  Same applies
on top of QEMU.

Jonathan

Re: A confusion about cxl.mem in CXL drivers

[Yuquan Wang]

On 2023-09-20 20:19,  jonathan.cameron wrote:

Thanks for your patient explanation.

>
> So from driver side of things, the CXL.IO stuff is either in ECAM (for config
> space) or mapped as PCIe BARs.  The CXL.mem stuff is mapped via the Host Physical
> Addresses described in a CXL Fixed Memory Window.
> 
> So the right type of access is used based on the underlying hardware performing
> the routing for the appropriate Host Physical Address range.  Same applies
> on top of QEMU.
> 

Therefore, from the view of kernel side, the kernel do not need to distinguish the physical cxl subprotocol
(cxl.io,cxl.cache,cxl.mem). In fact, the underlying hardware would directly finish this work so system software
don't care about it.

According to the instance of qemu virt machine, my understanding is below:
1) CXL.IO: finding, setting and enumerating CXL ECAM/BARs (programmed in cxl_acpi, cxl_pci drivers)
    Underlying hardware performing the routing : PCIe RC
    
2) CXL.MEM: host is going to access the memory mapped in CFMWs 
    Underlying hardware performing the routing : HDM decoders

Many thanks
Yuquan