A confusion about CXL in arm virt machine

A confusion about CXL in arm virt machine

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

Hi, Gregory

There is one confusion about CXL in QEMU I hope to consult. 
If you have some time to look at this email, I would have better understanding of CXL 
emulation in QEMU.

On docs/system/devices/cxl.rst ,  Gregory wrote:
A very simple setup with just one directly attached CXL Type 3 Volatile Memory device::
qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \
......

As the current master branch of QEMU has not yet complemented the CXL option/expansion 
in arm virt machine, how this example command lines worked? Or here used another branch 
rather than master?

Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 2769 bytes --]

Re: A confusion about CXL in arm virt machine

[Gregory Price]

On Fri, Jun 16, 2023 at 03:43:31PM +0800, Yuquan Wang wrote:
> Hi, Gregory
> 
> There is one confusion about CXL in QEMU I hope to consult. 
> If you have some time to look at this email, I would have better understanding of CXL 
> emulation in QEMU.
> 
> On docs/system/devices/cxl.rst ,  Gregory wrote:
> A very simple setup with just one directly attached CXL Type 3 Volatile Memory device::
> qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \
> ......
> 
> As the current master branch of QEMU has not yet complemented the CXL option/expansion 
> in arm virt machine, how this example command lines worked? Or here used another branch 
> rather than master?
> 
> Many thanks
> Yuquan

As of today, the qemu/qemu.git master branch does have the required
patch for volatile region support: adacc814f541af9281c922e750d8ba4b90c1a73e

however, the last time i tested it on x86, the master branch was
incapable of enabling these regions with the latest kernel (6.3.x)
despite that kernel having sufficient support to do so.  I have not dug
into what the discrepency between master and johnathan's working branch
are just yet.

Last I tested cxl-2023-05-25 branch of Johnathan's fork is working on x86: 

https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25

I have not worked with the ARM machine, but Johnathan may be able to
comment on the state of ARM support for this code.

~Gregory

Re: A confusion about CXL in arm virt machine

[Jonathan Cameron via]

On Fri, 16 Jun 2023 14:10:24 -0400
Gregory Price <gregory.price@memverge.com> wrote:

> On Fri, Jun 16, 2023 at 03:43:31PM +0800, Yuquan Wang wrote:
> > Hi, Gregory
> > 
> > There is one confusion about CXL in QEMU I hope to consult. 
> > If you have some time to look at this email, I would have better understanding of CXL 
> > emulation in QEMU.
> > 
> > On docs/system/devices/cxl.rst ,  Gregory wrote:
> > A very simple setup with just one directly attached CXL Type 3 Volatile Memory device::
> > qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \
> > ......
> > 
> > As the current master branch of QEMU has not yet complemented the CXL option/expansion 
> > in arm virt machine, how this example command lines worked? Or here used another branch 
> > rather than master?
> > 
> > Many thanks
> > Yuquan  
> 
> As of today, the qemu/qemu.git master branch does have the required
> patch for volatile region support: adacc814f541af9281c922e750d8ba4b90c1a73e
> 
> however, the last time i tested it on x86, the master branch was
> incapable of enabling these regions with the latest kernel (6.3.x)
> despite that kernel having sufficient support to do so.  I have not dug
> into what the discrepency between master and johnathan's working branch
> are just yet.

Events support is missing so the upstream kernel drivers won't probe
successfully.  That's queued up for merge but hasn't happened quite yet.
*fingers crossed* it should go in soon.

> 
> Last I tested cxl-2023-05-25 branch of Johnathan's fork is working on x86: 
> 
> https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25
> 
> I have not worked with the ARM machine, but Johnathan may be able to
> comment on the state of ARM support for this code.

ARM support is not yet upstream.  There are some precursor problems we still
have to solve because arm-virt should also support device tree bindings.
See talk I gave at Linaro connect that includes some of them:
https://resources.linaro.org/en/resource/hM986DSHfoTrZ98UjpvLg1

For now, I'm carrying the arm-virt + ACPI support on the tree above.
There are a lot of things we still need to provide support for in QEMU CXL
world so for now figuring out the path forward for upstreaming ARM support
isn't at the top of my list.  I'll get back to it at somepoint - probably
next month.

Jonathan


> 
> ~Gregory

Re: A confusion about CXL in arm virt machine

[Peter Maydell]

On Mon, 19 Jun 2023 at 10:58, Jonathan Cameron via <qemu-arm@nongnu.org> wrote:
>
> On Fri, 16 Jun 2023 14:10:24 -0400
> Gregory Price <gregory.price@memverge.com> wrote:
> >
> > Last I tested cxl-2023-05-25 branch of Johnathan's fork is working on x86:
> >
> > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25
> >
> > I have not worked with the ARM machine, but Johnathan may be able to
> > comment on the state of ARM support for this code.
>
> ARM support is not yet upstream.  There are some precursor problems we still
> have to solve because arm-virt should also support device tree bindings.
> See talk I gave at Linaro connect that includes some of them:
> https://resources.linaro.org/en/resource/hM986DSHfoTrZ98UjpvLg1
>
> For now, I'm carrying the arm-virt + ACPI support on the tree above.
> There are a lot of things we still need to provide support for in QEMU CXL
> world so for now figuring out the path forward for upstreaming ARM support
> isn't at the top of my list.  I'll get back to it at somepoint - probably
> next month.

Is the Arm CXL support still out-of-tree? I ask because at the
moment docs/system/devices/cxl.rst has some aarch64 virt command
lines which don't work, and we've had a bug report about it:
https://gitlab.com/qemu-project/qemu/-/issues/1892

Could you submit a patch to correct the documentation, please
(either fixing the command line, or just deleting any claims
to aarch64 support if it's not upstream) ?

thanks
-- PMM

Re: A confusion about CXL in arm virt machine

[Jonathan Cameron via]

On Mon, 18 Sep 2023 13:41:20 +0100
Peter Maydell <peter.maydell@linaro.org> wrote:

> On Mon, 19 Jun 2023 at 10:58, Jonathan Cameron via <qemu-arm@nongnu.org> wrote:
> >
> > On Fri, 16 Jun 2023 14:10:24 -0400
> > Gregory Price <gregory.price@memverge.com> wrote:  
> > >
> > > Last I tested cxl-2023-05-25 branch of Johnathan's fork is working on x86:
> > >
> > > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25
> > >
> > > I have not worked with the ARM machine, but Johnathan may be able to
> > > comment on the state of ARM support for this code.  
> >
> > ARM support is not yet upstream.  There are some precursor problems we still
> > have to solve because arm-virt should also support device tree bindings.
> > See talk I gave at Linaro connect that includes some of them:
> > https://resources.linaro.org/en/resource/hM986DSHfoTrZ98UjpvLg1
> >
> > For now, I'm carrying the arm-virt + ACPI support on the tree above.
> > There are a lot of things we still need to provide support for in QEMU CXL
> > world so for now figuring out the path forward for upstreaming ARM support
> > isn't at the top of my list.  I'll get back to it at somepoint - probably
> > next month.  
> 
> Is the Arm CXL support still out-of-tree? I ask because at the
> moment docs/system/devices/cxl.rst has some aarch64 virt command
> lines which don't work, and we've had a bug report about it:
> https://gitlab.com/qemu-project/qemu/-/issues/1892

It's still blocked on device tree support...
Step 1 to fixing that is working out that PXB device tree enumeration
dance. I've not yet had time to look at whether we can do
more of the enumeration part in the OS.

> 
> Could you submit a patch to correct the documentation, please
> (either fixing the command line, or just deleting any claims
> to aarch64 support if it's not upstream) ?

Strange, I thought we'd long fixed the docs for this.
Gah, I messed up a rebase of Gregory's series adding multiple
region support and put some back in again.

I'll roll this fix into a little series with some others I have queued up
and post it in a few minutes.

Jonathan





> 
> thanks
> -- PMM
> 

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Fri, 11 Aug 2023 18:31:28 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, 
> On 2023-08-10 21:56,  jonathan.cameron wrote:
>  
> So took a look at your issue - be it on the cxl-2023-08-07 branch rebased on qemu/master
> from today (side effect of looking at the segfault that was stopping me getting to this).
>  
> For me at least the branch does create an ACPI0017 DSDT entry and an ACPI0016 one
> and all the CXL devices turn up in /sys/bus/cxl/devices as expected.
>  
> 
> Oh, thanks for your guidance. It works so now I can get  ACPI0017 & ACPI0016 information in DSDT.   : )
> 
> By the way,  I found that if we add a pcie root port which create the same bus number as we assigned to pxb-cxl, 
> the enumeration of cxl and pcie would be different from what we expected.  In this case,  we cannot find 
> CXL devices in /sys/bus/cxl/devices. 

So this seems to be a case of shooting ourselves in the foot, but not catching the nonsensical configuration
(as you observer later! :)
pxb-pcie complains if you try and add two at the same bus number, but that doesn't protect against overlapping
ranges because they aren't known until after enumeration (which is done by the bios - and I guess the bios
doesn't sanity check for this insanity).  Qemu could take another look when it builds the ACPI tables a
second time though.

Looking at edk2 logs I can see it is happily populating the root bus 1 on my arm64 setup and that it
observes there are no subordinate buses available for the main PCIe bus (0) that QEMU is creating by
default. The _CRS entries look correct but the kernel ignores them it seems. 

It is very much not a valid configuration so there is no reason the kernel should cope with it.

Maybe it's worth considering some hardening code?


> 
> According to my test, the error happened in 
> "devm_cxl_register_pci_bus()"  of  "add_host_bridge_uport" in  "cxl_acpi_probe".
> Actually,  in above case, the incorrect enumeration of pcie will also occur with pxb-pcie except for pxb-cxl, 
> hence I guess the kernel did not deal with such case and users just need to avoid it if they need a correct
> enumeration result.

Agreed - Protecting against ever corner case of impossible configuration is tricky to
do.

> 
> My qemu script (which will cause the incorrect enumeration):
> qemu-system-x86_64 \
> -M q35,nvdimm=on,cxl=on \
> -m 4G,maxmem=8G,slots=8 \
> -smp 1 \
> -object memory-backend-file,id=cxl-mem1,share=on,mem-path=./memfile/cxltest3.raw,size=256M \
> -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=./memfile/lsa3.raw,size=256M \
> -device ioh3420,bus=pcie.0,id=root_port1,chassis=0,slot=1,addr=04 \
> -device qemu-xhci,bus=root_port1 \
> -device pxb-cxl,bus_nr=1,bus=pcie.0,id=cxl.1 \
> -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
> -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
> -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G \
> ......
> 
> Many thanks
> Yuquan