A confusion about CXL in arm virt machine

A confusion about CXL in arm virt machine

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

Hi, Gregory

There is one confusion about CXL in QEMU I hope to consult. 
If you have some time to look at this email, I would have better understanding of CXL 
emulation in QEMU.

On docs/system/devices/cxl.rst ,  Gregory wrote:
A very simple setup with just one directly attached CXL Type 3 Volatile Memory device::
qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \
......

As the current master branch of QEMU has not yet complemented the CXL option/expansion 
in arm virt machine, how this example command lines worked? Or here used another branch 
rather than master?

Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 2769 bytes --]

Re: A confusion about CXL in arm virt machine

[Gregory Price]

On Fri, Jun 16, 2023 at 03:43:31PM +0800, Yuquan Wang wrote:
> Hi, Gregory
> 
> There is one confusion about CXL in QEMU I hope to consult. 
> If you have some time to look at this email, I would have better understanding of CXL 
> emulation in QEMU.
> 
> On docs/system/devices/cxl.rst ,  Gregory wrote:
> A very simple setup with just one directly attached CXL Type 3 Volatile Memory device::
> qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \
> ......
> 
> As the current master branch of QEMU has not yet complemented the CXL option/expansion 
> in arm virt machine, how this example command lines worked? Or here used another branch 
> rather than master?
> 
> Many thanks
> Yuquan

As of today, the qemu/qemu.git master branch does have the required
patch for volatile region support: adacc814f541af9281c922e750d8ba4b90c1a73e

however, the last time i tested it on x86, the master branch was
incapable of enabling these regions with the latest kernel (6.3.x)
despite that kernel having sufficient support to do so.  I have not dug
into what the discrepency between master and johnathan's working branch
are just yet.

Last I tested cxl-2023-05-25 branch of Johnathan's fork is working on x86: 

https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25

I have not worked with the ARM machine, but Johnathan may be able to
comment on the state of ARM support for this code.

~Gregory

Re: A confusion about CXL in arm virt machine

[Jonathan Cameron via]

On Fri, 16 Jun 2023 14:10:24 -0400
Gregory Price <gregory.price@memverge.com> wrote:

> On Fri, Jun 16, 2023 at 03:43:31PM +0800, Yuquan Wang wrote:
> > Hi, Gregory
> > 
> > There is one confusion about CXL in QEMU I hope to consult. 
> > If you have some time to look at this email, I would have better understanding of CXL 
> > emulation in QEMU.
> > 
> > On docs/system/devices/cxl.rst ,  Gregory wrote:
> > A very simple setup with just one directly attached CXL Type 3 Volatile Memory device::
> > qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 -cpu max \
> > ......
> > 
> > As the current master branch of QEMU has not yet complemented the CXL option/expansion 
> > in arm virt machine, how this example command lines worked? Or here used another branch 
> > rather than master?
> > 
> > Many thanks
> > Yuquan  
> 
> As of today, the qemu/qemu.git master branch does have the required
> patch for volatile region support: adacc814f541af9281c922e750d8ba4b90c1a73e
> 
> however, the last time i tested it on x86, the master branch was
> incapable of enabling these regions with the latest kernel (6.3.x)
> despite that kernel having sufficient support to do so.  I have not dug
> into what the discrepency between master and johnathan's working branch
> are just yet.

Events support is missing so the upstream kernel drivers won't probe
successfully.  That's queued up for merge but hasn't happened quite yet.
*fingers crossed* it should go in soon.

> 
> Last I tested cxl-2023-05-25 branch of Johnathan's fork is working on x86: 
> 
> https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25
> 
> I have not worked with the ARM machine, but Johnathan may be able to
> comment on the state of ARM support for this code.

ARM support is not yet upstream.  There are some precursor problems we still
have to solve because arm-virt should also support device tree bindings.
See talk I gave at Linaro connect that includes some of them:
https://resources.linaro.org/en/resource/hM986DSHfoTrZ98UjpvLg1

For now, I'm carrying the arm-virt + ACPI support on the tree above.
There are a lot of things we still need to provide support for in QEMU CXL
world so for now figuring out the path forward for upstreaming ARM support
isn't at the top of my list.  I'll get back to it at somepoint - probably
next month.

Jonathan


> 
> ~Gregory

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Fri, 11 Aug 2023 18:31:28 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, 
> On 2023-08-10 21:56,  jonathan.cameron wrote:
>  
> So took a look at your issue - be it on the cxl-2023-08-07 branch rebased on qemu/master
> from today (side effect of looking at the segfault that was stopping me getting to this).
>  
> For me at least the branch does create an ACPI0017 DSDT entry and an ACPI0016 one
> and all the CXL devices turn up in /sys/bus/cxl/devices as expected.
>  
> 
> Oh, thanks for your guidance. It works so now I can get  ACPI0017 & ACPI0016 information in DSDT.   : )
> 
> By the way,  I found that if we add a pcie root port which create the same bus number as we assigned to pxb-cxl, 
> the enumeration of cxl and pcie would be different from what we expected.  In this case,  we cannot find 
> CXL devices in /sys/bus/cxl/devices. 

So this seems to be a case of shooting ourselves in the foot, but not catching the nonsensical configuration
(as you observer later! :)
pxb-pcie complains if you try and add two at the same bus number, but that doesn't protect against overlapping
ranges because they aren't known until after enumeration (which is done by the bios - and I guess the bios
doesn't sanity check for this insanity).  Qemu could take another look when it builds the ACPI tables a
second time though.

Looking at edk2 logs I can see it is happily populating the root bus 1 on my arm64 setup and that it
observes there are no subordinate buses available for the main PCIe bus (0) that QEMU is creating by
default. The _CRS entries look correct but the kernel ignores them it seems. 

It is very much not a valid configuration so there is no reason the kernel should cope with it.

Maybe it's worth considering some hardening code?


> 
> According to my test, the error happened in 
> "devm_cxl_register_pci_bus()"  of  "add_host_bridge_uport" in  "cxl_acpi_probe".
> Actually,  in above case, the incorrect enumeration of pcie will also occur with pxb-pcie except for pxb-cxl, 
> hence I guess the kernel did not deal with such case and users just need to avoid it if they need a correct
> enumeration result.

Agreed - Protecting against ever corner case of impossible configuration is tricky to
do.

> 
> My qemu script (which will cause the incorrect enumeration):
> qemu-system-x86_64 \
> -M q35,nvdimm=on,cxl=on \
> -m 4G,maxmem=8G,slots=8 \
> -smp 1 \
> -object memory-backend-file,id=cxl-mem1,share=on,mem-path=./memfile/cxltest3.raw,size=256M \
> -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=./memfile/lsa3.raw,size=256M \
> -device ioh3420,bus=pcie.0,id=root_port1,chassis=0,slot=1,addr=04 \
> -device qemu-xhci,bus=root_port1 \
> -device pxb-cxl,bus_nr=1,bus=pcie.0,id=cxl.1 \
> -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
> -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
> -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G \
> ......
> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Mon, 4 Sep 2023 18:27:10 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> 
Hi Yuquan

Given this question isn't just ARM specific included qemu-devel in the cc list as
that gets much wider reading than qemu-arm.

> Due to my poor experience and knowledge on cxl development history, I am sorry to continue
> to ask some very simple and fundamental questions  : (
> 
> In hw/arm/virt :
> [VIRT_CXL_HOST] =           { 0x0, 64 * KiB * 16 }, /* 16 UID */
> 
> It seems like the specific space for MMIO of cxl host bridges.  Why not just use the existing "VIRT_PCIE_MMIO" 
> space for them? 

At the system design level, MMIO space of Root complex register space via RCRB
does not map in a similar fashion to PCIE MMIO space (which is handled via
address decoding in the PCIE fabric). It is much more similar to MMIO for platform
devices - as such the implementation handles in like a platform device (well 16 of
them which seemed enough for any sane usecase).


Now the next bit I've only quickly explored so may have some errors!

For instance on a GPEX the main MMIO region is an alias to the sysbus mmio region 1.
https://elixir.bootlin.com/qemu/latest/source/hw/arm/virt.c#L1452
That region is then mapped to a generic pci_root_bus for use by bar
mappings etc.

https://elixir.bootlin.com/qemu/latest/source/hw/pci-host/gpex.c#L136

So in theory we could make some space for the CXL root bridge RCRB registers
but it would make various generic paths more complex.  In a real system
those registers are likely to be far from the PCI MMIO space anyway so the
way it's modeled is probably more realistic than pushing the RCRB into the
existing allocation.

I hope that clarifies our reasoning for handling this MMIO region separately.

Jonathan

> 
> I would be grateful if you have some free time to help check this issue : )
> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

Hi, Jonathan
On 2023-09-04 20:43,  jonathan.cameron wrote:
> 
> At the system design level, MMIO space of Root complex register space via RCRB
> does not map in a similar fashion to PCIE MMIO space (which is handled via
> address decoding in the PCIE fabric). It is much more similar to MMIO for platform
> devices - as such the implementation handles in like a platform device (well 16 of
> them which seemed enough for any sane usecase).
> 
> 

Oh,thanks! According to above, therefore, the core factor is the implementation of RCRB.

> 
> So in theory we could make some space for the CXL root bridge RCRB registers
> but it would make various generic paths more complex.  In a real system
> those registers are likely to be far from the PCI MMIO space anyway so the
> way it's modeled is probably more realistic than pushing the RCRB into the
> existing allocation.
> 

Here implies that all CXL root bridge will use RCRB registers.

From Table 8-17 and Figure 9-14 in CXL3.0 specification, I understood that only RCH DP &
RCD UP will use RCRBs, and CXL host bridges VH mode will use other way to realize
the CHBCR. I had tried to find more explanation in CXL spec, but I haven't found. Hence 
this is why I am confused.

Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 2493 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Tue, 5 Sep 2023 18:45:02 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> On 2023-09-04 20:43,  jonathan.cameron wrote:
> > 
> > At the system design level, MMIO space of Root complex register space via RCRB
> > does not map in a similar fashion to PCIE MMIO space (which is handled via
> > address decoding in the PCIE fabric). It is much more similar to MMIO for platform
> > devices - as such the implementation handles in like a platform device (well 16 of
> > them which seemed enough for any sane usecase).
> > 
> >   
> 
> Oh,thanks! According to above, therefore, the core factor is the implementation of RCRB.
> 
> > 
> > So in theory we could make some space for the CXL root bridge RCRB registers
> > but it would make various generic paths more complex.  In a real system
> > those registers are likely to be far from the PCI MMIO space anyway so the
> > way it's modeled is probably more realistic than pushing the RCRB into the
> > existing allocation.
> >   
> 
> Here implies that all CXL root bridge will use RCRB registers.
> 
> From Table 8-17 and Figure 9-14 in CXL3.0 specification, I understood that only RCH DP &
> RCD UP will use RCRBs, and CXL host bridges VH mode will use other way to realize
> the CHBCR. I had tried to find more explanation in CXL spec, but I haven't found. Hence 
> this is why I am confused.

Ah. That distinction is a bit messy.  Both an RCRB and CHBCR (CXL Host Bridge Component
Registers) are similar in the sense that they are mapped in host memory space. 
As I understand it the distinction is more about the format / contents of that memory
than how you access them. As an aside, they are described by a static ACPI table,
so they can't be in the MMIO space used for BARs etc.

Jonathan


> 
> Many thanks
> Yuquan

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Yuquan Wang]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

Hi, Jonathan
On 2023-09-05 22:34,  jonathan.cameron wrote:
> 
> As I understand it the distinction is more about the format / contents of that memory
> than how you access them. 

Yes, RCH DP RCRB includes registers from PCIe Type 1 Config Header and
PCIe capabilities and extended capabilities while CHBCR includes component registers 
with the same layout and discovery mechanism in other CXL components.

> As an aside, they are described by a static ACPI table,
> so they can't be in the MMIO space used for BARs etc.
> 

In CXL 3.0 Spec, the Figure 9-14 (CXL Link/Protocol Register Mapping in a CXL VH)
shows that CHBCR is mapped by "Host Proprietary Static Bar". According to your guidance,
it is not a standard PCIe BAR using PCIe MMIO Space, so I understand it is a special BAR for 
MMIO of a platform device?


Many thanks
Yuquan

[-- Attachment #2: Type: text/html, Size: 2501 bytes --]

Re: CXL Namespaces of ACPI disappearing in Qemu demo

[Jonathan Cameron via]

On Wed, 6 Sep 2023 19:22:19 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hi, Jonathan
> On 2023-09-05 22:34,  jonathan.cameron wrote:
> > 
> > As I understand it the distinction is more about the format / contents of that memory
> > than how you access them.   
> 
> Yes, RCH DP RCRB includes registers from PCIe Type 1 Config Header and
> PCIe capabilities and extended capabilities while CHBCR includes component registers 
> with the same layout and discovery mechanism in other CXL components.
> 
> > As an aside, they are described by a static ACPI table,
> > so they can't be in the MMIO space used for BARs etc.
> >   
> 
> In CXL 3.0 Spec, the Figure 9-14 (CXL Link/Protocol Register Mapping in a CXL VH)
> shows that CHBCR is mapped by "Host Proprietary Static Bar". According to your guidance,
> it is not a standard PCIe BAR using PCIe MMIO Space, so I understand it is a special BAR for 
> MMIO of a platform device?

Hmm. Host proprietary so basically you can in theory do anything you like.

In QEMU emulation at least it's not in the PCIe MMIO space.  I'd not rule out other
implementations putting it somewhere in that space.  For now I'm not seeing
a) Anything that says our choice is invalid.
b) Any advantage in making the handling of PCIe MMIO space more complex to shoe
   horn this in there.

Jonathan
> 
> 
> Many thanks
> Yuquan

Re: A confusion about CXL in arm virt machine

[Peter Maydell]

On Mon, 19 Jun 2023 at 10:58, Jonathan Cameron via <qemu-arm@nongnu.org> wrote:
>
> On Fri, 16 Jun 2023 14:10:24 -0400
> Gregory Price <gregory.price@memverge.com> wrote:
> >
> > Last I tested cxl-2023-05-25 branch of Johnathan's fork is working on x86:
> >
> > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25
> >
> > I have not worked with the ARM machine, but Johnathan may be able to
> > comment on the state of ARM support for this code.
>
> ARM support is not yet upstream.  There are some precursor problems we still
> have to solve because arm-virt should also support device tree bindings.
> See talk I gave at Linaro connect that includes some of them:
> https://resources.linaro.org/en/resource/hM986DSHfoTrZ98UjpvLg1
>
> For now, I'm carrying the arm-virt + ACPI support on the tree above.
> There are a lot of things we still need to provide support for in QEMU CXL
> world so for now figuring out the path forward for upstreaming ARM support
> isn't at the top of my list.  I'll get back to it at somepoint - probably
> next month.

Is the Arm CXL support still out-of-tree? I ask because at the
moment docs/system/devices/cxl.rst has some aarch64 virt command
lines which don't work, and we've had a bug report about it:
https://gitlab.com/qemu-project/qemu/-/issues/1892

Could you submit a patch to correct the documentation, please
(either fixing the command line, or just deleting any claims
to aarch64 support if it's not upstream) ?

thanks
-- PMM

Re: A confusion about CXL in arm virt machine

[Jonathan Cameron via]

On Mon, 18 Sep 2023 13:41:20 +0100
Peter Maydell <peter.maydell@linaro.org> wrote:

> On Mon, 19 Jun 2023 at 10:58, Jonathan Cameron via <qemu-arm@nongnu.org> wrote:
> >
> > On Fri, 16 Jun 2023 14:10:24 -0400
> > Gregory Price <gregory.price@memverge.com> wrote:  
> > >
> > > Last I tested cxl-2023-05-25 branch of Johnathan's fork is working on x86:
> > >
> > > https://gitlab.com/jic23/qemu/-/tree/cxl-2023-05-25
> > >
> > > I have not worked with the ARM machine, but Johnathan may be able to
> > > comment on the state of ARM support for this code.  
> >
> > ARM support is not yet upstream.  There are some precursor problems we still
> > have to solve because arm-virt should also support device tree bindings.
> > See talk I gave at Linaro connect that includes some of them:
> > https://resources.linaro.org/en/resource/hM986DSHfoTrZ98UjpvLg1
> >
> > For now, I'm carrying the arm-virt + ACPI support on the tree above.
> > There are a lot of things we still need to provide support for in QEMU CXL
> > world so for now figuring out the path forward for upstreaming ARM support
> > isn't at the top of my list.  I'll get back to it at somepoint - probably
> > next month.  
> 
> Is the Arm CXL support still out-of-tree? I ask because at the
> moment docs/system/devices/cxl.rst has some aarch64 virt command
> lines which don't work, and we've had a bug report about it:
> https://gitlab.com/qemu-project/qemu/-/issues/1892

It's still blocked on device tree support...
Step 1 to fixing that is working out that PXB device tree enumeration
dance. I've not yet had time to look at whether we can do
more of the enumeration part in the OS.

> 
> Could you submit a patch to correct the documentation, please
> (either fixing the command line, or just deleting any claims
> to aarch64 support if it's not upstream) ?

Strange, I thought we'd long fixed the docs for this.
Gah, I messed up a rebase of Gregory's series adding multiple
region support and put some back in again.

I'll roll this fix into a little series with some others I have queued up
and post it in a few minutes.

Jonathan





> 
> thanks
> -- PMM
>