CXL emulation on aarch64

CXL emulation on aarch64

[Itaru Kitayama]

Hi,
Is anybody working on the CXL emulation on aarch64?
If there’s a WIP branch, a pointer would be appreciated.

Itaru.

Re: CXL emulation on aarch64

[Zhijian Li (Fujitsu) via]

On 10/01/2025 13:29, Itaru Kitayama wrote:
> Hi,
> Is anybody working on the CXL emulation on aarch64?

I'm not currently working on the CXL emulation on aarch64.

However, IIRC the CXL maintainer's tree should work.
https://gitlab.com/jic23/qemu/


Thanks
Zhijian

> If there’s a WIP branch, a pointer would be appreciated.
> 
> Itaru.

Re: CXL emulation on aarch64

[Jonathan Cameron via]

On Fri, 10 Jan 2025 09:20:54 +0000
"Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:

> On 10/01/2025 13:29, Itaru Kitayama wrote:
> > Hi,
> > Is anybody working on the CXL emulation on aarch64?  
> 
> I'm not currently working on the CXL emulation on aarch64.
> 
> However, IIRC the CXL maintainer's tree should work.
> https://gitlab.com/jic23/qemu/

Pick up latest branch from there. I'm prepping a rebased version
with some new stuff but might take a few more days.

Note my main development work is on arm64 so that tends to work
more reliably than x86 which I only lightly test for stuff that
isn't ready for upstream yet.

Give me a shout if you run into any problems.

The main blocker on upstreaming this is resolving the missing device tree
support for PCI expander bridges.  I've not made any progress on this since
talk at Linaro connect in 2023.

Jonathan


> 
> 
> Thanks
> Zhijian
> 
> > If there’s a WIP branch, a pointer would be appreciated.
> > 
> > Itaru  

Re: CXL emulation on aarch64

[Itaru Kitayama]

Hi Jonathan, 

> On Jan 10, 2025, at 21:31, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> 
> On Fri, 10 Jan 2025 09:20:54 +0000
> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
> 
>> On 10/01/2025 13:29, Itaru Kitayama wrote:
>>> Hi,
>>> Is anybody working on the CXL emulation on aarch64?  
>> 
>> I'm not currently working on the CXL emulation on aarch64.
>> 
>> However, IIRC the CXL maintainer's tree should work.
>> https://gitlab.com/jic23/qemu/
> 
> Pick up latest branch from there. I'm prepping a rebased version
> with some new stuff but might take a few more days.

Thanks for sharing your work with us.  Your master and cxl-2024-11-27 branches give:

$ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not found

My commands are below:
$HOME/projects/qemu/build/qemu-system-aarch64 \
        -M virt,virtualization=on,gic-version=3 \
        -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
        -accel tcg,cxl=on \
        -nographic \
        -bios $HOME/cca-v4/out/bin/flash.bin \
        -kernel Image-cca \
        -drive format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
        -device virtio-blk-pci,drive=hd0 \
        -append root=/dev/vda \
        -nodefaults \
        --serial tcp:localhost:54320 \
         -serial tcp:localhost:54321 \
         -append "root=/dev/vda earlycon console=hvc0" \
         -device virtio-net-pci,netdev=net0 \
         -netdev user,id=net0 \
         -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
         -fsdev local,security_model=none,path=../../,id=shr0

Yes, I’m using Linaro’s CCA capable OP-TEE builds above.

Let me know which branch you were suggesting.

Thanks,
Itaru. 

> 
> Note my main development work is on arm64 so that tends to work
> more reliably than x86 which I only lightly test for stuff that
> isn't ready for upstream yet.
> 
> Give me a shout if you run into any problems.
> 
> The main blocker on upstreaming this is resolving the missing device tree
> support for PCI expander bridges.  I've not made any progress on this since
> talk at Linaro connect in 2023.
> 
> Jonathan
> 
> 
>> 
>> 
>> Thanks
>> Zhijian
>> 
>>> If there’s a WIP branch, a pointer would be appreciated.
>>> 
>>> Itaru  
> 

Re: CXL emulation on aarch64

[Jonathan Cameron via]

On Tue, 14 Jan 2025 12:03:03 +0900
Itaru Kitayama <itaru.kitayama@linux.dev> wrote:

> Hi Jonathan, 
> 
> > On Jan 10, 2025, at 21:31, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> > 
> > On Fri, 10 Jan 2025 09:20:54 +0000
> > "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
> >   
> >> On 10/01/2025 13:29, Itaru Kitayama wrote:  
> >>> Hi,
> >>> Is anybody working on the CXL emulation on aarch64?    
> >> 
> >> I'm not currently working on the CXL emulation on aarch64.
> >> 
> >> However, IIRC the CXL maintainer's tree should work.
> >> https://gitlab.com/jic23/qemu/  
> > 
> > Pick up latest branch from there. I'm prepping a rebased version
> > with some new stuff but might take a few more days.  
> 
> Thanks for sharing your work with us.  Your master and cxl-2024-11-27 branches give:
> 
> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not found

cxl is a machine property not a accel one. So needs to be after virt
There are tests in the tree for bios tables. Copy the command line from those.

> 
> My commands are below:
> $HOME/projects/qemu/build/qemu-system-aarch64 \
>         -M virt,virtualization=on,gic-version=3 \
>         -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
>         -accel tcg,cxl=on \
>         -nographic \
>         -bios $HOME/cca-v4/out/bin/flash.bin \
>         -kernel Image-cca \
>         -drive format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
>         -device virtio-blk-pci,drive=hd0 \
>         -append root=/dev/vda \
>         -nodefaults \
>         --serial tcp:localhost:54320 \
>          -serial tcp:localhost:54321 \
>          -append "root=/dev/vda earlycon console=hvc0" \
>          -device virtio-net-pci,netdev=net0 \
>          -netdev user,id=net0 \
>          -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
>          -fsdev local,security_model=none,path=../../,id=shr0
> 
> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.

I'm a little curious why optee is relevant for this but shouldn't matter as long
as an appropriate EDK2 is loaded.

Jonathan

> 
> Let me know which branch you were suggesting.
> 
> Thanks,
> Itaru. 
> 
> > 
> > Note my main development work is on arm64 so that tends to work
> > more reliably than x86 which I only lightly test for stuff that
> > isn't ready for upstream yet.
> > 
> > Give me a shout if you run into any problems.
> > 
> > The main blocker on upstreaming this is resolving the missing device tree
> > support for PCI expander bridges.  I've not made any progress on this since
> > talk at Linaro connect in 2023.
> > 
> > Jonathan
> > 
> >   
> >> 
> >> 
> >> Thanks
> >> Zhijian
> >>   
> >>> If there’s a WIP branch, a pointer would be appreciated.
> >>> 
> >>> Itaru    
> >   
> 

Re: CXL emulation on aarch64

[Itaru Kitayama]

Hi Jonathan,

> On Jan 14, 2025, at 19:26, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> 
> On Tue, 14 Jan 2025 12:03:03 +0900
> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> 
>> Hi Jonathan, 
>> 
>>> On Jan 10, 2025, at 21:31, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>> 
>>> On Fri, 10 Jan 2025 09:20:54 +0000
>>> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
>>> 
>>>> On 10/01/2025 13:29, Itaru Kitayama wrote:  
>>>>> Hi,
>>>>> Is anybody working on the CXL emulation on aarch64?    
>>>> 
>>>> I'm not currently working on the CXL emulation on aarch64.
>>>> 
>>>> However, IIRC the CXL maintainer's tree should work.
>>>> https://gitlab.com/jic23/qemu/  
>>> 
>>> Pick up latest branch from there. I'm prepping a rebased version
>>> with some new stuff but might take a few more days.  
>> 
>> Thanks for sharing your work with us.  Your master and cxl-2024-11-27 branches give:
>> 
>> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not found
> 
> cxl is a machine property not a accel one. So needs to be after virt
> There are tests in the tree for bios tables. Copy the command line from those.
> 
>> 
>> My commands are below:
>> $HOME/projects/qemu/build/qemu-system-aarch64 \
>>        -M virt,virtualization=on,gic-version=3 \
>>        -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
>>        -accel tcg,cxl=on \
>>        -nographic \
>>        -bios $HOME/cca-v4/out/bin/flash.bin \
>>        -kernel Image-cca \
>>        -drive format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
>>        -device virtio-blk-pci,drive=hd0 \
>>        -append root=/dev/vda \
>>        -nodefaults \
>>        --serial tcp:localhost:54320 \
>>         -serial tcp:localhost:54321 \
>>         -append "root=/dev/vda earlycon console=hvc0" \
>>         -device virtio-net-pci,netdev=net0 \
>>         -netdev user,id=net0 \
>>         -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
>>         -fsdev local,security_model=none,path=../../,id=shr0
>> 
>> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.
> 
> I'm a little curious why optee is relevant for this but shouldn't matter as long
> as an appropriate EDK2 is loaded.
> 

I picked up your tree’s “master” and “cxl-next” as of today, and only the latter at least booted.
The former gives:

qemu-system-aarch64: Property 'virt-9.2-machine.cxl' not found

Should I stick with the cxl-next? My concern is that the base QEMU version is a bit old
7.0.50.

Thanks,
Itaru.

> Jonathan
> 
>> 
>> Let me know which branch you were suggesting.
>> 
>> Thanks,
>> Itaru. 
>> 
>>> 
>>> Note my main development work is on arm64 so that tends to work
>>> more reliably than x86 which I only lightly test for stuff that
>>> isn't ready for upstream yet.
>>> 
>>> Give me a shout if you run into any problems.
>>> 
>>> The main blocker on upstreaming this is resolving the missing device tree
>>> support for PCI expander bridges.  I've not made any progress on this since
>>> talk at Linaro connect in 2023.
>>> 
>>> Jonathan
>>> 
>>> 
>>>> 
>>>> 
>>>> Thanks
>>>> Zhijian
>>>> 
>>>>> If there’s a WIP branch, a pointer would be appreciated.
>>>>> 
>>>>> Itaru    

Re: CXL emulation on aarch64

[Jonathan Cameron via]

On Thu, 16 Jan 2025 15:04:53 +0900
Itaru Kitayama <itaru.kitayama@linux.dev> wrote:

> Hi Jonathan,
> 
> > On Jan 14, 2025, at 19:26, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> > 
> > On Tue, 14 Jan 2025 12:03:03 +0900
> > Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> >   
> >> Hi Jonathan, 
> >>   
> >>> On Jan 10, 2025, at 21:31, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> >>> 
> >>> On Fri, 10 Jan 2025 09:20:54 +0000
> >>> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
> >>>   
> >>>> On 10/01/2025 13:29, Itaru Kitayama wrote:    
> >>>>> Hi,
> >>>>> Is anybody working on the CXL emulation on aarch64?      
> >>>> 
> >>>> I'm not currently working on the CXL emulation on aarch64.
> >>>> 
> >>>> However, IIRC the CXL maintainer's tree should work.
> >>>> https://gitlab.com/jic23/qemu/    
> >>> 
> >>> Pick up latest branch from there. I'm prepping a rebased version
> >>> with some new stuff but might take a few more days.    
> >> 
> >> Thanks for sharing your work with us.  Your master and cxl-2024-11-27 branches give:
> >> 
> >> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not found  
> > 
> > cxl is a machine property not a accel one. So needs to be after virt
> > There are tests in the tree for bios tables. Copy the command line from those.
> >   
> >> 
> >> My commands are below:
> >> $HOME/projects/qemu/build/qemu-system-aarch64 \
> >>        -M virt,virtualization=on,gic-version=3 \
> >>        -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
> >>        -accel tcg,cxl=on \
> >>        -nographic \
> >>        -bios $HOME/cca-v4/out/bin/flash.bin \
> >>        -kernel Image-cca \
> >>        -drive format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
> >>        -device virtio-blk-pci,drive=hd0 \
> >>        -append root=/dev/vda \
> >>        -nodefaults \
> >>        --serial tcp:localhost:54320 \
> >>         -serial tcp:localhost:54321 \
> >>         -append "root=/dev/vda earlycon console=hvc0" \
> >>         -device virtio-net-pci,netdev=net0 \
> >>         -netdev user,id=net0 \
> >>         -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
> >>         -fsdev local,security_model=none,path=../../,id=shr0
> >> 
> >> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.  
> > 
> > I'm a little curious why optee is relevant for this but shouldn't matter as long
> > as an appropriate EDK2 is loaded.
> >   
> 
> I picked up your tree’s “master” and “cxl-next” as of today, and only the latter at least booted.
> The former gives:
> 
> qemu-system-aarch64: Property 'virt-9.2-machine.cxl' not found
> 
> Should I stick with the cxl-next? My concern is that the base QEMU version is a bit old
> 7.0.50.

Always use the latest dated branch on that tree.  I release whenever there
is something new to carry or a major rebase needed.

cxl-<date> is the right branch to use. Hope that helps.

Jonathan

> 
> Thanks,
> Itaru.
> 
> > Jonathan
> >   
> >> 
> >> Let me know which branch you were suggesting.
> >> 
> >> Thanks,
> >> Itaru. 
> >>   
> >>> 
> >>> Note my main development work is on arm64 so that tends to work
> >>> more reliably than x86 which I only lightly test for stuff that
> >>> isn't ready for upstream yet.
> >>> 
> >>> Give me a shout if you run into any problems.
> >>> 
> >>> The main blocker on upstreaming this is resolving the missing device tree
> >>> support for PCI expander bridges.  I've not made any progress on this since
> >>> talk at Linaro connect in 2023.
> >>> 
> >>> Jonathan
> >>> 
> >>>   
> >>>> 
> >>>> 
> >>>> Thanks
> >>>> Zhijian
> >>>>   
> >>>>> If there’s a WIP branch, a pointer would be appreciated.
> >>>>> 
> >>>>> Itaru      
> 
> 

Re: CXL emulation on aarch64

[Itaru Kitayama]

> On Jan 16, 2025, at 19:58, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> 
> On Thu, 16 Jan 2025 15:04:53 +0900
> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> 
>> Hi Jonathan,
>> 
>>> On Jan 14, 2025, at 19:26, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>> 
>>> On Tue, 14 Jan 2025 12:03:03 +0900
>>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>>> 
>>>> Hi Jonathan, 
>>>> 
>>>>> On Jan 10, 2025, at 21:31, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>>>> 
>>>>> On Fri, 10 Jan 2025 09:20:54 +0000
>>>>> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
>>>>> 
>>>>>> On 10/01/2025 13:29, Itaru Kitayama wrote:    
>>>>>>> Hi,
>>>>>>> Is anybody working on the CXL emulation on aarch64?      
>>>>>> 
>>>>>> I'm not currently working on the CXL emulation on aarch64.
>>>>>> 
>>>>>> However, IIRC the CXL maintainer's tree should work.
>>>>>> https://gitlab.com/jic23/qemu/    
>>>>> 
>>>>> Pick up latest branch from there. I'm prepping a rebased version
>>>>> with some new stuff but might take a few more days.    
>>>> 
>>>> Thanks for sharing your work with us.  Your master and cxl-2024-11-27 branches give:
>>>> 
>>>> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not found  
>>> 
>>> cxl is a machine property not a accel one. So needs to be after virt
>>> There are tests in the tree for bios tables. Copy the command line from those.
>>> 
>>>> 
>>>> My commands are below:
>>>> $HOME/projects/qemu/build/qemu-system-aarch64 \
>>>>       -M virt,virtualization=on,gic-version=3 \
>>>>       -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
>>>>       -accel tcg,cxl=on \
>>>>       -nographic \
>>>>       -bios $HOME/cca-v4/out/bin/flash.bin \
>>>>       -kernel Image-cca \
>>>>       -drive format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
>>>>       -device virtio-blk-pci,drive=hd0 \
>>>>       -append root=/dev/vda \
>>>>       -nodefaults \
>>>>       --serial tcp:localhost:54320 \
>>>>        -serial tcp:localhost:54321 \
>>>>        -append "root=/dev/vda earlycon console=hvc0" \
>>>>        -device virtio-net-pci,netdev=net0 \
>>>>        -netdev user,id=net0 \
>>>>        -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
>>>>        -fsdev local,security_model=none,path=../../,id=shr0
>>>> 
>>>> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.  
>>> 
>>> I'm a little curious why optee is relevant for this but shouldn't matter as long
>>> as an appropriate EDK2 is loaded.
>>> 
>> 
>> I picked up your tree’s “master” and “cxl-next” as of today, and only the latter at least booted.
>> The former gives:
>> 
>> qemu-system-aarch64: Property 'virt-9.2-machine.cxl' not found
>> 
>> Should I stick with the cxl-next? My concern is that the base QEMU version is a bit old
>> 7.0.50.
> 
> Always use the latest dated branch on that tree.  I release whenever there
> is something new to carry or a major rebase needed.
> 
> cxl-<date> is the right branch to use. Hope that helps.
> 

Okay the cxl-2024-11-27 gives this:

qemu-system-aarch64: CFMWS does not fit under PA limit

Below is my QEMU options I use currently:

/home/itaru/projects/qemu/build/qemu-system-aarch64 \
         -M virt,virtualization=on,pflash0=rom,pflash1=efivars,gic-version=3,virtualization=on,cxl=on -m 8192 \
         -cpu cortex-a53 \
         -smp 2 \
         -accel tcg \
         -nographic \
         -display none \
         -kernel ${HOME}/projects/linux/arch/arm64/boot/Image \
         -append "root=/dev/vda rw earlycon acpi=force" \
         -drive format=raw,if=none,file=${HOME}/ubuntu24.img,id=hd0 \
         -device virtio-blk-pci,drive=hd0 \
         -nodefaults \
         -serial mon:stdio \
         -device virtio-net-pci,netdev=net0 \
         -netdev user,id=net0,hostfwd=tcp::8024-:22 \
         -blockdev node-name=rom,driver=file,filename=edk2-aarch64-code.fd,read-only=true \
         -blockdev node-name=efivars,driver=file,filename=qemu-arm64-efivars.test \
         -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \
         -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \
         -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
         -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
         -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
         -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G

> Jonathan
> 
>> 
>> Thanks,
>> Itaru.
>> 
>>> Jonathan
>>> 
>>>> 
>>>> Let me know which branch you were suggesting.
>>>> 
>>>> Thanks,
>>>> Itaru. 
>>>> 
>>>>> 
>>>>> Note my main development work is on arm64 so that tends to work
>>>>> more reliably than x86 which I only lightly test for stuff that
>>>>> isn't ready for upstream yet.
>>>>> 
>>>>> Give me a shout if you run into any problems.
>>>>> 
>>>>> The main blocker on upstreaming this is resolving the missing device tree
>>>>> support for PCI expander bridges.  I've not made any progress on this since
>>>>> talk at Linaro connect in 2023.
>>>>> 
>>>>> Jonathan
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Thanks
>>>>>> Zhijian
>>>>>> 
>>>>>>> If there’s a WIP branch, a pointer would be appreciated.
>>>>>>> 
>>>>>>> Itaru      

Re: CXL emulation on aarch64

[Itaru Kitayama]

> On Jan 16, 2025, at 19:58, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> 
> On Thu, 16 Jan 2025 15:04:53 +0900
> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> 
>> Hi Jonathan,
>> 
>>> On Jan 14, 2025, at 19:26, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>> 
>>> On Tue, 14 Jan 2025 12:03:03 +0900
>>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>>> 
>>>> Hi Jonathan, 
>>>> 
>>>>> On Jan 10, 2025, at 21:31, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>>>> 
>>>>> On Fri, 10 Jan 2025 09:20:54 +0000
>>>>> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
>>>>> 
>>>>>> On 10/01/2025 13:29, Itaru Kitayama wrote:    
>>>>>>> Hi,
>>>>>>> Is anybody working on the CXL emulation on aarch64?      
>>>>>> 
>>>>>> I'm not currently working on the CXL emulation on aarch64.
>>>>>> 
>>>>>> However, IIRC the CXL maintainer's tree should work.
>>>>>> https://gitlab.com/jic23/qemu/    
>>>>> 
>>>>> Pick up latest branch from there. I'm prepping a rebased version
>>>>> with some new stuff but might take a few more days.    
>>>> 
>>>> Thanks for sharing your work with us.  Your master and cxl-2024-11-27 branches give:
>>>> 
>>>> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not found  
>>> 
>>> cxl is a machine property not a accel one. So needs to be after virt
>>> There are tests in the tree for bios tables. Copy the command line from those.
>>> 
>>>> 
>>>> My commands are below:
>>>> $HOME/projects/qemu/build/qemu-system-aarch64 \
>>>>       -M virt,virtualization=on,gic-version=3 \
>>>>       -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
>>>>       -accel tcg,cxl=on \
>>>>       -nographic \
>>>>       -bios $HOME/cca-v4/out/bin/flash.bin \
>>>>       -kernel Image-cca \
>>>>       -drive format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
>>>>       -device virtio-blk-pci,drive=hd0 \
>>>>       -append root=/dev/vda \
>>>>       -nodefaults \
>>>>       --serial tcp:localhost:54320 \
>>>>        -serial tcp:localhost:54321 \
>>>>        -append "root=/dev/vda earlycon console=hvc0" \
>>>>        -device virtio-net-pci,netdev=net0 \
>>>>        -netdev user,id=net0 \
>>>>        -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
>>>>        -fsdev local,security_model=none,path=../../,id=shr0
>>>> 
>>>> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.  
>>> 
>>> I'm a little curious why optee is relevant for this but shouldn't matter as long
>>> as an appropriate EDK2 is loaded.
>>> 
>> 
>> I picked up your tree’s “master” and “cxl-next” as of today, and only the latter at least booted.
>> The former gives:
>> 
>> qemu-system-aarch64: Property 'virt-9.2-machine.cxl' not found
>> 
>> Should I stick with the cxl-next? My concern is that the base QEMU version is a bit old
>> 7.0.50.
> 
> Always use the latest dated branch on that tree.  I release whenever there
> is something new to carry or a major rebase needed.
> 
> cxl-<date> is the right branch to use. Hope that helps.

When do you think you want to get them (aarch64 specific?) merged mainline. Any reason you want to carry the patches by yourself?

Itaru.

> 
> Jonathan
> 
>> 
>> Thanks,
>> Itaru.
>> 
>>> Jonathan
>>> 
>>>> 
>>>> Let me know which branch you were suggesting.
>>>> 
>>>> Thanks,
>>>> Itaru. 
>>>> 
>>>>> 
>>>>> Note my main development work is on arm64 so that tends to work
>>>>> more reliably than x86 which I only lightly test for stuff that
>>>>> isn't ready for upstream yet.
>>>>> 
>>>>> Give me a shout if you run into any problems.
>>>>> 
>>>>> The main blocker on upstreaming this is resolving the missing device tree
>>>>> support for PCI expander bridges.  I've not made any progress on this since
>>>>> talk at Linaro connect in 2023.
>>>>> 
>>>>> Jonathan
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Thanks
>>>>>> Zhijian
>>>>>> 
>>>>>>> If there’s a WIP branch, a pointer would be appreciated.
>>>>>>> 
>>>>>>> Itaru      

Re: CXL emulation on aarch64

[Jonathan Cameron via]

On Fri, 17 Jan 2025 09:24:15 +0900
Itaru Kitayama <itaru.kitayama@linux.dev> wrote:

> > On Jan 16, 2025, at 19:58, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> > 
> > On Thu, 16 Jan 2025 15:04:53 +0900
> > Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> >   
> >> Hi Jonathan,
> >>   
> >>> On Jan 14, 2025, at 19:26, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> >>> 
> >>> On Tue, 14 Jan 2025 12:03:03 +0900
> >>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> >>>   
> >>>> Hi Jonathan, 
> >>>>   
> >>>>> On Jan 10, 2025, at 21:31, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> >>>>> 
> >>>>> On Fri, 10 Jan 2025 09:20:54 +0000
> >>>>> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
> >>>>>   
> >>>>>> On 10/01/2025 13:29, Itaru Kitayama wrote:      
> >>>>>>> Hi,
> >>>>>>> Is anybody working on the CXL emulation on aarch64?        
> >>>>>> 
> >>>>>> I'm not currently working on the CXL emulation on aarch64.
> >>>>>> 
> >>>>>> However, IIRC the CXL maintainer's tree should work.
> >>>>>> https://gitlab.com/jic23/qemu/      
> >>>>> 
> >>>>> Pick up latest branch from there. I'm prepping a rebased version
> >>>>> with some new stuff but might take a few more days.      
> >>>> 
> >>>> Thanks for sharing your work with us.  Your master and cxl-2024-11-27 branches give:
> >>>> 
> >>>> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not found    
> >>> 
> >>> cxl is a machine property not a accel one. So needs to be after virt
> >>> There are tests in the tree for bios tables. Copy the command line from those.
> >>>   
> >>>> 
> >>>> My commands are below:
> >>>> $HOME/projects/qemu/build/qemu-system-aarch64 \
> >>>>       -M virt,virtualization=on,gic-version=3 \
> >>>>       -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
> >>>>       -accel tcg,cxl=on \
> >>>>       -nographic \
> >>>>       -bios $HOME/cca-v4/out/bin/flash.bin \
> >>>>       -kernel Image-cca \
> >>>>       -drive format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
> >>>>       -device virtio-blk-pci,drive=hd0 \
> >>>>       -append root=/dev/vda \
> >>>>       -nodefaults \
> >>>>       --serial tcp:localhost:54320 \
> >>>>        -serial tcp:localhost:54321 \
> >>>>        -append "root=/dev/vda earlycon console=hvc0" \
> >>>>        -device virtio-net-pci,netdev=net0 \
> >>>>        -netdev user,id=net0 \
> >>>>        -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
> >>>>        -fsdev local,security_model=none,path=../../,id=shr0
> >>>> 
> >>>> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.    
> >>> 
> >>> I'm a little curious why optee is relevant for this but shouldn't matter as long
> >>> as an appropriate EDK2 is loaded.
> >>>   
> >> 
> >> I picked up your tree’s “master” and “cxl-next” as of today, and only the latter at least booted.
> >> The former gives:
> >> 
> >> qemu-system-aarch64: Property 'virt-9.2-machine.cxl' not found
> >> 
> >> Should I stick with the cxl-next? My concern is that the base QEMU version is a bit old
> >> 7.0.50.  
> > 
> > Always use the latest dated branch on that tree.  I release whenever there
> > is something new to carry or a major rebase needed.
> > 
> > cxl-<date> is the right branch to use. Hope that helps.
> >   
> 
> Okay the cxl-2024-11-27 gives this:
> 
> qemu-system-aarch64: CFMWS does not fit under PA limit
> 
> Below is my QEMU options I use currently:
> 

Try using max as the cpu (or n1 or later).  a53 is ancient
which is probably where the limit comes from.

> /home/itaru/projects/qemu/build/qemu-system-aarch64 \
>          -M virt,virtualization=on,pflash0=rom,pflash1=efivars,gic-version=3,virtualization=on,cxl=on -m 8192 \
>          -cpu cortex-a53 \
>          -smp 2 \
>          -accel tcg \
>          -nographic \
>          -display none \
>          -kernel ${HOME}/projects/linux/arch/arm64/boot/Image \
>          -append "root=/dev/vda rw earlycon acpi=force" \
>          -drive format=raw,if=none,file=${HOME}/ubuntu24.img,id=hd0 \
>          -device virtio-blk-pci,drive=hd0 \
>          -nodefaults \
>          -serial mon:stdio \
>          -device virtio-net-pci,netdev=net0 \
>          -netdev user,id=net0,hostfwd=tcp::8024-:22 \
>          -blockdev node-name=rom,driver=file,filename=edk2-aarch64-code.fd,read-only=true \
>          -blockdev node-name=efivars,driver=file,filename=qemu-arm64-efivars.test \
>          -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \
>          -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \
>          -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
>          -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
>          -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
>          -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G
> 
> > Jonathan
> >   
> >> 
> >> Thanks,
> >> Itaru.
> >>   
> >>> Jonathan
> >>>   
> >>>> 
> >>>> Let me know which branch you were suggesting.
> >>>> 
> >>>> Thanks,
> >>>> Itaru. 
> >>>>   
> >>>>> 
> >>>>> Note my main development work is on arm64 so that tends to work
> >>>>> more reliably than x86 which I only lightly test for stuff that
> >>>>> isn't ready for upstream yet.
> >>>>> 
> >>>>> Give me a shout if you run into any problems.
> >>>>> 
> >>>>> The main blocker on upstreaming this is resolving the missing device tree
> >>>>> support for PCI expander bridges.  I've not made any progress on this since
> >>>>> talk at Linaro connect in 2023.
> >>>>> 
> >>>>> Jonathan
> >>>>> 
> >>>>>   
> >>>>>> 
> >>>>>> 
> >>>>>> Thanks
> >>>>>> Zhijian
> >>>>>>   
> >>>>>>> If there’s a WIP branch, a pointer would be appreciated.
> >>>>>>> 
> >>>>>>> Itaru        
> 
> 

Re: CXL emulation on aarch64

[Jonathan Cameron via]

On Fri, 17 Jan 2025 10:13:41 +0900
Itaru Kitayama <itaru.kitayama@linux.dev> wrote:

> > On Jan 16, 2025, at 19:58, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> > 
> > On Thu, 16 Jan 2025 15:04:53 +0900
> > Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> >   
> >> Hi Jonathan,
> >>   
> >>> On Jan 14, 2025, at 19:26, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> >>> 
> >>> On Tue, 14 Jan 2025 12:03:03 +0900
> >>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> >>>   
> >>>> Hi Jonathan, 
> >>>>   
> >>>>> On Jan 10, 2025, at 21:31, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> >>>>> 
> >>>>> On Fri, 10 Jan 2025 09:20:54 +0000
> >>>>> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
> >>>>>   
> >>>>>> On 10/01/2025 13:29, Itaru Kitayama wrote:      
> >>>>>>> Hi,
> >>>>>>> Is anybody working on the CXL emulation on aarch64?        
> >>>>>> 
> >>>>>> I'm not currently working on the CXL emulation on aarch64.
> >>>>>> 
> >>>>>> However, IIRC the CXL maintainer's tree should work.
> >>>>>> https://gitlab.com/jic23/qemu/      
> >>>>> 
> >>>>> Pick up latest branch from there. I'm prepping a rebased version
> >>>>> with some new stuff but might take a few more days.      
> >>>> 
> >>>> Thanks for sharing your work with us.  Your master and cxl-2024-11-27 branches give:
> >>>> 
> >>>> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not found    
> >>> 
> >>> cxl is a machine property not a accel one. So needs to be after virt
> >>> There are tests in the tree for bios tables. Copy the command line from those.
> >>>   
> >>>> 
> >>>> My commands are below:
> >>>> $HOME/projects/qemu/build/qemu-system-aarch64 \
> >>>>       -M virt,virtualization=on,gic-version=3 \
> >>>>       -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
> >>>>       -accel tcg,cxl=on \
> >>>>       -nographic \
> >>>>       -bios $HOME/cca-v4/out/bin/flash.bin \
> >>>>       -kernel Image-cca \
> >>>>       -drive format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
> >>>>       -device virtio-blk-pci,drive=hd0 \
> >>>>       -append root=/dev/vda \
> >>>>       -nodefaults \
> >>>>       --serial tcp:localhost:54320 \
> >>>>        -serial tcp:localhost:54321 \
> >>>>        -append "root=/dev/vda earlycon console=hvc0" \
> >>>>        -device virtio-net-pci,netdev=net0 \
> >>>>        -netdev user,id=net0 \
> >>>>        -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
> >>>>        -fsdev local,security_model=none,path=../../,id=shr0
> >>>> 
> >>>> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.    
> >>> 
> >>> I'm a little curious why optee is relevant for this but shouldn't matter as long
> >>> as an appropriate EDK2 is loaded.
> >>>   
> >> 
> >> I picked up your tree’s “master” and “cxl-next” as of today, and only the latter at least booted.
> >> The former gives:
> >> 
> >> qemu-system-aarch64: Property 'virt-9.2-machine.cxl' not found
> >> 
> >> Should I stick with the cxl-next? My concern is that the base QEMU version is a bit old
> >> 7.0.50.  
> > 
> > Always use the latest dated branch on that tree.  I release whenever there
> > is something new to carry or a major rebase needed.
> > 
> > cxl-<date> is the right branch to use. Hope that helps.  
> 
> When do you think you want to get them (aarch64 specific?) merged mainline. Any reason you want to carry the patches by yourself?

Nothing much has changed since I presented on this at Linaro connect in 2023.
https://resources.linaro.org/en/resource/hM986DSHfoTrZ98UjpvLg1

The issue is device tree bindings for PCI Expander bridgess and the fact that
those need to be generated without the full enumeration that EDK2 is doing
prior to ACPI final table builds. In order to move forward with that it
needs a bunch of work to prove that we absolutely cannot get patches
upstream to support kernel base enumeration and breaking up of the
various resources (like EDK2 does).

Given PXB enumeration in kernel has some issues on ARM anyway (that you can paper
over with _DSM 5 - it self requiring an extra patch that isn't upstreamable because
of IO port issues) there is quite a bit of work needed, mostly not in QEMU.
Or convince Peter and others that not all virt support needs DT bindings
(note that PXB for PCIE has been supported for years without an DT support,
just no one noticed!)

After that we'd need to figure out CXL DT bindings in general and add kernel
code support - despite there being no known DT based CXL systems out there, so
that is going to be hard to do.  Various CXL kernel maintainers have expressed
they aren't against such support, but it's hardly going to be review priority
(other than for me if someone else does the work!)

For me this isn't particularly high priority. The ARM bit is fairly easy to rebase.
I would like to see it solved, but it is behind various other items on my
backlog.

There are SBSA machine patches on list, but it's not a useful platform for
CXL kernel code development because of the limited supported configurations
(in keeping with the more or less fixed model that SBSA-ref uses).

Jonathan



> 
> Itaru.
> 
> > 
> > Jonathan
> >   
> >> 
> >> Thanks,
> >> Itaru.
> >>   
> >>> Jonathan
> >>>   
> >>>> 
> >>>> Let me know which branch you were suggesting.
> >>>> 
> >>>> Thanks,
> >>>> Itaru. 
> >>>>   
> >>>>> 
> >>>>> Note my main development work is on arm64 so that tends to work
> >>>>> more reliably than x86 which I only lightly test for stuff that
> >>>>> isn't ready for upstream yet.
> >>>>> 
> >>>>> Give me a shout if you run into any problems.
> >>>>> 
> >>>>> The main blocker on upstreaming this is resolving the missing device tree
> >>>>> support for PCI expander bridges.  I've not made any progress on this since
> >>>>> talk at Linaro connect in 2023.
> >>>>> 
> >>>>> Jonathan
> >>>>> 
> >>>>>   
> >>>>>> 
> >>>>>> 
> >>>>>> Thanks
> >>>>>> Zhijian
> >>>>>>   
> >>>>>>> If there’s a WIP branch, a pointer would be appreciated.
> >>>>>>> 
> >>>>>>> Itaru        
> 
> 

Re: CXL emulation on aarch64

[Jonathan Cameron via]

On Fri, 17 Jan 2025 09:43:11 +0000
Jonathan Cameron via <qemu-devel@nongnu.org> wrote:

> On Fri, 17 Jan 2025 10:13:41 +0900
> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> 
> > > On Jan 16, 2025, at 19:58, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> > > 
> > > On Thu, 16 Jan 2025 15:04:53 +0900
> > > Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> > >     
> > >> Hi Jonathan,
> > >>     
> > >>> On Jan 14, 2025, at 19:26, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> > >>> 
> > >>> On Tue, 14 Jan 2025 12:03:03 +0900
> > >>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> > >>>     
> > >>>> Hi Jonathan, 
> > >>>>     
> > >>>>> On Jan 10, 2025, at 21:31, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> > >>>>> 
> > >>>>> On Fri, 10 Jan 2025 09:20:54 +0000
> > >>>>> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
> > >>>>>     
> > >>>>>> On 10/01/2025 13:29, Itaru Kitayama wrote:        
> > >>>>>>> Hi,
> > >>>>>>> Is anybody working on the CXL emulation on aarch64?          
> > >>>>>> 
> > >>>>>> I'm not currently working on the CXL emulation on aarch64.
> > >>>>>> 
> > >>>>>> However, IIRC the CXL maintainer's tree should work.
> > >>>>>> https://gitlab.com/jic23/qemu/        
> > >>>>> 
> > >>>>> Pick up latest branch from there. I'm prepping a rebased version
> > >>>>> with some new stuff but might take a few more days.        
> > >>>> 
> > >>>> Thanks for sharing your work with us.  Your master and cxl-2024-11-27 branches give:
> > >>>> 
> > >>>> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not found      
> > >>> 
> > >>> cxl is a machine property not a accel one. So needs to be after virt
> > >>> There are tests in the tree for bios tables. Copy the command line from those.
> > >>>     
> > >>>> 
> > >>>> My commands are below:
> > >>>> $HOME/projects/qemu/build/qemu-system-aarch64 \
> > >>>>       -M virt,virtualization=on,gic-version=3 \
> > >>>>       -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
> > >>>>       -accel tcg,cxl=on \
> > >>>>       -nographic \
> > >>>>       -bios $HOME/cca-v4/out/bin/flash.bin \
> > >>>>       -kernel Image-cca \
> > >>>>       -drive format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
> > >>>>       -device virtio-blk-pci,drive=hd0 \
> > >>>>       -append root=/dev/vda \
> > >>>>       -nodefaults \
> > >>>>       --serial tcp:localhost:54320 \
> > >>>>        -serial tcp:localhost:54321 \
> > >>>>        -append "root=/dev/vda earlycon console=hvc0" \
> > >>>>        -device virtio-net-pci,netdev=net0 \
> > >>>>        -netdev user,id=net0 \
> > >>>>        -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
> > >>>>        -fsdev local,security_model=none,path=../../,id=shr0
> > >>>> 
> > >>>> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.      
> > >>> 
> > >>> I'm a little curious why optee is relevant for this but shouldn't matter as long
> > >>> as an appropriate EDK2 is loaded.
> > >>>     
> > >> 
> > >> I picked up your tree’s “master” and “cxl-next” as of today, and only the latter at least booted.
> > >> The former gives:
> > >> 
> > >> qemu-system-aarch64: Property 'virt-9.2-machine.cxl' not found
> > >> 
> > >> Should I stick with the cxl-next? My concern is that the base QEMU version is a bit old
> > >> 7.0.50.    
> > > 
> > > Always use the latest dated branch on that tree.  I release whenever there
> > > is something new to carry or a major rebase needed.
> > > 
> > > cxl-<date> is the right branch to use. Hope that helps.    
> > 
> > When do you think you want to get them (aarch64 specific?) merged mainline. Any reason you want to carry the patches by yourself?  
> 
> Nothing much has changed since I presented on this at Linaro connect in 2023.
> https://resources.linaro.org/en/resource/hM986DSHfoTrZ98UjpvLg1
> 
> The issue is device tree bindings for PCI Expander bridgess and the fact that
> those need to be generated without the full enumeration that EDK2 is doing
> prior to ACPI final table builds. In order to move forward with that it
> needs a bunch of work to prove that we absolutely cannot get patches
> upstream to support kernel base enumeration and breaking up of the
> various resources (like EDK2 does).

I was talking to Peter Maydell earlier and given developments in the last couple
of years that have by necessity been ACPI only in arm virt he is less
opposed to ACPI only features being added where device tree is challenging.

So we may be able to move forwards without device tree support.

The PXB enumeration question is also relevant for managing multiple
vIOMMUs to represent multiple physical IOMMUs with the correct isolation
and do it efficiently which is probably a more pressing usecase than CXL emulation.
The discussion was mainly about that usecase, but maybe it also unblocks
upstreaming this support.

Thanks,

Jonathan

> 
> Given PXB enumeration in kernel has some issues on ARM anyway (that you can paper
> over with _DSM 5 - it self requiring an extra patch that isn't upstreamable because
> of IO port issues) there is quite a bit of work needed, mostly not in QEMU.
> Or convince Peter and others that not all virt support needs DT bindings
> (note that PXB for PCIE has been supported for years without an DT support,
> just no one noticed!)
> 
> After that we'd need to figure out CXL DT bindings in general and add kernel
> code support - despite there being no known DT based CXL systems out there, so
> that is going to be hard to do.  Various CXL kernel maintainers have expressed
> they aren't against such support, but it's hardly going to be review priority
> (other than for me if someone else does the work!)
> 
> For me this isn't particularly high priority. The ARM bit is fairly easy to rebase.
> I would like to see it solved, but it is behind various other items on my
> backlog.
> 
> There are SBSA machine patches on list, but it's not a useful platform for
> CXL kernel code development because of the limited supported configurations
> (in keeping with the more or less fixed model that SBSA-ref uses).
> 
> Jonathan
> 
> 
> 
> > 
> > Itaru.
> >   
> > > 
> > > Jonathan
> > >     
> > >> 
> > >> Thanks,
> > >> Itaru.
> > >>     
> > >>> Jonathan
> > >>>     
> > >>>> 
> > >>>> Let me know which branch you were suggesting.
> > >>>> 
> > >>>> Thanks,
> > >>>> Itaru. 
> > >>>>     
> > >>>>> 
> > >>>>> Note my main development work is on arm64 so that tends to work
> > >>>>> more reliably than x86 which I only lightly test for stuff that
> > >>>>> isn't ready for upstream yet.
> > >>>>> 
> > >>>>> Give me a shout if you run into any problems.
> > >>>>> 
> > >>>>> The main blocker on upstreaming this is resolving the missing device tree
> > >>>>> support for PCI expander bridges.  I've not made any progress on this since
> > >>>>> talk at Linaro connect in 2023.
> > >>>>> 
> > >>>>> Jonathan
> > >>>>> 
> > >>>>>     
> > >>>>>> 
> > >>>>>> 
> > >>>>>> Thanks
> > >>>>>> Zhijian
> > >>>>>>     
> > >>>>>>> If there’s a WIP branch, a pointer would be appreciated.
> > >>>>>>> 
> > >>>>>>> Itaru          
> > 
> >   
> 
> 

Re: CXL emulation on aarch64

[Itaru Kitayama]

> On Jan 22, 2025, at 23:07, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> 
> On Fri, 17 Jan 2025 09:43:11 +0000
> Jonathan Cameron via <qemu-devel@nongnu.org> wrote:
> 
>> On Fri, 17 Jan 2025 10:13:41 +0900
>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>> 
>>>> On Jan 16, 2025, at 19:58, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>>> 
>>>> On Thu, 16 Jan 2025 15:04:53 +0900
>>>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>>>> 
>>>>> Hi Jonathan,
>>>>> 
>>>>>> On Jan 14, 2025, at 19:26, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>>>>> 
>>>>>> On Tue, 14 Jan 2025 12:03:03 +0900
>>>>>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>>>>>> 
>>>>>>> Hi Jonathan, 
>>>>>>> 
>>>>>>>> On Jan 10, 2025, at 21:31, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>>>>>>> 
>>>>>>>> On Fri, 10 Jan 2025 09:20:54 +0000
>>>>>>>> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
>>>>>>>> 
>>>>>>>>> On 10/01/2025 13:29, Itaru Kitayama wrote:        
>>>>>>>>>> Hi,
>>>>>>>>>> Is anybody working on the CXL emulation on aarch64?          
>>>>>>>>> 
>>>>>>>>> I'm not currently working on the CXL emulation on aarch64.
>>>>>>>>> 
>>>>>>>>> However, IIRC the CXL maintainer's tree should work.
>>>>>>>>> https://gitlab.com/jic23/qemu/        
>>>>>>>> 
>>>>>>>> Pick up latest branch from there. I'm prepping a rebased version
>>>>>>>> with some new stuff but might take a few more days.        
>>>>>>> 
>>>>>>> Thanks for sharing your work with us.  Your master and cxl-2024-11-27 branches give:
>>>>>>> 
>>>>>>> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not found      
>>>>>> 
>>>>>> cxl is a machine property not a accel one. So needs to be after virt
>>>>>> There are tests in the tree for bios tables. Copy the command line from those.
>>>>>> 
>>>>>>> 
>>>>>>> My commands are below:
>>>>>>> $HOME/projects/qemu/build/qemu-system-aarch64 \
>>>>>>>      -M virt,virtualization=on,gic-version=3 \
>>>>>>>      -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
>>>>>>>      -accel tcg,cxl=on \
>>>>>>>      -nographic \
>>>>>>>      -bios $HOME/cca-v4/out/bin/flash.bin \
>>>>>>>      -kernel Image-cca \
>>>>>>>      -drive format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
>>>>>>>      -device virtio-blk-pci,drive=hd0 \
>>>>>>>      -append root=/dev/vda \
>>>>>>>      -nodefaults \
>>>>>>>      --serial tcp:localhost:54320 \
>>>>>>>       -serial tcp:localhost:54321 \
>>>>>>>       -append "root=/dev/vda earlycon console=hvc0" \
>>>>>>>       -device virtio-net-pci,netdev=net0 \
>>>>>>>       -netdev user,id=net0 \
>>>>>>>       -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
>>>>>>>       -fsdev local,security_model=none,path=../../,id=shr0
>>>>>>> 
>>>>>>> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.      
>>>>>> 
>>>>>> I'm a little curious why optee is relevant for this but shouldn't matter as long
>>>>>> as an appropriate EDK2 is loaded.
>>>>>> 
>>>>> 
>>>>> I picked up your tree’s “master” and “cxl-next” as of today, and only the latter at least booted.
>>>>> The former gives:
>>>>> 
>>>>> qemu-system-aarch64: Property 'virt-9.2-machine.cxl' not found
>>>>> 
>>>>> Should I stick with the cxl-next? My concern is that the base QEMU version is a bit old
>>>>> 7.0.50.    
>>>> 
>>>> Always use the latest dated branch on that tree.  I release whenever there
>>>> is something new to carry or a major rebase needed.
>>>> 
>>>> cxl-<date> is the right branch to use. Hope that helps.    
>>> 
>>> When do you think you want to get them (aarch64 specific?) merged mainline. Any reason you want to carry the patches by yourself?  
>> 
>> Nothing much has changed since I presented on this at Linaro connect in 2023.
>> https://resources.linaro.org/en/resource/hM986DSHfoTrZ98UjpvLg1
>> 
>> The issue is device tree bindings for PCI Expander bridgess and the fact that
>> those need to be generated without the full enumeration that EDK2 is doing
>> prior to ACPI final table builds. In order to move forward with that it
>> needs a bunch of work to prove that we absolutely cannot get patches
>> upstream to support kernel base enumeration and breaking up of the
>> various resources (like EDK2 does).
> 
> I was talking to Peter Maydell earlier and given developments in the last couple
> of years that have by necessity been ACPI only in arm virt he is less
> opposed to ACPI only features being added where device tree is challenging.
> 
> So we may be able to move forwards without device tree support.
> 
> The PXB enumeration question is also relevant for managing multiple
> vIOMMUs to represent multiple physical IOMMUs with the correct isolation
> and do it efficiently which is probably a more pressing usecase than CXL emulation.
> The discussion was mainly about that usecase, but maybe it also unblocks
> upstreaming this support.
> 
> Thanks,
> 
> Jonathan

I finally made some CXL tests ran within the ndctl test framework along with the kernel modules (QEMU is on your cxl-2024-11-27 branch) on aarch64. However, the recent rebase cxl-2025-01-24 fails to start the system emulation

qemu-system-aarch64: Property 'virt-10.0-machine.cxl' not found

The build did not complain, what kind of tests you run against your periodical QEMU rebase?

Itaru. 

> 
>> 
>> Given PXB enumeration in kernel has some issues on ARM anyway (that you can paper
>> over with _DSM 5 - it self requiring an extra patch that isn't upstreamable because
>> of IO port issues) there is quite a bit of work needed, mostly not in QEMU.
>> Or convince Peter and others that not all virt support needs DT bindings
>> (note that PXB for PCIE has been supported for years without an DT support,
>> just no one noticed!)
>> 
>> After that we'd need to figure out CXL DT bindings in general and add kernel
>> code support - despite there being no known DT based CXL systems out there, so
>> that is going to be hard to do.  Various CXL kernel maintainers have expressed
>> they aren't against such support, but it's hardly going to be review priority
>> (other than for me if someone else does the work!)
>> 
>> For me this isn't particularly high priority. The ARM bit is fairly easy to rebase.
>> I would like to see it solved, but it is behind various other items on my
>> backlog.
>> 
>> There are SBSA machine patches on list, but it's not a useful platform for
>> CXL kernel code development because of the limited supported configurations
>> (in keeping with the more or less fixed model that SBSA-ref uses).
>> 
>> Jonathan
>> 
>> 
>> 
>>> 
>>> Itaru.
>>> 
>>>> 
>>>> Jonathan
>>>> 
>>>>> 
>>>>> Thanks,
>>>>> Itaru.
>>>>> 
>>>>>> Jonathan
>>>>>> 
>>>>>>> 
>>>>>>> Let me know which branch you were suggesting.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Itaru. 
>>>>>>> 
>>>>>>>> 
>>>>>>>> Note my main development work is on arm64 so that tends to work
>>>>>>>> more reliably than x86 which I only lightly test for stuff that
>>>>>>>> isn't ready for upstream yet.
>>>>>>>> 
>>>>>>>> Give me a shout if you run into any problems.
>>>>>>>> 
>>>>>>>> The main blocker on upstreaming this is resolving the missing device tree
>>>>>>>> support for PCI expander bridges.  I've not made any progress on this since
>>>>>>>> talk at Linaro connect in 2023.
>>>>>>>> 
>>>>>>>> Jonathan
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Thanks
>>>>>>>>> Zhijian
>>>>>>>>> 
>>>>>>>>>> If there’s a WIP branch, a pointer would be appreciated.
>>>>>>>>>> 
>>>>>>>>>> Itaru          

Re: CXL emulation on aarch64

[Jonathan Cameron via]

On Wed, 29 Jan 2025 10:14:34 +0900
Itaru Kitayama <itaru.kitayama@linux.dev> wrote:

> > On Jan 22, 2025, at 23:07, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> > 
> > On Fri, 17 Jan 2025 09:43:11 +0000
> > Jonathan Cameron via <qemu-devel@nongnu.org> wrote:
> >   
> >> On Fri, 17 Jan 2025 10:13:41 +0900
> >> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> >>   
> >>>> On Jan 16, 2025, at 19:58, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> >>>> 
> >>>> On Thu, 16 Jan 2025 15:04:53 +0900
> >>>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> >>>>   
> >>>>> Hi Jonathan,
> >>>>>   
> >>>>>> On Jan 14, 2025, at 19:26, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> >>>>>> 
> >>>>>> On Tue, 14 Jan 2025 12:03:03 +0900
> >>>>>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> >>>>>>   
> >>>>>>> Hi Jonathan, 
> >>>>>>>   
> >>>>>>>> On Jan 10, 2025, at 21:31, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> >>>>>>>> 
> >>>>>>>> On Fri, 10 Jan 2025 09:20:54 +0000
> >>>>>>>> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
> >>>>>>>>   
> >>>>>>>>> On 10/01/2025 13:29, Itaru Kitayama wrote:          
> >>>>>>>>>> Hi,
> >>>>>>>>>> Is anybody working on the CXL emulation on aarch64?            
> >>>>>>>>> 
> >>>>>>>>> I'm not currently working on the CXL emulation on aarch64.
> >>>>>>>>> 
> >>>>>>>>> However, IIRC the CXL maintainer's tree should work.
> >>>>>>>>> https://gitlab.com/jic23/qemu/          
> >>>>>>>> 
> >>>>>>>> Pick up latest branch from there. I'm prepping a rebased version
> >>>>>>>> with some new stuff but might take a few more days.          
> >>>>>>> 
> >>>>>>> Thanks for sharing your work with us.  Your master and cxl-2024-11-27 branches give:
> >>>>>>> 
> >>>>>>> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not found        
> >>>>>> 
> >>>>>> cxl is a machine property not a accel one. So needs to be after virt
> >>>>>> There are tests in the tree for bios tables. Copy the command line from those.
> >>>>>>   
> >>>>>>> 
> >>>>>>> My commands are below:
> >>>>>>> $HOME/projects/qemu/build/qemu-system-aarch64 \
> >>>>>>>      -M virt,virtualization=on,gic-version=3 \
> >>>>>>>      -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
> >>>>>>>      -accel tcg,cxl=on \
> >>>>>>>      -nographic \
> >>>>>>>      -bios $HOME/cca-v4/out/bin/flash.bin \
> >>>>>>>      -kernel Image-cca \
> >>>>>>>      -drive format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
> >>>>>>>      -device virtio-blk-pci,drive=hd0 \
> >>>>>>>      -append root=/dev/vda \
> >>>>>>>      -nodefaults \
> >>>>>>>      --serial tcp:localhost:54320 \
> >>>>>>>       -serial tcp:localhost:54321 \
> >>>>>>>       -append "root=/dev/vda earlycon console=hvc0" \
> >>>>>>>       -device virtio-net-pci,netdev=net0 \
> >>>>>>>       -netdev user,id=net0 \
> >>>>>>>       -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
> >>>>>>>       -fsdev local,security_model=none,path=../../,id=shr0
> >>>>>>> 
> >>>>>>> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.        
> >>>>>> 
> >>>>>> I'm a little curious why optee is relevant for this but shouldn't matter as long
> >>>>>> as an appropriate EDK2 is loaded.
> >>>>>>   
> >>>>> 
> >>>>> I picked up your tree’s “master” and “cxl-next” as of today, and only the latter at least booted.
> >>>>> The former gives:
> >>>>> 
> >>>>> qemu-system-aarch64: Property 'virt-9.2-machine.cxl' not found
> >>>>> 
> >>>>> Should I stick with the cxl-next? My concern is that the base QEMU version is a bit old
> >>>>> 7.0.50.      
> >>>> 
> >>>> Always use the latest dated branch on that tree.  I release whenever there
> >>>> is something new to carry or a major rebase needed.
> >>>> 
> >>>> cxl-<date> is the right branch to use. Hope that helps.      
> >>> 
> >>> When do you think you want to get them (aarch64 specific?) merged mainline. Any reason you want to carry the patches by yourself?    
> >> 
> >> Nothing much has changed since I presented on this at Linaro connect in 2023.
> >> https://resources.linaro.org/en/resource/hM986DSHfoTrZ98UjpvLg1
> >> 
> >> The issue is device tree bindings for PCI Expander bridgess and the fact that
> >> those need to be generated without the full enumeration that EDK2 is doing
> >> prior to ACPI final table builds. In order to move forward with that it
> >> needs a bunch of work to prove that we absolutely cannot get patches
> >> upstream to support kernel base enumeration and breaking up of the
> >> various resources (like EDK2 does).  
> > 
> > I was talking to Peter Maydell earlier and given developments in the last couple
> > of years that have by necessity been ACPI only in arm virt he is less
> > opposed to ACPI only features being added where device tree is challenging.
> > 
> > So we may be able to move forwards without device tree support.
> > 
> > The PXB enumeration question is also relevant for managing multiple
> > vIOMMUs to represent multiple physical IOMMUs with the correct isolation
> > and do it efficiently which is probably a more pressing usecase than CXL emulation.
> > The discussion was mainly about that usecase, but maybe it also unblocks
> > upstreaming this support.
> > 
> > Thanks,
> > 
> > Jonathan  
> 
> I finally made some CXL tests ran within the ndctl test framework along with the kernel modules (QEMU is on your cxl-2024-11-27 branch) on aarch64. However, the recent rebase cxl-2025-01-24 fails to start the system emulation
> 
> qemu-system-aarch64: Property 'virt-10.0-machine.cxl' not found
> 
> The build did not complain, what kind of tests you run against your periodical QEMU rebase?

On this run I was focused on chmu testing which will have included what you
are running into here.  It's made a little tricky by some local networking issues
so getting a branch up on gitlab is far from straight forward.

Ah. Looks like the push went wrong as that has 0 patches on top of main qemu branch
so that branch is effectively empty. Typo in what I pushed.
Sorry about that.

Note this is far from a production tree as it carries many patches with outstanding
review comments etc.  Mainly exists for the purposes of testing specific kernel
patches where the support is not ready for upstream QEMU yet.  It is not in any
sense a 'release' with the sort of testing that would imply, so mostly a case
of local smoke tests.

Any help with more comprehensive tests would be much appreciated. I know some other
folk in the CXL community are talking about that, but I don't think anyone has
it in place yet. It's hard as there are typically a lot of moving parts.

Pushed now.

Jonathan




> 
> Itaru. 
> 
> >   
> >> 
> >> Given PXB enumeration in kernel has some issues on ARM anyway (that you can paper
> >> over with _DSM 5 - it self requiring an extra patch that isn't upstreamable because
> >> of IO port issues) there is quite a bit of work needed, mostly not in QEMU.
> >> Or convince Peter and others that not all virt support needs DT bindings
> >> (note that PXB for PCIE has been supported for years without an DT support,
> >> just no one noticed!)
> >> 
> >> After that we'd need to figure out CXL DT bindings in general and add kernel
> >> code support - despite there being no known DT based CXL systems out there, so
> >> that is going to be hard to do.  Various CXL kernel maintainers have expressed
> >> they aren't against such support, but it's hardly going to be review priority
> >> (other than for me if someone else does the work!)
> >> 
> >> For me this isn't particularly high priority. The ARM bit is fairly easy to rebase.
> >> I would like to see it solved, but it is behind various other items on my
> >> backlog.
> >> 
> >> There are SBSA machine patches on list, but it's not a useful platform for
> >> CXL kernel code development because of the limited supported configurations
> >> (in keeping with the more or less fixed model that SBSA-ref uses).
> >> 
> >> Jonathan
> >> 
> >> 
> >>   
> >>> 
> >>> Itaru.
> >>>   
> >>>> 
> >>>> Jonathan
> >>>>   
> >>>>> 
> >>>>> Thanks,
> >>>>> Itaru.
> >>>>>   
> >>>>>> Jonathan
> >>>>>>   
> >>>>>>> 
> >>>>>>> Let me know which branch you were suggesting.
> >>>>>>> 
> >>>>>>> Thanks,
> >>>>>>> Itaru. 
> >>>>>>>   
> >>>>>>>> 
> >>>>>>>> Note my main development work is on arm64 so that tends to work
> >>>>>>>> more reliably than x86 which I only lightly test for stuff that
> >>>>>>>> isn't ready for upstream yet.
> >>>>>>>> 
> >>>>>>>> Give me a shout if you run into any problems.
> >>>>>>>> 
> >>>>>>>> The main blocker on upstreaming this is resolving the missing device tree
> >>>>>>>> support for PCI expander bridges.  I've not made any progress on this since
> >>>>>>>> talk at Linaro connect in 2023.
> >>>>>>>> 
> >>>>>>>> Jonathan
> >>>>>>>> 
> >>>>>>>>   
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>>> Thanks
> >>>>>>>>> Zhijian
> >>>>>>>>>   
> >>>>>>>>>> If there’s a WIP branch, a pointer would be appreciated.
> >>>>>>>>>> 
> >>>>>>>>>> Itaru            
> 
> 

Re: CXL emulation on aarch64

[Itaru Kitayama]

> On Jan 30, 2025, at 2:16, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> 
> On Wed, 29 Jan 2025 10:14:34 +0900
> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> 
>>> On Jan 22, 2025, at 23:07, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>> 
>>> On Fri, 17 Jan 2025 09:43:11 +0000
>>> Jonathan Cameron via <qemu-devel@nongnu.org> wrote:
>>> 
>>>> On Fri, 17 Jan 2025 10:13:41 +0900
>>>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>>>> 
>>>>>> On Jan 16, 2025, at 19:58, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>>>>> 
>>>>>> On Thu, 16 Jan 2025 15:04:53 +0900
>>>>>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>>>>>> 
>>>>>>> Hi Jonathan,
>>>>>>> 
>>>>>>>> On Jan 14, 2025, at 19:26, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>>>>>>> 
>>>>>>>> On Tue, 14 Jan 2025 12:03:03 +0900
>>>>>>>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>>>>>>>> 
>>>>>>>>> Hi Jonathan, 
>>>>>>>>> 
>>>>>>>>>> On Jan 10, 2025, at 21:31, Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> On Fri, 10 Jan 2025 09:20:54 +0000
>>>>>>>>>> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
>>>>>>>>>> 
>>>>>>>>>>> On 10/01/2025 13:29, Itaru Kitayama wrote:          
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> Is anybody working on the CXL emulation on aarch64?            
>>>>>>>>>>> 
>>>>>>>>>>> I'm not currently working on the CXL emulation on aarch64.
>>>>>>>>>>> 
>>>>>>>>>>> However, IIRC the CXL maintainer's tree should work.
>>>>>>>>>>> https://gitlab.com/jic23/qemu/          
>>>>>>>>>> 
>>>>>>>>>> Pick up latest branch from there. I'm prepping a rebased version
>>>>>>>>>> with some new stuff but might take a few more days.          
>>>>>>>>> 
>>>>>>>>> Thanks for sharing your work with us.  Your master and cxl-2024-11-27 branches give:
>>>>>>>>> 
>>>>>>>>> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not found        
>>>>>>>> 
>>>>>>>> cxl is a machine property not a accel one. So needs to be after virt
>>>>>>>> There are tests in the tree for bios tables. Copy the command line from those.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> My commands are below:
>>>>>>>>> $HOME/projects/qemu/build/qemu-system-aarch64 \
>>>>>>>>>     -M virt,virtualization=on,gic-version=3 \
>>>>>>>>>     -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
>>>>>>>>>     -accel tcg,cxl=on \
>>>>>>>>>     -nographic \
>>>>>>>>>     -bios $HOME/cca-v4/out/bin/flash.bin \
>>>>>>>>>     -kernel Image-cca \
>>>>>>>>>     -drive format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
>>>>>>>>>     -device virtio-blk-pci,drive=hd0 \
>>>>>>>>>     -append root=/dev/vda \
>>>>>>>>>     -nodefaults \
>>>>>>>>>     --serial tcp:localhost:54320 \
>>>>>>>>>      -serial tcp:localhost:54321 \
>>>>>>>>>      -append "root=/dev/vda earlycon console=hvc0" \
>>>>>>>>>      -device virtio-net-pci,netdev=net0 \
>>>>>>>>>      -netdev user,id=net0 \
>>>>>>>>>      -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
>>>>>>>>>      -fsdev local,security_model=none,path=../../,id=shr0
>>>>>>>>> 
>>>>>>>>> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.        
>>>>>>>> 
>>>>>>>> I'm a little curious why optee is relevant for this but shouldn't matter as long
>>>>>>>> as an appropriate EDK2 is loaded.
>>>>>>>> 
>>>>>>> 
>>>>>>> I picked up your tree’s “master” and “cxl-next” as of today, and only the latter at least booted.
>>>>>>> The former gives:
>>>>>>> 
>>>>>>> qemu-system-aarch64: Property 'virt-9.2-machine.cxl' not found
>>>>>>> 
>>>>>>> Should I stick with the cxl-next? My concern is that the base QEMU version is a bit old
>>>>>>> 7.0.50.      
>>>>>> 
>>>>>> Always use the latest dated branch on that tree.  I release whenever there
>>>>>> is something new to carry or a major rebase needed.
>>>>>> 
>>>>>> cxl-<date> is the right branch to use. Hope that helps.      
>>>>> 
>>>>> When do you think you want to get them (aarch64 specific?) merged mainline. Any reason you want to carry the patches by yourself?    
>>>> 
>>>> Nothing much has changed since I presented on this at Linaro connect in 2023.
>>>> https://resources.linaro.org/en/resource/hM986DSHfoTrZ98UjpvLg1
>>>> 
>>>> The issue is device tree bindings for PCI Expander bridgess and the fact that
>>>> those need to be generated without the full enumeration that EDK2 is doing
>>>> prior to ACPI final table builds. In order to move forward with that it
>>>> needs a bunch of work to prove that we absolutely cannot get patches
>>>> upstream to support kernel base enumeration and breaking up of the
>>>> various resources (like EDK2 does).  
>>> 
>>> I was talking to Peter Maydell earlier and given developments in the last couple
>>> of years that have by necessity been ACPI only in arm virt he is less
>>> opposed to ACPI only features being added where device tree is challenging.
>>> 
>>> So we may be able to move forwards without device tree support.
>>> 
>>> The PXB enumeration question is also relevant for managing multiple
>>> vIOMMUs to represent multiple physical IOMMUs with the correct isolation
>>> and do it efficiently which is probably a more pressing usecase than CXL emulation.
>>> The discussion was mainly about that usecase, but maybe it also unblocks
>>> upstreaming this support.
>>> 
>>> Thanks,
>>> 
>>> Jonathan  
>> 
>> I finally made some CXL tests ran within the ndctl test framework along with the kernel modules (QEMU is on your cxl-2024-11-27 branch) on aarch64. However, the recent rebase cxl-2025-01-24 fails to start the system emulation
>> 
>> qemu-system-aarch64: Property 'virt-10.0-machine.cxl' not found
>> 
>> The build did not complain, what kind of tests you run against your periodical QEMU rebase?
> 
> On this run I was focused on chmu testing which will have included what you
> are running into here.  It's made a little tricky by some local networking issues
> so getting a branch up on gitlab is far from straight forward.
> 
> Ah. Looks like the push went wrong as that has 0 patches on top of main qemu branch
> so that branch is effectively empty. Typo in what I pushed.
> Sorry about that.

Not at all. I confirmed that the same branch now boots fine.
 
> 
> Note this is far from a production tree as it carries many patches with outstanding
> review comments etc.  Mainly exists for the purposes of testing specific kernel
> patches where the support is not ready for upstream QEMU yet.  It is not in any
> sense a 'release' with the sort of testing that would imply, so mostly a case
> of local smoke tests.

I understand this.

Thanks,
Itaru.

> 
> Any help with more comprehensive tests would be much appreciated. I know some other
> folk in the CXL community are talking about that, but I don't think anyone has
> it in place yet. It's hard as there are typically a lot of moving parts.
> 
> Pushed now.
> 
> Jonathan
> 
> 
> 
> 
>> 
>> Itaru. 
>> 
>>> 
>>>> 
>>>> Given PXB enumeration in kernel has some issues on ARM anyway (that you can paper
>>>> over with _DSM 5 - it self requiring an extra patch that isn't upstreamable because
>>>> of IO port issues) there is quite a bit of work needed, mostly not in QEMU.
>>>> Or convince Peter and others that not all virt support needs DT bindings
>>>> (note that PXB for PCIE has been supported for years without an DT support,
>>>> just no one noticed!)
>>>> 
>>>> After that we'd need to figure out CXL DT bindings in general and add kernel
>>>> code support - despite there being no known DT based CXL systems out there, so
>>>> that is going to be hard to do.  Various CXL kernel maintainers have expressed
>>>> they aren't against such support, but it's hardly going to be review priority
>>>> (other than for me if someone else does the work!)
>>>> 
>>>> For me this isn't particularly high priority. The ARM bit is fairly easy to rebase.
>>>> I would like to see it solved, but it is behind various other items on my
>>>> backlog.
>>>> 
>>>> There are SBSA machine patches on list, but it's not a useful platform for
>>>> CXL kernel code development because of the limited supported configurations
>>>> (in keeping with the more or less fixed model that SBSA-ref uses).
>>>> 
>>>> Jonathan
>>>> 
>>>> 
>>>> 
>>>>> 
>>>>> Itaru.
>>>>> 
>>>>>> 
>>>>>> Jonathan
>>>>>> 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Itaru.
>>>>>>> 
>>>>>>>> Jonathan
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Let me know which branch you were suggesting.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Itaru. 
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Note my main development work is on arm64 so that tends to work
>>>>>>>>>> more reliably than x86 which I only lightly test for stuff that
>>>>>>>>>> isn't ready for upstream yet.
>>>>>>>>>> 
>>>>>>>>>> Give me a shout if you run into any problems.
>>>>>>>>>> 
>>>>>>>>>> The main blocker on upstreaming this is resolving the missing device tree
>>>>>>>>>> support for PCI expander bridges.  I've not made any progress on this since
>>>>>>>>>> talk at Linaro connect in 2023.
>>>>>>>>>> 
>>>>>>>>>> Jonathan
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Thanks
>>>>>>>>>>> Zhijian
>>>>>>>>>>> 
>>>>>>>>>>>> If there’s a WIP branch, a pointer would be appreciated.
>>>>>>>>>>>> 
>>>>>>>>>>>> Itaru