CXL numa error on arm64 qemu virt machine

CXL numa error on arm64 qemu virt machine

[Yuquan Wang]

Hello, Jonathan

Recently I run some cxl tests on qemu virt(branch:cxl-2024-04-22-draft) but met some
problems.

Problems: 
1) the virt machine could not set the right numa topology from user input;

My Qemu numa set:
-object memory-backend-ram,size=2G,id=mem0 \
-numa node,nodeid=0,cpus=0-1,memdev=mem0 \
-object memory-backend-ram,size=2G,id=mem1 \
-numa node,nodeid=1,cpus=2-3,memdev=mem1 \

However, the system shows:
root@ubuntu-jammy-arm64:~# numactl -H
	available: 1 nodes (0)
	node 0 cpus: 0 1 2 3
	node 0 size: 4166 MB
	node 0 free: 3920 MB
	node distances:
	node   0 
	0:  10 

Boot Kernel print:
[    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x40000000-0xbfffffff]
[    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0xc0000000-0x13fffffff]
[    0.000000] ACPI: Unknown target node for memory at 0x10000000000, assuming node 0
[    0.000000] NUMA: Warning: invalid memblk node 16 [mem 0x0000000004000000-0x0000000007ffffff]
[    0.000000] NUMA: Faking a node at [mem 0x0000000004000000-0x000000013fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x13f7f89c0-0x13f7fafff]

2) it seems like the problem of allocating numa node in arm for cxl memory still exists;
Previous discussion: https://lore.kernel.org/linux-cxl/20231011150620.0000212a@Huawei.com/

root@debian-bullseye-arm64:~# cxl create-region -d decoder0.0 -t ram
[   68.653873] cxl region0: Bypassing cpu_cache_invalidate_memregion() for testing!
[   68.660568] Unknown target node for memory at 0x10000000000, assuming node 0

If not, maybe I could try to do something to help fix this problem.


My full qemu command line:
qemu-system-aarch64 \
-M virt,gic-version=3,cxl=on \
-m 4G \
-smp 4 \
-object memory-backend-ram,size=2G,id=mem0 \
-numa node,nodeid=0,cpus=0-1,memdev=mem0 \
-object memory-backend-ram,size=2G,id=mem1 \
-numa node,nodeid=1,cpus=2-3,memdev=mem1 \
-cpu cortex-a57 \
-bios QEMU_EFI.fd.bak \
-device virtio-blk-pci,drive=hd,bus=pcie.0 \
-drive if=none,id=hd,file=../disk/debos_arm64.ext \
-nographic \
-object memory-backend-file,id=mem2,mem-path=/tmp/mem2,size=256M,share=true \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
-device cxl-type3,bus=root_port13,volatile-memdev=mem2,id=cxl-mem1 \
-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G \
-qmp tcp:127.0.0.1:4444,server,nowait \

Qemu version: the lastest commit of branch cxl-2024-04-22-draft in "https://gitlab.com/jic23/qemu" 
Kernel version: 6.6.0

Many thanks
Yuquan

Re: CXL numa error on arm64 qemu virt machine

[Jonathan Cameron via]

On Wed, 8 May 2024 16:00:51 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> Hello, Jonathan
> 
> Recently I run some cxl tests on qemu virt(branch:cxl-2024-04-22-draft) but met some
> problems.
> 
> Problems: 
> 1) the virt machine could not set the right numa topology from user input;
> 
> My Qemu numa set:
> -object memory-backend-ram,size=2G,id=mem0 \
> -numa node,nodeid=0,cpus=0-1,memdev=mem0 \
> -object memory-backend-ram,size=2G,id=mem1 \
> -numa node,nodeid=1,cpus=2-3,memdev=mem1 \

That is setting up the main DRAM nodes, unrelated to
CXL memory. For CXL memory you need to use generic
port entries (in my gitlab qemu tree - with examples but not upstream yet)
However, if you get some breakage

> 
> However, the system shows:
> root@ubuntu-jammy-arm64:~# numactl -H
> 	available: 1 nodes (0)
> 	node 0 cpus: 0 1 2 3
> 	node 0 size: 4166 MB
> 	node 0 free: 3920 MB
> 	node distances:
> 	node   0 
> 	0:  10 
> 
> Boot Kernel print:
> [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x40000000-0xbfffffff]
> [    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0xc0000000-0x13fffffff]
> [    0.000000] ACPI: Unknown target node for memory at 0x10000000000, assuming node 0
> [    0.000000] NUMA: Warning: invalid memblk node 16 [mem 0x0000000004000000-0x0000000007ffffff]
> [    0.000000] NUMA: Faking a node at [mem 0x0000000004000000-0x000000013fffffff]
> [    0.000000] NUMA: NODE_DATA [mem 0x13f7f89c0-0x13f7fafff]
> 
> 2) it seems like the problem of allocating numa node in arm for cxl memory still exists;
> Previous discussion: https://lore.kernel.org/linux-cxl/20231011150620.0000212a@Huawei.com/
> 
> root@debian-bullseye-arm64:~# cxl create-region -d decoder0.0 -t ram
> [   68.653873] cxl region0: Bypassing cpu_cache_invalidate_memregion() for testing!
> [   68.660568] Unknown target node for memory at 0x10000000000, assuming node 0

You need a load of kernel changes for NUMA nodes to work correctly with
CXL memory on arm64 platforms.  I have some working code but need to tidy
up a few corners that came up in an internal review earlier this week.

I have some travel coming up so may take a week or so to get those out.

Curiously that invalid memblk has nothing to do with the CXL fixed memory window
Could you check if that is happening for you without the CXL patches?

> 
> If not, maybe I could try to do something to help fix this problem.
> 
> 
> My full qemu command line:
> qemu-system-aarch64 \
> -M virt,gic-version=3,cxl=on \
> -m 4G \
> -smp 4 \
> -object memory-backend-ram,size=2G,id=mem0 \
> -numa node,nodeid=0,cpus=0-1,memdev=mem0 \
> -object memory-backend-ram,size=2G,id=mem1 \
> -numa node,nodeid=1,cpus=2-3,memdev=mem1 \
> -cpu cortex-a57 \
> -bios QEMU_EFI.fd.bak \
> -device virtio-blk-pci,drive=hd,bus=pcie.0 \
> -drive if=none,id=hd,file=../disk/debos_arm64.ext \
> -nographic \
> -object memory-backend-file,id=mem2,mem-path=/tmp/mem2,size=256M,share=true \
> -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
> -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
> -device cxl-type3,bus=root_port13,volatile-memdev=mem2,id=cxl-mem1 \
> -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G \
> -qmp tcp:127.0.0.1:4444,server,nowait \
> 
> Qemu version: the lastest commit of branch cxl-2024-04-22-draft in "https://gitlab.com/jic23/qemu" 
> Kernel version: 6.6.0

Whilst it doesn't work yet (because of missing kernel support)
you'll need something that looks more like the generic ports test added in 
https://gitlab.com/jic23/qemu/-/commit/6589c527920ba22fe0923b60b58d33a8e9fd371e

Most importantly
-numa node,nodeid=2 -object acpi-generic-port,id=gp0,pci-bus-cxl.1,node=2
+ the bits setting distances etc.  Note CXL memory does not provide SLIT like
data at the moment, so the test above won't help you identify if it is correctly
set up.  That's a gap in general in the kernel support. Whilst we'd love
it if everyone moved to hmat derived information we may need to provide
some fallback.

Jonathan




> 
> Many thanks
> Yuquan
> 

Re: CXL numa error on arm64 qemu virt machine

[Yuquan Wang]

On Wed, May 08, 2024 at 01:02:52PM +0100, Jonathan Cameron wrote:
> 
> > [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x40000000-0xbfffffff]
> > [    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0xc0000000-0x13fffffff]
> > [    0.000000] ACPI: Unknown target node for memory at 0x10000000000, assuming node 0
> > [    0.000000] NUMA: Warning: invalid memblk node 16 [mem 0x0000000004000000-0x0000000007ffffff]
> > [    0.000000] NUMA: Faking a node at [mem 0x0000000004000000-0x000000013fffffff]
> > [    0.000000] NUMA: NODE_DATA [mem 0x13f7f89c0-0x13f7fafff]
> > 
> > Previous discussion: https://lore.kernel.org/linux-cxl/20231011150620.0000212a@Huawei.com/
> > 
> > root@debian-bullseye-arm64:~# cxl create-region -d decoder0.0 -t ram
> > [   68.653873] cxl region0: Bypassing cpu_cache_invalidate_memregion() for testing!
> > [   68.660568] Unknown target node for memory at 0x10000000000, assuming node 0
> 
> You need a load of kernel changes for NUMA nodes to work correctly with
> CXL memory on arm64 platforms.  I have some working code but need to tidy
> up a few corners that came up in an internal review earlier this week.
> 
> I have some travel coming up so may take a week or so to get those out.
> 
> Curiously that invalid memblk has nothing to do with the CXL fixed memory window
> Could you check if that is happening for you without the CXL patches?
> 

Thanks.

I have checked it, the problem is caused by my bios firmware file. I
change the bios file and the numa topology can works.

BTW, if it is convenient, could you help to post the link of the patches of CXL 
memory NUMA nodes on arm64 platforms?

> 
> Whilst it doesn't work yet (because of missing kernel support)
> you'll need something that looks more like the generic ports test added in 
> https://gitlab.com/jic23/qemu/-/commit/6589c527920ba22fe0923b60b58d33a8e9fd371e
> 
> Most importantly
> -numa node,nodeid=2 -object acpi-generic-port,id=gp0,pci-bus-cxl.1,node=2
> + the bits setting distances etc.  Note CXL memory does not provide SLIT like
> data at the moment, so the test above won't help you identify if it is correctly
> set up.  That's a gap in general in the kernel support. Whilst we'd love
> it if everyone moved to hmat derived information we may need to provide
> some fallback.
> 
> Jonathan
> 

Many thanks
Yuquan

Re: CXL numa error on arm64 qemu virt machine

[Jonathan Cameron via]

On Thu, 9 May 2024 16:35:34 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> On Wed, May 08, 2024 at 01:02:52PM +0100, Jonathan Cameron wrote:
> >   
> > > [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x40000000-0xbfffffff]
> > > [    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0xc0000000-0x13fffffff]
> > > [    0.000000] ACPI: Unknown target node for memory at 0x10000000000, assuming node 0
> > > [    0.000000] NUMA: Warning: invalid memblk node 16 [mem 0x0000000004000000-0x0000000007ffffff]
> > > [    0.000000] NUMA: Faking a node at [mem 0x0000000004000000-0x000000013fffffff]
> > > [    0.000000] NUMA: NODE_DATA [mem 0x13f7f89c0-0x13f7fafff]
> > > 
> > > Previous discussion: https://lore.kernel.org/linux-cxl/20231011150620.0000212a@Huawei.com/
> > > 
> > > root@debian-bullseye-arm64:~# cxl create-region -d decoder0.0 -t ram
> > > [   68.653873] cxl region0: Bypassing cpu_cache_invalidate_memregion() for testing!
> > > [   68.660568] Unknown target node for memory at 0x10000000000, assuming node 0  
> > 
> > You need a load of kernel changes for NUMA nodes to work correctly with
> > CXL memory on arm64 platforms.  I have some working code but need to tidy
> > up a few corners that came up in an internal review earlier this week.
> > 
> > I have some travel coming up so may take a week or so to get those out.
> > 
> > Curiously that invalid memblk has nothing to do with the CXL fixed memory window
> > Could you check if that is happening for you without the CXL patches?
> >   
> 
> Thanks.
> 
> I have checked it, the problem is caused by my bios firmware file. I
> change the bios file and the numa topology can works.
> 
> BTW, if it is convenient, could you help to post the link of the patches of CXL 
> memory NUMA nodes on arm64 platforms?

https://git.kernel.org/pub/scm/linux/kernel/git/jic23/cxl-staging.git/log/?h=arm-numa-fixes

I've run out of time to sort out cover letters and things + just before the merge
window is never a good time get anyone to pay attention to potentially controversial
patches.  So for now I've thrown up a branch on kernel.org with Robert's
series of fixes of related code (that's queued in the ACPI tree for the merge window)
and Dan Williams (from several years ago) + my additions that 'work' (lightly tested)
on qemu/arm64 with the generic port patches etc. 

I'll send out an RFC in a couple of weeks.  In meantime let me know if you
run into any problems or have suggestions to improve them.

Jonathan

p.s. Apparently my computer is in the future. (-28 minutes and counting!)
> 
> > 
> > Whilst it doesn't work yet (because of missing kernel support)
> > you'll need something that looks more like the generic ports test added in 
> > https://gitlab.com/jic23/qemu/-/commit/6589c527920ba22fe0923b60b58d33a8e9fd371e
> > 
> > Most importantly
> > -numa node,nodeid=2 -object acpi-generic-port,id=gp0,pci-bus-cxl.1,node=2
> > + the bits setting distances etc.  Note CXL memory does not provide SLIT like
> > data at the moment, so the test above won't help you identify if it is correctly
> > set up.  That's a gap in general in the kernel support. Whilst we'd love
> > it if everyone moved to hmat derived information we may need to provide
> > some fallback.
> > 
> > Jonathan
> >   
> 
> Many thanks
> Yuquan
> 

Re: CXL numa error on arm64 qemu virt machine

[Yuquan Wang]

On Fri, May 10, 2024 at 06:16:46PM +0100, Jonathan Cameron wrote:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/jic23/cxl-staging.git/log/?h=arm-numa-fixes
> 
Thank you :)
> I've run out of time to sort out cover letters and things + just before the merge
> window is never a good time get anyone to pay attention to potentially controversial
> patches.  So for now I've thrown up a branch on kernel.org with Robert's
> series of fixes of related code (that's queued in the ACPI tree for the merge window)
> and Dan Williams (from several years ago) + my additions that 'work' (lightly tested)
> on qemu/arm64 with the generic port patches etc. 
> 
> I'll send out an RFC in a couple of weeks.  In meantime let me know if you
> run into any problems or have suggestions to improve them.
> 
> Jonathan
>
With the latest commit(d077bf9) in the 'arm-numa-fixes', the qemu virt
could create a cxl region with a new numa node (node 2) just like x86.
At this stage(the first time to create cxl region), everything works
fine.

However, if I use below commands to delete the created cxl region:

`daxctl offline-memory dax0.0`
`cxl disable-region region0`
`cxl destroy-region region0`

and then recreate it by `cxl create-region -d decoder0.0 -t ram`, the
kernel could not create the numa node2 again, and the kernel will print:

[  589.458971] Fallback order for Node 0: 0 1
[  589.459136] Fallback order for Node 1: 1 0
[  589.459175] Fallback order for Node 2: 0 1
[  589.459213] Built 2 zonelists, mobility grouping on.  Total pages: 1009890
[  589.459284] Policy zone: Normal

Meanwhile, the qemu reports that: 

"qemu-system-aarch64: virtio: bogus descriptor or out of resources"

Many thanks
Yuquan

Re: CXL numa error on arm64 qemu virt machine

[Jonathan Cameron via]

On Fri, 17 May 2024 18:07:07 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:

> On Fri, May 10, 2024 at 06:16:46PM +0100, Jonathan Cameron wrote:
> > 
> > https://git.kernel.org/pub/scm/linux/kernel/git/jic23/cxl-staging.git/log/?h=arm-numa-fixes
> >   
> Thank you :)
> > I've run out of time to sort out cover letters and things + just before the merge
> > window is never a good time get anyone to pay attention to potentially controversial
> > patches.  So for now I've thrown up a branch on kernel.org with Robert's
> > series of fixes of related code (that's queued in the ACPI tree for the merge window)
> > and Dan Williams (from several years ago) + my additions that 'work' (lightly tested)
> > on qemu/arm64 with the generic port patches etc. 
> > 
> > I'll send out an RFC in a couple of weeks.  In meantime let me know if you
> > run into any problems or have suggestions to improve them.
> > 
> > Jonathan
> >  
> With the latest commit(d077bf9) in the 'arm-numa-fixes', the qemu virt
> could create a cxl region with a new numa node (node 2) just like x86.
> At this stage(the first time to create cxl region), everything works
> fine.
> 
> However, if I use below commands to delete the created cxl region:
> 
> `daxctl offline-memory dax0.0`
> `cxl disable-region region0`
> `cxl destroy-region region0`
> 
> and then recreate it by `cxl create-region -d decoder0.0 -t ram`, the
> kernel could not create the numa node2 again, and the kernel will print:
> 
> [  589.458971] Fallback order for Node 0: 0 1
> [  589.459136] Fallback order for Node 1: 1 0
> [  589.459175] Fallback order for Node 2: 0 1
> [  589.459213] Built 2 zonelists, mobility grouping on.  Total pages: 1009890
> [  589.459284] Policy zone: Normal

I'll see if I can figure out what is happening there.
> 
> Meanwhile, the qemu reports that: 
> 
> "qemu-system-aarch64: virtio: bogus descriptor or out of resources"

That sounds like another TCG issue, or possibly the DMA bounce buffer
problem resurfacing.  It's not directly related to his NUMA aspect unless
something very odd is going on.  I'm even more confused because I think
you are not using kmem with the above commands, so we shouldn't be using
the CXL memory for virtio.

Just to check, you aren't running with KVM I hope?  That opens a much
bigger problem set. :(

Jonathan



> 
> Many thanks
> Yuquan
> 

Re: CXL numa error on arm64 qemu virt machine

[Jonathan Cameron via]

On Fri, 17 May 2024 11:14:41 +0100
Jonathan Cameron <Jonathan.Cameron@Huawei.com> wrote:

> On Fri, 17 May 2024 18:07:07 +0800
> Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:
> 
> > On Fri, May 10, 2024 at 06:16:46PM +0100, Jonathan Cameron wrote:  
> > > 
> > > https://git.kernel.org/pub/scm/linux/kernel/git/jic23/cxl-staging.git/log/?h=arm-numa-fixes
> > >     
> > Thank you :)  
> > > I've run out of time to sort out cover letters and things + just before the merge
> > > window is never a good time get anyone to pay attention to potentially controversial
> > > patches.  So for now I've thrown up a branch on kernel.org with Robert's
> > > series of fixes of related code (that's queued in the ACPI tree for the merge window)
> > > and Dan Williams (from several years ago) + my additions that 'work' (lightly tested)
> > > on qemu/arm64 with the generic port patches etc. 
> > > 
> > > I'll send out an RFC in a couple of weeks.  In meantime let me know if you
> > > run into any problems or have suggestions to improve them.
> > > 
> > > Jonathan
> > >    
> > With the latest commit(d077bf9) in the 'arm-numa-fixes', the qemu virt
> > could create a cxl region with a new numa node (node 2) just like x86.
> > At this stage(the first time to create cxl region), everything works
> > fine.
> > 
> > However, if I use below commands to delete the created cxl region:
> > 
> > `daxctl offline-memory dax0.0`
> > `cxl disable-region region0`
> > `cxl destroy-region region0`
> > 
> > and then recreate it by `cxl create-region -d decoder0.0 -t ram`, the
> > kernel could not create the numa node2 again, and the kernel will print:
> > 
> > [  589.458971] Fallback order for Node 0: 0 1
> > [  589.459136] Fallback order for Node 1: 1 0
> > [  589.459175] Fallback order for Node 2: 0 1
> > [  589.459213] Built 2 zonelists, mobility grouping on.  Total pages: 1009890
> > [  589.459284] Policy zone: Normal  
> 
> I'll see if I can figure out what is happening there.

So I know what is happening but not sure on the solution yet.
The issue is on unbind of the region there is a call to try_remove_memory()
and that calls memblock_phys_free(). That removes the reserved memblocks being used
for tracking the numa node, so when you bind a region at that HPA again, there
is no tracking information.

So far I haven't figured out why that call is there in the first place
which isn't helping me solve this.

https://elixir.bootlin.com/linux/v6.9.1/source/mm/memory_hotplug.c#L2286

Until I get this code out there, kind of hard to ask the mm folk
- for now I may just have to say it only works once and point at that
line as the problem in an RFC.

Long shot, but Dan, did you run into this when you were doing your 
[PATCH v2 08/22] memblock: Introduce a generic phys_addr_to_target_node()
stuff?  I assume that ultimately called try_remove_memory() in a remove
path somewhere and similarly to this if you try putting it back it
would be missing.  Or alternatively, any idea why what that memblock_phys_free()
is balancing with?

Jonathan




> > 
> > Meanwhile, the qemu reports that: 
> > 
> > "qemu-system-aarch64: virtio: bogus descriptor or out of resources"  
> 
> That sounds like another TCG issue, or possibly the DMA bounce buffer
> problem resurfacing.  It's not directly related to his NUMA aspect unless
> something very odd is going on.  I'm even more confused because I think
> you are not using kmem with the above commands, so we shouldn't be using
> the CXL memory for virtio.
> 
> Just to check, you aren't running with KVM I hope?  That opens a much
> bigger problem set. :(
> 
> Jonathan
> 
> 
> 
> > 
> > Many thanks
> > Yuquan
> >   
> 
>