Questions about the memory of CXL Type3 device offline

Questions about the memory of CXL Type3 device offline

[Yuquan Wang]

Hello, everyone!

I have used Qemu to do some CXL tests and I failed with configuring
a CXL region and numa node. 

My qemu command line:
qemu-system-x86_64 \
-M q35,nvdimm=on,cxl=on \
-m 4G \
-smp 4 \
-object memory-backend-ram,size=2G,id=mem0 \
-numa node,nodeid=0,cpus=0-1,memdev=mem0 \
-object memory-backend-ram,size=2G,id=mem1 \
-numa node,nodeid=1,cpus=2-3,memdev=mem1 \
-object memory-backend-ram,size=256M,id=cxl-mem1 \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1,addr=04 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
-device cxl-type3,bus=root_port13,volatile-memdev=cxl-mem1,id=cxl-mem1 \
-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G \
-hda ./disk/ubuntu_x86_test_dup.qcow2 \
-nographic 

Qemu version: 8.1.50, commit-9ef497755afc252fb8e060c9ea6b0987abfd20b6
Kernel version: 6.5.0
ndctl version: 77

My steps in the Linux system:
1) root@mali:/home/mali# ./cxl list
[
  {
    "memdev":"mem0",
    "ram_size":268435456,
    "serial":0,
    "host":"0000:0d:00.0"
  }
]

2) root@mali:/home/mali# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1
node 0 size: 1921 MB
node 0 free: 1705 MB
node 1 cpus: 2 3
node 1 size: 2014 MB
node 1 free: 1834 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 

3) root@mali:/home/mali# ./cxl create-region -d decoder0.0 -t ram
{
  "region":"region0",
  "resource":"0x190000000",
  "size":"256.00 MiB (268.44 MB)",
  "type":"ram",
  "interleave_ways":1,
  "interleave_granularity":256,
  "decode_state":"commit",
  "mappings":[
    {
      "position":0,
      "memdev":"mem0",
      "decoder":"decoder2.0"
    }
  ]
}
cxl region: cmd_create_region: created 1 region
[  126.862782] cxl region0: Bypassing cpu_cache_invalidate_memregion() for testing!
[  126.916152] Fallback order for Node 2: 2 0 1 
[  126.916699] Built 2 zonelists, mobility grouping on.  Total pages: 982290
[  126.916877] Policy zone: Normal

The results:
root@mali:/home/mali# numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0 1
node 0 size: 1950 MB
node 0 free: 1785 MB
node 1 cpus: 2 3
node 1 size: 1985 MB
node 1 free: 1790 MB
node 2 cpus:
node 2 size: 0 MB
node 2 free: 0 MB
node distances:
node   0   1   2 
  0:  10  20  20 
  1:  20  10  20 
  2:  20  20  10 

root@mali:/home/mali# lsmem --output-all
RANGE                                  SIZE   STATE REMOVABLE BLOCK NODE          ZONES
0x0000000000000000-0x0000000007ffffff  128M  online       yes     0    0           None
0x0000000008000000-0x000000007fffffff  1.9G  online       yes  1-15    0          DMA32
0x0000000100000000-0x000000017fffffff    2G  online       yes 32-47    1         Normal
0x0000000190000000-0x000000019fffffff  256M offline           50-51    2 Normal/Movable

root@mali:/home/mali# cat /proc/iomem
190000000-28fffffff : CXL Window 0
  190000000-19fffffff : region0
    190000000-19fffffff : dax0.0
      190000000-19fffffff : System RAM (kmem)
2c0000000-abfffffff : PCI Bus 0000:00
fd00000000-ffffffffff : Reserved

Questions:
1)Why the meomory of CXL node is still offline?
2)Is my configuration process standard?
Hope I can get some helps here, any help will be greatly appreciated.

Many thanks
Yuquan

Re: Questions about the memory of CXL Type3 device offline

[Gregory Price]

On Mon, Oct 09, 2023 at 08:39:47PM +0800, Yuquan Wang wrote:
> Hello, everyone!
> 
> Questions:
> 1)Why the meomory of CXL node is still offline?

because you likely have auto-online of new memory blocks sets to false.

use `daxctl list` and  `daxctl online-memory [dax-device]` to online the memory

> 2)Is my configuration process standard?

looks fine to me 

Re: Questions about the memory of CXL Type3 device offline

[Yuquan Wang]

On 2023-10-07 18:56,  gregory.price@memverge.com wrote:
> because you likely have auto-online of new memory blocks sets to false.
> 
> use `daxctl list` and  `daxctl online-memory [dax-device]` to online the memory

Thank you Gregory. It works! :)

BTW, I aslo did the same test in 'virt'(arm64) on Jonathan's development branch, 
however, it would automatically allcate CXL memory to numa node 0:

root@debian-bullseye-arm64:~# cxl create-region -d decoder0.0 -t ram
[   68.653873] cxl region0: Bypassing cpu_cache_invalidate_memregion() for testing!
[   68.660568] Unknown target node for memory at 0x10000000000, assuming node 0
{
  "region":"region0",
  "resource":"0x10000000000",
  "size":"256.00 MiB (268.44 MB)",
  "type":"ram",
  "interleave_ways":1,
  "interleave_granularity":256,
  "decode_state":"commit",
  "mappings":[
    {
      "position":0,
      "memdev":"mem0",
      "decoder":"decoder2.0"
    }
  ]
}
cxl region: cmd_create_region: created 1 region

In addition, with 'daxclt online-memory' I can online the memory on node 0:
1)root@debian-bullseye-arm64:~# daxctl online-memory dax0.0
[  144.137685] Fallback order for Node 0: 0 1 
[  144.137819] Fallback order for Node 1: 1 0 
[  144.137872] Built 2 zonelists, mobility grouping on.  Total pages: 1011494
[  144.139050] Policy zone: Normal
onlined memory for 1 device

2)root@debian-bullseye-arm64:~# lsmem --output-all
RANGE                                  SIZE  STATE REMOVABLE     BLOCK NODE ZONES
0x0000000040000000-0x00000000bfffffff    2G online       yes      8-23    0 
0x00000000c0000000-0x000000013fffffff    2G online       yes     24-39    1 
0x0000010000000000-0x000001000fffffff  256M online       yes 8192-8193    0 

Memory block size:       128M
Total online memory:     4.3G
Total offline memory:      0B

With Jonathan's qemu branch(cxl-2023-09-26):https://gitlab.com/jic23/qemu.git
My qemu command line:
qemu-system-aarch64 \   
-M virt,gic-version=3,cxl=on \
-m 4G \
-object memory-backend-ram,size=2G,id=mem0 \
-numa node,nodeid=0,cpus=0-3,memdev=mem0 \
-object memory-backend-ram,size=2G,id=mem1 \
-numa node,nodeid=1,cpus=4-7,memdev=mem1 \
-cpu cortex-a57 \
-bios QEMU_EFI.fd \
-smp 8 \
-device virtio-blk-pci,drive=hd,bus=pcie.0 \
-drive if=none,id=hd,file=./disk/full-rootfs-debian-bullseye-arm64.ext4 \
-nographic \
-object memory-backend-ram,size=256M,id=cxl-mem1 \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
-device cxl-type3,bus=root_port13,volatile-memdev=cxl-mem1,id=cxl-mem1 \
-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G \

Is there anything else I missed in this environment?
Maybe Jonathan has already helped someone else with a similar problem :)


Many thanks
Yuquan

Re: Questions about the memory of CXL Type3 device offline

[Gregory Price]

On Tue, Oct 10, 2023 at 02:46:48PM +0800, Yuquan Wang wrote:
> On 2023-10-07 18:56,  gregory.price@memverge.com wrote:
> > because you likely have auto-online of new memory blocks sets to false.
> > 
> > use `daxctl list` and  `daxctl online-memory [dax-device]` to online the memory
> 
> Thank you Gregory. It works! :)
> 
> BTW, I aslo did the same test in 'virt'(arm64) on Jonathan's development branch, 
> however, it would automatically allcate CXL memory to numa node 0:
> 

I haven't done any work with the arm machine.  Jonathan might have
input.

But if the memory comes up, it's probably fine.  If the memory comes up
automatically it's because you have auto-online turned on.

~Gregory

Re: Questions about the memory of CXL Type3 device offline

[Jonathan Cameron]

On Sun, 8 Oct 2023 09:30:33 -0400
Gregory Price <gregory.price@memverge.com> wrote:

> On Tue, Oct 10, 2023 at 02:46:48PM +0800, Yuquan Wang wrote:
> > On 2023-10-07 18:56,  gregory.price@memverge.com wrote:  
> > > because you likely have auto-online of new memory blocks sets to false.
> > > 
> > > use `daxctl list` and  `daxctl online-memory [dax-device]` to online the memory  
> > 
> > Thank you Gregory. It works! :)
> > 
> > BTW, I aslo did the same test in 'virt'(arm64) on Jonathan's development branch, 
> > however, it would automatically allcate CXL memory to numa node 0:
> >   
> 
> I haven't done any work with the arm machine.  Jonathan might have
> input.
> 
> But if the memory comes up, it's probably fine.  If the memory comes up
> automatically it's because you have auto-online turned on.

ARMs management of numa nodes is a bit simplistic currently as it doesn't
keep as much information around as x86 does.  That's something I've been
meaning to fix (in kernel) but not done so yet.

J
> 
> ~Gregory