Re: [PATCH v8 00/14] Enabling DCD emulation support in Qemu

[Alireza Sanaee <alireza.sanaee@huawei.com>]

On Thu, 23 May 2024 10:44:40 -0700
nifan.cxl@gmail.com wrote:

> From: Fan Ni <nifan.cxl@gmail.com>
> 
> A git tree of this series can be found here (with one extra commit on
> top for printing out accepted/pending extent list for testing): 
> https://github.com/moking/qemu/tree/dcd-v8-qapi
> 
> v7->v8:
> 
> This version carries over the following two patches from Gregory.
> 1. hw/cxl/mailbox: change CCI cmd set structure to be a member, not a
> reference https://gitlab.com/jic23/qemu/-/commit/f44ebc5a455ccdd6535879b0c5824e0d76b04da5
> 2. hw/cxl/mailbox: interface to add CCI commands to an existing CCI
> https://gitlab.com/jic23/qemu/-/commit/00a4dd8b388add03c588298f665ee918626296a5
> 
> Note, the above two patches are not directly related to DCD emulation.
> 
> All the following patches in this series are built on top of
> mainstream QEMU and the above two patches.
> 
> The most significant changes of v8 is in Patch 11 (Patch 9 in v7).
> Based on feedback from Markus and Jonathan, the QMP interfaces for
> adding and releasing DC extents have been redesigned and now they
> look like below,
> 
> # add a 128MB extent at offset 0 to region 0
> 	{ "execute": "cxl-add-dynamic-capacity",
> 	  "arguments": {
> 		  "path": "/machine/peripheral/cxl-memdev0",
>           "host-id":0,
>           "selection-policy": 'prescriptive',
> 		  "region": 0,
>           "tag": "",
> 		  "extents": [
> 		  {
> 			  "offset": 0,
> 			  "len": 134217728
> 		  }
> 		  ]
> 	  }
> 	}
> 
> Note: tag is optional.
> 
> # Release a 128MB extent at offset 0 from region 0
>  { "execute": "cxl-release-dynamic-capacity",
> 	  "arguments": {
> 		  "path": "/machine/peripheral/cxl-memdev0",
>           "host-id":0,
>           "removal-policy":"prescriptive",
>           "forced-removal": false,
>           "sanitize-on-release": false,
>           "region": 0,
>           "tag": "",
> 		  "extents": [
> 		  {
> 			  "offset": 0,
> 			  "len": 134217728
> 		  }
> 		  ]
> 	  }
> 	}
>     
> Note: removal-policy, sanitize-on-release and tag are optional.
>     
> Other changes include,
> 1. Applied tags to patches. 
> 2. Replaced error_setq with error_append_hint for
> cxl_create_dc_region error case in Patch 6 (Patch 4 in v7); (Zhijian
> Li) 3. Updated the error message to include region size information in
>     cxl_create_dc_region.
> 4. set range1_size_hi to 0 for DCD in build_dvsec. (Jonathan)
> 5. Several minor format fixes.
> 
> Thanks Markus, Jonathan, Gregory, and Zhijian for reviewing v7 and
> svetly Todorov for testing v7.
> 
> This series pass the same tests as v7 check the cover letter of v7 for
> more details. Additionally, we tested the QAPI interface for
> adding/releasing DC extents with optional input parameters.
>   
> 
> v7: https://lore.kernel.org/linux-cxl/5856b7a4-4082-465f-9f61-b1ec6c35ef0f@fujitsu.com/T/#mec4c85022ce28c80b241aaf2d5431cadaa45f097
> 
> 
> Fan Ni (12):
>   hw/cxl/cxl-mailbox-utils: Add dc_event_log_size field to output
>     payload of identify memory device command
>   hw/cxl/cxl-mailbox-utils: Add dynamic capacity region representative
>     and mailbox command support
>   include/hw/cxl/cxl_device: Rename mem_size as static_mem_size for
>     type3 memory devices
>   hw/mem/cxl_type3: Add support to create DC regions to type3 memory
>     devices
>   hw/mem/cxl-type3: Refactor ct3_build_cdat_entries_for_mr to take mr
>     size instead of mr as argument
>   hw/mem/cxl_type3: Add host backend and address space handling for DC
>     regions
>   hw/mem/cxl_type3: Add DC extent list representative and get DC
> extent list mailbox support
>   hw/cxl/cxl-mailbox-utils: Add mailbox commands to support
> add/release dynamic capacity response
>   hw/cxl/events: Add qmp interfaces to add/release dynamic capacity
>     extents
>   hw/mem/cxl_type3: Add DPA range validation for accesses to DC
> regions hw/cxl/cxl-mailbox-utils: Add superset extent release mailbox
> support hw/mem/cxl_type3: Allow to release extent superset in QMP
> interface
> 
> Gregory Price (2):
>   hw/cxl/mailbox: change CCI cmd set structure to be a member, not a
>     reference
>   hw/cxl/mailbox: interface to add CCI commands to an existing CCI
> 
>  hw/cxl/cxl-mailbox-utils.c  | 658
> +++++++++++++++++++++++++++++++++++- hw/mem/cxl_type3.c          |
> 634 ++++++++++++++++++++++++++++++++-- hw/mem/cxl_type3_stubs.c    |
> 25 ++ include/hw/cxl/cxl_device.h |  85 ++++-
>  include/hw/cxl/cxl_events.h |  18 +
>  qapi/cxl.json               | 143 ++++++++
>  6 files changed, 1511 insertions(+), 52 deletions(-)
> 

Hi Nifan,

I am trying to test this patchset with Ira's set on DCD, and I am
trying to work out everything by using sysfs rather than using tools
instead. I am not sure where things are going of the rail.

I am using this patchset
(https://lore.kernel.org/qemu-devel/20240523174651.1089554-2-nifan.cxl@gmail.com/) with Ira's v9 https://lore.kernel.org/all/20250413-dcd-type2-upstream-v9-0-1d4911a0b365@intel.com/

This my CXL config:
return [
    "-m",      "6G,maxmem=20G,slots=10",
    "-object", "memory-backend-ram,id=vmem0,share=on,size=2G",
    "-device", "pxb-cxl,bus_nr=23,bus=pcie.0,id=cxl.1",
    "-device","cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2", 
    "-device","cxl-type3,bus=root_port13,volatile-dc-memdev=vmem0,id=cxl-vmem0,num-dc-regions=2",
    "-M", "cxl=on,cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G" 
]


This is how I create a new CXL DC Region ready to work with.

# creating a region
region=$(cat
/sys/bus/cxl/devices/decoder0.0/create_dynamic_ram_a_region) 
echo $region

echo $region >
/sys/bus/cxl/devices/decoder0.0/create_dynamic_ram_a_region 
echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity 
echo 1 > /sys/bus/cxl/devices/$region/interleave_ways

echo "dynamic_ram_a" > /sys/bus/cxl/devices/decoder2.0/mode
echo $((256 << 20)) > /sys/bus/cxl/devices/decoder2.0/dpa_size

echo $((256 << 20)) > /sys/bus/cxl/devices/$region/size
echo "decoder2.0" > /sys/bus/cxl/devices/$region/target0
echo 1 > /sys/bus/cxl/devices/$region/commit

echo $region > /sys/bus/cxl/drivers/cxl_region/bind

After this I have two things created, region0 and dax_region0:

root@localhost:~# ls /sys/bus/cxl/devices/
dax_region0/    decoder1.0/     decoder1.2/     decoder2.0/
decoder2.2/     endpoint2/      nvdimm-bridge0/ region0/ decoder0.0/
 decoder1.1/     decoder1.3/     decoder2.1/     decoder2.3/     mem0/
         port1/          root0/

This is what I have in dax_region0:
root@localhost:~# ls /sys/bus/cxl/devices/dax_region0/
dax0.0  dax_region  devtype  driver  modalias  subsystem  uevent

Now I want to add an extent, and this is how I am doing it:
	
	{ "execute": "cxl-add-dynamic-capacity",
	  "arguments": {
		  "path": "/machine/peripheral/cxl-memdev0",
          "host-id":0,
          "selection-policy": 'prescriptive',
		  "region": 0,
          "tag": "",
		  "extents": [
		  {
			  "offset": 0,
			  "len": 134217728
		  }
		  ]
	  }
	}

Now I see the extent added in my region device:
root@localhost:~# ls /sys/bus/cxl/devices/dax_region0/
dax0.0  dax_region  devtype  driver  extent0.0  modalias  subsystem
uevent root@localhost:~# ls /sys/bus/cxl/devices/dax_region0/extent0.0/
length  offset  uevent

This is where things will go off the rails, at this point I want to
create a new dax device to use, but this part is unsuccessful. Here I
first add some size to the dax region created before: 

root@localhost:~# echo 67108864 > /sys/bus/dax/devices/dax0.0/size 
[264.539280] dax:alloc_dev_dax_range:1015: dax dax0.0: alloc range[0]:
0x0000000810000000:0x0000000813ffffff 

When I check the size everything looks OK: 
root@localhost:~# cat /sys/bus/dax/devices/dax0.0/size 
67108864

But when I want to bind it then it does not work:
root@localhost:~# echo "dax0.0" > /sys/bus/dax/drivers/device_dax/bind
-bash: echo: write error: No such device

I believe I am missing something here. Would be good if you can help
out here.

Thanks,
Alireza