Re: [RFC PATCH 0/4] cxl: introduce CXL Virtualization module

[Dongsheng Yang <dongsheng.yang@easystack.cn>]

在 2024/1/25 星期四 上午 11:49, Dan Williams 写道:
> Dongsheng Yang wrote:
>>
>>
>> 在 2024/1/4 星期四 上午 4:48, Dan Williams 写道:
>>> Dongsheng Yang wrote:
>>>> Hi all:
>>>> 	This patchset introduce cxlv module to allow user to
>>>> create virtual cxl device. it's based linux6.7-rc5, you can
>>>> get the code from https://github.com/DataTravelGuide/linux
>>>>
>>>> 	As the real CXL device is not widely available now, we need
>>>> some virtual cxl device to do uplayer software developing or
>>>> testing. Qemu is good for functional testing, but not good
>>>> for some performance testing.
>>>
>>> How is it performance testing if it's just using host-DRAM? Is the use
>>> case something like pinning the benchmark on Socket0 and target DRAM on
>>> Socket1 as emulated CXL to approximate CXL bus latency?
>>
>> Hi Dan,
>> 	I give an example as below, please check it inline.
>>>
>>>>
>>>> 	The new CXLV module allow user to use the reserved RAM[1], to
>>>> create virtual cxl device. When the cxlv module load, it will
>>>> create a directory named as "cxl_virt" under /sys/devices/virtual:
>>>>
>>>> 	"/sys/devices/virtual/cxl_virt/"
>>>>
>>>> that's the top level device for all cxlv devices.
>>>> At the same time, cxlv module will create a debugfs directory:
>>>>
>>>> /sys/kernel/debug/cxl/cxlv
>>>> ├── create
>>>> └── remove
>>>>
>>>> the create and remove debugfs file is the cxlv entry to create or remove
>>>> a cxlv device.
>>>>
>>>> 	Each cxlv device have its owned virtual pci related bridge and bus, cxlv
>>>> will create a new root_port for the new cxlv device, setup cxl ports for
>>>> dport and nvdimm-bridge. After that, we will add the virtual pci device,
>>>> that will go into the cxl_pci_probe to setup new memdev.
>>>>
>>>> 	Then we can see the cxl device with cxl list and use it as a real cxl
>>>> device.
>>>>
>>>>    $ echo "memstart=$((8*1024*1024*1024)),cxltype=3,pmem=1,memsize=$((2*1024*1024*1024))" > /sys/kernel/debug/cxl/cxlv/create
>>>
>>> Are these ranges reserved out of the mmap at boot time?
>>
>> Yes, it is reserved by memmap option in boot cmdline. I use memmap=8G$8G.
> 
> A faster way to get to a device-dax interface fronting reserved memory
> is to use the efi_fake_mem= command line option.
> 
> For example:
> 
>      efi_fake_mem=4G@13G:0x40000
> 
> ...assigns 4GB of System-RAM starting at the 13G physical offset with
> the EFI_MEMORY_SP attribute. By default the kernel creates device-dax
> devices for that dedicated memory.
> 
> For dax mapping performance testing you don't need any of the CXL
> driver infrastructure since the CXL driver has nothing to do with the
> data path.

Thanx for your information, I create cxlv because I think there could be 
some other use cases other than device-dax to use cxl memdev. In that 
way, we need emulate cxl memdev in cxl driver level.

If we always use cxl memdev by creating a region and creating a 
device-dax, I agree we dont need to emulate it in cxl driver level.

Thanx
>