RE: [RFC PATCH 0/4] cxl: introduce CXL Virtualization module

[Dan Williams <dan.j.williams@intel.com>]

Dongsheng Yang wrote:
> Hi all:
> 	This patchset introduce cxlv module to allow user to
> create virtual cxl device. it's based linux6.7-rc5, you can
> get the code from https://github.com/DataTravelGuide/linux
> 
> 	As the real CXL device is not widely available now, we need
> some virtual cxl device to do uplayer software developing or
> testing. Qemu is good for functional testing, but not good
> for some performance testing.

How is it performance testing if it's just using host-DRAM? Is the use
case something like pinning the benchmark on Socket0 and target DRAM on
Socket1 as emulated CXL to approximate CXL bus latency?

> 
> 	The new CXLV module allow user to use the reserved RAM[1], to
> create virtual cxl device. When the cxlv module load, it will
> create a directory named as "cxl_virt" under /sys/devices/virtual:
> 
> 	"/sys/devices/virtual/cxl_virt/"
> 
> that's the top level device for all cxlv devices.
> At the same time, cxlv module will create a debugfs directory:
> 
> /sys/kernel/debug/cxl/cxlv
> ├── create
> └── remove
> 
> the create and remove debugfs file is the cxlv entry to create or remove
> a cxlv device.
> 
> 	Each cxlv device have its owned virtual pci related bridge and bus, cxlv
> will create a new root_port for the new cxlv device, setup cxl ports for
> dport and nvdimm-bridge. After that, we will add the virtual pci device,
> that will go into the cxl_pci_probe to setup new memdev.
> 
> 	Then we can see the cxl device with cxl list and use it as a real cxl
> device.
> 
>  $ echo "memstart=$((8*1024*1024*1024)),cxltype=3,pmem=1,memsize=$((2*1024*1024*1024))" > /sys/kernel/debug/cxl/cxlv/create

Are these ranges reserved out of the mmap at boot time? 

[..]
>  14 files changed, 2388 insertions(+), 148 deletions(-)

This seems like a lot of code for something that is mostly already
supported by tools/testing/cxl/ (cxl_test). That too creates virtual CXL
devices that support ABI flows that are difficult to support in QEMU.
The only thing missing for "performance / functional emulation" testing
today is backing the memory regions with accessible memory rather than
unusable address space.

It is also the case that the static nature of cxl_test topology
definition has already started to prove too limiting for some tests. So
an enhancement to make cxl_test more dynamic like your proposed command
interface is appealing.

One change to get cxl_test to get it to emulate with DRAM rather than
fake address space is to just fill cxl_mock_pool with addresses backed
by DRAM rather than addresses from an unused portion of the physical
address map.

Currently cxl_test defines address ranges that may be larger than what a
host or VM can support, so this would be a new cxl_test mode limited by
available / reserved memory capacity.

See cxl-create-region.sh for an example of the virtual CXL regions that
cxl_test creates today:

https://github.com/pmem/ndctl/blob/main/test/cxl-create-region.sh