Re: [RFC PATCH 0/4] cxl: introduce CXL Virtualization module

[Jonathan Cameron <Jonathan.Cameron@Huawei.com>]

On Wed, 3 Jan 2024 09:22:36 -0800
Ira Weiny <ira.weiny@intel.com> wrote:

> Dongsheng Yang wrote:
> > Hi all:
> > 	This patchset introduce cxlv module to allow user to
> > create virtual cxl device. it's based linux6.7-rc5, you can
> > get the code from https://github.com/DataTravelGuide/linux
> > 
> > 	As the real CXL device is not widely available now, we need
> > some virtual cxl device to do uplayer software developing or
> > testing. Qemu is good for functional testing, but not good
> > for some performance testing.  
> 
> Do you have more details on what performance is missing from Qemu and why
> this solution is better than a solution to fix Qemu?
> 
> Long term it seems better to fix Qemu for this type of work.

I plan to look at this sometime soon, but note that the fix will be special
cases only (no interleave!) in the short term.  Emulating interleave
is always going to be costly - can probably be better than we have it today
for large granularity (pages) but I'm not sure we will ever care enough
to implement that.

For virtualization usecases, if we go with CXL emulation as the path for
DCD then we'll just emulate direct connected devices and patch up the
perf characteristics to cover interleave, switches etc.

With such limitations we can get QEMU to perform well. I'm not keen on
separating QEMU for functional testing from QEMU for workload testing
but meh, we can at least make it automatic to use a higher perf root
if the interleave config allows it.

> 
> Are their other advantages to having this additional test infrastructure
> in the kernel?  We already have cxl_test.
> 
> Ira
> 
> > 
> > 	The new CXLV module allow user to use the reserved RAM[1], to
> > create virtual cxl device. When the cxlv module load, it will
> > create a directory named as "cxl_virt" under /sys/devices/virtual:
> > 
> > 	"/sys/devices/virtual/cxl_virt/"
> > 
> > that's the top level device for all cxlv devices.
> > At the same time, cxlv module will create a debugfs directory:
> > 
> > /sys/kernel/debug/cxl/cxlv
> > ├── create
> > └── remove
> > 
> > the create and remove debugfs file is the cxlv entry to create or remove
> > a cxlv device.
> > 
> > 	Each cxlv device have its owned virtual pci related bridge and bus, cxlv
> > will create a new root_port for the new cxlv device, setup cxl ports for
> > dport and nvdimm-bridge. After that, we will add the virtual pci device,
> > that will go into the cxl_pci_probe to setup new memdev.
> > 
> > 	Then we can see the cxl device with cxl list and use it as a real cxl
> > device.
> > 
> >  $ echo "memstart=$((8*1024*1024*1024)),cxltype=3,pmem=1,memsize=$((2*1024*1024*1024))" > /sys/kernel/debug/cxl/cxlv/create
> >  $ cxl list
> > [
> >   {
> >     "memdev":"mem0",
> >     "pmem_size":1879048192,
> >     "serial":0,
> >     "numa_node":0,
> >     "host":"0010:01:00.0"
> >   }
> > ]
> >  $ cxl create-region -m mem0 -d decoder0.0 -t pmem
> > {
> >   "region":"region0",
> >   "resource":"0x210000000",
> >   "size":"1792.00 MiB (1879.05 MB)",
> >   "type":"pmem",
> >   "interleave_ways":1,
> >   "interleave_granularity":256,
> >   "decode_state":"commit",
> >   "mappings":[
> >     {
> >       "position":0,
> >       "memdev":"mem0",
> >       "decoder":"decoder2.0"
> >     }
> >   ]
> > }
> > cxl region: cmd_create_region: created 1 region
> > 
> >  $ ndctl create-namespace -r region0 -m fsdax --map dev -t pmem -b 0
> > {
> >   "dev":"namespace0.0",
> >   "mode":"fsdax",
> >   "map":"dev",
> >   "size":"1762.00 MiB (1847.59 MB)",
> >   "uuid":"686fd289-a252-42cf-a3a5-95a39ed5c9d5",
> >   "sector_size":512,
> >   "align":2097152,
> >   "blockdev":"pmem0"
> > }
> > 
> >  $ mkfs.xfs -f /dev/pmem0 
> > meta-data=/dev/pmem0             isize=512    agcount=4, agsize=112768
> > blks
> >          =                       sectsz=4096  attr=2, projid32bit=1
> >          =                       crc=1        finobt=1, sparse=1,
> > rmapbt=0
> >          =                       reflink=1    bigtime=0 inobtcount=0
> > data     =                       bsize=4096   blocks=451072, imaxpct=25
> >          =                       sunit=0      swidth=0 blks
> > naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
> > log      =internal log           bsize=4096   blocks=2560, version=2
> >          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> > realtime =none                   extsz=4096   blocks=0, rtextents=0
> > 
> > Any comment is welcome!
> > 
> > TODO: implement cxlv command in ndctl to do cxlv device management.
> > 
> > [1]: Add argument in kernel command line: "memmap=nn[KMG]$ss[KMG]",
> > detail in Documentation/driver-api/cxl/memory-devices.rst
> > 
> > Thanx
> > 
> > Dongsheng Yang (4):
> >   cxl: move some function from acpi module to core module
> >   cxl/port: allow dport host to be driver-less device
> >   cxl/port: introduce cxl_disable_port() function
> >   cxl: introduce CXL Virtualization module
> > 
> >  MAINTAINERS                         |   6 +
> >  drivers/cxl/Kconfig                 |  11 +
> >  drivers/cxl/Makefile                |   1 +
> >  drivers/cxl/acpi.c                  | 143 +-----
> >  drivers/cxl/core/port.c             | 231 ++++++++-
> >  drivers/cxl/cxl.h                   |   6 +
> >  drivers/cxl/cxl_virt/Makefile       |   5 +
> >  drivers/cxl/cxl_virt/cxlv.h         |  87 ++++
> >  drivers/cxl/cxl_virt/cxlv_debugfs.c | 260 ++++++++++
> >  drivers/cxl/cxl_virt/cxlv_device.c  | 311 ++++++++++++
> >  drivers/cxl/cxl_virt/cxlv_main.c    |  67 +++
> >  drivers/cxl/cxl_virt/cxlv_pci.c     | 710 ++++++++++++++++++++++++++++
> >  drivers/cxl/cxl_virt/cxlv_pci.h     | 549 +++++++++++++++++++++
> >  drivers/cxl/cxl_virt/cxlv_port.c    | 149 ++++++
> >  14 files changed, 2388 insertions(+), 148 deletions(-)
> >  create mode 100644 drivers/cxl/cxl_virt/Makefile
> >  create mode 100644 drivers/cxl/cxl_virt/cxlv.h
> >  create mode 100644 drivers/cxl/cxl_virt/cxlv_debugfs.c
> >  create mode 100644 drivers/cxl/cxl_virt/cxlv_device.c
> >  create mode 100644 drivers/cxl/cxl_virt/cxlv_main.c
> >  create mode 100644 drivers/cxl/cxl_virt/cxlv_pci.c
> >  create mode 100644 drivers/cxl/cxl_virt/cxlv_pci.h
> >  create mode 100644 drivers/cxl/cxl_virt/cxlv_port.c
> > 
> > -- 
> > 2.34.1
> >   
> 
> 
>