[RFC 0/3] Qemu FM emulation

[nifan.cxl@gmail.com]

From: Fan Ni <fan.ni@samsung.com>

The RFC provides a way for FM emulation in Qemu. The goal is to provide
a context where we can have more FM emulation discussions and share solutions
for a reasonable FM implementation in Qemu.

The basic idea is,

We have two VMs, one is the VM we want to test (named Target VM) and one is the
FM VM. The target VM has the kernel which we are interested (for example, DCD
or RAS feature enabled). The FM VM can be VM with any kernel version as long as
OOB communication support is enabled.

An application running in the FM VM issues FM commands to the underlying device
with OOB channel (e.g., MCTP over I2C), when the device receives the message,
it will not response to the request locally, instead the request will be stored
in a share buffer (implemented with /dev/shm), and a QMP request will be sent
to the target VM to notify there is a MCTP message in the shared buffer,
which needs to be processed. The FM will wait the completion of the request.
The target VM will read the buffer and process the message.
When the process completes, the output payload and any information needs to
return is stored in buffer, and a state field will be reset to notify the FM of
the completion of the processing.

The nice points of the method:
1. It is simple model (consumer-produce model with shm as shared buffer).
2. The communication between the two VMs through the qmp interface is simple.
One qmp interface works for all MCTP messages. Moreover, the qmp interface may
be able to use as a way for the communication between two VMs in different
context.

How we run the test?
Step 1: Start the VM we want to Target VM.
The device interested having "allow-fm-attach=on,mctp-buf-init=on"
For example, for my test, it is the DCD device.

In our test, the kernel run on the target VM is Ira's DCD branch:
https://github.com/weiny2/linux-kernel/tree/dcd-v4-2024-12-11.

qemu-system-x86_64 -gdb tcp::1235  -kernel bzImage -append "root=/dev/sda rw console=ttyS0,115200 ignore_loglevel nokaslr" \
-smp 8 -accel kvm -serial mon:stdio  -nographic  -qmp tcp:localhost:4445,server,wait=off \
-netdev user,id=network0,hostfwd=tcp::2024-:22    \
-device e1000,netdev=network0  -monitor telnet:127.0.0.1:12346,server,nowait \
-drive file=/home/fan/cxl/images/qemu-image.img,index=0,media=disk,format=raw \
-machine q35,cxl=on -cpu qemu64,mce=on -m 8G,maxmem=64G,slots=8 \
-virtfs local,path=/opt/lib/modules,mount_tag=modshare,security_model=mapped  \
-virtfs local,path=/home/fan,mount_tag=homeshare,security_model=mapped \
-object memory-backend-file,id=cxl-mem2,mem-path=/tmp/host0/t3_cxl2.raw,size=4G \
-object memory-backend-file,id=cxl-lsa2,mem-path=/tmp/host0/t3_lsa2.raw,size=1M \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1,hdm_for_passthrough=true \
-device cxl-rp,port=0,bus=cxl.1,id=cxl_rp_port0,chassis=0,slot=2 \
-device cxl-upstream,port=2,sn=1234,bus=cxl_rp_port0,id=us0,addr=0.0,multifunction=on, \
-device cxl-switch-mailbox-cci,bus=cxl_rp_port0,addr=0.1,target=us0 \
-device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \
-device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \
-device cxl-downstream,port=3,bus=us0,id=swport2,chassis=0,slot=6 \
-device cxl-type3,bus=swport2,volatile-dc-memdev=cxl-mem2,id=cxl-dcd0,lsa=cxl-lsa2,num-dc-regions=2,sn=99,allow-fm-attach=on,mctp-buf-init=on \
-machine cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=1k \
-device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=4,target=us0 \
-device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=6,target=cxl-dcd0 \
-device virtio-rng-pci,bus=swport1

Step 2: Start the FM VM and run the test program to send MCTP requests and
forward to the target VM for processing.

Note: the kernel for FM VM should have MCTP support.

In the test, we use linux-v6.6-rc6 with Jonathan's MCTP hack patches:
https://github.com/moking/cxl-test-tool/blob/main/test-workflows/mctp/mctp-patches-kernel.patch

qemu-system-x86_64 -gdb tcp::1236 -kernel fm-bzImage -append "root=/dev/sda rw console=ttyS0,115200 ignore_loglevel nokaslr " \
-smp 8 -accel kvm -serial mon:stdio  -nographic  -qmp tcp:localhost:4446,server,wait=off \
-netdev user,id=network0,hostfwd=tcp::2025-:22    \
-device e1000,netdev=network0  -monitor telnet:127.0.0.1:12347,server,nowait \
-drive file=/home/fan/cxl/images/qemu-image-fm.img,index=0,media=disk,format=raw \
-machine q35,cxl=on -cpu qemu64,mce=on -m 8G,maxmem=64G,slots=8  \
-virtfs local,path=/opt/lib/modules,mount_tag=modshare,security_model=mapped  \
-virtfs local,path=/home/fan,mount_tag=homeshare,security_model=mapped \
-object memory-backend-file,id=cxl-mem2,mem-path=/tmp/host1/t3_cxl2.raw,size=4G \
-object memory-backend-file,id=cxl-lsa2,mem-path=/tmp/host1/t3_lsa2.raw,size=1M \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1,hdm_for_passthrough=true \
-device cxl-rp,port=0,bus=cxl.1,id=cxl_rp_port0,chassis=0,slot=2 \
-device cxl-upstream,port=2,sn=1234,bus=cxl_rp_port0,id=us0,addr=0.0,multifunction=on, \
-device cxl-switch-mailbox-cci,bus=cxl_rp_port0,addr=0.1,target=us0 \
-device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \
-device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \
-device cxl-downstream,port=3,bus=us0,id=swport2,chassis=0,slot=6 \
-device cxl-type3,bus=swport2,volatile-dc-memdev=cxl-mem2,id=cxl-dcd0,lsa=cxl-lsa2,num-dc-regions=2,sn=99,allow-fm-attach=on \
-machine cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=1k \
-device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=4,target=us0 \
-device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=6,target=cxl-dcd0,qmp=127.0.0.1:4445,mctp-msg-forward=on \
-device virtio-rng-pci,bus=swport1

Currently, the code is not clean at all, it is a POC to prove the idea. Only
type3 (including DCD) devices can accept requests from the FM, which should be
easy to extend to support switch-targeted FM command processing.

The code is based on Jonathan's cxl-2025-03-20 branch.
A qemu branch with the code: https://github.com/moking/qemu-jic-clone/tree/fm-qmp

FYI.
I have a tool to make the test easier.
https://github.com/moking/cxl-test-tool/tree/main

Part of .var.config, see run_vars.example

QEMU_ROOT=~/cxl/jic/qemu
# for FM VM
FM_KERNEL_ROOT=~/cxl/linux-v6.6-rc6/
FM_QEMU_IMG=~/cxl/images/qemu-image-fm.img

# for Target VM
KERNEL_ROOT=~/cxl/linux-dcd/
QEMU_IMG=~/cxl/images/qemu-image.img

command:
1. cxl-tool.py --run -T FM_TARGET
2. cxl-tool.py --attach-VM -T FM_CLIENT
3. cxl-tool.py --install-libcxlmi-fm
4. cxl-tool.py --setup-mctp-fm
5. cxl-tool.py --login-fm (run the test program with libcxlmi)

Fan Ni (3):
  cxl_type3: Preparing information sharing between VMs
  cxl_type3: Add qmp_cxl_process_mctp_message qmp interface
  cxl/i2c_mctp_cxl: Add support to process MCTP command remotely

 hw/cxl/cxl-mctp-qmp.c             |  85 +++++++++++++++
 hw/cxl/i2c_mctp_cxl.c             |  68 ++++++++++--
 hw/cxl/meson.build                |   2 +-
 hw/mem/cxl_type3.c                | 166 +++++++++++++++++++++++++++++-
 hw/mem/cxl_type3_stubs.c          |   5 +
 include/hw/cxl/cxl_device.h       |   8 ++
 include/hw/cxl/cxl_mctp_message.h |  43 ++++++++
 qapi/cxl.json                     |  18 ++++
 8 files changed, 387 insertions(+), 8 deletions(-)
 create mode 100644 hw/cxl/cxl-mctp-qmp.c
 create mode 100644 include/hw/cxl/cxl_mctp_message.h

-- 
2.47.2