[SPDK] Using cmb_copy in QEMU environment

[SPDK] Using cmb_copy in QEMU environment

[archiemorningstar88]

[-- Attachment #1: Type: text/plain, Size: 1621 bytes --]

Hi,

I've been trying to get cmb_copy (examples/nvme/cmb_copy) to work using an emulated SSD with CMB in a QEMU environment, but cmb_copy cannot allocate a CMB buffer. The identify (examples/nvme/identify) reports the two emulated SSD's I created have 128MB of CMB. I'm running QEMU v4.2.0 with the guest OS Ubuntu 18.04. I've also tried various revisions of SPDK (20.01.1, 19.10.1, 19.04.1, 18.04.1) with similar results.

I see these two errors print out which I believe couldn't register the memory so the call to spdk_nvme_ctrlr_alloc_cmb_io_buffer() fails (pctrlr->cmb_bar_virt_addr is NULL):
could not get phys addr for 0x1088200000
nvme_pcie.c: 554:nvme_pcie_ctrlr_map_cmb: *ERROR*: spdk_mem_register() failed

I tried to trace the code and see that when it searches for the physical address g_vtophys_pci_devices list is empty when trying to register the CMB memory space. It only gets later populated with the two SSDs when cleanup_pci_devices() is called at the end of spdk_pci_device_attach().

My qemu startup:
x86_64-softmmu/qemu-system-x86_64 \
-cpu host \
-machine q35,accel=kvm,kernel-irqchip=split \
-smp 8 \
-m 32768 \
-enable-kvm \
-drive if=virtio,file=test1.qcow2,cache=none \
-vnc :1 \
-device e1000,netdev=net0 \
-netdev user,id=net0,hostfwd=tcp::5555-:22 \
-drive file=nvme0.img,if=none,id=D22,format=raw \
-device nvme,drive=D22,serial=1234,cmb_size_mb=128 \
-drive file=nvme1.img,if=none,id=D23,format=raw \
-device nvme,drive=D23,serial=5678,cmb_size_mb=128 \
-device intel-iommu,intremap=on \

If anyone has any suggestions it would be greatly appreciated.

Archie