Hi All,

I am using SPDK and DPDK.

I have single intel nvme SSD connected to x4 pci slot.

Setup.sh runs fine, and able to unbind kernel driver okay.

But when I run fio benchmark : as follows:

fio --randrepeat=1 --ioengine=/home/oza/SPDK/spdk/examples/nvme/fio_plugin/fio_plugin --direct=1 --numjobs=1 --gtod_reduce=1 --name=readtest --filename=0000.01.00.00/1 --bs=4096   --iodepth=128 --size=4G --readwrite=read

I get pci_map_resource failed.

I have debugged this to an extent where:

I figure that pci_map_resource trying to map msi-x memory and vfio is denying it.

I see there is a call in pci_vfio_map_resource which asks for pci_vfio_get_msix_bar.

But following log suggests that pci_vfio_map_resource is being called twice which is strange.

On top of that, I am getting different value for msix_bar second time, which is causing the failure.

So first time bar0 as msi-x which seems okay with respect to kernel vfio (msix-bar)

But second time DPDK says msix_bar is 1, but kernel vfio still says it is 0, and denies to map msi-x memory to user space.

EAL: Detected 8 lcore(s)

EAL: Auto-detected process type: PRIMARY

EAL: Probing VFIO support...

EAL: VFIO support initialized

EAL: cannot open /proc/self/numa_maps, consider that all memory is in socket_id 0

EAL: PCI device 0000:01:00.0 on NUMA socket 0

EAL:   probe driver: 8086:953 spdk_nvme

EAL:   using IOMMU type 1 (Type 1)

EAL: the msi-x bar number is 0

EAL: Hotplug doesn't support vfio yet

EAL: PCI device 0000:01:00.0 on NUMA socket 0

EAL:   probe driver: 8086:953 spdk_nvme

EAL: the msi-x bar number is 1

EAL: pci_map_resource(): cannot mmap(16, 0x1080004000, 0x4000, 0x0): Invalid argument (0xffffffffffffffff)

EAL:   0000:01:00.0 mapping BAR0 failed: Invalid argument

EAL: Error - exiting with code: 1

Regards,

Oza.