[SPDK] EAL: pci_map_resource(): cannot mmap(16, 0x1080004000, 0x4000, 0x0): Invalid argument (0xffffffffffffffff)

[SPDK] EAL: pci_map_resource(): cannot mmap(16, 0x1080004000, 0x4000, 0x0): Invalid argument (0xffffffffffffffff)

[Oza Oza]

[-- Attachment #1: Type: text/plain, Size: 1930 bytes --]

Hi All,



I am using SPDK and DPDK.



I have single intel nvme SSD connected to x4 pci slot.



Setup.sh runs fine, and able to unbind kernel driver okay.

But when I run fio benchmark : as follows:



fio --randrepeat=1
--ioengine=/home/oza/SPDK/spdk/examples/nvme/fio_plugin/fio_plugin
--direct=1 --numjobs=1 --gtod_reduce=1 --name=readtest --filename=
0000.01.00.00/1 --bs=4096   --iodepth=128 --size=4G --readwrite=read



I get pci_map_resource failed.

I have debugged this to an extent where:



I figure that pci_map_resource trying to map msi-x memory and vfio is
denying it.

I see there is a call in pci_vfio_map_resource which asks for
pci_vfio_get_msix_bar.



But following log suggests that pci_vfio_map_resource is being called twice
which is strange.

On top of that, I am getting different value for msix_bar second time,
which is causing the failure.



So first time bar0 as msi-x which seems okay with respect to kernel vfio
(msix-bar)

But second time DPDK says msix_bar is 1, but kernel vfio still says it is
0, and denies to map msi-x memory to user space.



EAL: Detected 8 lcore(s)

EAL: Auto-detected process type: PRIMARY

EAL: Probing VFIO support...

EAL: VFIO support initialized

EAL: cannot open /proc/self/numa_maps, consider that all memory is in
socket_id 0

EAL: PCI device 0000:01:00.0 on NUMA socket 0

EAL:   probe driver: 8086:953 spdk_nvme

EAL:   using IOMMU type 1 (Type 1)

EAL: the msi-x bar number is 0

EAL: Hotplug doesn't support vfio yet

EAL: PCI device 0000:01:00.0 on NUMA socket 0

EAL:   probe driver: 8086:953 spdk_nvme

EAL: the msi-x bar number is 1

EAL: pci_map_resource(): cannot mmap(16, 0x1080004000, 0x4000, 0x0):
Invalid argument (0xffffffffffffffff)

EAL:   0000:01:00.0 mapping BAR0 failed: Invalid argument

EAL: Error - exiting with code: 1





Regards,

Oza.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 3939 bytes --]

Re: [SPDK] EAL: pci_map_resource(): cannot mmap(16, 0x1080004000, 0x4000, 0x0): Invalid argument (0xffffffffffffffff)

[Oza Oza]

[-- Attachment #1: Type: text/plain, Size: 1856 bytes --]

Hi All,

I am using SPDK and DPDK.

I have single intel nvme SSD connected to x4 pci slot.

Setup.sh runs fine, and able to unbind kernel driver okay.
But when I run fio benchmark : as follows:

fio --randrepeat=1
--ioengine=/home/oza/SPDK/spdk/examples/nvme/fio_plugin/fio_plugin
--direct=1 --numjobs=1 --gtod_reduce=1 --name=readtest
--filename=0000.01.00.00/1 --bs=4096   --iodepth=128 --size=4G
--readwrite=read

I get pci_map_resource failed.
I have debugged this to an extent where:

I figure that pci_map_resource trying to map msi-x memory and vfio is
denying it.
I see there is a call in pci_vfio_map_resource which asks for
pci_vfio_get_msix_bar.

But following log suggests that pci_vfio_map_resource is being called
twice which is strange.
On top of that, I am getting different value for msix_bar second time,
which is causing the failure.

So first time bar0 as msi-x which seems okay with respect to kernel vfio
(msix-bar)
But second time DPDK says msix_bar is 1, but kernel vfio still says it is
0, and denies to map msi-x memory to user space.

EAL: Detected 8 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: cannot open /proc/self/numa_maps, consider that all memory is in
socket_id 0
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 8086:953 spdk_nvme
EAL:   using IOMMU type 1 (Type 1)
EAL: the msi-x bar number is 0
EAL: Hotplug doesn't support vfio yet
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 8086:953 spdk_nvme
EAL: the msi-x bar number is 1
EAL: pci_map_resource(): cannot mmap(16, 0x1080004000, 0x4000, 0x0):
Invalid argument (0xffffffffffffffff)
EAL:   0000:01:00.0 mapping BAR0 failed: Invalid argument
EAL: Error - exiting with code: 1


Regards,
Oza.

Re: [SPDK] EAL: pci_map_resource(): cannot mmap(16, 0x1080004000, 0x4000, 0x0): Invalid argument (0xffffffffffffffff)

[Oza Oza]

[-- Attachment #1: Type: text/plain, Size: 2637 bytes --]

I have ported SPDK for ARMv8.
And DPDK is compiled with version: 16.11.1
And with following config

My config is:
defconfig_arm64-armv8a-linuxapp-gcc

#include "common_linuxapp"

CONFIG_RTE_MACHINE="armv8a"

CONFIG_RTE_ARCH="arm64"
CONFIG_RTE_ARCH_ARM64=y
CONFIG_RTE_ARCH_64=y

CONFIG_RTE_FORCE_INTRINSICS=y

CONFIG_RTE_TOOLCHAIN="gcc"
CONFIG_RTE_TOOLCHAIN_GCC=y

CONFIG_RTE_EAL_IGB_UIO=n

CONFIG_RTE_LIBRTE_FM10K_PMD=n

CONFIG_RTE_SCHED_VECTOR=n
CONFIG_RTE_KNI_KMOD=n
EXTRA_CFLAGS=-fPIC

-----Original Message-----
From: Oza Oza [mailto:oza.oza(a)broadcom.com]
Sent: Sunday, April 02, 2017 11:37 AM
To: 'spdk(a)lists.01.org'
Subject: RE: EAL: pci_map_resource(): cannot mmap(16, 0x1080004000,
0x4000, 0x0): Invalid argument (0xffffffffffffffff)

Hi All,

I am using SPDK and DPDK.

I have single intel nvme SSD connected to x4 pci slot.

Setup.sh runs fine, and able to unbind kernel driver okay.
But when I run fio benchmark : as follows:

fio --randrepeat=1
--ioengine=/home/oza/SPDK/spdk/examples/nvme/fio_plugin/fio_plugin
--direct=1 --numjobs=1 --gtod_reduce=1 --name=readtest
--filename=0000.01.00.00/1 --bs=4096   --iodepth=128 --size=4G
--readwrite=read

I get pci_map_resource failed.
I have debugged this to an extent where:

I figure that pci_map_resource trying to map msi-x memory and vfio is
denying it.
I see there is a call in pci_vfio_map_resource which asks for
pci_vfio_get_msix_bar.

But following log suggests that pci_vfio_map_resource is being called
twice which is strange.
On top of that, I am getting different value for msix_bar second time,
which is causing the failure.

So first time bar0 as msi-x which seems okay with respect to kernel vfio
(msix-bar) But second time DPDK says msix_bar is 1, but kernel vfio still
says it is 0, and denies to map msi-x memory to user space.

EAL: Detected 8 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: cannot open /proc/self/numa_maps, consider that all memory is in
socket_id 0
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 8086:953 spdk_nvme
EAL:   using IOMMU type 1 (Type 1)
EAL: the msi-x bar number is 0
EAL: Hotplug doesn't support vfio yet
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 8086:953 spdk_nvme
EAL: the msi-x bar number is 1
EAL: pci_map_resource(): cannot mmap(16, 0x1080004000, 0x4000, 0x0):
Invalid argument (0xffffffffffffffff)
EAL:   0000:01:00.0 mapping BAR0 failed: Invalid argument
EAL: Error - exiting with code: 1


Regards,
Oza.