Re: [SPDK] NVMf target configuration issue

[Harris, James R <james.r.harris at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 8728 bytes --]

Hi,

Can you confirm that your NVMe device is bound to a uio or vfio driver, and not the kernel NVMe driver?  SPDK will not try to use devices that are still bound to the kernel nvme driver.  You can use scripts/setup.sh to bind all NVMe devices to uio_pci_generic or vfio-pci (based on whether your system has an IOMMU enabled).

lspci –v –s 0000:07:00.0 will show you which kernel module your NVMe device is bound to currently.

Regards,

-Jim


From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Kirubakaran Kaliannan <kirubak(a)zadarastorage.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Wednesday, September 6, 2017 at 12:28 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] NVMf target configuration issue


Hi,

I have the SPDK target working for AIO and Malloc.

I now have configured NVMe device on the same setup, but not able to start the target, as the target is not finding the nvme0 bdev.

Do we have any other dependency for configuring Nvme devices ?

Nvmf.conf

[Global]

[Rpc]
  Enable No
  Listen 127.0.0.1

[Nvmf]
  MaxQueuesPerSession 4
  AcceptorPollRate 10000

[Nvme]
  TransportId "trtype:PCIe traddr:0000:07:00.0" Nvme0
  RetryCount 4
  Timeout 0
  AdminPollRate 100000
  HotplugEnable Yes

[Subsystem1]
  NQN nqn.2016-06.io.spdk:cnode1
  Core 0
  Listen RDMA 10.3.7.2:4420<http://10.3.7.2:4420>
  AllowAnyHost Yes
  SN SPDK00000000000001
  Namespace Nvme0n1 1

# app/nvmf_tgt/nvmf_tgt -t all -c /etc/spdk/nvmf.conf
Starting DPDK 17.05.0 initialization...
[ DPDK EAL parameters: nvmf -c 0x1 --file-prefix=spdk_pid97433 ]
EAL: Detected 20 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
Total cores available: 1
Occupied cpu socket mask is 0x1
reactor.c: 314:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0
EAL: PCI device 0000:00:04.0 on NUMA socket 0
EAL:   probe driver: 8086:6f20 spdk_ioat
EAL: PCI device 0000:00:04.1 on NUMA socket 0
EAL:   probe driver: 8086:6f21 spdk_ioat
EAL: PCI device 0000:00:04.2 on NUMA socket 0
EAL:   probe driver: 8086:6f22 spdk_ioat
EAL: PCI device 0000:00:04.3 on NUMA socket 0
EAL:   probe driver: 8086:6f23 spdk_ioat
EAL: PCI device 0000:00:04.4 on NUMA socket 0
EAL:   probe driver: 8086:6f24 spdk_ioat
EAL: PCI device 0000:00:04.5 on NUMA socket 0
EAL:   probe driver: 8086:6f25 spdk_ioat
EAL: PCI device 0000:00:04.6 on NUMA socket 0
EAL:   probe driver: 8086:6f26 spdk_ioat
EAL: PCI device 0000:00:04.7 on NUMA socket 0
EAL:   probe driver: 8086:6f27 spdk_ioat
copy_engine_ioat.c: 306:copy_engine_ioat_init: *NOTICE*: Ioat Copy Engine Offload Enabled
EAL: PCI device 0000:00:04.0 on NUMA socket 0
EAL:   probe driver: 8086:6f20 spdk_ioat
EAL: PCI device 0000:00:04.1 on NUMA socket 0
EAL:   probe driver: 8086:6f21 spdk_ioat
EAL: PCI device 0000:00:04.2 on NUMA socket 0
EAL:   probe driver: 8086:6f22 spdk_ioat
EAL: PCI device 0000:00:04.3 on NUMA socket 0
EAL:   probe driver: 8086:6f23 spdk_ioat
EAL: PCI device 0000:00:04.4 on NUMA socket 0
EAL:   probe driver: 8086:6f24 spdk_ioat
EAL: PCI device 0000:00:04.5 on NUMA socket 0
EAL:   probe driver: 8086:6f25 spdk_ioat
EAL: PCI device 0000:00:04.6 on NUMA socket 0
EAL:   probe driver: 8086:6f26 spdk_ioat
EAL: PCI device 0000:00:04.7 on NUMA socket 0
EAL:   probe driver: 8086:6f27 spdk_ioat
EAL: PCI device 0000:07:00.0 on NUMA socket 0
EAL:   probe driver: 8086:953 spdk_nvme
nvmf.c:  67:spdk_nvmf_tgt_init: *INFO*: Max Queue Pairs Per Controller: 4
nvmf.c:  68:spdk_nvmf_tgt_init: *INFO*: Max Queue Depth: 128
nvmf.c:  69:spdk_nvmf_tgt_init: *INFO*: Max In Capsule Data: 4096 bytes
nvmf.c:  70:spdk_nvmf_tgt_init: *INFO*: Max I/O Size: 131072 bytes
nvmf_tgt.c: 215:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 0 on socket 0
nvmf_tgt.c: 215:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2016-06.io.spdk:cnode1 on lcore 0 on socket 0
transport.c:  63:spdk_nvmf_transport_create: *ERROR*: inside spdk_nvmf_transport_create 1
rdma.c: 955:spdk_nvmf_rdma_create: *NOTICE*: *** RDMA Transport Init ***
rdma.c:1120:spdk_nvmf_rdma_listen: *NOTICE*: *** NVMf Target Listening on 10.3.7.2 port 4420 ***
bdev.c: 214:spdk_bdev_get_by_name: *ERROR*: bdev's returned null 'Nvme0n1'
conf.c: 492:spdk_nvmf_construct_subsystem: *ERROR*: Could not find namespace bdev 'Nvme0n1'
subsystem.c: 232:spdk_nvmf_delete_subsystem: *INFO*: subsystem is 0x1218ef0
nvmf_tgt.c: 276:spdk_nvmf_startup: *ERROR*: spdk_nvmf_parse_conf() failed
subsystem.c: 232:spdk_nvmf_delete_subsystem: *INFO*: subsystem is 0x1218bf0

# lspci  |grep -i vola
07:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)

Thanks
-kiru



From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Yang, Ziye
Sent: Tuesday, September 05, 2017 10:41 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] NVMf target configuration issue

And the other choice is to remove the Host related line, which means any host can connect.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Yang, Ziye
Sent: Tuesday, September 5, 2017 1:04 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] NVMf target configuration issue

Hi Kiru,

According to the error message, I suggest that you add the Host info in the following section.


[Subsystem1]
  NQN nqn.2016-06.io.spdk:cnode1
  Core 0
  Listen RDMA 10.3.7.2:4420<http://10.3.7.2:4420>
  Host nqn.2016-06.io.spdk:init
                 Host nqn.2014-08.org.nvmexpress:NVMf:uuid:5e5e0064-2ad6-4a43-9c26-063e3ef6cf14
  SN SPDK00000000000001
  Namespace AIO0


Thanks.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Kirubakaran Kaliannan
Sent: Tuesday, September 5, 2017 12:52 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] NVMf target configuration issue



Hi All,

I am trying to get the nvmf configuration going with SPDK.

I have the following configuration,

# ofed 2.4 on both target and initiator
# SPDK + 3.18 kernel on target
# 4.9 kernel on initiator
# mellanox connect3X-Pro

/etc/spdk/nvmf.conf

[Global]
[Rpc]
  Enable No
  Listen 127.0.0.1
[AIO]
  AIO /dev/sdd AIO0

[Nvmf]
  MaxQueuesPerSession 4
  AcceptorPollRate 10000

[Subsystem1]
  NQN nqn.2016-06.io.spdk:cnode1
  Core 0
  Listen RDMA 10.3.7.2:4420<http://10.3.7.2:4420>
  Host nqn.2016-06.io.spdk:init
  SN SPDK00000000000001
  Namespace AIO0


From initiator I tried discover and connect

# nvme discover -t rdma -a 10.3.7.2 -s 4420

Discovery Log Number of Records 1, Generation counter 4
=====Discovery Log Entry 0======
trtype:  rdma
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified
portid:  0
trsvcid: 4420
subnqn:  nqn.2016-06.io.spdk:cnode1
traddr:  10.3.7.2
rdma_prtype: not specified
rdma_qptype: connected
rdma_cms:    rdma-cm
rdma_pkey: 0x0000

# nvme connect -t rdma -n "nqn.2016-06.io.spdk:cnode1" -a 10.3.7.2 -s 4420
Failed to write to /dev/nvme-fabrics: Input/output error

[dmesg]
[59909.541392] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.3.7.2:4420<http://10.3.7.2:4420>
[59940.389952] nvme nvme0: Connect command failed, error wo/DNR bit: 388

On target I get the following error

# app/nvmf_tgt/nvmf_tgt -c /etc/spdk/nvmf.conf  > /tmp/1
EAL: Detected 20 lcore(s)
reactor.c: 314:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0
copy_engine_ioat.c: 306:copy_engine_ioat_init: *NOTICE*: Ioat Copy Engine Offload Enabled
nvmf_tgt.c: 215:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 0 on socket 0
nvmf_tgt.c: 215:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2016-06.io.spdk:cnode1 on lcore 0 on socket 0
rdma.c: 955:spdk_nvmf_rdma_create: *NOTICE*: *** RDMA Transport Init ***
rdma.c:1120:spdk_nvmf_rdma_listen: *NOTICE*: *** NVMf Target Listening on 10.3.7.2 port 4420 ***
conf.c: 500:spdk_nvmf_construct_subsystem: *NOTICE*: Attaching block device AIO0 to subsystem nqn.2016-06.io.spdk:cnode1
nvmf_tgt.c: 290:spdk_nvmf_startup: *NOTICE*: Acceptor running on core 0 on socket 0
request.c: 171:nvmf_process_connect: *ERROR*: Subsystem 'nqn.2016-06.io.spdk:cnode1' does not allow host 'nqn.2014-08.org.nvmexpress:NVMf:uuid:5e5e0064-2ad6-4a43-9c26-063e3ef6cf14'


can you please help on why I am not able to connect in this configuration.

Regards,
-kiru


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 31030 bytes --]