Hi,
I have the SPDK target working for AIO and Malloc.
I now have configured NVMe device on the same setup, but not able to start the target, as the target is not finding the nvme0 bdev.
Do we have any other dependency for configuring Nvme devices ?
Nvmf.conf
[Global]
[Rpc]
Enable No
Listen 127.0.0.1
[Nvmf]
MaxQueuesPerSession 4
AcceptorPollRate 10000
[Nvme]
TransportId "trtype:PCIe traddr:0000:07:00.0" Nvme0
RetryCount 4
Timeout 0
AdminPollRate 100000
HotplugEnable Yes
[Subsystem1]
NQN nqn.2016-06.io.spdk:cnode1
Core 0
Listen RDMA 10.3.7.2:4420
AllowAnyHost Yes
SN SPDK00000000000001
Namespace Nvme0n1 1
# app/nvmf_tgt/nvmf_tgt -t all -c /etc/spdk/nvmf.conf
Starting DPDK 17.05.0 initialization...
[ DPDK EAL parameters: nvmf -c 0x1 --file-prefix=spdk_pid97433 ]
EAL: Detected 20 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
Total cores available: 1
Occupied cpu socket mask is 0x1
reactor.c: 314:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0
EAL: PCI device 0000:00:04.0 on NUMA socket 0
EAL: probe driver: 8086:6f20 spdk_ioat
EAL: PCI device 0000:00:04.1 on NUMA socket 0
EAL: probe driver: 8086:6f21 spdk_ioat
EAL: PCI device 0000:00:04.2 on NUMA socket 0
EAL: probe driver: 8086:6f22 spdk_ioat
EAL: PCI device 0000:00:04.3 on NUMA socket 0
EAL: probe driver: 8086:6f23 spdk_ioat
EAL: PCI device 0000:00:04.4 on NUMA socket 0
EAL: probe driver: 8086:6f24 spdk_ioat
EAL: PCI device 0000:00:04.5 on NUMA socket 0
EAL: probe driver: 8086:6f25 spdk_ioat
EAL: PCI device 0000:00:04.6 on NUMA socket 0
EAL: probe driver: 8086:6f26 spdk_ioat
EAL: PCI device 0000:00:04.7 on NUMA socket 0
EAL: probe driver: 8086:6f27 spdk_ioat
copy_engine_ioat.c: 306:copy_engine_ioat_init: *NOTICE*: Ioat Copy Engine Offload Enabled
EAL: PCI device 0000:00:04.0 on NUMA socket 0
EAL: probe driver: 8086:6f20 spdk_ioat
EAL: PCI device 0000:00:04.1 on NUMA socket 0
EAL: probe driver: 8086:6f21 spdk_ioat
EAL: PCI device 0000:00:04.2 on NUMA socket 0
EAL: probe driver: 8086:6f22 spdk_ioat
EAL: PCI device 0000:00:04.3 on NUMA socket 0
EAL: probe driver: 8086:6f23 spdk_ioat
EAL: PCI device 0000:00:04.4 on NUMA socket 0
EAL: probe driver: 8086:6f24 spdk_ioat
EAL: PCI device 0000:00:04.5 on NUMA socket 0
EAL: probe driver: 8086:6f25 spdk_ioat
EAL: PCI device 0000:00:04.6 on NUMA socket 0
EAL: probe driver: 8086:6f26 spdk_ioat
EAL: PCI device 0000:00:04.7 on NUMA socket 0
EAL: probe driver: 8086:6f27 spdk_ioat
EAL: PCI device 0000:07:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
nvmf.c: 67:spdk_nvmf_tgt_init: *INFO*: Max Queue Pairs Per Controller: 4
nvmf.c: 68:spdk_nvmf_tgt_init: *INFO*: Max Queue Depth: 128
nvmf.c: 69:spdk_nvmf_tgt_init: *INFO*: Max In Capsule Data: 4096 bytes
nvmf.c: 70:spdk_nvmf_tgt_init: *INFO*: Max I/O Size: 131072 bytes
nvmf_tgt.c: 215:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 0 on socket 0
nvmf_tgt.c: 215:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2016-06.io.spdk:cnode1 on lcore 0 on socket 0
transport.c: 63:spdk_nvmf_transport_create: *ERROR*: inside spdk_nvmf_transport_create 1
rdma.c: 955:spdk_nvmf_rdma_create: *NOTICE*: *** RDMA Transport Init ***
rdma.c:1120:spdk_nvmf_rdma_listen: *NOTICE*: *** NVMf Target Listening on 10.3.7.2 port 4420 ***
bdev.c: 214:spdk_bdev_get_by_name: *ERROR*: bdev's returned null 'Nvme0n1'
conf.c: 492:spdk_nvmf_construct_subsystem: *ERROR*: Could not find namespace bdev 'Nvme0n1'
subsystem.c: 232:spdk_nvmf_delete_subsystem: *INFO*: subsystem is 0x1218ef0
nvmf_tgt.c: 276:spdk_nvmf_startup: *ERROR*: spdk_nvmf_parse_conf() failed
subsystem.c: 232:spdk_nvmf_delete_subsystem: *INFO*: subsystem is 0x1218bf0
# lspci |grep -i vola
07:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)
Thanks
-kiru
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Yang, Ziye
Sent: Tuesday, September 05, 2017 10:41 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] NVMf target configuration issue
And the other choice is to remove the Host related line, which means any host can connect.
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Yang, Ziye
Sent: Tuesday, September 5, 2017 1:04 PM
To: Storage Performance Development Kit <spdk@lists.01.org>
Subject: Re: [SPDK] NVMf target configuration issue
Hi Kiru,
According to the error message, I suggest that you add the Host info in the following section.
[Subsystem1]
NQN nqn.2016-06.io.spdk:cnode1
Core 0
Listen RDMA 10.3.7.2:4420
Host nqn.2016-06.io.spdk:init
Host nqn.2014-08.org.nvmexpress:NVMf:uuid:5e5e0064-2ad6-4a43-9c26-063e3ef6cf14
SN SPDK00000000000001
Namespace AIO0
Thanks.
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Kirubakaran Kaliannan
Sent: Tuesday, September 5, 2017 12:52 PM
To: Storage Performance Development Kit <spdk@lists.01.org>
Subject: [SPDK] NVMf target configuration issue
Hi All,
I am trying to get the nvmf configuration going with SPDK.
I have the following configuration,
# ofed 2.4 on both target and initiator
# SPDK + 3.18 kernel on target
# 4.9 kernel on initiator
# mellanox connect3X-Pro
/etc/spdk/nvmf.conf
[Global]
[Rpc]
Enable No
Listen 127.0.0.1
[AIO]
AIO /dev/sdd AIO0
[Nvmf]
MaxQueuesPerSession 4
AcceptorPollRate 10000
[Subsystem1]
NQN nqn.2016-06.io.spdk:cnode1
Core 0
Listen RDMA 10.3.7.2:4420
Host nqn.2016-06.io.spdk:init
SN SPDK00000000000001
Namespace AIO0
From initiator I tried discover and connect
# nvme discover -t rdma -a 10.3.7.2 -s 4420
Discovery Log Number of Records 1, Generation counter 4
=====Discovery Log Entry 0======
trtype: rdma
adrfam: ipv4
subtype: nvme subsystem
treq: not specified
portid: 0
trsvcid: 4420
subnqn: nqn.2016-06.io.spdk:cnode1
traddr: 10.3.7.2
rdma_prtype: not specified
rdma_qptype: connected
rdma_cms: rdma-cm
rdma_pkey: 0x0000
# nvme connect -t rdma -n "nqn.2016-06.io.spdk:cnode1" -a 10.3.7.2 -s 4420
Failed to write to /dev/nvme-fabrics: Input/output error
[dmesg]
[59909.541392] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 10.3.7.2:4420
[59940.389952] nvme nvme0: Connect command failed, error wo/DNR bit: 388
On target I get the following error
# app/nvmf_tgt/nvmf_tgt -c /etc/spdk/nvmf.conf > /tmp/1
EAL: Detected 20 lcore(s)
reactor.c: 314:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0
copy_engine_ioat.c: 306:copy_engine_ioat_init: *NOTICE*: Ioat Copy Engine Offload Enabled
nvmf_tgt.c: 215:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 0 on socket 0
nvmf_tgt.c: 215:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2016-06.io.spdk:cnode1 on lcore 0 on socket 0
rdma.c: 955:spdk_nvmf_rdma_create: *NOTICE*: *** RDMA Transport Init ***
rdma.c:1120:spdk_nvmf_rdma_listen: *NOTICE*: *** NVMf Target Listening on 10.3.7.2 port 4420 ***
conf.c: 500:spdk_nvmf_construct_subsystem: *NOTICE*: Attaching block device AIO0 to subsystem nqn.2016-06.io.spdk:cnode1
nvmf_tgt.c: 290:spdk_nvmf_startup: *NOTICE*: Acceptor running on core 0 on socket 0
request.c: 171:nvmf_process_connect: *ERROR*: Subsystem 'nqn.2016-06.io.spdk:cnode1' does not allow host 'nqn.2014-08.org.nvmexpress:NVMf:uuid:5e5e0064-2ad6-4a43-9c26-063e3ef6cf14'
can you please help on why I am not able to connect in this configuration.
Regards,
-kiru