Re: [SPDK] NVMf target configuration issue

[Kirubakaran Kaliannan <kirubak at zadarastorage.com>]

[-- Attachment #1: Type: text/plain, Size: 9496 bytes --]

Hi Harris,



Yes, I missed this part. Thanks, I could get going with NVMe now.



Thanks

-kiru



*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
James R
*Sent:* Wednesday, September 06, 2017 8:59 PM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] NVMf target configuration issue



Hi,



Can you confirm that your NVMe device is bound to a uio or vfio driver, and
not the kernel NVMe driver?  SPDK will not try to use devices that are
still bound to the kernel nvme driver.  You can use scripts/setup.sh to
bind all NVMe devices to uio_pci_generic or vfio-pci (based on whether your
system has an IOMMU enabled).



lspci –v –s 0000:07:00.0 will show you which kernel module your NVMe device
is bound to currently.



Regards,



-Jim





*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Kirubakaran Kaliannan
<kirubak(a)zadarastorage.com>
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Wednesday, September 6, 2017 at 12:28 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *Re: [SPDK] NVMf target configuration issue





Hi,



I have the SPDK target working for AIO and Malloc.



I now have configured NVMe device on the same setup, but not able to start
the target, as the target is not finding the nvme0 bdev.



Do we have any other dependency for configuring Nvme devices ?



*Nvmf.conf*



[Global]



[Rpc]

  Enable No

  Listen 127.0.0.1



[Nvmf]

  MaxQueuesPerSession 4

  AcceptorPollRate 10000



[Nvme]

  TransportId "trtype:PCIe traddr:0000:07:00.0" Nvme0

  RetryCount 4

  Timeout 0

  AdminPollRate 100000

  HotplugEnable Yes



[Subsystem1]

  NQN nqn.2016-06.io.spdk:cnode1

  Core 0

  Listen RDMA 10.3.7.2:4420

  AllowAnyHost Yes

  SN SPDK00000000000001

  Namespace Nvme0n1 1



*# app/nvmf_tgt/nvmf_tgt -t all -c /etc/spdk/nvmf.conf *

Starting DPDK 17.05.0 initialization...

[ DPDK EAL parameters: nvmf -c 0x1 --file-prefix=spdk_pid97433 ]

EAL: Detected 20 lcore(s)

EAL: Probing VFIO support...

EAL: VFIO support initialized

Total cores available: 1

Occupied cpu socket mask is 0x1

reactor.c: 314:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on
socket 0

EAL: PCI device 0000:00:04.0 on NUMA socket 0

EAL:   probe driver: 8086:6f20 spdk_ioat

EAL: PCI device 0000:00:04.1 on NUMA socket 0

EAL:   probe driver: 8086:6f21 spdk_ioat

EAL: PCI device 0000:00:04.2 on NUMA socket 0

EAL:   probe driver: 8086:6f22 spdk_ioat

EAL: PCI device 0000:00:04.3 on NUMA socket 0

EAL:   probe driver: 8086:6f23 spdk_ioat

EAL: PCI device 0000:00:04.4 on NUMA socket 0

EAL:   probe driver: 8086:6f24 spdk_ioat

EAL: PCI device 0000:00:04.5 on NUMA socket 0

EAL:   probe driver: 8086:6f25 spdk_ioat

EAL: PCI device 0000:00:04.6 on NUMA socket 0

EAL:   probe driver: 8086:6f26 spdk_ioat

EAL: PCI device 0000:00:04.7 on NUMA socket 0

EAL:   probe driver: 8086:6f27 spdk_ioat

copy_engine_ioat.c: 306:copy_engine_ioat_init: *NOTICE*: Ioat Copy Engine
Offload Enabled

EAL: PCI device 0000:00:04.0 on NUMA socket 0

EAL:   probe driver: 8086:6f20 spdk_ioat

EAL: PCI device 0000:00:04.1 on NUMA socket 0

EAL:   probe driver: 8086:6f21 spdk_ioat

EAL: PCI device 0000:00:04.2 on NUMA socket 0

EAL:   probe driver: 8086:6f22 spdk_ioat

EAL: PCI device 0000:00:04.3 on NUMA socket 0

EAL:   probe driver: 8086:6f23 spdk_ioat

EAL: PCI device 0000:00:04.4 on NUMA socket 0

EAL:   probe driver: 8086:6f24 spdk_ioat

EAL: PCI device 0000:00:04.5 on NUMA socket 0

EAL:   probe driver: 8086:6f25 spdk_ioat

EAL: PCI device 0000:00:04.6 on NUMA socket 0

EAL:   probe driver: 8086:6f26 spdk_ioat

EAL: PCI device 0000:00:04.7 on NUMA socket 0

EAL:   probe driver: 8086:6f27 spdk_ioat

EAL: PCI device 0000:07:00.0 on NUMA socket 0

EAL:   probe driver: 8086:953 spdk_nvme

nvmf.c:  67:spdk_nvmf_tgt_init: *INFO*: Max Queue Pairs Per Controller: 4

nvmf.c:  68:spdk_nvmf_tgt_init: *INFO*: Max Queue Depth: 128

nvmf.c:  69:spdk_nvmf_tgt_init: *INFO*: Max In Capsule Data: 4096 bytes

nvmf.c:  70:spdk_nvmf_tgt_init: *INFO*: Max I/O Size: 131072 bytes

nvmf_tgt.c: 215:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem
nqn.2014-08.org.nvmexpress.discovery on lcore 0 on socket 0

nvmf_tgt.c: 215:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem
nqn.2016-06.io.spdk:cnode1 on lcore 0 on socket 0

transport.c:  63:spdk_nvmf_transport_create: *ERROR*: inside
spdk_nvmf_transport_create 1

rdma.c: 955:spdk_nvmf_rdma_create: *NOTICE*: *** RDMA Transport Init ***

rdma.c:1120:spdk_nvmf_rdma_listen: *NOTICE*: *** NVMf Target Listening on
10.3.7.2 port 4420 ***

bdev.c: 214:spdk_bdev_get_by_name: *ERROR*: bdev's returned null 'Nvme0n1'

*conf.c: 492:spdk_nvmf_construct_subsystem: *ERROR*: Could not find
namespace bdev 'Nvme0n1'*

subsystem.c: 232:spdk_nvmf_delete_subsystem: *INFO*: subsystem is 0x1218ef0

nvmf_tgt.c: 276:spdk_nvmf_startup: *ERROR*: spdk_nvmf_parse_conf() failed

subsystem.c: 232:spdk_nvmf_delete_subsystem: *INFO*: subsystem is 0x1218bf0



*# lspci  |grep -i vola*

07:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center
SSD (rev 01)



Thanks

-kiru







*From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Yang, Ziye
*Sent:* Tuesday, September 05, 2017 10:41 AM
*To:* Storage Performance Development Kit
*Subject:* Re: [SPDK] NVMf target configuration issue



And the other choice is to remove the Host related line, which means any
host can connect.



*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Yang, Ziye
*Sent:* Tuesday, September 5, 2017 1:04 PM
*To:* Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject:* Re: [SPDK] NVMf target configuration issue



Hi Kiru,



According to the error message, I suggest that you add the Host info in the
following section.





[Subsystem1]

  NQN nqn.2016-06.io.spdk:cnode1

  Core 0

  Listen RDMA 10.3.7.2:4420

  Host nqn.2016-06.io.spdk:init

*                 Host
nqn.2014-08.org.nvmexpress:NVMf:uuid:5e5e0064-2ad6-4a43-9c26-063e3ef6cf14*

  SN SPDK00000000000001

  Namespace AIO0





Thanks.



*From:* SPDK [mailto:spdk-bounces(a)lists.01.org <spdk-bounces(a)lists.01.org>] *On
Behalf Of *Kirubakaran Kaliannan
*Sent:* Tuesday, September 5, 2017 12:52 PM
*To:* Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject:* [SPDK] NVMf target configuration issue







Hi All,



I am trying to get the nvmf configuration going with SPDK.



I have the following configuration,



*# ofed 2.4 on both target and initiator*

*# SPDK + 3.18 kernel on target *

*# 4.9 kernel on initiator*

*# mellanox connect3X-Pro*



*/etc/spdk/nvmf.conf*



[Global]

[Rpc]

  Enable No

  Listen 127.0.0.1

[AIO]

  AIO /dev/sdd AIO0



[Nvmf]

  MaxQueuesPerSession 4

  AcceptorPollRate 10000



[Subsystem1]

  NQN nqn.2016-06.io.spdk:cnode1

  Core 0

  Listen RDMA 10.3.7.2:4420

  Host nqn.2016-06.io.spdk:init

  SN SPDK00000000000001

  Namespace AIO0





*From initiator I tried discover and connect*



*# nvme discover -t rdma -a 10.3.7.2 -s 4420*



Discovery Log Number of Records 1, Generation counter 4

=====Discovery Log Entry 0======

trtype:  rdma

adrfam:  ipv4

subtype: nvme subsystem

treq:    not specified

portid:  0

trsvcid: 4420

subnqn:  nqn.2016-06.io.spdk:cnode1

traddr:  10.3.7.2

rdma_prtype: not specified

rdma_qptype: connected

rdma_cms:    rdma-cm

rdma_pkey: 0x0000



*# nvme connect -t rdma -n "nqn.2016-06.io.spdk:cnode1" -a 10.3.7.2 -s 4420*

Failed to write to /dev/nvme-fabrics: Input/output error



[dmesg]

[59909.541392] nvme nvme0: new ctrl: NQN
"nqn.2014-08.org.nvmexpress.discovery", addr 10.3.7.2:4420

[59940.389952] nvme nvme0: Connect command failed, error wo/DNR bit: 388



*On target I get the following error*



# app/nvmf_tgt/nvmf_tgt -c /etc/spdk/nvmf.conf  > /tmp/1

EAL: Detected 20 lcore(s)

reactor.c: 314:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on
socket 0

copy_engine_ioat.c: 306:copy_engine_ioat_init: *NOTICE*: Ioat Copy Engine
Offload Enabled

nvmf_tgt.c: 215:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem
nqn.2014-08.org.nvmexpress.discovery on lcore 0 on socket 0

nvmf_tgt.c: 215:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem
nqn.2016-06.io.spdk:cnode1 on lcore 0 on socket 0

rdma.c: 955:spdk_nvmf_rdma_create: *NOTICE*: *** RDMA Transport Init ***

rdma.c:1120:spdk_nvmf_rdma_listen: *NOTICE*: *** NVMf Target Listening on
10.3.7.2 port 4420 ***

conf.c: 500:spdk_nvmf_construct_subsystem: *NOTICE*: Attaching block device
AIO0 to subsystem nqn.2016-06.io.spdk:cnode1

nvmf_tgt.c: 290:spdk_nvmf_startup: *NOTICE*: Acceptor running on core 0 on
socket 0

*request.c: 171:nvmf_process_connect: *ERROR*: Subsystem
'nqn.2016-06.io.spdk:cnode1' does not allow host
'nqn.2014-08.org.nvmexpress:NVMf:uuid:5e5e0064-2ad6-4a43-9c26-063e3ef6cf14'*





can you please help on why I am not able to connect in this configuration.



Regards,

-kiru

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 29075 bytes --]