Re: [SPDK] using nvmf target/initiator in spdk

[Kirubakaran Kaliannan <kirubak at zadarastorage.com>]

[-- Attachment #1: Type: text/plain, Size: 6334 bytes --]

Hi All,

I may need your help to resolve this issue (with spdk (3.18 kernel) vs
nvme-fabrics  (4.17+rc2 kernel)

I have the nvme controller from Intel on server1, and exported using
nvmf_tgt.

On server 2, I have the latest kernel with nvme-fabrics installed. Trying to
run discover, and I am hitting the following error. (using the native
rdma/mlx drivers)

# nvme discover -t rdma -a 150.50.3.22 -s 4420
Aug  5 14:05:50.202036 server-0000027c-vc-1 kernel: [ 6071.642654] nvme
nvme0: Connect command failed, error wo/DNR bit: 6 (NVME_SC_INTERNAL)


On server1, strace of nvmf_tgt, I am getting following error continuously

write(26, "\f\0\0\0\10\0H\1 hRT\375\177\0\0", 16) = -1 EAGAIN (Resource
temporarily unavailable)
write(26, "\f\0\0\0\10\0H\1 hRT\375\177\0\0", 16) = -1 EAGAIN (Resource
temporarily unavailable)



Thanks,
-kiru

-----Original Message-----
From: Kirubakaran Kaliannan [mailto:kirubak(a)zadarastorage.com]
Sent: Thursday, August 04, 2016 1:08 PM
To: 'Storage Performance Development Kit'
Subject: RE: [SPDK] using nvmf target/initiator in spdk



Hi,

I have the following EAGAIN issue, when I start the nvmf_tgt on the server.

Do we have any OFED dependencies or any other package/configuration ? I
followed the nvmf_getting started document.

# tail -f /etc/spdk/nvmf.conf
[Subsystem1]
  NQN nqn.2016-06.io.spdk:cnode1
  Mode Direct
  Listen RDMA 150.50.3.22:4420
  Host nqn.2016-06.io.spdk:init
  NVMe 0000:05:00.0

# strace app/nvmf_tgt/nvmf_tgt -c /etc/spdk/nvmf.conf > /tmp/out 2>&1

open("/dev/infiniband/rdma_cm", O_RDWR) = 26 write(2, "*** NVMf Target
Listening on por"..., 43*** NVMf Target Listening on port 4420 ***
) = 43
sendto(4, "<29>Aug  4 10:22:41 nvmf[11287]:"..., 76, MSG_NOSIGNAL, NULL, 0)
= 76 write(26, "\f\0\0\0\10\0H\1\220\335\241\236\374\177\0\0", 16) = -1
EAGAIN (Resource temporarily unavailable) write(26,
"\f\0\0\0\10\0H\1\220\335\241\236\374\177\0\0", 16) = -1 EAGAIN (Resource
temporarily unavailable) write(26,
"\f\0\0\0\10\0H\1\220\335\241\236\374\177\0\0", 16) = -1 EAGAIN (Resource
temporarily unavailable) write(26,
"\f\0\0\0\10\0H\1\220\335\241\236\374\177\0\0", 16) = -1 EAGAIN (Resource
temporarily unavailable) ......

# dpkg -l | grep -i ofed
ii  ibverbs-utils                           1.1.8mlnx1-OFED.2.4.1.0.7
amd64        Examples for the libibverbs library
ii  libibverbs1                             1.1.8mlnx1-OFED.2.4.1.0.7
amd64        Library for direct userspace use of RDMA (InfiniBand/iWARP)
ii  libmlx4-1                               1.0.6mlnx1-OFED.2.4.0.1.2
amd64        Userspace driver for Mellanox ConnectX InfiniBand HCAs
ii  librdmacm-utils                         1.0.19mlnx-OFED.2.4.28.g50d12b8
amd64        Examples for the librdmacm library
ii  librdmacm1                              1.0.19mlnx-OFED.2.4.28.g50d12b8
amd64        Userspace RDMA Connection Manager
ii  mlnx-ofed-kernel-utils                  2.4-OFED.2.4.1.1.2.1.g5b97871
amd64        Userspace tools to restart and tune mlnx-ofed kernel modules
ii  ofed-scripts                            2.4-OFED.2.4.1.1.2
amd64        MLNX_OFED utilities

# nvme_manage list-controller

NVMe Controller:        0000:05:00.00
============================
Controller Capabilities/Features
Controller ID:          0
Serial Number:          CVMD439300582P0EGN

Admin Command Set Attributes
============================
Namespace Manage And Attach:            Not Supported
Namespace Format:                       Supported

NVM Command Set Attributes
============================
Namespace format operation applies to all namespaces

Namespace Attributes
============================
Namespace ID:1
Size (in LBAs):              3907029168 (3726M)
Capacity (in LBAs):          3907029168 (3726M)
Utilization (in LBAs):       3907029168 (3726M)
Format Progress Indicator:   Not Supported
Number of LBA Formats:       7
Current LBA Format:          LBA Format #00
LBA Format #00: Data Size:   512  Metadata Size:     0
LBA Format #01: Data Size:   512  Metadata Size:     8
LBA Format #02: Data Size:   512  Metadata Size:    16
LBA Format #03: Data Size:  4096  Metadata Size:     0
LBA Format #04: Data Size:  4096  Metadata Size:     8
LBA Format #05: Data Size:  4096  Metadata Size:    64
LBA Format #06: Data Size:  4096  Metadata Size:   128
Data Protection Capabilities:PIT1 Location: Tail
Data Protection Setting:     N/A
Multipath IO and Sharing:    Not Supported

Thanks
-kiru

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Tuesday, August 02, 2016 9:36 PM
To: spdk(a)lists.01.org
Subject: Re: [SPDK] using nvmf target/initiator in spdk

On Tue, 2016-08-02 at 15:52 +0530, Kirubakaran Kaliannan wrote:
>
>
> Hi,
>
> Do we have a document on, how to use the spdk to perform the following
>
> 1.       List the devices that SPDK can see

You can run the identify or nvme_manage example to get a nice list of all of
the devices that SPDK can attach to. They are in examples/nvme.

> 2.       Start nvmf target driver with nvme devices

We have this page: http://www.spdk.io/spdk/doc/nvmf_getting_started.html,
which is accurate for our last tagged release but has gone stale for the
code on 'master'. We'll have an updated one shortly.
Right now it's fairly early stages, so you mostly have to just read the
code. Our configuration file is reasonably well documented too.

> 3.       What nvmf initiator that we can use to connect this on the host
> machine (ubuntu)?

We test against the NVMe-oF initiator that will be added to the Linux
kernel. It currently exists as a set of patches against kernel 4.8. Here is
an article from Mellanox explaining how to set up your Linux kernel for
NVMe-oF: https://community.mellanox.com/docs/DOC-2508. SPDK can run against
essentially any modern Linux kernel, so you only have to put kernel 4.8 on
the initiator system.

>
>
> Thanks,
> -kiru
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk