From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============4797047555578942618==" MIME-Version: 1.0 From: Kirubakaran Kaliannan Subject: Re: [SPDK] using nvmf target/initiator in spdk Date: Sun, 07 Aug 2016 10:04:39 +0530 Message-ID: <8ec52eeb74a909c6a2d4e550cee61345@mail.gmail.com> In-Reply-To: 22cc1b66764baccc4307f2bf09d499e9@mail.gmail.com List-ID: To: spdk@lists.01.org --===============4797047555578942618== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi All, I may need your help to resolve this issue (with spdk (3.18 kernel) vs nvme-fabrics (4.17+rc2 kernel) I have the nvme controller from Intel on server1, and exported using nvmf_tgt. On server 2, I have the latest kernel with nvme-fabrics installed. Trying to run discover, and I am hitting the following error. (using the native rdma/mlx drivers) # nvme discover -t rdma -a 150.50.3.22 -s 4420 Aug 5 14:05:50.202036 server-0000027c-vc-1 kernel: [ 6071.642654] nvme nvme0: Connect command failed, error wo/DNR bit: 6 (NVME_SC_INTERNAL) On server1, strace of nvmf_tgt, I am getting following error continuously write(26, "\f\0\0\0\10\0H\1 hRT\375\177\0\0", 16) =3D -1 EAGAIN (Resource temporarily unavailable) write(26, "\f\0\0\0\10\0H\1 hRT\375\177\0\0", 16) =3D -1 EAGAIN (Resource temporarily unavailable) Thanks, -kiru -----Original Message----- From: Kirubakaran Kaliannan [mailto:kirubak(a)zadarastorage.com] Sent: Thursday, August 04, 2016 1:08 PM To: 'Storage Performance Development Kit' Subject: RE: [SPDK] using nvmf target/initiator in spdk Hi, I have the following EAGAIN issue, when I start the nvmf_tgt on the server. Do we have any OFED dependencies or any other package/configuration ? I followed the nvmf_getting started document. # tail -f /etc/spdk/nvmf.conf [Subsystem1] NQN nqn.2016-06.io.spdk:cnode1 Mode Direct Listen RDMA 150.50.3.22:4420 Host nqn.2016-06.io.spdk:init NVMe 0000:05:00.0 # strace app/nvmf_tgt/nvmf_tgt -c /etc/spdk/nvmf.conf > /tmp/out 2>&1 open("/dev/infiniband/rdma_cm", O_RDWR) =3D 26 write(2, "*** NVMf Target Listening on por"..., 43*** NVMf Target Listening on port 4420 *** ) =3D 43 sendto(4, "<29>Aug 4 10:22:41 nvmf[11287]:"..., 76, MSG_NOSIGNAL, NULL, 0) =3D 76 write(26, "\f\0\0\0\10\0H\1\220\335\241\236\374\177\0\0", 16) =3D -1 EAGAIN (Resource temporarily unavailable) write(26, "\f\0\0\0\10\0H\1\220\335\241\236\374\177\0\0", 16) =3D -1 EAGAIN (Resource temporarily unavailable) write(26, "\f\0\0\0\10\0H\1\220\335\241\236\374\177\0\0", 16) =3D -1 EAGAIN (Resource temporarily unavailable) write(26, "\f\0\0\0\10\0H\1\220\335\241\236\374\177\0\0", 16) =3D -1 EAGAIN (Resource temporarily unavailable) ...... # dpkg -l | grep -i ofed ii ibverbs-utils 1.1.8mlnx1-OFED.2.4.1.0.7 amd64 Examples for the libibverbs library ii libibverbs1 1.1.8mlnx1-OFED.2.4.1.0.7 amd64 Library for direct userspace use of RDMA (InfiniBand/iWARP) ii libmlx4-1 1.0.6mlnx1-OFED.2.4.0.1.2 amd64 Userspace driver for Mellanox ConnectX InfiniBand HCAs ii librdmacm-utils 1.0.19mlnx-OFED.2.4.28.g50d12b8 amd64 Examples for the librdmacm library ii librdmacm1 1.0.19mlnx-OFED.2.4.28.g50d12b8 amd64 Userspace RDMA Connection Manager ii mlnx-ofed-kernel-utils 2.4-OFED.2.4.1.1.2.1.g5b97871 amd64 Userspace tools to restart and tune mlnx-ofed kernel modules ii ofed-scripts 2.4-OFED.2.4.1.1.2 amd64 MLNX_OFED utilities # nvme_manage list-controller NVMe Controller: 0000:05:00.00 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D Controller Capabilities/Features Controller ID: 0 Serial Number: CVMD439300582P0EGN Admin Command Set Attributes =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D Namespace Manage And Attach: Not Supported Namespace Format: Supported NVM Command Set Attributes =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D Namespace format operation applies to all namespaces Namespace Attributes =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D Namespace ID:1 Size (in LBAs): 3907029168 (3726M) Capacity (in LBAs): 3907029168 (3726M) Utilization (in LBAs): 3907029168 (3726M) Format Progress Indicator: Not Supported Number of LBA Formats: 7 Current LBA Format: LBA Format #00 LBA Format #00: Data Size: 512 Metadata Size: 0 LBA Format #01: Data Size: 512 Metadata Size: 8 LBA Format #02: Data Size: 512 Metadata Size: 16 LBA Format #03: Data Size: 4096 Metadata Size: 0 LBA Format #04: Data Size: 4096 Metadata Size: 8 LBA Format #05: Data Size: 4096 Metadata Size: 64 LBA Format #06: Data Size: 4096 Metadata Size: 128 Data Protection Capabilities:PIT1 Location: Tail Data Protection Setting: N/A Multipath IO and Sharing: Not Supported Thanks -kiru -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjam= in Sent: Tuesday, August 02, 2016 9:36 PM To: spdk(a)lists.01.org Subject: Re: [SPDK] using nvmf target/initiator in spdk On Tue, 2016-08-02 at 15:52 +0530, Kirubakaran Kaliannan wrote: > > > Hi, > > Do we have a document on, how to use the spdk to perform the following > > 1. List the devices that SPDK can see You can run the identify or nvme_manage example to get a nice list of all of the devices that SPDK can attach to. They are in examples/nvme. > 2. Start nvmf target driver with nvme devices We have this page: http://www.spdk.io/spdk/doc/nvmf_getting_started.html, which is accurate for our last tagged release but has gone stale for the code on 'master'. We'll have an updated one shortly. Right now it's fairly early stages, so you mostly have to just read the code. Our configuration file is reasonably well documented too. > 3. What nvmf initiator that we can use to connect this on the host > machine (ubuntu)? We test against the NVMe-oF initiator that will be added to the Linux kernel. It currently exists as a set of patches against kernel 4.8. Here is an article from Mellanox explaining how to set up your Linux kernel for NVMe-oF: https://community.mellanox.com/docs/DOC-2508. SPDK can run against essentially any modern Linux kernel, so you only have to put kernel 4.8 on the initiator system. > > > Thanks, > -kiru > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk --===============4797047555578942618==--