* [SPDK] ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem 'nqn.2016-06.io.spdk:cnode2' does not allow host '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543'
@ 2019-07-26 17:19 sushilkumar gangadharan
0 siblings, 0 replies; 5+ messages in thread
From: sushilkumar gangadharan @ 2019-07-26 17:19 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5885 bytes --]
Hi
I am trying to setup a docker container which then talks to Name drive
using SPDK. In the first phase of the experimenting, I did the following
within the docker container and it seems to work:
508 app/nvmf_tgt/nvmf_tgt
509 scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
510 scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
511 scripts/rpc.py construct_malloc_bdev -b Malloc0 512 512
512 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode1 -a
-s SPDK00000000000001 -d SPDK_Controller1
513 scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1
Malloc0
514 scripts/rpc.py nvmf_subsystem_add_listener
nqn.2016-06.io.spdk:cnode1 -t TCP -a 172.20.0.2 -s 4420
Was able to verify that Identify works fine:
sh-4.2# ./identify -r 'trtype:TCP adrfam:IPv4 traddr:172.20.0.2
trsvcid:4420'
Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
[ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --no-pci
--log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
--file-prefix=spdk_pid498 ]
EAL: No available hugepages reported in hugepages-1048576kB
EAL: VFIO support initialized
=====================================================
NVMe over Fabrics controller at 172.20.0.2:4420:
nqn.2014-08.org.nvmexpress.discovery
=====================================================
Controller Capabilities/Features
================================
Vendor ID: 0000
Subsystem Vendor ID: 0000
Serial Number: ....................
Model Number:
........................................
Firmware Version: 19.04.1
Recommended Arb Burst: 0
IEEE OUI Identifier: 00 00 00
Multi-path I/O
May have multiple subsystem ports: No
May be connected to multiple hosts: No
Associated with SR-IOV VF: No
Max Data Transfer Size: 131072
NVMe Specification Version (VS): 1.3
NVMe Specification Version (Identify): 1.3
Maximum Queue Entries: 128
Contiguous Queues Required: Yes
Arbitration Mechanisms Supported
....
Then I tried I/O :
sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10 -r
'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
subnqn:nqn.2016-06.io.spdk:cnode1'
Ignoring -M option... Please use -M option only when using rw or randrw.
Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
[ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
--log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
--file-prefix=spdk_pid549 ]
EAL: No available hugepages reported in hugepages-1048576kB
EAL: VFIO support initialized
Initializing NVMe Controllers
Attaching to NVMe over Fabrics controller at 172.20.0.2:4420:
nqn.2016-06.io.spdk:cnode1
Attached to NVMe over Fabrics controller at 172.20.0.2:4420:
nqn.2016-06.io.spdk:cnode1
Associating SPDK_Controller1 (SPDK00000000000001 ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS
MiB/s Average min max
SPDK_Controller1 (SPDK00000000000001 ) from core 0: 37.60
0.15 26709.88 14961.21 39904.69
========================================================
Total : 37.60
0.15 26709.88 14961.21 39904.69
Then on the same container I tried the following steps :
604 scripts/rpc.py construct_nvme_bdev -b nvme1 -t PCIe -a 0000:1d:00.00
605 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 a -s
SPDK00000000000002 -d SPDK_Controller2
606 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 -sa
-s SPDK00000000000002 -d SPDK_Controller2
607 scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2
nvme1n1
608 scripts/rpc.py nvmf_subsystem_add_listener
nqn.2016-06.io.spdk:cnode2 -t TCP -a 172.20.0.2 -s 4421
This time when I try I/O I get this error :
sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10 -r
'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4421
subnqn:nqn.2016-06.io.spdk:cnode2'
Ignoring -M option... Please use -M option only when using rw or randrw.
Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
[ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
--log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
--file-prefix=spdk_pid561 ]
EAL: No available hugepages reported in hugepages-1048576kB
EAL: VFIO support initialized
Initializing NVMe Controllers
Attaching to NVMe over Fabrics controller at 172.20.0.2:4421:
nqn.2016-06.io.spdk:cnode2
ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem
'nqn.2016-06.io.spdk:cnode2' does not allow host
'2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543'
nvme_fabric.c: 327:nvme_fabric_qpair_connect: *ERROR*: Connect command
failed
nvme_tcp.c:1749:nvme_tcp_qpair_connect: *ERROR*: Failed to send an NVMe-oF
Fabric CONNECT command
nvme_tcp.c:1826:nvme_tcp_ctrlr_construct: *ERROR*: failed to create admin
qpair
nvme.c: 408:nvme_ctrlr_probe: *ERROR*: Failed to construct NVMe controller
for SSD: 172.20.0.2
nvme.c: 535:spdk_nvme_probe_internal: *ERROR*: NVMe ctrlr scan failed
nvme.c: 608:spdk_nvme_probe: *ERROR*: Create probe context failed
spdk_nvme_probe() failed for transport address '172.20.0.2'
/usr/local/bin/spdk_nvme_perf: errors occured
What does the above error mean and how do I correct this ? Is my second
test expected to work ? Apologies if this sort of newbie query. I am just
starting on this.
Thanks
Sushil
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [SPDK] ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem 'nqn.2016-06.io.spdk:cnode2' does not allow host '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543'
@ 2019-07-26 17:34 Walker, Benjamin
0 siblings, 0 replies; 5+ messages in thread
From: Walker, Benjamin @ 2019-07-26 17:34 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 7353 bytes --]
On Fri, 2019-07-26 at 10:19 -0700, sushilkumar gangadharan wrote:
> Hi
> I am trying to setup a docker container which then talks to Name drive
> using SPDK. In the first phase of the experimenting, I did the following
> within the docker container and it seems to work:
>
> 508 app/nvmf_tgt/nvmf_tgt
> 509 scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
> 510 scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
Creating the tcp transport a second time doesn't fail? It should. You can only
have one of each type of transport.
> 511 scripts/rpc.py construct_malloc_bdev -b Malloc0 512 512
> 512 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode1 -a
> -s SPDK00000000000001 -d SPDK_Controller1
Note for later the use of the '-a' parameter when creating the subsystem, which
means allow any host to connect.
> 513 scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1
> Malloc0
> 514 scripts/rpc.py nvmf_subsystem_add_listener
> nqn.2016-06.io.spdk:cnode1 -t TCP -a 172.20.0.2 -s 4420
>
> Was able to verify that Identify works fine:
> sh-4.2# ./identify -r 'trtype:TCP adrfam:IPv4 traddr:172.20.0.2
> trsvcid:4420'
> Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> [ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --no-pci
> --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> --file-prefix=spdk_pid498 ]
> EAL: No available hugepages reported in hugepages-1048576kB
> EAL: VFIO support initialized
> =====================================================
> NVMe over Fabrics controller at 172.20.0.2:4420:
> nqn.2014-08.org.nvmexpress.discovery
> =====================================================
> Controller Capabilities/Features
> ================================
> Vendor ID: 0000
> Subsystem Vendor ID: 0000
> Serial Number: ....................
> Model Number:
> ........................................
> Firmware Version: 19.04.1
> Recommended Arb Burst: 0
> IEEE OUI Identifier: 00 00 00
> Multi-path I/O
> May have multiple subsystem ports: No
> May be connected to multiple hosts: No
> Associated with SR-IOV VF: No
> Max Data Transfer Size: 131072
> NVMe Specification Version (VS): 1.3
> NVMe Specification Version (Identify): 1.3
> Maximum Queue Entries: 128
> Contiguous Queues Required: Yes
> Arbitration Mechanisms Supported
> ....
>
> Then I tried I/O :
> sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10 -r
> 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
> subnqn:nqn.2016-06.io.spdk:cnode1'
> Ignoring -M option... Please use -M option only when using rw or randrw.
> Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> --file-prefix=spdk_pid549 ]
> EAL: No available hugepages reported in hugepages-1048576kB
> EAL: VFIO support initialized
> Initializing NVMe Controllers
> Attaching to NVMe over Fabrics controller at 172.20.0.2:4420:
> nqn.2016-06.io.spdk:cnode1
> Attached to NVMe over Fabrics controller at 172.20.0.2:4420:
> nqn.2016-06.io.spdk:cnode1
> Associating SPDK_Controller1 (SPDK00000000000001 ) with lcore 0
> Initialization complete. Launching workers.
> Starting thread on core 0
> ========================================================
>
> Latency(us)
> Device Information : IOPS
> MiB/s Average min max
> SPDK_Controller1 (SPDK00000000000001 ) from core 0: 37.60
> 0.15 26709.88 14961.21 39904.69
> ========================================================
> Total : 37.60
> 0.15 26709.88 14961.21 39904.69
>
>
> Then on the same container I tried the following steps :
>
> 604 scripts/rpc.py construct_nvme_bdev -b nvme1 -t PCIe -a 0000:1d:00.00
> 605 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 a -s
> SPDK00000000000002 -d SPDK_Controller2
You forgot the '-' on the a. I believe this command failed in your script. Try
adding set -e to the top so that your script exits on error. If this doesn't
fail, please file a GitHub issue. It should not have allowed you to proceed.
> 606 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 -sa
> -s SPDK00000000000002 -d SPDK_Controller2
This should be failing - you can't create a subsystem twice. Note also that you
have the -sa option, which is not valid.
> 607 scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2
> nvme1n1
> 608 scripts/rpc.py nvmf_subsystem_add_listener
> nqn.2016-06.io.spdk:cnode2 -t TCP -a 172.20.0.2 -s 4421
>
>
>
>
> This time when I try I/O I get this error :
> sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10 -r
> 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4421
> subnqn:nqn.2016-06.io.spdk:cnode2'
> Ignoring -M option... Please use -M option only when using rw or randrw.
> Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> --file-prefix=spdk_pid561 ]
> EAL: No available hugepages reported in hugepages-1048576kB
> EAL: VFIO support initialized
> Initializing NVMe Controllers
> Attaching to NVMe over Fabrics controller at 172.20.0.2:4421:
> nqn.2016-06.io.spdk:cnode2
> ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem
> 'nqn.2016-06.io.spdk:cnode2' does not allow host
> '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543'
> nvme_fabric.c: 327:nvme_fabric_qpair_connect: *ERROR*: Connect command
> failed
> nvme_tcp.c:1749:nvme_tcp_qpair_connect: *ERROR*: Failed to send an NVMe-oF
> Fabric CONNECT command
> nvme_tcp.c:1826:nvme_tcp_ctrlr_construct: *ERROR*: failed to create admin
> qpair
> nvme.c: 408:nvme_ctrlr_probe: *ERROR*: Failed to construct NVMe controller
> for SSD: 172.20.0.2
> nvme.c: 535:spdk_nvme_probe_internal: *ERROR*: NVMe ctrlr scan failed
> nvme.c: 608:spdk_nvme_probe: *ERROR*: Create probe context failed
> spdk_nvme_probe() failed for transport address '172.20.0.2'
> /usr/local/bin/spdk_nvme_perf: errors occured
>
> What does the above error mean and how do I correct this ? Is my second
> test expected to work ? Apologies if this sort of newbie query. I am just
> starting on this.
It means you didn't set up the access control on the subsystem to allow this
host to connect. "Subsystem [xyz] does not allow host [ijk]". That's because you
didn't pass valid parameters to the command that creates the subsystem, and
those commands should have failed. If they don't fail today, please do file an
issue on GitHub and we'll get that sorted out.
Thanks,
Ben
>
> Thanks
> Sushil
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [SPDK] ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem 'nqn.2016-06.io.spdk:cnode2' does not allow host '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543'
@ 2019-07-26 18:48 sushilkumar gangadharan
0 siblings, 0 replies; 5+ messages in thread
From: sushilkumar gangadharan @ 2019-07-26 18:48 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 13650 bytes --]
Thanks Ben for the response. I am trying to talk to an NvMe drive and with
the "-a" option it looks to be working.
This is the corrected sequence of commands.
sh-4.2# scripts/setup.sh
0000:0f:00.0 (144d a802): Already using the vfio-pci driver
0000:11:00.0 (144d a802): Already using the vfio-pci driver
0000:13:00.0 (144d a802): Already using the vfio-pci driver
0000:15:00.0 (144d a802): Already using the vfio-pci driver
0000:17:00.0 (144d a802): Already using the vfio-pci driver
0000:19:00.0 (144d a802): Already using the vfio-pci driver
0000:1b:00.0 (144d a802): Already using the vfio-pci driver
0000:1d:00.0 (144d a802): Already using the vfio-pci driver
Mounting hugetlbfs at /mnt/huge
Current user memlock limit: 0 MB
This is the maximum amount of memory you will be
able to use with DPDK and VFIO if run as current user.
To change this, please adjust limits.conf memlock limit for current user.
## WARNING: memlock limit is less than 64MB
## DPDK with VFIO may not be able to initialize if run as current user.
sh-4.2# app/nvmf_tgt/nvmf_tgt &
[1] 457
sh-4.2# Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
[ DPDK EAL parameters: nvmf --no-shconf -c 0x1 --log-level=lib.eal:6
--base-virtaddr=0x200000000000 --match-allocations
--file-prefix=spdk_pid457 ]
EAL: No available hugepages reported in hugepages-1048576kB
EAL: VFIO support initialized
app.c: 627:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 251:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
tcp.c: 543:spdk_nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
sh-4.2# scripts/rpc.py construct_nvme_bdev -b nvme1 -t PCIe -a 0000:1d:00.00
EAL: using IOMMU type 1 (Type 1)
nvme1n1
sh-4.2# scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 -a
-s SPDK00000000000001 -d SPDK_Controller1
sh-4.2# scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2
nvme1n1
sh-4.2# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.20.0.2 netmask 255.255.0.0 broadcast 172.20.255.255
ether 02:42:ac:14:00:02 txqueuelen 0 (Ethernet)
RX packets 14 bytes 1156 (1.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
sh-4.2# scripts/rpc.py nvmf_subsystem_add_listener
nqn.2016-06.io.spdk:cnode2 -t TCP -a 172.20.0.2 -s 4420
tcp.c: 726:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on
172.20.0.2 port 4420 ***
sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10 -r
'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
subnqn:nqn.2016-06.io.spdk:cnode2'
sh: /usr/local/bin/spdk_nvme_perf: No such file or directory
sh-4.2# cd examples/nvme/perf/make
sh: cd: examples/nvme/perf/make: No such file or directory
sh-4.2# cd examples/nvme/perf
sh-4.2# make
INSTALL /usr/local/bin/spdk_nvme_perf
sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -t 10 -r
'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
subnqn:nqn.2016-06.io.spdk:cnode2'
Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
[ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
--log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
--file-prefix=spdk_pid475 ]
EAL: No available hugepages reported in hugepages-1048576kB
EAL: VFIO support initialized
Initializing NVMe Controllers
Attaching to NVMe over Fabrics controller at 172.20.0.2:4420:
nqn.2016-06.io.spdk:cnode2
Attached to NVMe over Fabrics controller at 172.20.0.2:4420:
nqn.2016-06.io.spdk:cnode2
Associating SPDK_Controller1 (SPDK00000000000001 ) with lcore 0
Initialization complete. Launching workers.
Starting thread on core 0
========================================================
Latency(us)
Device Information : IOPS
MiB/s Average min max
SPDK_Controller1 (SPDK00000000000001 ) from core 0: 37.50
0.15 26709.32 14964.37 47885.26
========================================================
Total : 37.50
0.15 26709.32 14964.37 47885.26
sh-4.2#
So did "-a" make the difference ? If I don't give the "-a" and gave only
sh-4.2# scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 -s
SPDK00000000000001 -d SPDK_Controller1
will the above mean that no hosts are allowed to connect ?
Also you mentioned :
It means you didn't set up the access control on the subsystem to allow this
host to connect. "Subsystem [xyz] does not allow host [ijk]". That's
because you
didn't pass valid parameters to the command that creates the subsystem, and
those commands should have failed. If they don't fail today, please do file
an
issue on GitHub and we'll get that sorted out.
How does one create a subsystem such that it permits certain hosts ? Is
there some documentation for this ?
Also added some comments inline ( regarding whether the commands failed or
not )..
Thanks
Sushil
On Fri, Jul 26, 2019 at 10:34 AM Walker, Benjamin <benjamin.walker(a)intel.com>
wrote:
> On Fri, 2019-07-26 at 10:19 -0700, sushilkumar gangadharan wrote:
> > Hi
> > I am trying to setup a docker container which then talks to Name drive
>
[Sushil] I meant Nvme ( autocorrect)
> > using SPDK. In the first phase of the experimenting, I did the following
> > within the docker container and it seems to work:
> >
> > 508 app/nvmf_tgt/nvmf_tgt
> > 509 scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
> > 510 scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
>
> Creating the tcp transport a second time doesn't fail? It should. You can
> only
> have one of each type of transport.
>
[Sushil] it did fail.
>
> > 511 scripts/rpc.py construct_malloc_bdev -b Malloc0 512 512
> > 512 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode1 -a
> > -s SPDK00000000000001 -d SPDK_Controller1
>
> Note for later the use of the '-a' parameter when creating the subsystem,
> which
> means allow any host to connect.
>
> > 513 scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1
> > Malloc0
> > 514 scripts/rpc.py nvmf_subsystem_add_listener
> > nqn.2016-06.io.spdk:cnode1 -t TCP -a 172.20.0.2 -s 4420
> >
> > Was able to verify that Identify works fine:
> > sh-4.2# ./identify -r 'trtype:TCP adrfam:IPv4 traddr:172.20.0.2
> > trsvcid:4420'
> > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> > [ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --no-pci
> > --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> > --file-prefix=spdk_pid498 ]
> > EAL: No available hugepages reported in hugepages-1048576kB
> > EAL: VFIO support initialized
> > =====================================================
> > NVMe over Fabrics controller at 172.20.0.2:4420:
> > nqn.2014-08.org.nvmexpress.discovery
> > =====================================================
> > Controller Capabilities/Features
> > ================================
> > Vendor ID: 0000
> > Subsystem Vendor ID: 0000
> > Serial Number: ....................
> > Model Number:
> > ........................................
> > Firmware Version: 19.04.1
> > Recommended Arb Burst: 0
> > IEEE OUI Identifier: 00 00 00
> > Multi-path I/O
> > May have multiple subsystem ports: No
> > May be connected to multiple hosts: No
> > Associated with SR-IOV VF: No
> > Max Data Transfer Size: 131072
> > NVMe Specification Version (VS): 1.3
> > NVMe Specification Version (Identify): 1.3
> > Maximum Queue Entries: 128
> > Contiguous Queues Required: Yes
> > Arbitration Mechanisms Supported
> > ....
> >
> > Then I tried I/O :
> > sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10 -r
> > 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
> > subnqn:nqn.2016-06.io.spdk:cnode1'
> > Ignoring -M option... Please use -M option only when using rw or randrw.
> > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> > --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> > --file-prefix=spdk_pid549 ]
> > EAL: No available hugepages reported in hugepages-1048576kB
> > EAL: VFIO support initialized
> > Initializing NVMe Controllers
> > Attaching to NVMe over Fabrics controller at 172.20.0.2:4420:
> > nqn.2016-06.io.spdk:cnode1
> > Attached to NVMe over Fabrics controller at 172.20.0.2:4420:
> > nqn.2016-06.io.spdk:cnode1
> > Associating SPDK_Controller1 (SPDK00000000000001 ) with lcore 0
> > Initialization complete. Launching workers.
> > Starting thread on core 0
> > ========================================================
> >
> > Latency(us)
> > Device Information : IOPS
> > MiB/s Average min max
> > SPDK_Controller1 (SPDK00000000000001 ) from core 0: 37.60
> > 0.15 26709.88 14961.21 39904.69
> > ========================================================
> > Total : 37.60
> > 0.15 26709.88 14961.21 39904.69
> >
> >
> > Then on the same container I tried the following steps :
> >
> > 604 scripts/rpc.py construct_nvme_bdev -b nvme1 -t PCIe -a
> 0000:1d:00.00
> > 605 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 a
> -s
> > SPDK00000000000002 -d SPDK_Controller2
>
> You forgot the '-' on the a. I believe this command failed in your script.
> Try
> adding set -e to the top so that your script exits on error. If this
> doesn't
> fail, please file a GitHub issue. It should not have allowed you to
> proceed.
> > 606 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2
> -sa
> > -s SPDK00000000000002 -d SPDK_Controller2
>
[Sushil] I dumping the history and the command did fail.
>
> This should be failing - you can't create a subsystem twice. Note also
> that you
> have the -sa option, which is not valid.
>
> > 607 scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2
> > nvme1n1
> > 608 scripts/rpc.py nvmf_subsystem_add_listener
> > nqn.2016-06.io.spdk:cnode2 -t TCP -a 172.20.0.2 -s 4421
> >
> >
> >
> >
> > This time when I try I/O I get this error :
> > sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10
> -r
> > 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4421
> > subnqn:nqn.2016-06.io.spdk:cnode2'
> > Ignoring -M option... Please use -M option only when using rw or randrw.
> > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> > --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> > --file-prefix=spdk_pid561 ]
> > EAL: No available hugepages reported in hugepages-1048576kB
> > EAL: VFIO support initialized
> > Initializing NVMe Controllers
> > Attaching to NVMe over Fabrics controller at 172.20.0.2:4421:
> > nqn.2016-06.io.spdk:cnode2
> > ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem
> > 'nqn.2016-06.io.spdk:cnode2' does not allow host
> > '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543'
> > nvme_fabric.c: 327:nvme_fabric_qpair_connect: *ERROR*: Connect command
> > failed
> > nvme_tcp.c:1749:nvme_tcp_qpair_connect: *ERROR*: Failed to send an
> NVMe-oF
> > Fabric CONNECT command
> > nvme_tcp.c:1826:nvme_tcp_ctrlr_construct: *ERROR*: failed to create admin
> > qpair
> > nvme.c: 408:nvme_ctrlr_probe: *ERROR*: Failed to construct NVMe
> controller
> > for SSD: 172.20.0.2
> > nvme.c: 535:spdk_nvme_probe_internal: *ERROR*: NVMe ctrlr scan failed
> > nvme.c: 608:spdk_nvme_probe: *ERROR*: Create probe context failed
> > spdk_nvme_probe() failed for transport address '172.20.0.2'
> > /usr/local/bin/spdk_nvme_perf: errors occured
> >
> > What does the above error mean and how do I correct this ? Is my second
> > test expected to work ? Apologies if this sort of newbie query. I am just
> > starting on this.
>
> It means you didn't set up the access control on the subsystem to allow
> this
> host to connect. "Subsystem [xyz] does not allow host [ijk]". That's
> because you
> didn't pass valid parameters to the command that creates the subsystem, and
> those commands should have failed. If they don't fail today, please do
> file an
> issue on GitHub and we'll get that sorted out.
>
> Thanks,
> Ben
>
> >
> > Thanks
> > Sushil
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [SPDK] ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem 'nqn.2016-06.io.spdk:cnode2' does not allow host '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543'
@ 2019-07-26 18:56 Walker, Benjamin
0 siblings, 0 replies; 5+ messages in thread
From: Walker, Benjamin @ 2019-07-26 18:56 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 14859 bytes --]
On Fri, 2019-07-26 at 11:48 -0700, sushilkumar gangadharan wrote:
> Thanks Ben for the response. I am trying to talk to an NvMe drive and with
> the "-a" option it looks to be working.
>
> This is the corrected sequence of commands.
>
> sh-4.2# scripts/setup.sh
> 0000:0f:00.0 (144d a802): Already using the vfio-pci driver
> 0000:11:00.0 (144d a802): Already using the vfio-pci driver
> 0000:13:00.0 (144d a802): Already using the vfio-pci driver
> 0000:15:00.0 (144d a802): Already using the vfio-pci driver
> 0000:17:00.0 (144d a802): Already using the vfio-pci driver
> 0000:19:00.0 (144d a802): Already using the vfio-pci driver
> 0000:1b:00.0 (144d a802): Already using the vfio-pci driver
> 0000:1d:00.0 (144d a802): Already using the vfio-pci driver
> Mounting hugetlbfs at /mnt/huge
>
> Current user memlock limit: 0 MB
>
> This is the maximum amount of memory you will be
> able to use with DPDK and VFIO if run as current user.
> To change this, please adjust limits.conf memlock limit for current user.
>
> ## WARNING: memlock limit is less than 64MB
> ## DPDK with VFIO may not be able to initialize if run as current user.
>
> sh-4.2# app/nvmf_tgt/nvmf_tgt &
> [1] 457
> sh-4.2# Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> [ DPDK EAL parameters: nvmf --no-shconf -c 0x1 --log-level=lib.eal:6
> --base-virtaddr=0x200000000000 --match-allocations
> --file-prefix=spdk_pid457 ]
> EAL: No available hugepages reported in hugepages-1048576kB
> EAL: VFIO support initialized
> app.c: 627:spdk_app_start: *NOTICE*: Total cores available: 1
> reactor.c: 251:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
> tcp.c: 543:spdk_nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
> sh-4.2# scripts/rpc.py construct_nvme_bdev -b nvme1 -t PCIe -a 0000:1d:00.00
> EAL: using IOMMU type 1 (Type 1)
> nvme1n1
>
> sh-4.2# scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 -a
> -s SPDK00000000000001 -d SPDK_Controller1
>
> sh-4.2# scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2
> nvme1n1
>
> sh-4.2# ifconfig
> eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet 172.20.0.2 netmask 255.255.0.0 broadcast 172.20.255.255
> ether 02:42:ac:14:00:02 txqueuelen 0 (Ethernet)
> RX packets 14 bytes 1156 (1.1 KiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 0 bytes 0 (0.0 B)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
> inet 127.0.0.1 netmask 255.0.0.0
> loop txqueuelen 1000 (Local Loopback)
> RX packets 0 bytes 0 (0.0 B)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 0 bytes 0 (0.0 B)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> sh-4.2# scripts/rpc.py nvmf_subsystem_add_listener
> nqn.2016-06.io.spdk:cnode2 -t TCP -a 172.20.0.2 -s 4420
>
> tcp.c: 726:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on
> 172.20.0.2 port 4420 ***
>
> sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10 -r
> 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
> subnqn:nqn.2016-06.io.spdk:cnode2'
> sh: /usr/local/bin/spdk_nvme_perf: No such file or directory
> sh-4.2# cd examples/nvme/perf/make
> sh: cd: examples/nvme/perf/make: No such file or directory
> sh-4.2# cd examples/nvme/perf
> sh-4.2# make
> INSTALL /usr/local/bin/spdk_nvme_perf
>
> sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -t 10 -r
> 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
> subnqn:nqn.2016-06.io.spdk:cnode2'
>
> Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> --file-prefix=spdk_pid475 ]
> EAL: No available hugepages reported in hugepages-1048576kB
> EAL: VFIO support initialized
> Initializing NVMe Controllers
> Attaching to NVMe over Fabrics controller at 172.20.0.2:4420:
> nqn.2016-06.io.spdk:cnode2
> Attached to NVMe over Fabrics controller at 172.20.0.2:4420:
> nqn.2016-06.io.spdk:cnode2
> Associating SPDK_Controller1 (SPDK00000000000001 ) with lcore 0
> Initialization complete. Launching workers.
> Starting thread on core 0
> ========================================================
>
> Latency(us)
> Device Information : IOPS
> MiB/s Average min max
> SPDK_Controller1 (SPDK00000000000001 ) from core 0: 37.50
> 0.15 26709.32 14964.37 47885.26
> ========================================================
> Total : 37.50
> 0.15 26709.32 14964.37 47885.26
>
> sh-4.2#
>
>
> So did "-a" make the difference ? If I don't give the "-a" and gave only
>
> sh-4.2# scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 -s
> SPDK00000000000001 -d SPDK_Controller1
>
> will the above mean that no hosts are allowed to connect ?
Correct. By default, the access control on a subsystem allows no hosts to
connect to it. If you specify the -a option, it allows any host by default
instead.
>
> Also you mentioned :
> It means you didn't set up the access control on the subsystem to allow this
> host to connect. "Subsystem [xyz] does not allow host [ijk]". That's
> because you
> didn't pass valid parameters to the command that creates the subsystem, and
> those commands should have failed. If they don't fail today, please do file
> an
> issue on GitHub and we'll get that sorted out.
>
> How does one create a subsystem such that it permits certain hosts ? Is
> there some documentation for this ?
Create a subsystem without the '-a' option. Then use the
"nvmf_subsystem_add_host" RPC to specifically list out the hosts that you want
to allow for that subsystem.
>
> Also added some comments inline ( regarding whether the commands failed or
> not )..
>
> Thanks
> Sushil
>
>
> On Fri, Jul 26, 2019 at 10:34 AM Walker, Benjamin <benjamin.walker(a)intel.com>
> wrote:
>
> > On Fri, 2019-07-26 at 10:19 -0700, sushilkumar gangadharan wrote:
> > > Hi
> > > I am trying to setup a docker container which then talks to Name drive
> [Sushil] I meant Nvme ( autocorrect)
>
> > > using SPDK. In the first phase of the experimenting, I did the following
> > > within the docker container and it seems to work:
> > >
> > > 508 app/nvmf_tgt/nvmf_tgt
> > > 509 scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
> > > 510 scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
> >
> > Creating the tcp transport a second time doesn't fail? It should. You can
> > only
> > have one of each type of transport.
> >
> [Sushil] it did fail.
>
> > > 511 scripts/rpc.py construct_malloc_bdev -b Malloc0 512 512
> > > 512 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode1 -a
> > > -s SPDK00000000000001 -d SPDK_Controller1
> >
> > Note for later the use of the '-a' parameter when creating the subsystem,
> > which
> > means allow any host to connect.
> >
> > > 513 scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1
> > > Malloc0
> > > 514 scripts/rpc.py nvmf_subsystem_add_listener
> > > nqn.2016-06.io.spdk:cnode1 -t TCP -a 172.20.0.2 -s 4420
> > >
> > > Was able to verify that Identify works fine:
> > > sh-4.2# ./identify -r 'trtype:TCP adrfam:IPv4 traddr:172.20.0.2
> > > trsvcid:4420'
> > > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> > > [ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --no-pci
> > > --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> > > --file-prefix=spdk_pid498 ]
> > > EAL: No available hugepages reported in hugepages-1048576kB
> > > EAL: VFIO support initialized
> > > =====================================================
> > > NVMe over Fabrics controller at 172.20.0.2:4420:
> > > nqn.2014-08.org.nvmexpress.discovery
> > > =====================================================
> > > Controller Capabilities/Features
> > > ================================
> > > Vendor ID: 0000
> > > Subsystem Vendor ID: 0000
> > > Serial Number: ....................
> > > Model Number:
> > > ........................................
> > > Firmware Version: 19.04.1
> > > Recommended Arb Burst: 0
> > > IEEE OUI Identifier: 00 00 00
> > > Multi-path I/O
> > > May have multiple subsystem ports: No
> > > May be connected to multiple hosts: No
> > > Associated with SR-IOV VF: No
> > > Max Data Transfer Size: 131072
> > > NVMe Specification Version (VS): 1.3
> > > NVMe Specification Version (Identify): 1.3
> > > Maximum Queue Entries: 128
> > > Contiguous Queues Required: Yes
> > > Arbitration Mechanisms Supported
> > > ....
> > >
> > > Then I tried I/O :
> > > sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10 -r
> > > 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
> > > subnqn:nqn.2016-06.io.spdk:cnode1'
> > > Ignoring -M option... Please use -M option only when using rw or randrw.
> > > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> > > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> > > --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> > > --file-prefix=spdk_pid549 ]
> > > EAL: No available hugepages reported in hugepages-1048576kB
> > > EAL: VFIO support initialized
> > > Initializing NVMe Controllers
> > > Attaching to NVMe over Fabrics controller at 172.20.0.2:4420:
> > > nqn.2016-06.io.spdk:cnode1
> > > Attached to NVMe over Fabrics controller at 172.20.0.2:4420:
> > > nqn.2016-06.io.spdk:cnode1
> > > Associating SPDK_Controller1 (SPDK00000000000001 ) with lcore 0
> > > Initialization complete. Launching workers.
> > > Starting thread on core 0
> > > ========================================================
> > >
> > > Latency(us)
> > > Device Information : IOPS
> > > MiB/s Average min max
> > > SPDK_Controller1 (SPDK00000000000001 ) from core 0: 37.60
> > > 0.15 26709.88 14961.21 39904.69
> > > ========================================================
> > > Total : 37.60
> > > 0.15 26709.88 14961.21 39904.69
> > >
> > >
> > > Then on the same container I tried the following steps :
> > >
> > > 604 scripts/rpc.py construct_nvme_bdev -b nvme1 -t PCIe -a
> > 0000:1d:00.00
> > > 605 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 a
> > -s
> > > SPDK00000000000002 -d SPDK_Controller2
> >
> > You forgot the '-' on the a. I believe this command failed in your script.
> > Try
> > adding set -e to the top so that your script exits on error. If this
> > doesn't
> > fail, please file a GitHub issue. It should not have allowed you to
> > proceed.
> > > 606 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2
> > -sa
> > > -s SPDK00000000000002 -d SPDK_Controller2
> [Sushil] I dumping the history and the command did fail.
>
> > This should be failing - you can't create a subsystem twice. Note also
> > that you
> > have the -sa option, which is not valid.
> >
> > > 607 scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2
> > > nvme1n1
> > > 608 scripts/rpc.py nvmf_subsystem_add_listener
> > > nqn.2016-06.io.spdk:cnode2 -t TCP -a 172.20.0.2 -s 4421
> > >
> > >
> > >
> > >
> > > This time when I try I/O I get this error :
> > > sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10
> > -r
> > > 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4421
> > > subnqn:nqn.2016-06.io.spdk:cnode2'
> > > Ignoring -M option... Please use -M option only when using rw or randrw.
> > > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> > > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> > > --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> > > --file-prefix=spdk_pid561 ]
> > > EAL: No available hugepages reported in hugepages-1048576kB
> > > EAL: VFIO support initialized
> > > Initializing NVMe Controllers
> > > Attaching to NVMe over Fabrics controller at 172.20.0.2:4421:
> > > nqn.2016-06.io.spdk:cnode2
> > > ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem
> > > 'nqn.2016-06.io.spdk:cnode2' does not allow host
> > > '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543'
> > > nvme_fabric.c: 327:nvme_fabric_qpair_connect: *ERROR*: Connect command
> > > failed
> > > nvme_tcp.c:1749:nvme_tcp_qpair_connect: *ERROR*: Failed to send an
> > NVMe-oF
> > > Fabric CONNECT command
> > > nvme_tcp.c:1826:nvme_tcp_ctrlr_construct: *ERROR*: failed to create admin
> > > qpair
> > > nvme.c: 408:nvme_ctrlr_probe: *ERROR*: Failed to construct NVMe
> > controller
> > > for SSD: 172.20.0.2
> > > nvme.c: 535:spdk_nvme_probe_internal: *ERROR*: NVMe ctrlr scan failed
> > > nvme.c: 608:spdk_nvme_probe: *ERROR*: Create probe context failed
> > > spdk_nvme_probe() failed for transport address '172.20.0.2'
> > > /usr/local/bin/spdk_nvme_perf: errors occured
> > >
> > > What does the above error mean and how do I correct this ? Is my second
> > > test expected to work ? Apologies if this sort of newbie query. I am just
> > > starting on this.
> >
> > It means you didn't set up the access control on the subsystem to allow
> > this
> > host to connect. "Subsystem [xyz] does not allow host [ijk]". That's
> > because you
> > didn't pass valid parameters to the command that creates the subsystem, and
> > those commands should have failed. If they don't fail today, please do
> > file an
> > issue on GitHub and we'll get that sorted out.
> >
> > Thanks,
> > Ben
> >
> > > Thanks
> > > Sushil
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> >
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [SPDK] ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem 'nqn.2016-06.io.spdk:cnode2' does not allow host '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543'
@ 2019-07-26 19:51 sushilkumar gangadharan
0 siblings, 0 replies; 5+ messages in thread
From: sushilkumar gangadharan @ 2019-07-26 19:51 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 15848 bytes --]
Thanks Ben.
On Fri, Jul 26, 2019 at 11:56 AM Walker, Benjamin <benjamin.walker(a)intel.com>
wrote:
> On Fri, 2019-07-26 at 11:48 -0700, sushilkumar gangadharan wrote:
> > Thanks Ben for the response. I am trying to talk to an NvMe drive and
> with
> > the "-a" option it looks to be working.
> >
> > This is the corrected sequence of commands.
> >
> > sh-4.2# scripts/setup.sh
> > 0000:0f:00.0 (144d a802): Already using the vfio-pci driver
> > 0000:11:00.0 (144d a802): Already using the vfio-pci driver
> > 0000:13:00.0 (144d a802): Already using the vfio-pci driver
> > 0000:15:00.0 (144d a802): Already using the vfio-pci driver
> > 0000:17:00.0 (144d a802): Already using the vfio-pci driver
> > 0000:19:00.0 (144d a802): Already using the vfio-pci driver
> > 0000:1b:00.0 (144d a802): Already using the vfio-pci driver
> > 0000:1d:00.0 (144d a802): Already using the vfio-pci driver
> > Mounting hugetlbfs at /mnt/huge
> >
> > Current user memlock limit: 0 MB
> >
> > This is the maximum amount of memory you will be
> > able to use with DPDK and VFIO if run as current user.
> > To change this, please adjust limits.conf memlock limit for current user.
> >
> > ## WARNING: memlock limit is less than 64MB
> > ## DPDK with VFIO may not be able to initialize if run as current user.
> >
> > sh-4.2# app/nvmf_tgt/nvmf_tgt &
> > [1] 457
> > sh-4.2# Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> > [ DPDK EAL parameters: nvmf --no-shconf -c 0x1 --log-level=lib.eal:6
> > --base-virtaddr=0x200000000000 --match-allocations
> > --file-prefix=spdk_pid457 ]
> > EAL: No available hugepages reported in hugepages-1048576kB
> > EAL: VFIO support initialized
> > app.c: 627:spdk_app_start: *NOTICE*: Total cores available: 1
> > reactor.c: 251:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> > scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
> > tcp.c: 543:spdk_nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
> > sh-4.2# scripts/rpc.py construct_nvme_bdev -b nvme1 -t PCIe -a
> 0000:1d:00.00
> > EAL: using IOMMU type 1 (Type 1)
> > nvme1n1
> >
> > sh-4.2# scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2
> -a
> > -s SPDK00000000000001 -d SPDK_Controller1
> >
> > sh-4.2# scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2
> > nvme1n1
> >
> > sh-4.2# ifconfig
> > eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> > inet 172.20.0.2 netmask 255.255.0.0 broadcast 172.20.255.255
> > ether 02:42:ac:14:00:02 txqueuelen 0 (Ethernet)
> > RX packets 14 bytes 1156 (1.1 KiB)
> > RX errors 0 dropped 0 overruns 0 frame 0
> > TX packets 0 bytes 0 (0.0 B)
> > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
> >
> > lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
> > inet 127.0.0.1 netmask 255.0.0.0
> > loop txqueuelen 1000 (Local Loopback)
> > RX packets 0 bytes 0 (0.0 B)
> > RX errors 0 dropped 0 overruns 0 frame 0
> > TX packets 0 bytes 0 (0.0 B)
> > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
> >
> > sh-4.2# scripts/rpc.py nvmf_subsystem_add_listener
> > nqn.2016-06.io.spdk:cnode2 -t TCP -a 172.20.0.2 -s 4420
> >
> > tcp.c: 726:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening
> on
> > 172.20.0.2 port 4420 ***
> >
> > sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10 -r
> > 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
> > subnqn:nqn.2016-06.io.spdk:cnode2'
> > sh: /usr/local/bin/spdk_nvme_perf: No such file or directory
> > sh-4.2# cd examples/nvme/perf/make
> > sh: cd: examples/nvme/perf/make: No such file or directory
> > sh-4.2# cd examples/nvme/perf
> > sh-4.2# make
> > INSTALL /usr/local/bin/spdk_nvme_perf
> >
> > sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -t 10 -r
> > 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
> > subnqn:nqn.2016-06.io.spdk:cnode2'
> >
> > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> > --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> > --file-prefix=spdk_pid475 ]
> > EAL: No available hugepages reported in hugepages-1048576kB
> > EAL: VFIO support initialized
> > Initializing NVMe Controllers
> > Attaching to NVMe over Fabrics controller at 172.20.0.2:4420:
> > nqn.2016-06.io.spdk:cnode2
> > Attached to NVMe over Fabrics controller at 172.20.0.2:4420:
> > nqn.2016-06.io.spdk:cnode2
> > Associating SPDK_Controller1 (SPDK00000000000001 ) with lcore 0
> > Initialization complete. Launching workers.
> > Starting thread on core 0
> > ========================================================
> >
> > Latency(us)
> > Device Information : IOPS
> > MiB/s Average min max
> > SPDK_Controller1 (SPDK00000000000001 ) from core 0: 37.50
> > 0.15 26709.32 14964.37 47885.26
> > ========================================================
> > Total : 37.50
> > 0.15 26709.32 14964.37 47885.26
> >
> > sh-4.2#
> >
> >
> > So did "-a" make the difference ? If I don't give the "-a" and gave only
> >
> > sh-4.2# scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2
> -s
> > SPDK00000000000001 -d SPDK_Controller1
> >
> > will the above mean that no hosts are allowed to connect ?
>
> Correct. By default, the access control on a subsystem allows no hosts to
> connect to it. If you specify the -a option, it allows any host by default
> instead.
>
> >
> > Also you mentioned :
> > It means you didn't set up the access control on the subsystem to allow
> this
> > host to connect. "Subsystem [xyz] does not allow host [ijk]". That's
> > because you
> > didn't pass valid parameters to the command that creates the subsystem,
> and
> > those commands should have failed. If they don't fail today, please do
> file
> > an
> > issue on GitHub and we'll get that sorted out.
> >
> > How does one create a subsystem such that it permits certain hosts ? Is
> > there some documentation for this ?
>
> Create a subsystem without the '-a' option. Then use the
> "nvmf_subsystem_add_host" RPC to specifically list out the hosts that you
> want
> to allow for that subsystem.
>
> >
> > Also added some comments inline ( regarding whether the commands failed
> or
> > not )..
> >
> > Thanks
> > Sushil
> >
> >
> > On Fri, Jul 26, 2019 at 10:34 AM Walker, Benjamin <
> benjamin.walker(a)intel.com>
> > wrote:
> >
> > > On Fri, 2019-07-26 at 10:19 -0700, sushilkumar gangadharan wrote:
> > > > Hi
> > > > I am trying to setup a docker container which then talks to Name
> drive
> > [Sushil] I meant Nvme ( autocorrect)
> >
> > > > using SPDK. In the first phase of the experimenting, I did the
> following
> > > > within the docker container and it seems to work:
> > > >
> > > > 508 app/nvmf_tgt/nvmf_tgt
> > > > 509 scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c
> 8192
> > > > 510 scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c
> 8192
> > >
> > > Creating the tcp transport a second time doesn't fail? It should. You
> can
> > > only
> > > have one of each type of transport.
> > >
> > [Sushil] it did fail.
> >
> > > > 511 scripts/rpc.py construct_malloc_bdev -b Malloc0 512 512
> > > > 512 scripts/rpc.py nvmf_subsystem_create
> nqn.2016-06.io.spdk:cnode1 -a
> > > > -s SPDK00000000000001 -d SPDK_Controller1
> > >
> > > Note for later the use of the '-a' parameter when creating the
> subsystem,
> > > which
> > > means allow any host to connect.
> > >
> > > > 513 scripts/rpc.py nvmf_subsystem_add_ns
> nqn.2016-06.io.spdk:cnode1
> > > > Malloc0
> > > > 514 scripts/rpc.py nvmf_subsystem_add_listener
> > > > nqn.2016-06.io.spdk:cnode1 -t TCP -a 172.20.0.2 -s 4420
> > > >
> > > > Was able to verify that Identify works fine:
> > > > sh-4.2# ./identify -r 'trtype:TCP adrfam:IPv4 traddr:172.20.0.2
> > > > trsvcid:4420'
> > > > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> > > > [ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --no-pci
> > > > --log-level=lib.eal:6 --base-virtaddr=0x200000000000
> --match-allocations
> > > > --file-prefix=spdk_pid498 ]
> > > > EAL: No available hugepages reported in hugepages-1048576kB
> > > > EAL: VFIO support initialized
> > > > =====================================================
> > > > NVMe over Fabrics controller at 172.20.0.2:4420:
> > > > nqn.2014-08.org.nvmexpress.discovery
> > > > =====================================================
> > > > Controller Capabilities/Features
> > > > ================================
> > > > Vendor ID: 0000
> > > > Subsystem Vendor ID: 0000
> > > > Serial Number: ....................
> > > > Model Number:
> > > > ........................................
> > > > Firmware Version: 19.04.1
> > > > Recommended Arb Burst: 0
> > > > IEEE OUI Identifier: 00 00 00
> > > > Multi-path I/O
> > > > May have multiple subsystem ports: No
> > > > May be connected to multiple hosts: No
> > > > Associated with SR-IOV VF: No
> > > > Max Data Transfer Size: 131072
> > > > NVMe Specification Version (VS): 1.3
> > > > NVMe Specification Version (Identify): 1.3
> > > > Maximum Queue Entries: 128
> > > > Contiguous Queues Required: Yes
> > > > Arbitration Mechanisms Supported
> > > > ....
> > > >
> > > > Then I tried I/O :
> > > > sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t
> 10 -r
> > > > 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
> > > > subnqn:nqn.2016-06.io.spdk:cnode1'
> > > > Ignoring -M option... Please use -M option only when using rw or
> randrw.
> > > > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> > > > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> > > > --log-level=lib.eal:6 --base-virtaddr=0x200000000000
> --match-allocations
> > > > --file-prefix=spdk_pid549 ]
> > > > EAL: No available hugepages reported in hugepages-1048576kB
> > > > EAL: VFIO support initialized
> > > > Initializing NVMe Controllers
> > > > Attaching to NVMe over Fabrics controller at 172.20.0.2:4420:
> > > > nqn.2016-06.io.spdk:cnode1
> > > > Attached to NVMe over Fabrics controller at 172.20.0.2:4420:
> > > > nqn.2016-06.io.spdk:cnode1
> > > > Associating SPDK_Controller1 (SPDK00000000000001 ) with lcore 0
> > > > Initialization complete. Launching workers.
> > > > Starting thread on core 0
> > > > ========================================================
> > > >
> > > > Latency(us)
> > > > Device Information : IOPS
> > > > MiB/s Average min max
> > > > SPDK_Controller1 (SPDK00000000000001 ) from core 0: 37.60
> > > > 0.15 26709.88 14961.21 39904.69
> > > > ========================================================
> > > > Total : 37.60
> > > > 0.15 26709.88 14961.21 39904.69
> > > >
> > > >
> > > > Then on the same container I tried the following steps :
> > > >
> > > > 604 scripts/rpc.py construct_nvme_bdev -b nvme1 -t PCIe -a
> > > 0000:1d:00.00
> > > > 605 scripts/rpc.py nvmf_subsystem_create
> nqn.2016-06.io.spdk:cnode2 a
> > > -s
> > > > SPDK00000000000002 -d SPDK_Controller2
> > >
> > > You forgot the '-' on the a. I believe this command failed in your
> script.
> > > Try
> > > adding set -e to the top so that your script exits on error. If this
> > > doesn't
> > > fail, please file a GitHub issue. It should not have allowed you to
> > > proceed.
> > > > 606 scripts/rpc.py nvmf_subsystem_create
> nqn.2016-06.io.spdk:cnode2
> > > -sa
> > > > -s SPDK00000000000002 -d SPDK_Controller2
> > [Sushil] I dumping the history and the command did fail.
> >
> > > This should be failing - you can't create a subsystem twice. Note also
> > > that you
> > > have the -sa option, which is not valid.
> > >
> > > > 607 scripts/rpc.py nvmf_subsystem_add_ns
> nqn.2016-06.io.spdk:cnode2
> > > > nvme1n1
> > > > 608 scripts/rpc.py nvmf_subsystem_add_listener
> > > > nqn.2016-06.io.spdk:cnode2 -t TCP -a 172.20.0.2 -s 4421
> > > >
> > > >
> > > >
> > > >
> > > > This time when I try I/O I get this error :
> > > > sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t
> 10
> > > -r
> > > > 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4421
> > > > subnqn:nqn.2016-06.io.spdk:cnode2'
> > > > Ignoring -M option... Please use -M option only when using rw or
> randrw.
> > > > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> > > > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> > > > --log-level=lib.eal:6 --base-virtaddr=0x200000000000
> --match-allocations
> > > > --file-prefix=spdk_pid561 ]
> > > > EAL: No available hugepages reported in hugepages-1048576kB
> > > > EAL: VFIO support initialized
> > > > Initializing NVMe Controllers
> > > > Attaching to NVMe over Fabrics controller at 172.20.0.2:4421:
> > > > nqn.2016-06.io.spdk:cnode2
> > > > ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem
> > > > 'nqn.2016-06.io.spdk:cnode2' does not allow host
> > > > '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543'
> > > > nvme_fabric.c: 327:nvme_fabric_qpair_connect: *ERROR*: Connect
> command
> > > > failed
> > > > nvme_tcp.c:1749:nvme_tcp_qpair_connect: *ERROR*: Failed to send an
> > > NVMe-oF
> > > > Fabric CONNECT command
> > > > nvme_tcp.c:1826:nvme_tcp_ctrlr_construct: *ERROR*: failed to create
> admin
> > > > qpair
> > > > nvme.c: 408:nvme_ctrlr_probe: *ERROR*: Failed to construct NVMe
> > > controller
> > > > for SSD: 172.20.0.2
> > > > nvme.c: 535:spdk_nvme_probe_internal: *ERROR*: NVMe ctrlr scan failed
> > > > nvme.c: 608:spdk_nvme_probe: *ERROR*: Create probe context failed
> > > > spdk_nvme_probe() failed for transport address '172.20.0.2'
> > > > /usr/local/bin/spdk_nvme_perf: errors occured
> > > >
> > > > What does the above error mean and how do I correct this ? Is my
> second
> > > > test expected to work ? Apologies if this sort of newbie query. I am
> just
> > > > starting on this.
> > >
> > > It means you didn't set up the access control on the subsystem to allow
> > > this
> > > host to connect. "Subsystem [xyz] does not allow host [ijk]". That's
> > > because you
> > > didn't pass valid parameters to the command that creates the
> subsystem, and
> > > those commands should have failed. If they don't fail today, please do
> > > file an
> > > issue on GitHub and we'll get that sorted out.
> > >
> > > Thanks,
> > > Ben
> > >
> > > > Thanks
> > > > Sushil
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > >
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-07-26 19:51 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-07-26 17:34 [SPDK] ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem 'nqn.2016-06.io.spdk:cnode2' does not allow host '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543' Walker, Benjamin
-- strict thread matches above, loose matches on Subject: below --
2019-07-26 19:51 sushilkumar gangadharan
2019-07-26 18:56 Walker, Benjamin
2019-07-26 18:48 sushilkumar gangadharan
2019-07-26 17:19 sushilkumar gangadharan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox