Re: [SPDK] ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem 'nqn.2016-06.io.spdk:cnode2' does not allow host '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543'

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 7353 bytes --]

On Fri, 2019-07-26 at 10:19 -0700, sushilkumar gangadharan wrote:
> Hi
> I am trying to setup a docker container which then talks to Name drive
> using SPDK. In the first phase of the experimenting, I did the following
> within the docker container and it seems to work:
> 
>   508  app/nvmf_tgt/nvmf_tgt
>   509  scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
>   510  scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192

Creating the tcp transport a second time doesn't fail? It should. You can only
have one of each type of transport.

>   511  scripts/rpc.py construct_malloc_bdev -b Malloc0 512 512
>   512  scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode1 -a
> -s SPDK00000000000001 -d SPDK_Controller1

Note for later the use of the '-a' parameter when creating the subsystem, which
means allow any host to connect.

>   513  scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1
> Malloc0
>   514  scripts/rpc.py nvmf_subsystem_add_listener
> nqn.2016-06.io.spdk:cnode1 -t TCP -a 172.20.0.2 -s 4420
> 
> Was able to verify that Identify works fine:
> sh-4.2# ./identify -r 'trtype:TCP adrfam:IPv4 traddr:172.20.0.2
> trsvcid:4420'
> Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> [ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --no-pci
> --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> --file-prefix=spdk_pid498 ]
> EAL: No available hugepages reported in hugepages-1048576kB
> EAL: VFIO support initialized
> =====================================================
> NVMe over Fabrics controller at 172.20.0.2:4420:
> nqn.2014-08.org.nvmexpress.discovery
> =====================================================
> Controller Capabilities/Features
> ================================
> Vendor ID:                             0000
> Subsystem Vendor ID:                   0000
> Serial Number:                         ....................
> Model Number:
>  ........................................
> Firmware Version:                      19.04.1
> Recommended Arb Burst:                 0
> IEEE OUI Identifier:                   00 00 00
> Multi-path I/O
>   May have multiple subsystem ports:   No
>   May be connected to multiple hosts:  No
>   Associated with SR-IOV VF:           No
> Max Data Transfer Size:                131072
> NVMe Specification Version (VS):       1.3
> NVMe Specification Version (Identify): 1.3
> Maximum Queue Entries:                 128
> Contiguous Queues Required:            Yes
> Arbitration Mechanisms Supported
> ....
> 
> Then I tried I/O :
> sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10 -r
> 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
> subnqn:nqn.2016-06.io.spdk:cnode1'
> Ignoring -M option... Please use -M option only when using rw or randrw.
> Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> --file-prefix=spdk_pid549 ]
> EAL: No available hugepages reported in hugepages-1048576kB
> EAL: VFIO support initialized
> Initializing NVMe Controllers
> Attaching to NVMe over Fabrics controller at 172.20.0.2:4420:
> nqn.2016-06.io.spdk:cnode1
> Attached to NVMe over Fabrics controller at 172.20.0.2:4420:
> nqn.2016-06.io.spdk:cnode1
> Associating SPDK_Controller1     (SPDK00000000000001  ) with lcore 0
> Initialization complete. Launching workers.
> Starting thread on core 0
> ========================================================
> 
>                 Latency(us)
> Device Information                                     :       IOPS
>  MiB/s    Average        min        max
> SPDK_Controller1     (SPDK00000000000001  ) from core 0:      37.60
> 0.15   26709.88   14961.21   39904.69
> ========================================================
> Total                                                  :      37.60
> 0.15   26709.88   14961.21   39904.69
> 
> 
> Then on the same container I tried the following steps :
> 
>   604  scripts/rpc.py construct_nvme_bdev -b nvme1 -t PCIe -a 0000:1d:00.00
>   605  scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 a -s
> SPDK00000000000002 -d SPDK_Controller2

You forgot the '-' on the a. I believe this command failed in your script. Try
adding set -e to the top so that your script exits on error. If this doesn't
fail, please file a GitHub issue. It should not have allowed you to proceed.

>   606  scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 -sa
> -s SPDK00000000000002 -d SPDK_Controller2

This should be failing - you can't create a subsystem twice. Note also that you
have the -sa option, which is not valid.

>   607  scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2
> nvme1n1
>   608  scripts/rpc.py nvmf_subsystem_add_listener
> nqn.2016-06.io.spdk:cnode2 -t TCP -a 172.20.0.2 -s 4421
> 
> 
> 
> 
> This time when I try I/O  I get this error :
> sh-4.2#  /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10 -r
> 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4421
> subnqn:nqn.2016-06.io.spdk:cnode2'
> Ignoring -M option... Please use -M option only when using rw or randrw.
> Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> --file-prefix=spdk_pid561 ]
> EAL: No available hugepages reported in hugepages-1048576kB
> EAL: VFIO support initialized
> Initializing NVMe Controllers
> Attaching to NVMe over Fabrics controller at 172.20.0.2:4421:
> nqn.2016-06.io.spdk:cnode2
> ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem
> 'nqn.2016-06.io.spdk:cnode2' does not allow host
> '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543'
> nvme_fabric.c: 327:nvme_fabric_qpair_connect: *ERROR*: Connect command
> failed
> nvme_tcp.c:1749:nvme_tcp_qpair_connect: *ERROR*: Failed to send an NVMe-oF
> Fabric CONNECT command
> nvme_tcp.c:1826:nvme_tcp_ctrlr_construct: *ERROR*: failed to create admin
> qpair
> nvme.c: 408:nvme_ctrlr_probe: *ERROR*: Failed to construct NVMe controller
> for SSD: 172.20.0.2
> nvme.c: 535:spdk_nvme_probe_internal: *ERROR*: NVMe ctrlr scan failed
> nvme.c: 608:spdk_nvme_probe: *ERROR*: Create probe context failed
> spdk_nvme_probe() failed for transport address '172.20.0.2'
> /usr/local/bin/spdk_nvme_perf: errors occured
> 
> What does the above error mean and how do I correct this ? Is my second
> test expected to work ? Apologies if this sort of newbie query. I am just
> starting on this.

It means you didn't set up the access control on the subsystem to allow this
host to connect. "Subsystem [xyz] does not allow host [ijk]". That's because you
didn't pass valid parameters to the command that creates the subsystem, and
those commands should have failed. If they don't fail today, please do file an
issue on GitHub and we'll get that sorted out.

Thanks,
Ben

> 
> Thanks
> Sushil
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk