Re: [SPDK] ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem 'nqn.2016-06.io.spdk:cnode2' does not allow host '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543'

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 14859 bytes --]

On Fri, 2019-07-26 at 11:48 -0700, sushilkumar gangadharan wrote:
> Thanks Ben for the response. I am trying to talk to an NvMe drive  and with
> the  "-a" option it looks to be working.
> 
> This is the corrected sequence of commands.
> 
> sh-4.2# scripts/setup.sh
> 0000:0f:00.0 (144d a802): Already using the vfio-pci driver
> 0000:11:00.0 (144d a802): Already using the vfio-pci driver
> 0000:13:00.0 (144d a802): Already using the vfio-pci driver
> 0000:15:00.0 (144d a802): Already using the vfio-pci driver
> 0000:17:00.0 (144d a802): Already using the vfio-pci driver
> 0000:19:00.0 (144d a802): Already using the vfio-pci driver
> 0000:1b:00.0 (144d a802): Already using the vfio-pci driver
> 0000:1d:00.0 (144d a802): Already using the vfio-pci driver
> Mounting hugetlbfs at /mnt/huge
> 
> Current user memlock limit: 0 MB
> 
> This is the maximum amount of memory you will be
> able to use with DPDK and VFIO if run as current user.
> To change this, please adjust limits.conf memlock limit for current user.
> 
> ## WARNING: memlock limit is less than 64MB
> ## DPDK with VFIO may not be able to initialize if run as current user.
> 
> sh-4.2# app/nvmf_tgt/nvmf_tgt &
> [1] 457
> sh-4.2# Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> [ DPDK EAL parameters: nvmf --no-shconf -c 0x1 --log-level=lib.eal:6
> --base-virtaddr=0x200000000000 --match-allocations
> --file-prefix=spdk_pid457 ]
> EAL: No available hugepages reported in hugepages-1048576kB
> EAL: VFIO support initialized
> app.c: 627:spdk_app_start: *NOTICE*: Total cores available: 1
> reactor.c: 251:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
> tcp.c: 543:spdk_nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
> sh-4.2# scripts/rpc.py construct_nvme_bdev -b nvme1 -t PCIe -a 0000:1d:00.00
> EAL:   using IOMMU type 1 (Type 1)
> nvme1n1
> 
> sh-4.2# scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 -a
> -s SPDK00000000000001 -d SPDK_Controller1
> 
> sh-4.2# scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2
> nvme1n1
> 
> sh-4.2# ifconfig
> eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet 172.20.0.2  netmask 255.255.0.0  broadcast 172.20.255.255
>         ether 02:42:ac:14:00:02  txqueuelen 0  (Ethernet)
>         RX packets 14  bytes 1156 (1.1 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 0  bytes 0 (0.0 B)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
>         inet 127.0.0.1  netmask 255.0.0.0
>         loop  txqueuelen 1000  (Local Loopback)
>         RX packets 0  bytes 0 (0.0 B)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 0  bytes 0 (0.0 B)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> sh-4.2# scripts/rpc.py nvmf_subsystem_add_listener
> nqn.2016-06.io.spdk:cnode2 -t TCP -a 172.20.0.2 -s 4420
> 
> tcp.c: 726:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on
> 172.20.0.2 port 4420 ***
> 
> sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10 -r
> 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
> subnqn:nqn.2016-06.io.spdk:cnode2'
> sh: /usr/local/bin/spdk_nvme_perf: No such file or directory
> sh-4.2# cd examples/nvme/perf/make
> sh: cd: examples/nvme/perf/make: No such file or directory
> sh-4.2# cd examples/nvme/perf
> sh-4.2# make
>   INSTALL /usr/local/bin/spdk_nvme_perf
> 
> sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -t 10 -r
> 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
> subnqn:nqn.2016-06.io.spdk:cnode2'
> 
> Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> --file-prefix=spdk_pid475 ]
> EAL: No available hugepages reported in hugepages-1048576kB
> EAL: VFIO support initialized
> Initializing NVMe Controllers
> Attaching to NVMe over Fabrics controller at 172.20.0.2:4420:
> nqn.2016-06.io.spdk:cnode2
> Attached to NVMe over Fabrics controller at 172.20.0.2:4420:
> nqn.2016-06.io.spdk:cnode2
> Associating SPDK_Controller1     (SPDK00000000000001  ) with lcore 0
> Initialization complete. Launching workers.
> Starting thread on core 0
> ========================================================
> 
>                 Latency(us)
> Device Information                                     :       IOPS
>  MiB/s    Average        min        max
> SPDK_Controller1     (SPDK00000000000001  ) from core 0:      37.50
> 0.15   26709.32   14964.37   47885.26
> ========================================================
> Total                                                  :      37.50
> 0.15   26709.32   14964.37   47885.26
> 
> sh-4.2#
> 
> 
> So did "-a" make the difference ? If I don't give the "-a" and gave only
> 
> sh-4.2# scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 -s
> SPDK00000000000001 -d SPDK_Controller1
> 
> will the above mean that no hosts are allowed to connect ?

Correct. By default, the access control on a subsystem allows no hosts to
connect to it. If you specify the -a option, it allows any host by default
instead.

> 
> Also you mentioned :
> It means you didn't set up the access control on the subsystem to allow this
> host to connect. "Subsystem [xyz] does not allow host [ijk]". That's
> because you
> didn't pass valid parameters to the command that creates the subsystem, and
> those commands should have failed. If they don't fail today, please do file
> an
> issue on GitHub and we'll get that sorted out.
> 
> How does one create a subsystem such that it permits certain hosts ?  Is
> there some documentation for this ?

Create a subsystem without the '-a' option. Then use the
"nvmf_subsystem_add_host" RPC to specifically list out the hosts that you want
to allow for that subsystem.

> 
> Also added some comments inline ( regarding whether the commands failed or
> not )..
> 
> Thanks
> Sushil
> 
> 
> On Fri, Jul 26, 2019 at 10:34 AM Walker, Benjamin <benjamin.walker(a)intel.com>
> wrote:
> 
> > On Fri, 2019-07-26 at 10:19 -0700, sushilkumar gangadharan wrote:
> > > Hi
> > > I am trying to setup a docker container which then talks to Name drive
> [Sushil] I meant Nvme ( autocorrect)
> 
> > > using SPDK. In the first phase of the experimenting, I did the following
> > > within the docker container and it seems to work:
> > > 
> > >   508  app/nvmf_tgt/nvmf_tgt
> > >   509  scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
> > >   510  scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192
> > 
> > Creating the tcp transport a second time doesn't fail? It should. You can
> > only
> > have one of each type of transport.
> > 
> [Sushil] it did fail.
> 
> > >   511  scripts/rpc.py construct_malloc_bdev -b Malloc0 512 512
> > >   512  scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode1 -a
> > > -s SPDK00000000000001 -d SPDK_Controller1
> > 
> > Note for later the use of the '-a' parameter when creating the subsystem,
> > which
> > means allow any host to connect.
> > 
> > >   513  scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1
> > > Malloc0
> > >   514  scripts/rpc.py nvmf_subsystem_add_listener
> > > nqn.2016-06.io.spdk:cnode1 -t TCP -a 172.20.0.2 -s 4420
> > > 
> > > Was able to verify that Identify works fine:
> > > sh-4.2# ./identify -r 'trtype:TCP adrfam:IPv4 traddr:172.20.0.2
> > > trsvcid:4420'
> > > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> > > [ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --no-pci
> > > --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> > > --file-prefix=spdk_pid498 ]
> > > EAL: No available hugepages reported in hugepages-1048576kB
> > > EAL: VFIO support initialized
> > > =====================================================
> > > NVMe over Fabrics controller at 172.20.0.2:4420:
> > > nqn.2014-08.org.nvmexpress.discovery
> > > =====================================================
> > > Controller Capabilities/Features
> > > ================================
> > > Vendor ID:                             0000
> > > Subsystem Vendor ID:                   0000
> > > Serial Number:                         ....................
> > > Model Number:
> > >  ........................................
> > > Firmware Version:                      19.04.1
> > > Recommended Arb Burst:                 0
> > > IEEE OUI Identifier:                   00 00 00
> > > Multi-path I/O
> > >   May have multiple subsystem ports:   No
> > >   May be connected to multiple hosts:  No
> > >   Associated with SR-IOV VF:           No
> > > Max Data Transfer Size:                131072
> > > NVMe Specification Version (VS):       1.3
> > > NVMe Specification Version (Identify): 1.3
> > > Maximum Queue Entries:                 128
> > > Contiguous Queues Required:            Yes
> > > Arbitration Mechanisms Supported
> > > ....
> > > 
> > > Then I tried I/O :
> > > sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10 -r
> > > 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420
> > > subnqn:nqn.2016-06.io.spdk:cnode1'
> > > Ignoring -M option... Please use -M option only when using rw or randrw.
> > > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> > > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> > > --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> > > --file-prefix=spdk_pid549 ]
> > > EAL: No available hugepages reported in hugepages-1048576kB
> > > EAL: VFIO support initialized
> > > Initializing NVMe Controllers
> > > Attaching to NVMe over Fabrics controller at 172.20.0.2:4420:
> > > nqn.2016-06.io.spdk:cnode1
> > > Attached to NVMe over Fabrics controller at 172.20.0.2:4420:
> > > nqn.2016-06.io.spdk:cnode1
> > > Associating SPDK_Controller1     (SPDK00000000000001  ) with lcore 0
> > > Initialization complete. Launching workers.
> > > Starting thread on core 0
> > > ========================================================
> > > 
> > >                 Latency(us)
> > > Device Information                                     :       IOPS
> > >  MiB/s    Average        min        max
> > > SPDK_Controller1     (SPDK00000000000001  ) from core 0:      37.60
> > > 0.15   26709.88   14961.21   39904.69
> > > ========================================================
> > > Total                                                  :      37.60
> > > 0.15   26709.88   14961.21   39904.69
> > > 
> > > 
> > > Then on the same container I tried the following steps :
> > > 
> > >   604  scripts/rpc.py construct_nvme_bdev -b nvme1 -t PCIe -a
> > 0000:1d:00.00
> > >   605  scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 a
> > -s
> > > SPDK00000000000002 -d SPDK_Controller2
> > 
> > You forgot the '-' on the a. I believe this command failed in your script.
> > Try
> > adding set -e to the top so that your script exits on error. If this
> > doesn't
> > fail, please file a GitHub issue. It should not have allowed you to
> > proceed.
> > >   606  scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2
> > -sa
> > > -s SPDK00000000000002 -d SPDK_Controller2
> [Sushil]  I dumping the history and the command did fail.
> 
> > This should be failing - you can't create a subsystem twice. Note also
> > that you
> > have the -sa option, which is not valid.
> > 
> > >   607  scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2
> > > nvme1n1
> > >   608  scripts/rpc.py nvmf_subsystem_add_listener
> > > nqn.2016-06.io.spdk:cnode2 -t TCP -a 172.20.0.2 -s 4421
> > > 
> > > 
> > > 
> > > 
> > > This time when I try I/O  I get this error :
> > > sh-4.2#  /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10
> > -r
> > > 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4421
> > > subnqn:nqn.2016-06.io.spdk:cnode2'
> > > Ignoring -M option... Please use -M option only when using rw or randrw.
> > > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization...
> > > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci
> > > --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations
> > > --file-prefix=spdk_pid561 ]
> > > EAL: No available hugepages reported in hugepages-1048576kB
> > > EAL: VFIO support initialized
> > > Initializing NVMe Controllers
> > > Attaching to NVMe over Fabrics controller at 172.20.0.2:4421:
> > > nqn.2016-06.io.spdk:cnode2
> > > ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem
> > > 'nqn.2016-06.io.spdk:cnode2' does not allow host
> > > '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543'
> > > nvme_fabric.c: 327:nvme_fabric_qpair_connect: *ERROR*: Connect command
> > > failed
> > > nvme_tcp.c:1749:nvme_tcp_qpair_connect: *ERROR*: Failed to send an
> > NVMe-oF
> > > Fabric CONNECT command
> > > nvme_tcp.c:1826:nvme_tcp_ctrlr_construct: *ERROR*: failed to create admin
> > > qpair
> > > nvme.c: 408:nvme_ctrlr_probe: *ERROR*: Failed to construct NVMe
> > controller
> > > for SSD: 172.20.0.2
> > > nvme.c: 535:spdk_nvme_probe_internal: *ERROR*: NVMe ctrlr scan failed
> > > nvme.c: 608:spdk_nvme_probe: *ERROR*: Create probe context failed
> > > spdk_nvme_probe() failed for transport address '172.20.0.2'
> > > /usr/local/bin/spdk_nvme_perf: errors occured
> > > 
> > > What does the above error mean and how do I correct this ? Is my second
> > > test expected to work ? Apologies if this sort of newbie query. I am just
> > > starting on this.
> > 
> > It means you didn't set up the access control on the subsystem to allow
> > this
> > host to connect. "Subsystem [xyz] does not allow host [ijk]". That's
> > because you
> > didn't pass valid parameters to the command that creates the subsystem, and
> > those commands should have failed. If they don't fail today, please do
> > file an
> > issue on GitHub and we'll get that sorted out.
> > 
> > Thanks,
> > Ben
> > 
> > > Thanks
> > > Sushil
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > 
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> > 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk