From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============0022076018743622590==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem 'nqn.2016-06.io.spdk:cnode2' does not allow host '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543' Date: Fri, 26 Jul 2019 18:56:42 +0000 Message-ID: <9838fe502618bfb31608ca329807380ab10408a7.camel@intel.com> In-Reply-To: CAPhUAXEGtX9fNkW-TpTHaiRZD17o5B-PTwLOc-voG9Fp-aVscQ@mail.gmail.com List-ID: To: spdk@lists.01.org --===============0022076018743622590== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Fri, 2019-07-26 at 11:48 -0700, sushilkumar gangadharan wrote: > Thanks Ben for the response. I am trying to talk to an NvMe drive and wi= th > the "-a" option it looks to be working. > = > This is the corrected sequence of commands. > = > sh-4.2# scripts/setup.sh > 0000:0f:00.0 (144d a802): Already using the vfio-pci driver > 0000:11:00.0 (144d a802): Already using the vfio-pci driver > 0000:13:00.0 (144d a802): Already using the vfio-pci driver > 0000:15:00.0 (144d a802): Already using the vfio-pci driver > 0000:17:00.0 (144d a802): Already using the vfio-pci driver > 0000:19:00.0 (144d a802): Already using the vfio-pci driver > 0000:1b:00.0 (144d a802): Already using the vfio-pci driver > 0000:1d:00.0 (144d a802): Already using the vfio-pci driver > Mounting hugetlbfs at /mnt/huge > = > Current user memlock limit: 0 MB > = > This is the maximum amount of memory you will be > able to use with DPDK and VFIO if run as current user. > To change this, please adjust limits.conf memlock limit for current user. > = > ## WARNING: memlock limit is less than 64MB > ## DPDK with VFIO may not be able to initialize if run as current user. > = > sh-4.2# app/nvmf_tgt/nvmf_tgt & > [1] 457 > sh-4.2# Starting SPDK v19.04.1 / DPDK 19.05.0 initialization... > [ DPDK EAL parameters: nvmf --no-shconf -c 0x1 --log-level=3Dlib.eal:6 > --base-virtaddr=3D0x200000000000 --match-allocations > --file-prefix=3Dspdk_pid457 ] > EAL: No available hugepages reported in hugepages-1048576kB > EAL: VFIO support initialized > app.c: 627:spdk_app_start: *NOTICE*: Total cores available: 1 > reactor.c: 251:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 > scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192 > tcp.c: 543:spdk_nvmf_tcp_create: *NOTICE*: *** TCP Transport Init *** > sh-4.2# scripts/rpc.py construct_nvme_bdev -b nvme1 -t PCIe -a 0000:1d:00= .00 > EAL: using IOMMU type 1 (Type 1) > nvme1n1 > = > sh-4.2# scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 -a > -s SPDK00000000000001 -d SPDK_Controller1 > = > sh-4.2# scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 > nvme1n1 > = > sh-4.2# ifconfig > eth0: flags=3D4163 mtu 1500 > inet 172.20.0.2 netmask 255.255.0.0 broadcast 172.20.255.255 > ether 02:42:ac:14:00:02 txqueuelen 0 (Ethernet) > RX packets 14 bytes 1156 (1.1 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 0 bytes 0 (0.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > = > lo: flags=3D73 mtu 65536 > inet 127.0.0.1 netmask 255.0.0.0 > loop txqueuelen 1000 (Local Loopback) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 0 bytes 0 (0.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > = > sh-4.2# scripts/rpc.py nvmf_subsystem_add_listener > nqn.2016-06.io.spdk:cnode2 -t TCP -a 172.20.0.2 -s 4420 > = > tcp.c: 726:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening = on > 172.20.0.2 port 4420 *** > = > sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 10 -r > 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420 > subnqn:nqn.2016-06.io.spdk:cnode2' > sh: /usr/local/bin/spdk_nvme_perf: No such file or directory > sh-4.2# cd examples/nvme/perf/make > sh: cd: examples/nvme/perf/make: No such file or directory > sh-4.2# cd examples/nvme/perf > sh-4.2# make > INSTALL /usr/local/bin/spdk_nvme_perf > = > sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -t 10 -r > 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420 > subnqn:nqn.2016-06.io.spdk:cnode2' > = > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization... > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci > --log-level=3Dlib.eal:6 --base-virtaddr=3D0x200000000000 --match-allocati= ons > --file-prefix=3Dspdk_pid475 ] > EAL: No available hugepages reported in hugepages-1048576kB > EAL: VFIO support initialized > Initializing NVMe Controllers > Attaching to NVMe over Fabrics controller at 172.20.0.2:4420: > nqn.2016-06.io.spdk:cnode2 > Attached to NVMe over Fabrics controller at 172.20.0.2:4420: > nqn.2016-06.io.spdk:cnode2 > Associating SPDK_Controller1 (SPDK00000000000001 ) with lcore 0 > Initialization complete. Launching workers. > Starting thread on core 0 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > = > Latency(us) > Device Information : IOPS > MiB/s Average min max > SPDK_Controller1 (SPDK00000000000001 ) from core 0: 37.50 > 0.15 26709.32 14964.37 47885.26 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > Total : 37.50 > 0.15 26709.32 14964.37 47885.26 > = > sh-4.2# > = > = > So did "-a" make the difference ? If I don't give the "-a" and gave only > = > sh-4.2# scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 -s > SPDK00000000000001 -d SPDK_Controller1 > = > will the above mean that no hosts are allowed to connect ? Correct. By default, the access control on a subsystem allows no hosts to connect to it. If you specify the -a option, it allows any host by default instead. > = > Also you mentioned : > It means you didn't set up the access control on the subsystem to allow t= his > host to connect. "Subsystem [xyz] does not allow host [ijk]". That's > because you > didn't pass valid parameters to the command that creates the subsystem, a= nd > those commands should have failed. If they don't fail today, please do fi= le > an > issue on GitHub and we'll get that sorted out. > = > How does one create a subsystem such that it permits certain hosts ? Is > there some documentation for this ? Create a subsystem without the '-a' option. Then use the "nvmf_subsystem_add_host" RPC to specifically list out the hosts that you w= ant to allow for that subsystem. > = > Also added some comments inline ( regarding whether the commands failed or > not ).. > = > Thanks > Sushil > = > = > On Fri, Jul 26, 2019 at 10:34 AM Walker, Benjamin > wrote: > = > > On Fri, 2019-07-26 at 10:19 -0700, sushilkumar gangadharan wrote: > > > Hi > > > I am trying to setup a docker container which then talks to Name drive > [Sushil] I meant Nvme ( autocorrect) > = > > > using SPDK. In the first phase of the experimenting, I did the follow= ing > > > within the docker container and it seems to work: > > > = > > > 508 app/nvmf_tgt/nvmf_tgt > > > 509 scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8= 192 > > > 510 scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8= 192 > > = > > Creating the tcp transport a second time doesn't fail? It should. You c= an > > only > > have one of each type of transport. > > = > [Sushil] it did fail. > = > > > 511 scripts/rpc.py construct_malloc_bdev -b Malloc0 512 512 > > > 512 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode= 1 -a > > > -s SPDK00000000000001 -d SPDK_Controller1 > > = > > Note for later the use of the '-a' parameter when creating the subsyste= m, > > which > > means allow any host to connect. > > = > > > 513 scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 > > > Malloc0 > > > 514 scripts/rpc.py nvmf_subsystem_add_listener > > > nqn.2016-06.io.spdk:cnode1 -t TCP -a 172.20.0.2 -s 4420 > > > = > > > Was able to verify that Identify works fine: > > > sh-4.2# ./identify -r 'trtype:TCP adrfam:IPv4 traddr:172.20.0.2 > > > trsvcid:4420' > > > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization... > > > [ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --no-pci > > > --log-level=3Dlib.eal:6 --base-virtaddr=3D0x200000000000 --match-allo= cations > > > --file-prefix=3Dspdk_pid498 ] > > > EAL: No available hugepages reported in hugepages-1048576kB > > > EAL: VFIO support initialized > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D > > > NVMe over Fabrics controller at 172.20.0.2:4420: > > > nqn.2014-08.org.nvmexpress.discovery > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D > > > Controller Capabilities/Features > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D > > > Vendor ID: 0000 > > > Subsystem Vendor ID: 0000 > > > Serial Number: .................... > > > Model Number: > > > ........................................ > > > Firmware Version: 19.04.1 > > > Recommended Arb Burst: 0 > > > IEEE OUI Identifier: 00 00 00 > > > Multi-path I/O > > > May have multiple subsystem ports: No > > > May be connected to multiple hosts: No > > > Associated with SR-IOV VF: No > > > Max Data Transfer Size: 131072 > > > NVMe Specification Version (VS): 1.3 > > > NVMe Specification Version (Identify): 1.3 > > > Maximum Queue Entries: 128 > > > Contiguous Queues Required: Yes > > > Arbitration Mechanisms Supported > > > .... > > > = > > > Then I tried I/O : > > > sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t 1= 0 -r > > > 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4420 > > > subnqn:nqn.2016-06.io.spdk:cnode1' > > > Ignoring -M option... Please use -M option only when using rw or rand= rw. > > > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization... > > > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci > > > --log-level=3Dlib.eal:6 --base-virtaddr=3D0x200000000000 --match-allo= cations > > > --file-prefix=3Dspdk_pid549 ] > > > EAL: No available hugepages reported in hugepages-1048576kB > > > EAL: VFIO support initialized > > > Initializing NVMe Controllers > > > Attaching to NVMe over Fabrics controller at 172.20.0.2:4420: > > > nqn.2016-06.io.spdk:cnode1 > > > Attached to NVMe over Fabrics controller at 172.20.0.2:4420: > > > nqn.2016-06.io.spdk:cnode1 > > > Associating SPDK_Controller1 (SPDK00000000000001 ) with lcore 0 > > > Initialization complete. Launching workers. > > > Starting thread on core 0 > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > > = > > > Latency(us) > > > Device Information : IOPS > > > MiB/s Average min max > > > SPDK_Controller1 (SPDK00000000000001 ) from core 0: 37.60 > > > 0.15 26709.88 14961.21 39904.69 > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > > Total : 37.60 > > > 0.15 26709.88 14961.21 39904.69 > > > = > > > = > > > Then on the same container I tried the following steps : > > > = > > > 604 scripts/rpc.py construct_nvme_bdev -b nvme1 -t PCIe -a > > 0000:1d:00.00 > > > 605 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode= 2 a > > -s > > > SPDK00000000000002 -d SPDK_Controller2 > > = > > You forgot the '-' on the a. I believe this command failed in your scri= pt. > > Try > > adding set -e to the top so that your script exits on error. If this > > doesn't > > fail, please file a GitHub issue. It should not have allowed you to > > proceed. > > > 606 scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode2 > > -sa > > > -s SPDK00000000000002 -d SPDK_Controller2 > [Sushil] I dumping the history and the command did fail. > = > > This should be failing - you can't create a subsystem twice. Note also > > that you > > have the -sa option, which is not valid. > > = > > > 607 scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode2 > > > nvme1n1 > > > 608 scripts/rpc.py nvmf_subsystem_add_listener > > > nqn.2016-06.io.spdk:cnode2 -t TCP -a 172.20.0.2 -s 4421 > > > = > > > = > > > = > > > = > > > This time when I try I/O I get this error : > > > sh-4.2# /usr/local/bin/spdk_nvme_perf -q 1 -o 4096 -w write -M 0 -t = 10 > > -r > > > 'trtype:TCP traddr:172.20.0.2 adrfam:IPv4 trsvcid:4421 > > > subnqn:nqn.2016-06.io.spdk:cnode2' > > > Ignoring -M option... Please use -M option only when using rw or rand= rw. > > > Starting SPDK v19.04.1 / DPDK 19.05.0 initialization... > > > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci > > > --log-level=3Dlib.eal:6 --base-virtaddr=3D0x200000000000 --match-allo= cations > > > --file-prefix=3Dspdk_pid561 ] > > > EAL: No available hugepages reported in hugepages-1048576kB > > > EAL: VFIO support initialized > > > Initializing NVMe Controllers > > > Attaching to NVMe over Fabrics controller at 172.20.0.2:4421: > > > nqn.2016-06.io.spdk:cnode2 > > > ctrlr.c: 534:spdk_nvmf_ctrlr_connect: *ERROR*: Subsystem > > > 'nqn.2016-06.io.spdk:cnode2' does not allow host > > > '2014-08.org.nvmexpress:uuid:f7124f51-379b-4752-b195-0487d39d4543' > > > nvme_fabric.c: 327:nvme_fabric_qpair_connect: *ERROR*: Connect command > > > failed > > > nvme_tcp.c:1749:nvme_tcp_qpair_connect: *ERROR*: Failed to send an > > NVMe-oF > > > Fabric CONNECT command > > > nvme_tcp.c:1826:nvme_tcp_ctrlr_construct: *ERROR*: failed to create a= dmin > > > qpair > > > nvme.c: 408:nvme_ctrlr_probe: *ERROR*: Failed to construct NVMe > > controller > > > for SSD: 172.20.0.2 > > > nvme.c: 535:spdk_nvme_probe_internal: *ERROR*: NVMe ctrlr scan failed > > > nvme.c: 608:spdk_nvme_probe: *ERROR*: Create probe context failed > > > spdk_nvme_probe() failed for transport address '172.20.0.2' > > > /usr/local/bin/spdk_nvme_perf: errors occured > > > = > > > What does the above error mean and how do I correct this ? Is my seco= nd > > > test expected to work ? Apologies if this sort of newbie query. I am = just > > > starting on this. > > = > > It means you didn't set up the access control on the subsystem to allow > > this > > host to connect. "Subsystem [xyz] does not allow host [ijk]". That's > > because you > > didn't pass valid parameters to the command that creates the subsystem,= and > > those commands should have failed. If they don't fail today, please do > > file an > > issue on GitHub and we'll get that sorted out. > > = > > Thanks, > > Ben > > = > > > Thanks > > > Sushil > > > _______________________________________________ > > > SPDK mailing list > > > SPDK(a)lists.01.org > > > https://lists.01.org/mailman/listinfo/spdk > > = > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============0022076018743622590==--