From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============8114005486870835505==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] nvmf.conf: AcceptorCore Vs Core Date: Wed, 23 Nov 2016 16:37:26 +0000 Message-ID: <1479919045.4630.1.camel@intel.com> In-Reply-To: FEB349DD341C264E8FE626BE236D26ED1CC2F885@SSIEXCH-MB3.ssi.samsung.com List-ID: To: spdk@lists.01.org --===============8114005486870835505== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Tue, 2016-11-22 at 18:14 +0000, Raj (Rajinikanth) Pandurangan wrote: > Just to update you all, latest DPDK (16.11) fixes RDMA init issue that I = was > facing with two RNICs. The latest version of SPDK on master requires DPDK 16.11 to correctly enume= rate more than one NVMe device. This may be addressed prior to the next release - we're looking at it now. The most recent SPDK release can run with older versions of DPDK though. I think what you are seeing is that because only o= ne NVMe device is found, the system only initializes the NIC that it is going = to use with that device. > =C2=A0 > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Raj (Rajinik= anth) > Pandurangan > Sent: Monday, November 21, 2016 12:07 PM > To: Storage Performance Development Kit > Subject: Re: [SPDK] nvmf.conf: AcceptorCore Vs Core > =C2=A0 > Hello John et al, > =C2=A0 > Looking at your log below, you got =E2=80=9CRDMA transport init=E2=80=9D = twice as you had two > RNICs.=C2=A0 In my case, =C2=A0=E2=80=9CNUM_TRANSPORTS=E2=80=9D in transp= ort.c has value of 1.=C2=A0 Thus it > only one of my RDMA gets initialized. > =C2=A0 > Not quite sure what would be the right fix so that it can initialize both= of > my RNIC devices? > =C2=A0 > Thanks, > =C2=A0 > From: Raj (Rajinikanth) Pandurangan=C2=A0 > Sent: Thursday, November 17, 2016 3:12 PM > To: Storage Performance Development Kit > Subject: RE: nvmf.conf: AcceptorCore Vs Core > =C2=A0 > Thanks for the details John.=C2=A0 Though it helps, I think I=E2=80=99m s= till missing > something more. > =C2=A0 > Here is the latest output from nvmf_tgt. > =C2=A0 > /rajp/spdk# app/nvmf_tgt/nvmf_tgt -c etc/spdk/nvmf.conf -p 15 > Starting Intel(R) DPDK initialization ... > [ DPDK EAL parameters: nvmf -c f000 -n 4 -m 2048 --master-lcore=3D15 --fi= le- > prefix=3Drte0 --proc-type=3Dauto ] > EAL: Detected 48 lcore(s) > EAL: Auto-detected process type: PRIMARY > EAL: No free hugepages reported in hugepages-1048576kB > EAL: Probing VFIO support... > done. > Occupied cpu core mask is 0xf000 > Occupied cpu socket mask is 0x3 > Ioat Copy Engine Offload Enabled > Total cores available: 4 > Reactor started on core 0xc > Reactor started on core 0xd > Reactor started on core 0xe > Reactor started on core 0xf > *** RDMA Transport Init *** > allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 15 > allocated subsystem nqn.2016-06.io.spdk:cnode1 on lcore 13 > *** NVMf Target Listening on 101.10.10.180 port 4420 *** > EAL: PCI device 0000:06:00.0 on NUMA socket 0 > EAL:=C2=A0=C2=A0 probe driver: 144d:a821 SPDK NVMe > EAL: PCI device 0000:07:00.0 on NUMA socket 0 > EAL:=C2=A0=C2=A0 probe driver: 144d:a821 SPDK NVMe > EAL: PCI device 0000:88:00.0 on NUMA socket 1 > EAL:=C2=A0=C2=A0 probe driver: 144d:a821 SPDK NVMe > EAL: PCI device 0000:89:00.0 on NUMA socket 1 > EAL:=C2=A0=C2=A0 probe driver: 144d:a821 SPDK NVMe > Attaching NVMe device 0x7ff98d65b6c0 at 0:6:0.0 to subsystem 0x1acf750 > allocated subsystem nqn.2016-06.io.spdk:cnode2 on lcore 14 > Subsystem nqn.2016-06.io.spdk:cnode2 is configured to run on a CPU core > belonging to a different NUMA node than the associated NIC. This may resu= lt in > reduced performance. > *** NVMf Target Listening on 100.10.10.180 port 4420 *** > EAL: PCI device 0000:06:00.0 on NUMA socket 0 > EAL:=C2=A0=C2=A0 probe driver: 144d:a821 SPDK NVMe > EAL: PCI device 0000:07:00.0 on NUMA socket 0 > EAL:=C2=A0=C2=A0 probe driver: 144d:a821 SPDK NVMe > EAL: PCI device 0000:88:00.0 on NUMA socket 1 > EAL:=C2=A0=C2=A0 probe driver: 144d:a821 SPDK NVMe > EAL: PCI device 0000:89:00.0 on NUMA socket 1 > EAL:=C2=A0=C2=A0 probe driver: 144d:a821 SPDK NVMe > Attaching NVMe device 0x7ff98d637700 at 0:88:0.0 to subsystem 0x1ad5900 > Subsystem nqn.2016-06.io.spdk:cnode2 is configured to run on a CPU core > belonging to a different NUMA node than the associated NVMe device. This = may > result in reduced performance. > Acceptor running on core 12 > =C2=A0 > =C2=A0 > Here is conf file: > =C2=A0 > # NVMf Target Configuration File > # > # Please write all parameters using ASCII. > # The parameter must be quoted if it includes whitespace. > # > # Configuration syntax: > # Leading whitespace is ignored. > # Lines starting with '#' are comments. > # Lines ending with '\' are concatenated with the next line. > # Bracketed ([]) names define sections > =C2=A0 > [Global] > =C2=A0 # Users can restrict work items to only run on certain cores by > =C2=A0 #=C2=A0 specifying a ReactorMask.=C2=A0 Default ReactorMask mask i= s defined as > =C2=A0 #=C2=A0 -c option in the 'ealargs' setting at beginning of file nv= mf_tgt.c. > =C2=A0 #ReactorMask 0x00FF > =C2=A0 ReactorMask 0x00F000 > =C2=A0 > =C2=A0 # Tracepoint group mask for spdk trace buffers > =C2=A0 # Default: 0x0 (all tracepoint groups disabled) > =C2=A0 # Set to 0xFFFFFFFFFFFFFFFF to enable all tracepoint groups. > =C2=A0 #TpointGroupMask 0x0 > =C2=A0 > =C2=A0 # syslog facility > =C2=A0 LogFacility "local7" > =C2=A0 > [Rpc] > =C2=A0 # Defines whether to enable configuration via RPC. > =C2=A0 # Default is disabled.=C2=A0 Note that the RPC interface is not > =C2=A0 # authenticated, so users should be careful about enabling > =C2=A0 # RPC in non-trusted environments. > =C2=A0 Enable No > =C2=A0 > # Users may change this section to create a different number or size of > #=C2=A0 malloc LUNs. > # This will generate 8 LUNs with a malloc-allocated backend. > # Each LUN will be size 64MB and these will be named > # Malloc0 through Malloc7.=C2=A0 Not all LUNs defined here are necessarily > #=C2=A0 used below. > [Malloc] > =C2=A0 NumberOfLuns 8 > =C2=A0 LunSizeInMB 64 > =C2=A0 > # Define NVMf protocol global options > [Nvmf] > =C2=A0 # Set the maximum number of submission and completion queues per s= ession. > =C2=A0 # Setting this to '8', for example, allows for 8 submission and 8 = completion > queues > =C2=A0 # per session. > =C2=A0 MaxQueuesPerSession 128 > =C2=A0 > =C2=A0 # Set the maximum number of outstanding I/O per queue. > =C2=A0 #MaxQueueDepth 128 > =C2=A0 > =C2=A0 # Set the maximum in-capsule data size. Must be a multiple of 16. > =C2=A0 #InCapsuleDataSize 4096 > =C2=A0 > =C2=A0 # Set the maximum I/O size. Must be a multiple of 4096. > =C2=A0 #MaxIOSize 131072 > =C2=A0 > =C2=A0 # Set the global acceptor lcore ID, lcores are numbered starting a= t 0. > =C2=A0 AcceptorCore 12 > =C2=A0 > =C2=A0 # Set how often the acceptor polls for incoming connections. The a= cceptor is > also > =C2=A0 # responsible for polling existing connections that have gone idle= . 0 means > continuously > =C2=A0 # poll. Units in microseconds. > =C2=A0 #AcceptorPollRate=C2=A0 1000 > =C2=A0 AcceptorPollRate=C2=A0 0 > =C2=A0 > # Define an NVMf Subsystem. > # Direct controller > [Subsystem1] > =C2=A0 NQN nqn.2016-06.io.spdk:cnode1 > =C2=A0 Core 13 > =C2=A0 Mode Direct > =C2=A0 Listen RDMA 101.10.10.180:4420 > #=C2=A0 Host nqn.2016-06.io.spdk:init > =C2=A0 NVMe 0000:06:00.0 > =C2=A0 > [Subsystem2] > =C2=A0 NQN nqn.2016-06.io.spdk:cnode2 > =C2=A0 Core 14 > =C2=A0 Mode Direct > =C2=A0 Listen RDMA 100.10.10.180:4420 > #=C2=A0 Host nqn.2016-06.io.spdk:init > =C2=A0 NVMe 0000:88:00.0 > =C2=A0 > =C2=A0 > My NUMA nodes and core: > NUMA node0 CPU(s):=C2=A0=C2=A0=C2=A0=C2=A0 > 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46 > NUMA node1 CPU(s):=C2=A0=C2=A0=C2=A0=C2=A0 > 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47 > =C2=A0 > Thanks, > -Rajinikanth > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Kariuki, Joh= n K > Sent: Thursday, November 17, 2016 1:34 PM > To: Storage Performance Development Kit > Subject: Re: [SPDK] nvmf.conf: AcceptorCore Vs Core > =C2=A0 > Raj > What is your Reactor Mask? > Here is an example of settings that I have used in my system to successfu= lly > assign work items to different cores. > 1)=C2=A0=C2=A0=C2=A0=C2=A0 Set the reactor mask to ReactorMask 0xF000000 = in the conf file to use > cores 24, 25, 26 and 27 for SPDK. > The ReactorMask restricts work items to only run on certain cores. > 2)=C2=A0=C2=A0=C2=A0=C2=A0 Put the acceptor on core 24 in conf file: Acce= ptorCore 24 > 3)=C2=A0=C2=A0=C2=A0=C2=A0 Put my subsystems on Core 25 and 26 > [Subsystem1] > =C2=A0 NQN nqn.2016-06.io.spdk:cnode1 > =C2=A0 Core 25 > =C2=A0 Mode Direct > =C2=A0 Listen RDMA 192.168.100.8:4420 > =C2=A0 Host nqn.2016-06.io.spdk:init > =C2=A0 NVMe 0000:81:00.0 > =C2=A0 > # Multiple subsystems are allowed. > [Subsystem2] > =C2=A0 NQN nqn.2016-06.io.spdk:cnode2 > =C2=A0 Core 26 > =C2=A0 Mode Direct > =C2=A0 Listen RDMA 192.168.100.9:4420 > =C2=A0 Host nqn.2016-06.io.spdk:init > =C2=A0 NVMe 0000:86:00.0 > =C2=A0 > 4)=C2=A0=C2=A0=C2=A0=C2=A0 Put the master on core 27 using =E2=80=93p at = command line ./nvmf_tgt -c > nvmf.conf.coreaffinity -p 27 > =C2=A0 > When the nvmf target starts I get the following output > Starting Intel(R) DPDK initialization ... > [ DPDK EAL parameters: nvmf -c f000000 -n 4 -m 2048 --master-lcore=3D27 -= -file- > prefix=3Drte0 --proc-type=3Dauto ] > EAL: Detected 96 lcore(s) > EAL: Auto-detected process type: PRIMARY > EAL: No free hugepages reported in hugepages-1048576kB > EAL: Probing VFIO support... > done. > Occupied cpu core mask is 0xf000000 > Occupied cpu socket mask is 0x2 > Ioat Copy Engine Offload Enabled > Total cores available: 4 > Reactor started on core 24 on socket 1 > Reactor started on core 25 on socket 1 > Reactor started on core 27 on socket 1 > Reactor started on core 26 on socket 1 > *** RDMA Transport Init *** > *** RDMA Transport Init *** > allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 27 on s= ocket > 1 > allocated subsystem nqn.2016-06.io.spdk:cnode1 on lcore 25 on socket 1 > *** NVMf Target Listening on 192.168.100.8 port 4420 *** > EAL: PCI device 0000:81:00.0 on NUMA socket 1 > EAL:=C2=A0=C2=A0 probe driver: 8086:953 SPDK NVMe > EAL: PCI device 0000:86:00.0 on NUMA socket 1 > EAL:=C2=A0=C2=A0 probe driver: 8086:953 SPDK NVMe > Attaching NVMe device 0x7f9256c38b80 at 0:81:0.0 to subsystem nqn.2016- > 06.io.spdk:cnode1 > allocated subsystem nqn.2016-06.io.spdk:cnode2 on lcore 26 on socket 1 > *** NVMf Target Listening on 192.168.100.9 port 4420 *** > EAL: PCI device 0000:81:00.0 on NUMA socket 1 > EAL:=C2=A0=C2=A0 probe driver: 8086:953 SPDK NVMe > EAL: PCI device 0000:86:00.0 on NUMA socket 1 > EAL:=C2=A0=C2=A0 probe driver: 8086:953 SPDK NVMe > Attaching NVMe device 0x7f9256c17880 at 0:86:0.0 to subsystem nqn.2016- > 06.io.spdk:cnode2 > Acceptor running on core 24 on socket 1 > =C2=A0 > Hope this helps. > =C2=A0 > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Raj (Rajinik= anth) > Pandurangan > Sent: Thursday, November 17, 2016 12:52 PM > To: Storage Performance Development Kit > Subject: [SPDK] nvmf.conf: AcceptorCore Vs Core > =C2=A0 > Hello, > =C2=A0 > I have a server with two NUMA nodes.=C2=A0 On each node, configured a NIC. > =C2=A0 > In nvmf.conf file, based on the node configuration, would like to assign = right > lcore. > =C2=A0 > Here is snippet of nvmf.conf: > =E2=80=A6 > .. > [Subsystem1] > NQN nqn.2016-06.io.spdk:cnode1 > Core 0 > Mode Direct > Listen RDMA 100.10.10.180:4420 > NVMe 0000:06:00.0 > =C2=A0 > =C2=A0 > [Subsystem2] > NQN nqn.2016-06.io.spdk:cnode2 > Core 1 > Mode Direct > Listen RDMA 101.10.10.180:4420 > NVMe 0000:86:00.0 > =C2=A0 > =C2=A0 > But noticed that it=E2=80=99s always uses =E2=80=9Ccore 0=E2=80=9D for bo= th the Subsystems no matter > what the value assigned to =E2=80=9CCore=E2=80=9D under =E2=80=9Csubsyste= m=E2=80=9D section. > =C2=A0 > Following warning confirms it=E2=80=99s uses lcore 0. > =C2=A0 > allocated subsystem nqn.2016-06.io.spdk:cnode2 on lcore 0 > =E2=80=9CSubsystem nqn.2016-06.io.spdk:cnode2 is configured to run on a C= PU core > belonging to a different NUMA node than the associated NVMe device. This = may > result in reduced performance.=E2=80=9D > =C2=A0 > Also getting =E2=80=9CSegment Fault=E2=80=9D if I try to set any non-zero= value to > =E2=80=9CAcceptorCore=E2=80=9D. > =C2=A0 > It would be nice if any of you could give more insights about =E2=80=9CAc= ceptorCore=E2=80=9D > and =E2=80=9CCore =E2=80=9D. > =C2=A0 > Thanks, > =C2=A0 > =C2=A0 > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============8114005486870835505==--