From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============1524372207590947206==" MIME-Version: 1.0 From: Sasha Kotchubievsky Subject: Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core Date: Fri, 30 Nov 2018 08:32:36 +0200 Message-ID: In-Reply-To: FA6C2217B01E9D48A581BB48660210143E09E9D9@shsmsx102.ccr.corp.intel.com List-ID: To: spdk@lists.01.org --===============1524372207590947206== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Great Thanks I'll review it and test next week. We're in weekend already On 11/30/2018 6:41 AM, Yang, Ziye wrote: > Hi Sasha, > > You may try this patch: https://review.gerrithub.io/#/c/spdk/spdk/+/43557= 7/ > > > > > Best Regards > Ziye Yang > > > -----Original Message----- > From: Yang, Ziye > Sent: Friday, November 30, 2018 11:53 AM > To: 'Storage Performance Development Kit' > Subject: RE: [SPDK] [NVME-OF TCP] Crash in perf running on more than sing= le core > > It is a definitely a bug, which is not related with the I/O size, but rel= ated with the multiple threads. > > > > > Best Regards > Ziye Yang > > > -----Original Message----- > From: Yang, Ziye > Sent: Friday, November 30, 2018 9:30 AM > To: spdk(a)lists.01.org > Subject: RE: [SPDK] [NVME-OF TCP] Crash in perf running on more than sing= le core > > Hi Sasha, > > Will look at this issue. > > > > > Best Regards > Ziye Yang > > > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sasha Kotchu= bievsky > Sent: Thursday, November 29, 2018 8:45 PM > To: spdk(a)lists.01.org > Subject: [SPDK] [NVME-OF TCP] Crash in perf running on more than single c= ore > > Hi, > > I'm testing NVME-OF TCP. I can run "perf" applications only on single cor= e. If I try to use 2 or more core, it crashes immediately. > > I tried different block sizes, number of cores in the target, read/write = operations. > > Do I miss something in configuration, or that's just a bug? > > Details: > > Version: "7a39a68" > > Command lines: > > Target: sudo=C2=A0 ./app/nvmf_tgt/nvmf_tgt -c ./nvmf.conf=C2=A0 -m 0x3 > > Client:=C2=A0 sudo=C2=A0=C2=A0=C2=A0 examples/nvme/perf/perf -q 5 -o 1036= 288=C2=A0 -w randwrite -t 60 -c 0x1100 -D -r 'trtype:TCP adrfam:IPv4 traddr= :1.1.75.1 > trsvcid:1023 nqn.2016-06.io.spdk.r-dcs75:rd0' > > Backtrace: > > #0=C2=A0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue > (tqpair=3D0xe19040) at nvme_tcp.c:447 > 447=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pdu_length =3D pdu->hdr= .common.plen - > pdu->writev_offset; > Missing separate debuginfos, use: debuginfo-install > glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64 > libgcc-4.8.5-28.el7_5.1.x86_64 > libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64 > libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64 > libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64 > openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64 > (gdb) bt > #0=C2=A0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue > (tqpair=3D0xe19040) at nvme_tcp.c:447 > #1=C2=A0 0x000000000042322c in nvme_tcp_qpair_process_completions > (qpair=3D0xe19040, max_completions=3D0) at nvme_tcp.c:1555 > #2=C2=A0 0x000000000041e01a in nvme_transport_qpair_process_completions > (qpair=3D0xe19040, max_completions=3D0) at nvme_transport.c:224 > #3=C2=A0 0x000000000041a78d in spdk_nvme_qpair_process_completions > (qpair=3D0xe19040, max_completions=3D0) at nvme_qpair.c:413 > #4=C2=A0 0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock > (qpair=3D0xe19040, status=3D0x7ffe66d56dd0, robust_mutex=3D0x0) at nvme.c= :128 > #5=C2=A0 0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=3D0xe= 19040, > status=3D0x7ffe66d56dd0) at nvme.c:142 > #6=C2=A0 0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=3D0xe1ed60= , offset=3D20, size=3D0 '\000', value=3D0x7ffe66d56e88) at nvme_fabric.c:97 > #7=C2=A0 0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=3D0xe1e= d60, offset=3D20, value=3D0x7ffe66d56f40) at nvme_fabric.c:130 > #8=C2=A0 0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=3D0xe1ed60= , offset=3D20, value=3D0x7ffe66d56f40) at nvme_tcp.c:369 > #9=C2=A0 0x000000000041d91f in nvme_transport_ctrlr_get_reg_4 (ctrlr=3D0x= e1ed60, offset=3D20, value=3D0x7ffe66d56f40) at nvme_transport.c:139 > #10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=3D0xe1ed60, > cc=3D0x7ffe66d56f40) at nvme_ctrlr.c:49 > #11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair (ctrlr=3D0xe1ed6= 0, user_opts=3D0x7ffe66d56f80, opts_size=3D12) at nvme_ctrlr.c:251 > #12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=3D0xe18d20) at perf.= c:826 > #13 0x0000000000406739 in work_fn (arg=3D0xe198c0) at perf.c:862 > #14 0x00000000004088ff in main (argc=3D14, argv=3D0x7ffe66d57148) at perf= .c:1709 > > Environment: > > OS: CentOS Linux release 7.5.1804 (Core) > > Kernel: 3.10.0-862.el7.x86_64 > > Target output" > > Starting SPDK v19.01-pre / DPDK 18.08.0 initialization... > [ DPDK EAL parameters: nvmf --no-shconf -c 0x3 > --base-virtaddr=3D0x200000000000 --file-prefix=3Dspdk_pid30815 ] > EAL: Detected 24 lcore(s) > EAL: Detected 2 NUMA nodes > EAL: No free hugepages reported in hugepages-1048576kB > EAL: Probing VFIO support... > app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2 > reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1 > reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 > EAL: PCI device 0000:81:00.0 on NUMA socket 1 > EAL:=C2=A0=C2=A0 probe driver: 8086:2700 spdk_nvme > conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options detecte= d for the NVMe-oF target. > The following options are no longer controlled by the target and should b= e set in the transport on a per-transport basis: > MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize, IOUnitS= ize This can be accomplished by setting the options through the create_nvmf= _transport RPC. > You may also continue to configure these options in the conf file under e= ach transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP Transport = Init *** > tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening = on 1.1.75.1 port 1023 *** > > Thanks > > Sasha > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============1524372207590947206==--