From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============2224009900026266488==" MIME-Version: 1.0 From: Harris, James R Subject: Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core Date: Thu, 29 Nov 2018 14:37:29 +0000 Message-ID: <154C733C-9DB0-4858-BBC2-578174DDDF7A@intel.com> In-Reply-To: 6ae02999-cda6-23db-249f-4e4169c57b60@dev.mellanox.co.il List-ID: To: spdk@lists.01.org --===============2224009900026266488== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable =EF=BB=BFOn 11/29/18, 7:31 AM, "SPDK on behalf of Sasha Kotchubievsky" wrote: = The crash happens also with 4K blocks. = I just play with different block size and test the solution, regardless = how the scenario is realistic. = BTW, RDMA based solution works fine with large blocks. Hi Sasha, Could you file an issue in GitHub for this? https://github.com/spdk/spdk/issues Thanks, -Jim = = Best regards = Sasha = = > > Regards, > Andrey > > >> Backtrace: >> >> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue >> (tqpair=3D0xe19040) at nvme_tcp.c:447 >> 447 pdu_length =3D pdu->hdr.common.plen - >> pdu->writev_offset; >> Missing separate debuginfos, use: debuginfo-install >> glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64 >> libgcc-4.8.5-28.el7_5.1.x86_64 >> libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64 >> libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x8= 6_64 >> libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64 >> openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64 >> (gdb) bt >> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue >> (tqpair=3D0xe19040) at nvme_tcp.c:447 >> #1 0x000000000042322c in nvme_tcp_qpair_process_completions >> (qpair=3D0xe19040, max_completions=3D0) at nvme_tcp.c:1555 >> #2 0x000000000041e01a in nvme_transport_qpair_process_completions >> (qpair=3D0xe19040, max_completions=3D0) at nvme_transport.c:224 >> #3 0x000000000041a78d in spdk_nvme_qpair_process_completions >> (qpair=3D0xe19040, max_completions=3D0) at nvme_qpair.c:413 >> #4 0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock >> (qpair=3D0xe19040, status=3D0x7ffe66d56dd0, robust_mutex=3D0x0) at n= vme.c:128 >> #5 0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=3D0xe= 19040, >> status=3D0x7ffe66d56dd0) at nvme.c:142 >> #6 0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=3D0xe1ed60, >> offset=3D20, size=3D0 '\000', value=3D0x7ffe66d56e88) at nvme_fabric= .c:97 >> #7 0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=3D0xe1e= d60, >> offset=3D20, value=3D0x7ffe66d56f40) at nvme_fabric.c:130 >> #8 0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=3D0xe1ed60, >> offset=3D20, value=3D0x7ffe66d56f40) at nvme_tcp.c:369 >> #9 0x000000000041d91f in nvme_transport_ctrlr_get_reg_4 >> (ctrlr=3D0xe1ed60, offset=3D20, value=3D0x7ffe66d56f40) at nvme_tran= sport.c:139 >> #10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=3D0xe1ed60, >> cc=3D0x7ffe66d56f40) at nvme_ctrlr.c:49 >> #11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair >> (ctrlr=3D0xe1ed60, user_opts=3D0x7ffe66d56f80, opts_size=3D12) at >> nvme_ctrlr.c:251 >> #12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=3D0xe18d20) at >> perf.c:826 >> #13 0x0000000000406739 in work_fn (arg=3D0xe198c0) at perf.c:862 >> #14 0x00000000004088ff in main (argc=3D14, argv=3D0x7ffe66d57148) at >> perf.c:1709 >> >> Environment: >> >> OS: CentOS Linux release 7.5.1804 (Core) >> >> Kernel: 3.10.0-862.el7.x86_64 >> >> Target output" >> >> Starting SPDK v19.01-pre / DPDK 18.08.0 initialization... >> [ DPDK EAL parameters: nvmf --no-shconf -c 0x3 >> --base-virtaddr=3D0x200000000000 --file-prefix=3Dspdk_pid30815 ] >> EAL: Detected 24 lcore(s) >> EAL: Detected 2 NUMA nodes >> EAL: No free hugepages reported in hugepages-1048576kB >> EAL: Probing VFIO support... >> app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2 >> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1 >> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 >> EAL: PCI device 0000:81:00.0 on NUMA socket 1 >> EAL: probe driver: 8086:2700 spdk_nvme >> conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options >> detected for the NVMe-oF target. >> The following options are no longer controlled by the target >> and should be set in the transport on a per-transport basis: >> MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize, >> IOUnitSize >> This can be accomplished by setting the options through the >> create_nvmf_transport RPC. >> You may also continue to configure these options in the conf file un= der >> each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP >> Transport Init *** >> tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Liste= ning >> on 1.1.75.1 port 1023 *** >> >> Thanks >> >> Sasha >> >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk = --===============2224009900026266488==--