Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core

[Sasha Kotchubievsky <sashakot at dev.mellanox.co.il>]

[-- Attachment #1: Type: text/plain, Size: 5971 bytes --]

Great

Thanks

I'll review it and test next week. We're in weekend already



On 11/30/2018 6:41 AM, Yang, Ziye wrote:
> Hi Sasha,
>
> You may try this patch: https://review.gerrithub.io/#/c/spdk/spdk/+/435577/
>
>
>
>
> Best Regards
> Ziye Yang
>
>
> -----Original Message-----
> From: Yang, Ziye
> Sent: Friday, November 30, 2018 11:53 AM
> To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
> Subject: RE: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
>
> It is a definitely a bug, which is not related with the I/O size, but related with the multiple threads.
>
>
>
>
> Best Regards
> Ziye Yang
>
>
> -----Original Message-----
> From: Yang, Ziye
> Sent: Friday, November 30, 2018 9:30 AM
> To: spdk(a)lists.01.org
> Subject: RE: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
>
> Hi Sasha,
>
> Will look at this issue.
>
>
>
>
> Best Regards
> Ziye Yang
>
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sasha Kotchubievsky
> Sent: Thursday, November 29, 2018 8:45 PM
> To: spdk(a)lists.01.org
> Subject: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
>
> Hi,
>
> I'm testing NVME-OF TCP. I can run "perf" applications only on single core. If I try to use 2 or more core, it crashes immediately.
>
> I tried different block sizes, number of cores in the target, read/write operations.
>
> Do I miss something in configuration, or that's just a bug?
>
> Details:
>
> Version: "7a39a68"
>
> Command lines:
>
> Target: sudo  ./app/nvmf_tgt/nvmf_tgt -c ./nvmf.conf  -m 0x3
>
> Client:  sudo    examples/nvme/perf/perf -q 5 -o 1036288  -w randwrite -t 60 -c 0x1100 -D -r 'trtype:TCP adrfam:IPv4 traddr:1.1.75.1
> trsvcid:1023 nqn.2016-06.io.spdk.r-dcs75:rd0'
>
> Backtrace:
>
> #0  0x0000000000420515 in nvme_tcp_qpair_process_send_queue
> (tqpair=0xe19040) at nvme_tcp.c:447
> 447                     pdu_length = pdu->hdr.common.plen -
> pdu->writev_offset;
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64
> libgcc-4.8.5-28.el7_5.1.x86_64
> libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64
> libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64
> libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
> openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
> (gdb) bt
> #0  0x0000000000420515 in nvme_tcp_qpair_process_send_queue
> (tqpair=0xe19040) at nvme_tcp.c:447
> #1  0x000000000042322c in nvme_tcp_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
> #2  0x000000000041e01a in nvme_transport_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_transport.c:224
> #3  0x000000000041a78d in spdk_nvme_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
> #4  0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock
> (qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
> #5  0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040,
> status=0x7ffe66d56dd0) at nvme.c:142
> #6  0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60, offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
> #7  0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
> #8  0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
> #9  0x000000000041d91f in nvme_transport_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
> #10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60,
> cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
> #11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair (ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at nvme_ctrlr.c:251
> #12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at perf.c:826
> #13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
> #14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at perf.c:1709
>
> Environment:
>
> OS: CentOS Linux release 7.5.1804 (Core)
>
> Kernel: 3.10.0-862.el7.x86_64
>
> Target output"
>
> Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
> [ DPDK EAL parameters: nvmf --no-shconf -c 0x3
> --base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
> EAL: Detected 24 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> EAL: PCI device 0000:81:00.0 on NUMA socket 1
> EAL:   probe driver: 8086:2700 spdk_nvme
> conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options detected for the NVMe-oF target.
> The following options are no longer controlled by the target and should be set in the transport on a per-transport basis:
> MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize, IOUnitSize This can be accomplished by setting the options through the create_nvmf_transport RPC.
> You may also continue to configure these options in the conf file under each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
> tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 1.1.75.1 port 1023 ***
>
> Thanks
>
> Sasha
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk