Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core

[Sasha Kotchubievsky <sashakot at dev.mellanox.co.il>]

[-- Attachment #1: Type: text/plain, Size: 5229 bytes --]

PSB[SK]

On 11/29/2018 3:45 PM, Andrey Kuzmin wrote:
> On Thu, Nov 29, 2018, 15:44 Sasha Kotchubievsky <sashakot(a)dev.mellanox.co.il>
> wrote:
>
>> Hi,
>>
>> I'm testing NVME-OF TCP. I can run "perf" applications only on single
>> core. If I try to use 2 or more core, it crashes immediately.
>>
>> I tried different block sizes, number of cores in the target, read/write
>> operations.
>>
>> Do I miss something in configuration, or that's just a bug?
>>
>> Details:
>>
>> Version: "7a39a68"
>>
>> Command lines:
>>
>> Target: sudo  ./app/nvmf_tgt/nvmf_tgt -c ./nvmf.conf  -m 0x3
>>
>> Client:  sudo    examples/nvme/perf/perf -q 5 -o 1036288  -w randwrite
>> -t 60 -c 0x1100 -D -r 'trtype:TCP adrfam:IPv4 traddr:1.1.75.1
>> trsvcid:1023 nqn.2016-06.io.spdk.r-dcs75:rd0'
>>
> Just curios, do you truly believe 1MB+ random writes to be a realistic
> workload?

The crash happens also with 4K blocks.

I just play with different block size and test the solution, regardless 
how the scenario is realistic.

BTW, RDMA based solution works fine with large blocks.


Best regards

Sasha


>
> Regards,
> Andrey
>
>
>> Backtrace:
>>
>> #0  0x0000000000420515 in nvme_tcp_qpair_process_send_queue
>> (tqpair=0xe19040) at nvme_tcp.c:447
>> 447                     pdu_length = pdu->hdr.common.plen -
>> pdu->writev_offset;
>> Missing separate debuginfos, use: debuginfo-install
>> glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64
>> libgcc-4.8.5-28.el7_5.1.x86_64
>> libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64
>> libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64
>> libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
>> openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
>> (gdb) bt
>> #0  0x0000000000420515 in nvme_tcp_qpair_process_send_queue
>> (tqpair=0xe19040) at nvme_tcp.c:447
>> #1  0x000000000042322c in nvme_tcp_qpair_process_completions
>> (qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
>> #2  0x000000000041e01a in nvme_transport_qpair_process_completions
>> (qpair=0xe19040, max_completions=0) at nvme_transport.c:224
>> #3  0x000000000041a78d in spdk_nvme_qpair_process_completions
>> (qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
>> #4  0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock
>> (qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
>> #5  0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040,
>> status=0x7ffe66d56dd0) at nvme.c:142
>> #6  0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60,
>> offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
>> #7  0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60,
>> offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
>> #8  0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60,
>> offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
>> #9  0x000000000041d91f in nvme_transport_ctrlr_get_reg_4
>> (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
>> #10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60,
>> cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
>> #11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair
>> (ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at
>> nvme_ctrlr.c:251
>> #12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at
>> perf.c:826
>> #13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
>> #14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at
>> perf.c:1709
>>
>> Environment:
>>
>> OS: CentOS Linux release 7.5.1804 (Core)
>>
>> Kernel: 3.10.0-862.el7.x86_64
>>
>> Target output"
>>
>> Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
>> [ DPDK EAL parameters: nvmf --no-shconf -c 0x3
>> --base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
>> EAL: Detected 24 lcore(s)
>> EAL: Detected 2 NUMA nodes
>> EAL: No free hugepages reported in hugepages-1048576kB
>> EAL: Probing VFIO support...
>> app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
>> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
>> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
>> EAL: PCI device 0000:81:00.0 on NUMA socket 1
>> EAL:   probe driver: 8086:2700 spdk_nvme
>> conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options
>> detected for the NVMe-oF target.
>> The following options are no longer controlled by the target
>> and should be set in the transport on a per-transport basis:
>> MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize,
>> IOUnitSize
>> This can be accomplished by setting the options through the
>> create_nvmf_transport RPC.
>> You may also continue to configure these options in the conf file under
>> each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP
>> Transport Init ***
>> tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening
>> on 1.1.75.1 port 1023 ***
>>
>> Thanks
>>
>> Sasha
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>