Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core

[Harris, James R <james.r.harris at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 5195 bytes --]



On 11/29/18, 7:31 AM, "SPDK on behalf of Sasha Kotchubievsky" <spdk-bounces(a)lists.01.org on behalf of sashakot(a)dev.mellanox.co.il> wrote:

    
    The crash happens also with 4K blocks.
    
    I just play with different block size and test the solution, regardless 
    how the scenario is realistic.
    
    BTW, RDMA based solution works fine with large blocks.

Hi Sasha,

Could you file an issue in GitHub for this?

https://github.com/spdk/spdk/issues

Thanks,

-Jim

    
    
    Best regards
    
    Sasha
    
    
    >
    > Regards,
    > Andrey
    >
    >
    >> Backtrace:
    >>
    >> #0  0x0000000000420515 in nvme_tcp_qpair_process_send_queue
    >> (tqpair=0xe19040) at nvme_tcp.c:447
    >> 447                     pdu_length = pdu->hdr.common.plen -
    >> pdu->writev_offset;
    >> Missing separate debuginfos, use: debuginfo-install
    >> glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64
    >> libgcc-4.8.5-28.el7_5.1.x86_64
    >> libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64
    >> libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64
    >> libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
    >> openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
    >> (gdb) bt
    >> #0  0x0000000000420515 in nvme_tcp_qpair_process_send_queue
    >> (tqpair=0xe19040) at nvme_tcp.c:447
    >> #1  0x000000000042322c in nvme_tcp_qpair_process_completions
    >> (qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
    >> #2  0x000000000041e01a in nvme_transport_qpair_process_completions
    >> (qpair=0xe19040, max_completions=0) at nvme_transport.c:224
    >> #3  0x000000000041a78d in spdk_nvme_qpair_process_completions
    >> (qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
    >> #4  0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock
    >> (qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
    >> #5  0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040,
    >> status=0x7ffe66d56dd0) at nvme.c:142
    >> #6  0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60,
    >> offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
    >> #7  0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60,
    >> offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
    >> #8  0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60,
    >> offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
    >> #9  0x000000000041d91f in nvme_transport_ctrlr_get_reg_4
    >> (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
    >> #10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60,
    >> cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
    >> #11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair
    >> (ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at
    >> nvme_ctrlr.c:251
    >> #12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at
    >> perf.c:826
    >> #13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
    >> #14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at
    >> perf.c:1709
    >>
    >> Environment:
    >>
    >> OS: CentOS Linux release 7.5.1804 (Core)
    >>
    >> Kernel: 3.10.0-862.el7.x86_64
    >>
    >> Target output"
    >>
    >> Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
    >> [ DPDK EAL parameters: nvmf --no-shconf -c 0x3
    >> --base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
    >> EAL: Detected 24 lcore(s)
    >> EAL: Detected 2 NUMA nodes
    >> EAL: No free hugepages reported in hugepages-1048576kB
    >> EAL: Probing VFIO support...
    >> app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
    >> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
    >> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
    >> EAL: PCI device 0000:81:00.0 on NUMA socket 1
    >> EAL:   probe driver: 8086:2700 spdk_nvme
    >> conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options
    >> detected for the NVMe-oF target.
    >> The following options are no longer controlled by the target
    >> and should be set in the transport on a per-transport basis:
    >> MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize,
    >> IOUnitSize
    >> This can be accomplished by setting the options through the
    >> create_nvmf_transport RPC.
    >> You may also continue to configure these options in the conf file under
    >> each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP
    >> Transport Init ***
    >> tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening
    >> on 1.1.75.1 port 1023 ***
    >>
    >> Thanks
    >>
    >> Sasha
    >>
    >> _______________________________________________
    >> SPDK mailing list
    >> SPDK(a)lists.01.org
    >> https://lists.01.org/mailman/listinfo/spdk
    >>
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk