* [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
@ 2018-11-29 12:44 Sasha Kotchubievsky
0 siblings, 0 replies; 11+ messages in thread
From: Sasha Kotchubievsky @ 2018-11-29 12:44 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 4342 bytes --]
Hi,
I'm testing NVME-OF TCP. I can run "perf" applications only on single
core. If I try to use 2 or more core, it crashes immediately.
I tried different block sizes, number of cores in the target, read/write
operations.
Do I miss something in configuration, or that's just a bug?
Details:
Version: "7a39a68"
Command lines:
Target: sudo ./app/nvmf_tgt/nvmf_tgt -c ./nvmf.conf -m 0x3
Client: sudo examples/nvme/perf/perf -q 5 -o 1036288 -w randwrite
-t 60 -c 0x1100 -D -r 'trtype:TCP adrfam:IPv4 traddr:1.1.75.1
trsvcid:1023 nqn.2016-06.io.spdk.r-dcs75:rd0'
Backtrace:
#0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
(tqpair=0xe19040) at nvme_tcp.c:447
447 pdu_length = pdu->hdr.common.plen -
pdu->writev_offset;
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64
libgcc-4.8.5-28.el7_5.1.x86_64
libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64
libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64
libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
(tqpair=0xe19040) at nvme_tcp.c:447
#1 0x000000000042322c in nvme_tcp_qpair_process_completions
(qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
#2 0x000000000041e01a in nvme_transport_qpair_process_completions
(qpair=0xe19040, max_completions=0) at nvme_transport.c:224
#3 0x000000000041a78d in spdk_nvme_qpair_process_completions
(qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
#4 0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock
(qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
#5 0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040,
status=0x7ffe66d56dd0) at nvme.c:142
#6 0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60,
offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
#7 0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60,
offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
#8 0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60,
offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
#9 0x000000000041d91f in nvme_transport_ctrlr_get_reg_4
(ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
#10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60,
cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
#11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair
(ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at nvme_ctrlr.c:251
#12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at perf.c:826
#13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
#14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at perf.c:1709
Environment:
OS: CentOS Linux release 7.5.1804 (Core)
Kernel: 3.10.0-862.el7.x86_64
Target output"
Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: nvmf --no-shconf -c 0x3
--base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL: probe driver: 8086:2700 spdk_nvme
conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options
detected for the NVMe-oF target.
The following options are no longer controlled by the target
and should be set in the transport on a per-transport basis:
MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize, IOUnitSize
This can be accomplished by setting the options through the
create_nvmf_transport RPC.
You may also continue to configure these options in the conf file under
each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP
Transport Init ***
tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening
on 1.1.75.1 port 1023 ***
Thanks
Sasha
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
@ 2018-11-29 13:45 Andrey Kuzmin
0 siblings, 0 replies; 11+ messages in thread
From: Andrey Kuzmin @ 2018-11-29 13:45 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 4835 bytes --]
On Thu, Nov 29, 2018, 15:44 Sasha Kotchubievsky <sashakot(a)dev.mellanox.co.il>
wrote:
> Hi,
>
> I'm testing NVME-OF TCP. I can run "perf" applications only on single
> core. If I try to use 2 or more core, it crashes immediately.
>
> I tried different block sizes, number of cores in the target, read/write
> operations.
>
> Do I miss something in configuration, or that's just a bug?
>
> Details:
>
> Version: "7a39a68"
>
> Command lines:
>
> Target: sudo ./app/nvmf_tgt/nvmf_tgt -c ./nvmf.conf -m 0x3
>
> Client: sudo examples/nvme/perf/perf -q 5 -o 1036288 -w randwrite
> -t 60 -c 0x1100 -D -r 'trtype:TCP adrfam:IPv4 traddr:1.1.75.1
> trsvcid:1023 nqn.2016-06.io.spdk.r-dcs75:rd0'
>
Just curios, do you truly believe 1MB+ random writes to be a realistic
workload?
Regards,
Andrey
> Backtrace:
>
> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
> (tqpair=0xe19040) at nvme_tcp.c:447
> 447 pdu_length = pdu->hdr.common.plen -
> pdu->writev_offset;
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64
> libgcc-4.8.5-28.el7_5.1.x86_64
> libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64
> libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64
> libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
> openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
> (gdb) bt
> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
> (tqpair=0xe19040) at nvme_tcp.c:447
> #1 0x000000000042322c in nvme_tcp_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
> #2 0x000000000041e01a in nvme_transport_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_transport.c:224
> #3 0x000000000041a78d in spdk_nvme_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
> #4 0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock
> (qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
> #5 0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040,
> status=0x7ffe66d56dd0) at nvme.c:142
> #6 0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60,
> offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
> #7 0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60,
> offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
> #8 0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60,
> offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
> #9 0x000000000041d91f in nvme_transport_ctrlr_get_reg_4
> (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
> #10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60,
> cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
> #11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair
> (ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at
> nvme_ctrlr.c:251
> #12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at
> perf.c:826
> #13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
> #14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at
> perf.c:1709
>
> Environment:
>
> OS: CentOS Linux release 7.5.1804 (Core)
>
> Kernel: 3.10.0-862.el7.x86_64
>
> Target output"
>
> Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
> [ DPDK EAL parameters: nvmf --no-shconf -c 0x3
> --base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
> EAL: Detected 24 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> EAL: PCI device 0000:81:00.0 on NUMA socket 1
> EAL: probe driver: 8086:2700 spdk_nvme
> conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options
> detected for the NVMe-oF target.
> The following options are no longer controlled by the target
> and should be set in the transport on a per-transport basis:
> MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize,
> IOUnitSize
> This can be accomplished by setting the options through the
> create_nvmf_transport RPC.
> You may also continue to configure these options in the conf file under
> each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP
> Transport Init ***
> tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening
> on 1.1.75.1 port 1023 ***
>
> Thanks
>
> Sasha
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
--
Regards,
Andrey
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
@ 2018-11-29 14:31 Sasha Kotchubievsky
0 siblings, 0 replies; 11+ messages in thread
From: Sasha Kotchubievsky @ 2018-11-29 14:31 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5229 bytes --]
PSB[SK]
On 11/29/2018 3:45 PM, Andrey Kuzmin wrote:
> On Thu, Nov 29, 2018, 15:44 Sasha Kotchubievsky <sashakot(a)dev.mellanox.co.il>
> wrote:
>
>> Hi,
>>
>> I'm testing NVME-OF TCP. I can run "perf" applications only on single
>> core. If I try to use 2 or more core, it crashes immediately.
>>
>> I tried different block sizes, number of cores in the target, read/write
>> operations.
>>
>> Do I miss something in configuration, or that's just a bug?
>>
>> Details:
>>
>> Version: "7a39a68"
>>
>> Command lines:
>>
>> Target: sudo ./app/nvmf_tgt/nvmf_tgt -c ./nvmf.conf -m 0x3
>>
>> Client: sudo examples/nvme/perf/perf -q 5 -o 1036288 -w randwrite
>> -t 60 -c 0x1100 -D -r 'trtype:TCP adrfam:IPv4 traddr:1.1.75.1
>> trsvcid:1023 nqn.2016-06.io.spdk.r-dcs75:rd0'
>>
> Just curios, do you truly believe 1MB+ random writes to be a realistic
> workload?
The crash happens also with 4K blocks.
I just play with different block size and test the solution, regardless
how the scenario is realistic.
BTW, RDMA based solution works fine with large blocks.
Best regards
Sasha
>
> Regards,
> Andrey
>
>
>> Backtrace:
>>
>> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
>> (tqpair=0xe19040) at nvme_tcp.c:447
>> 447 pdu_length = pdu->hdr.common.plen -
>> pdu->writev_offset;
>> Missing separate debuginfos, use: debuginfo-install
>> glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64
>> libgcc-4.8.5-28.el7_5.1.x86_64
>> libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64
>> libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64
>> libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
>> openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
>> (gdb) bt
>> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
>> (tqpair=0xe19040) at nvme_tcp.c:447
>> #1 0x000000000042322c in nvme_tcp_qpair_process_completions
>> (qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
>> #2 0x000000000041e01a in nvme_transport_qpair_process_completions
>> (qpair=0xe19040, max_completions=0) at nvme_transport.c:224
>> #3 0x000000000041a78d in spdk_nvme_qpair_process_completions
>> (qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
>> #4 0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock
>> (qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
>> #5 0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040,
>> status=0x7ffe66d56dd0) at nvme.c:142
>> #6 0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60,
>> offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
>> #7 0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60,
>> offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
>> #8 0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60,
>> offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
>> #9 0x000000000041d91f in nvme_transport_ctrlr_get_reg_4
>> (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
>> #10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60,
>> cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
>> #11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair
>> (ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at
>> nvme_ctrlr.c:251
>> #12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at
>> perf.c:826
>> #13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
>> #14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at
>> perf.c:1709
>>
>> Environment:
>>
>> OS: CentOS Linux release 7.5.1804 (Core)
>>
>> Kernel: 3.10.0-862.el7.x86_64
>>
>> Target output"
>>
>> Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
>> [ DPDK EAL parameters: nvmf --no-shconf -c 0x3
>> --base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
>> EAL: Detected 24 lcore(s)
>> EAL: Detected 2 NUMA nodes
>> EAL: No free hugepages reported in hugepages-1048576kB
>> EAL: Probing VFIO support...
>> app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
>> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
>> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
>> EAL: PCI device 0000:81:00.0 on NUMA socket 1
>> EAL: probe driver: 8086:2700 spdk_nvme
>> conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options
>> detected for the NVMe-oF target.
>> The following options are no longer controlled by the target
>> and should be set in the transport on a per-transport basis:
>> MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize,
>> IOUnitSize
>> This can be accomplished by setting the options through the
>> create_nvmf_transport RPC.
>> You may also continue to configure these options in the conf file under
>> each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP
>> Transport Init ***
>> tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening
>> on 1.1.75.1 port 1023 ***
>>
>> Thanks
>>
>> Sasha
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
@ 2018-11-29 14:37 Harris, James R
0 siblings, 0 replies; 11+ messages in thread
From: Harris, James R @ 2018-11-29 14:37 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5195 bytes --]
On 11/29/18, 7:31 AM, "SPDK on behalf of Sasha Kotchubievsky" <spdk-bounces(a)lists.01.org on behalf of sashakot(a)dev.mellanox.co.il> wrote:
The crash happens also with 4K blocks.
I just play with different block size and test the solution, regardless
how the scenario is realistic.
BTW, RDMA based solution works fine with large blocks.
Hi Sasha,
Could you file an issue in GitHub for this?
https://github.com/spdk/spdk/issues
Thanks,
-Jim
Best regards
Sasha
>
> Regards,
> Andrey
>
>
>> Backtrace:
>>
>> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
>> (tqpair=0xe19040) at nvme_tcp.c:447
>> 447 pdu_length = pdu->hdr.common.plen -
>> pdu->writev_offset;
>> Missing separate debuginfos, use: debuginfo-install
>> glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64
>> libgcc-4.8.5-28.el7_5.1.x86_64
>> libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64
>> libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64
>> libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
>> openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
>> (gdb) bt
>> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
>> (tqpair=0xe19040) at nvme_tcp.c:447
>> #1 0x000000000042322c in nvme_tcp_qpair_process_completions
>> (qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
>> #2 0x000000000041e01a in nvme_transport_qpair_process_completions
>> (qpair=0xe19040, max_completions=0) at nvme_transport.c:224
>> #3 0x000000000041a78d in spdk_nvme_qpair_process_completions
>> (qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
>> #4 0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock
>> (qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
>> #5 0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040,
>> status=0x7ffe66d56dd0) at nvme.c:142
>> #6 0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60,
>> offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
>> #7 0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60,
>> offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
>> #8 0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60,
>> offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
>> #9 0x000000000041d91f in nvme_transport_ctrlr_get_reg_4
>> (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
>> #10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60,
>> cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
>> #11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair
>> (ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at
>> nvme_ctrlr.c:251
>> #12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at
>> perf.c:826
>> #13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
>> #14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at
>> perf.c:1709
>>
>> Environment:
>>
>> OS: CentOS Linux release 7.5.1804 (Core)
>>
>> Kernel: 3.10.0-862.el7.x86_64
>>
>> Target output"
>>
>> Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
>> [ DPDK EAL parameters: nvmf --no-shconf -c 0x3
>> --base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
>> EAL: Detected 24 lcore(s)
>> EAL: Detected 2 NUMA nodes
>> EAL: No free hugepages reported in hugepages-1048576kB
>> EAL: Probing VFIO support...
>> app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
>> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
>> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
>> EAL: PCI device 0000:81:00.0 on NUMA socket 1
>> EAL: probe driver: 8086:2700 spdk_nvme
>> conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options
>> detected for the NVMe-oF target.
>> The following options are no longer controlled by the target
>> and should be set in the transport on a per-transport basis:
>> MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize,
>> IOUnitSize
>> This can be accomplished by setting the options through the
>> create_nvmf_transport RPC.
>> You may also continue to configure these options in the conf file under
>> each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP
>> Transport Init ***
>> tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening
>> on 1.1.75.1 port 1023 ***
>>
>> Thanks
>>
>> Sasha
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
@ 2018-11-30 1:29 Yang, Ziye
0 siblings, 0 replies; 11+ messages in thread
From: Yang, Ziye @ 2018-11-30 1:29 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 4764 bytes --]
Hi Sasha,
Will look at this issue.
Best Regards
Ziye Yang
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sasha Kotchubievsky
Sent: Thursday, November 29, 2018 8:45 PM
To: spdk(a)lists.01.org
Subject: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
Hi,
I'm testing NVME-OF TCP. I can run "perf" applications only on single core. If I try to use 2 or more core, it crashes immediately.
I tried different block sizes, number of cores in the target, read/write operations.
Do I miss something in configuration, or that's just a bug?
Details:
Version: "7a39a68"
Command lines:
Target: sudo ./app/nvmf_tgt/nvmf_tgt -c ./nvmf.conf -m 0x3
Client: sudo examples/nvme/perf/perf -q 5 -o 1036288 -w randwrite -t 60 -c 0x1100 -D -r 'trtype:TCP adrfam:IPv4 traddr:1.1.75.1
trsvcid:1023 nqn.2016-06.io.spdk.r-dcs75:rd0'
Backtrace:
#0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
(tqpair=0xe19040) at nvme_tcp.c:447
447 pdu_length = pdu->hdr.common.plen -
pdu->writev_offset;
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64
libgcc-4.8.5-28.el7_5.1.x86_64
libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64
libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64
libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
(tqpair=0xe19040) at nvme_tcp.c:447
#1 0x000000000042322c in nvme_tcp_qpair_process_completions
(qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
#2 0x000000000041e01a in nvme_transport_qpair_process_completions
(qpair=0xe19040, max_completions=0) at nvme_transport.c:224
#3 0x000000000041a78d in spdk_nvme_qpair_process_completions
(qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
#4 0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock
(qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
#5 0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040,
status=0x7ffe66d56dd0) at nvme.c:142
#6 0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60, offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
#7 0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
#8 0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
#9 0x000000000041d91f in nvme_transport_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
#10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60,
cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
#11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair (ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at nvme_ctrlr.c:251
#12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at perf.c:826
#13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
#14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at perf.c:1709
Environment:
OS: CentOS Linux release 7.5.1804 (Core)
Kernel: 3.10.0-862.el7.x86_64
Target output"
Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: nvmf --no-shconf -c 0x3
--base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL: probe driver: 8086:2700 spdk_nvme
conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options detected for the NVMe-oF target.
The following options are no longer controlled by the target and should be set in the transport on a per-transport basis:
MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize, IOUnitSize This can be accomplished by setting the options through the create_nvmf_transport RPC.
You may also continue to configure these options in the conf file under each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 1.1.75.1 port 1023 ***
Thanks
Sasha
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
@ 2018-11-30 3:53 Yang, Ziye
0 siblings, 0 replies; 11+ messages in thread
From: Yang, Ziye @ 2018-11-30 3:53 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5105 bytes --]
It is a definitely a bug, which is not related with the I/O size, but related with the multiple threads.
Best Regards
Ziye Yang
-----Original Message-----
From: Yang, Ziye
Sent: Friday, November 30, 2018 9:30 AM
To: spdk(a)lists.01.org
Subject: RE: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
Hi Sasha,
Will look at this issue.
Best Regards
Ziye Yang
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sasha Kotchubievsky
Sent: Thursday, November 29, 2018 8:45 PM
To: spdk(a)lists.01.org
Subject: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
Hi,
I'm testing NVME-OF TCP. I can run "perf" applications only on single core. If I try to use 2 or more core, it crashes immediately.
I tried different block sizes, number of cores in the target, read/write operations.
Do I miss something in configuration, or that's just a bug?
Details:
Version: "7a39a68"
Command lines:
Target: sudo ./app/nvmf_tgt/nvmf_tgt -c ./nvmf.conf -m 0x3
Client: sudo examples/nvme/perf/perf -q 5 -o 1036288 -w randwrite -t 60 -c 0x1100 -D -r 'trtype:TCP adrfam:IPv4 traddr:1.1.75.1
trsvcid:1023 nqn.2016-06.io.spdk.r-dcs75:rd0'
Backtrace:
#0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
(tqpair=0xe19040) at nvme_tcp.c:447
447 pdu_length = pdu->hdr.common.plen -
pdu->writev_offset;
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64
libgcc-4.8.5-28.el7_5.1.x86_64
libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64
libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64
libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
(tqpair=0xe19040) at nvme_tcp.c:447
#1 0x000000000042322c in nvme_tcp_qpair_process_completions
(qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
#2 0x000000000041e01a in nvme_transport_qpair_process_completions
(qpair=0xe19040, max_completions=0) at nvme_transport.c:224
#3 0x000000000041a78d in spdk_nvme_qpair_process_completions
(qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
#4 0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock
(qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
#5 0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040,
status=0x7ffe66d56dd0) at nvme.c:142
#6 0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60, offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
#7 0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
#8 0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
#9 0x000000000041d91f in nvme_transport_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
#10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60,
cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
#11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair (ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at nvme_ctrlr.c:251
#12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at perf.c:826
#13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
#14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at perf.c:1709
Environment:
OS: CentOS Linux release 7.5.1804 (Core)
Kernel: 3.10.0-862.el7.x86_64
Target output"
Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: nvmf --no-shconf -c 0x3
--base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL: probe driver: 8086:2700 spdk_nvme
conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options detected for the NVMe-oF target.
The following options are no longer controlled by the target and should be set in the transport on a per-transport basis:
MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize, IOUnitSize This can be accomplished by setting the options through the create_nvmf_transport RPC.
You may also continue to configure these options in the conf file under each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 1.1.75.1 port 1023 ***
Thanks
Sasha
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
@ 2018-11-30 4:41 Yang, Ziye
0 siblings, 0 replies; 11+ messages in thread
From: Yang, Ziye @ 2018-11-30 4:41 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5471 bytes --]
Hi Sasha,
You may try this patch: https://review.gerrithub.io/#/c/spdk/spdk/+/435577/
Best Regards
Ziye Yang
-----Original Message-----
From: Yang, Ziye
Sent: Friday, November 30, 2018 11:53 AM
To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
Subject: RE: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
It is a definitely a bug, which is not related with the I/O size, but related with the multiple threads.
Best Regards
Ziye Yang
-----Original Message-----
From: Yang, Ziye
Sent: Friday, November 30, 2018 9:30 AM
To: spdk(a)lists.01.org
Subject: RE: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
Hi Sasha,
Will look at this issue.
Best Regards
Ziye Yang
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sasha Kotchubievsky
Sent: Thursday, November 29, 2018 8:45 PM
To: spdk(a)lists.01.org
Subject: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
Hi,
I'm testing NVME-OF TCP. I can run "perf" applications only on single core. If I try to use 2 or more core, it crashes immediately.
I tried different block sizes, number of cores in the target, read/write operations.
Do I miss something in configuration, or that's just a bug?
Details:
Version: "7a39a68"
Command lines:
Target: sudo ./app/nvmf_tgt/nvmf_tgt -c ./nvmf.conf -m 0x3
Client: sudo examples/nvme/perf/perf -q 5 -o 1036288 -w randwrite -t 60 -c 0x1100 -D -r 'trtype:TCP adrfam:IPv4 traddr:1.1.75.1
trsvcid:1023 nqn.2016-06.io.spdk.r-dcs75:rd0'
Backtrace:
#0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
(tqpair=0xe19040) at nvme_tcp.c:447
447 pdu_length = pdu->hdr.common.plen -
pdu->writev_offset;
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64
libgcc-4.8.5-28.el7_5.1.x86_64
libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64
libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64
libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
(tqpair=0xe19040) at nvme_tcp.c:447
#1 0x000000000042322c in nvme_tcp_qpair_process_completions
(qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
#2 0x000000000041e01a in nvme_transport_qpair_process_completions
(qpair=0xe19040, max_completions=0) at nvme_transport.c:224
#3 0x000000000041a78d in spdk_nvme_qpair_process_completions
(qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
#4 0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock
(qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
#5 0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040,
status=0x7ffe66d56dd0) at nvme.c:142
#6 0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60, offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
#7 0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
#8 0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
#9 0x000000000041d91f in nvme_transport_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
#10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60,
cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
#11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair (ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at nvme_ctrlr.c:251
#12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at perf.c:826
#13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
#14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at perf.c:1709
Environment:
OS: CentOS Linux release 7.5.1804 (Core)
Kernel: 3.10.0-862.el7.x86_64
Target output"
Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: nvmf --no-shconf -c 0x3
--base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL: probe driver: 8086:2700 spdk_nvme
conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options detected for the NVMe-oF target.
The following options are no longer controlled by the target and should be set in the transport on a per-transport basis:
MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize, IOUnitSize This can be accomplished by setting the options through the create_nvmf_transport RPC.
You may also continue to configure these options in the conf file under each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 1.1.75.1 port 1023 ***
Thanks
Sasha
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
@ 2018-11-30 6:32 Sasha Kotchubievsky
0 siblings, 0 replies; 11+ messages in thread
From: Sasha Kotchubievsky @ 2018-11-30 6:32 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5971 bytes --]
Great
Thanks
I'll review it and test next week. We're in weekend already
On 11/30/2018 6:41 AM, Yang, Ziye wrote:
> Hi Sasha,
>
> You may try this patch: https://review.gerrithub.io/#/c/spdk/spdk/+/435577/
>
>
>
>
> Best Regards
> Ziye Yang
>
>
> -----Original Message-----
> From: Yang, Ziye
> Sent: Friday, November 30, 2018 11:53 AM
> To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
> Subject: RE: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
>
> It is a definitely a bug, which is not related with the I/O size, but related with the multiple threads.
>
>
>
>
> Best Regards
> Ziye Yang
>
>
> -----Original Message-----
> From: Yang, Ziye
> Sent: Friday, November 30, 2018 9:30 AM
> To: spdk(a)lists.01.org
> Subject: RE: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
>
> Hi Sasha,
>
> Will look at this issue.
>
>
>
>
> Best Regards
> Ziye Yang
>
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sasha Kotchubievsky
> Sent: Thursday, November 29, 2018 8:45 PM
> To: spdk(a)lists.01.org
> Subject: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
>
> Hi,
>
> I'm testing NVME-OF TCP. I can run "perf" applications only on single core. If I try to use 2 or more core, it crashes immediately.
>
> I tried different block sizes, number of cores in the target, read/write operations.
>
> Do I miss something in configuration, or that's just a bug?
>
> Details:
>
> Version: "7a39a68"
>
> Command lines:
>
> Target: sudo ./app/nvmf_tgt/nvmf_tgt -c ./nvmf.conf -m 0x3
>
> Client: sudo examples/nvme/perf/perf -q 5 -o 1036288 -w randwrite -t 60 -c 0x1100 -D -r 'trtype:TCP adrfam:IPv4 traddr:1.1.75.1
> trsvcid:1023 nqn.2016-06.io.spdk.r-dcs75:rd0'
>
> Backtrace:
>
> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
> (tqpair=0xe19040) at nvme_tcp.c:447
> 447 pdu_length = pdu->hdr.common.plen -
> pdu->writev_offset;
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64
> libgcc-4.8.5-28.el7_5.1.x86_64
> libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64
> libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64
> libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
> openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
> (gdb) bt
> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
> (tqpair=0xe19040) at nvme_tcp.c:447
> #1 0x000000000042322c in nvme_tcp_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
> #2 0x000000000041e01a in nvme_transport_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_transport.c:224
> #3 0x000000000041a78d in spdk_nvme_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
> #4 0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock
> (qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
> #5 0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040,
> status=0x7ffe66d56dd0) at nvme.c:142
> #6 0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60, offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
> #7 0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
> #8 0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
> #9 0x000000000041d91f in nvme_transport_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
> #10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60,
> cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
> #11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair (ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at nvme_ctrlr.c:251
> #12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at perf.c:826
> #13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
> #14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at perf.c:1709
>
> Environment:
>
> OS: CentOS Linux release 7.5.1804 (Core)
>
> Kernel: 3.10.0-862.el7.x86_64
>
> Target output"
>
> Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
> [ DPDK EAL parameters: nvmf --no-shconf -c 0x3
> --base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
> EAL: Detected 24 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> EAL: PCI device 0000:81:00.0 on NUMA socket 1
> EAL: probe driver: 8086:2700 spdk_nvme
> conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options detected for the NVMe-oF target.
> The following options are no longer controlled by the target and should be set in the transport on a per-transport basis:
> MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize, IOUnitSize This can be accomplished by setting the options through the create_nvmf_transport RPC.
> You may also continue to configure these options in the conf file under each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
> tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 1.1.75.1 port 1023 ***
>
> Thanks
>
> Sasha
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
@ 2018-11-30 10:59 Sasha Kotchubievsky
0 siblings, 0 replies; 11+ messages in thread
From: Sasha Kotchubievsky @ 2018-11-30 10:59 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5792 bytes --]
Done
https://github.com/spdk/spdk/issues/528
On 11/29/2018 4:37 PM, Harris, James R wrote:
>
> On 11/29/18, 7:31 AM, "SPDK on behalf of Sasha Kotchubievsky" <spdk-bounces(a)lists.01.org on behalf of sashakot(a)dev.mellanox.co.il> wrote:
>
>
> The crash happens also with 4K blocks.
>
> I just play with different block size and test the solution, regardless
> how the scenario is realistic.
>
> BTW, RDMA based solution works fine with large blocks.
>
> Hi Sasha,
>
> Could you file an issue in GitHub for this?
>
> https://github.com/spdk/spdk/issues
>
> Thanks,
>
> -Jim
>
>
>
> Best regards
>
> Sasha
>
>
> >
> > Regards,
> > Andrey
> >
> >
> >> Backtrace:
> >>
> >> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
> >> (tqpair=0xe19040) at nvme_tcp.c:447
> >> 447 pdu_length = pdu->hdr.common.plen -
> >> pdu->writev_offset;
> >> Missing separate debuginfos, use: debuginfo-install
> >> glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64
> >> libgcc-4.8.5-28.el7_5.1.x86_64
> >> libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64
> >> libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64
> >> libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
> >> openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
> >> (gdb) bt
> >> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
> >> (tqpair=0xe19040) at nvme_tcp.c:447
> >> #1 0x000000000042322c in nvme_tcp_qpair_process_completions
> >> (qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
> >> #2 0x000000000041e01a in nvme_transport_qpair_process_completions
> >> (qpair=0xe19040, max_completions=0) at nvme_transport.c:224
> >> #3 0x000000000041a78d in spdk_nvme_qpair_process_completions
> >> (qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
> >> #4 0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock
> >> (qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
> >> #5 0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040,
> >> status=0x7ffe66d56dd0) at nvme.c:142
> >> #6 0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60,
> >> offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
> >> #7 0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60,
> >> offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
> >> #8 0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60,
> >> offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
> >> #9 0x000000000041d91f in nvme_transport_ctrlr_get_reg_4
> >> (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
> >> #10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60,
> >> cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
> >> #11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair
> >> (ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at
> >> nvme_ctrlr.c:251
> >> #12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at
> >> perf.c:826
> >> #13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
> >> #14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at
> >> perf.c:1709
> >>
> >> Environment:
> >>
> >> OS: CentOS Linux release 7.5.1804 (Core)
> >>
> >> Kernel: 3.10.0-862.el7.x86_64
> >>
> >> Target output"
> >>
> >> Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
> >> [ DPDK EAL parameters: nvmf --no-shconf -c 0x3
> >> --base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
> >> EAL: Detected 24 lcore(s)
> >> EAL: Detected 2 NUMA nodes
> >> EAL: No free hugepages reported in hugepages-1048576kB
> >> EAL: Probing VFIO support...
> >> app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
> >> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
> >> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> >> EAL: PCI device 0000:81:00.0 on NUMA socket 1
> >> EAL: probe driver: 8086:2700 spdk_nvme
> >> conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options
> >> detected for the NVMe-oF target.
> >> The following options are no longer controlled by the target
> >> and should be set in the transport on a per-transport basis:
> >> MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize,
> >> IOUnitSize
> >> This can be accomplished by setting the options through the
> >> create_nvmf_transport RPC.
> >> You may also continue to configure these options in the conf file under
> >> each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP
> >> Transport Init ***
> >> tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening
> >> on 1.1.75.1 port 1023 ***
> >>
> >> Thanks
> >>
> >> Sasha
> >>
> >> _______________________________________________
> >> SPDK mailing list
> >> SPDK(a)lists.01.org
> >> https://lists.01.org/mailman/listinfo/spdk
> >>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
@ 2018-12-02 7:11 Sasha Kotchubievsky
0 siblings, 0 replies; 11+ messages in thread
From: Sasha Kotchubievsky @ 2018-12-02 7:11 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5942 bytes --]
Hi,
The patch solves the issue.
Thanks
Sasha
On 11/30/2018 6:41 AM, Yang, Ziye wrote:
> Hi Sasha,
>
> You may try this patch: https://review.gerrithub.io/#/c/spdk/spdk/+/435577/
>
>
>
>
> Best Regards
> Ziye Yang
>
>
> -----Original Message-----
> From: Yang, Ziye
> Sent: Friday, November 30, 2018 11:53 AM
> To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
> Subject: RE: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
>
> It is a definitely a bug, which is not related with the I/O size, but related with the multiple threads.
>
>
>
>
> Best Regards
> Ziye Yang
>
>
> -----Original Message-----
> From: Yang, Ziye
> Sent: Friday, November 30, 2018 9:30 AM
> To: spdk(a)lists.01.org
> Subject: RE: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
>
> Hi Sasha,
>
> Will look at this issue.
>
>
>
>
> Best Regards
> Ziye Yang
>
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sasha Kotchubievsky
> Sent: Thursday, November 29, 2018 8:45 PM
> To: spdk(a)lists.01.org
> Subject: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
>
> Hi,
>
> I'm testing NVME-OF TCP. I can run "perf" applications only on single core. If I try to use 2 or more core, it crashes immediately.
>
> I tried different block sizes, number of cores in the target, read/write operations.
>
> Do I miss something in configuration, or that's just a bug?
>
> Details:
>
> Version: "7a39a68"
>
> Command lines:
>
> Target: sudo ./app/nvmf_tgt/nvmf_tgt -c ./nvmf.conf -m 0x3
>
> Client: sudo examples/nvme/perf/perf -q 5 -o 1036288 -w randwrite -t 60 -c 0x1100 -D -r 'trtype:TCP adrfam:IPv4 traddr:1.1.75.1
> trsvcid:1023 nqn.2016-06.io.spdk.r-dcs75:rd0'
>
> Backtrace:
>
> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
> (tqpair=0xe19040) at nvme_tcp.c:447
> 447 pdu_length = pdu->hdr.common.plen -
> pdu->writev_offset;
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64
> libgcc-4.8.5-28.el7_5.1.x86_64
> libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64
> libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64
> libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
> openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
> (gdb) bt
> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
> (tqpair=0xe19040) at nvme_tcp.c:447
> #1 0x000000000042322c in nvme_tcp_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
> #2 0x000000000041e01a in nvme_transport_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_transport.c:224
> #3 0x000000000041a78d in spdk_nvme_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
> #4 0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock
> (qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
> #5 0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040,
> status=0x7ffe66d56dd0) at nvme.c:142
> #6 0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60, offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
> #7 0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
> #8 0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
> #9 0x000000000041d91f in nvme_transport_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
> #10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60,
> cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
> #11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair (ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at nvme_ctrlr.c:251
> #12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at perf.c:826
> #13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
> #14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at perf.c:1709
>
> Environment:
>
> OS: CentOS Linux release 7.5.1804 (Core)
>
> Kernel: 3.10.0-862.el7.x86_64
>
> Target output"
>
> Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
> [ DPDK EAL parameters: nvmf --no-shconf -c 0x3
> --base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
> EAL: Detected 24 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> EAL: PCI device 0000:81:00.0 on NUMA socket 1
> EAL: probe driver: 8086:2700 spdk_nvme
> conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options detected for the NVMe-oF target.
> The following options are no longer controlled by the target and should be set in the transport on a per-transport basis:
> MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize, IOUnitSize This can be accomplished by setting the options through the create_nvmf_transport RPC.
> You may also continue to configure these options in the conf file under each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
> tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 1.1.75.1 port 1023 ***
>
> Thanks
>
> Sasha
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
@ 2018-12-03 1:29 Yang, Ziye
0 siblings, 0 replies; 11+ messages in thread
From: Yang, Ziye @ 2018-12-03 1:29 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 6403 bytes --]
Hi Sasha,
Glad to hear that.
Best Regards
Ziye Yang
-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sasha Kotchubievsky
Sent: Sunday, December 2, 2018 3:12 PM
To: spdk(a)lists.01.org
Subject: Re: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
Hi,
The patch solves the issue.
Thanks
Sasha
On 11/30/2018 6:41 AM, Yang, Ziye wrote:
> Hi Sasha,
>
> You may try this patch: https://review.gerrithub.io/#/c/spdk/spdk/+/435577/
>
>
>
>
> Best Regards
> Ziye Yang
>
>
> -----Original Message-----
> From: Yang, Ziye
> Sent: Friday, November 30, 2018 11:53 AM
> To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
> Subject: RE: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
>
> It is a definitely a bug, which is not related with the I/O size, but related with the multiple threads.
>
>
>
>
> Best Regards
> Ziye Yang
>
>
> -----Original Message-----
> From: Yang, Ziye
> Sent: Friday, November 30, 2018 9:30 AM
> To: spdk(a)lists.01.org
> Subject: RE: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
>
> Hi Sasha,
>
> Will look at this issue.
>
>
>
>
> Best Regards
> Ziye Yang
>
>
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sasha Kotchubievsky
> Sent: Thursday, November 29, 2018 8:45 PM
> To: spdk(a)lists.01.org
> Subject: [SPDK] [NVME-OF TCP] Crash in perf running on more than single core
>
> Hi,
>
> I'm testing NVME-OF TCP. I can run "perf" applications only on single core. If I try to use 2 or more core, it crashes immediately.
>
> I tried different block sizes, number of cores in the target, read/write operations.
>
> Do I miss something in configuration, or that's just a bug?
>
> Details:
>
> Version: "7a39a68"
>
> Command lines:
>
> Target: sudo ./app/nvmf_tgt/nvmf_tgt -c ./nvmf.conf -m 0x3
>
> Client: sudo examples/nvme/perf/perf -q 5 -o 1036288 -w randwrite -t 60 -c 0x1100 -D -r 'trtype:TCP adrfam:IPv4 traddr:1.1.75.1
> trsvcid:1023 nqn.2016-06.io.spdk.r-dcs75:rd0'
>
> Backtrace:
>
> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
> (tqpair=0xe19040) at nvme_tcp.c:447
> 447 pdu_length = pdu->hdr.common.plen -
> pdu->writev_offset;
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64
> libgcc-4.8.5-28.el7_5.1.x86_64
> libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64
> libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64
> libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64
> openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
> (gdb) bt
> #0 0x0000000000420515 in nvme_tcp_qpair_process_send_queue
> (tqpair=0xe19040) at nvme_tcp.c:447
> #1 0x000000000042322c in nvme_tcp_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
> #2 0x000000000041e01a in nvme_transport_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_transport.c:224
> #3 0x000000000041a78d in spdk_nvme_qpair_process_completions
> (qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
> #4 0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock
> (qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
> #5 0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040,
> status=0x7ffe66d56dd0) at nvme.c:142
> #6 0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60, offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
> #7 0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
> #8 0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
> #9 0x000000000041d91f in nvme_transport_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
> #10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60,
> cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
> #11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair (ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at nvme_ctrlr.c:251
> #12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at perf.c:826
> #13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
> #14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at perf.c:1709
>
> Environment:
>
> OS: CentOS Linux release 7.5.1804 (Core)
>
> Kernel: 3.10.0-862.el7.x86_64
>
> Target output"
>
> Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
> [ DPDK EAL parameters: nvmf --no-shconf -c 0x3
> --base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
> EAL: Detected 24 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
> reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> EAL: PCI device 0000:81:00.0 on NUMA socket 1
> EAL: probe driver: 8086:2700 spdk_nvme
> conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options detected for the NVMe-oF target.
> The following options are no longer controlled by the target and should be set in the transport on a per-transport basis:
> MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize, IOUnitSize This can be accomplished by setting the options through the create_nvmf_transport RPC.
> You may also continue to configure these options in the conf file under each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
> tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 1.1.75.1 port 1023 ***
>
> Thanks
>
> Sasha
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2018-12-03 1:29 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-11-29 14:37 [SPDK] [NVME-OF TCP] Crash in perf running on more than single core Harris, James R
-- strict thread matches above, loose matches on Subject: below --
2018-12-03 1:29 Yang, Ziye
2018-12-02 7:11 Sasha Kotchubievsky
2018-11-30 10:59 Sasha Kotchubievsky
2018-11-30 6:32 Sasha Kotchubievsky
2018-11-30 4:41 Yang, Ziye
2018-11-30 3:53 Yang, Ziye
2018-11-30 1:29 Yang, Ziye
2018-11-29 14:31 Sasha Kotchubievsky
2018-11-29 13:45 Andrey Kuzmin
2018-11-29 12:44 Sasha Kotchubievsky
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox