* Re: [SPDK] Program core dumps when I/O is issued in thread.
@ 2017-11-21 20:14 Sreeni Busam
0 siblings, 0 replies; 5+ messages in thread
From: Sreeni Busam @ 2017-11-21 20:14 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 8418 bytes --]
Did any of you get a chance to look it this? I have modified the code for hello_world.c to issue I/O on device in a thread. The program crashes in qpair allocation.
Please let me know your thoughts on why it fails. Thanks for your help.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Monday, November 20, 2017 12:50 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Program core dumps when I/O is issued in thread.
Hi Paul/Jim,
I am trying to create some threads and issue I/O write and read by creating qpair in each of the thread. If the I/O commands are run in main thread then the program works fine, but if I create a thread, the programs crashes and fails in qpair allocation spdk_nvme_ctrlr_alloc_io_qpair().
I have attached the complete program. Here is the trace where crash happens.
0x0000000000412844 in nvme_pcie_qpair_process_completions (
qpair=0x103ffe9310, max_completions=127) at nvme_pcie.c:1893
#1 0x000000000040da10 in nvme_transport_qpair_process_completions (
qpair=0x103ffe9310, max_completions=0) at nvme_transport.c:201
#2 0x000000000040b4d4 in spdk_nvme_qpair_process_completions (
qpair=0x103ffe9310, max_completions=0) at nvme_qpair.c:368
#3 0x000000000041166f in _nvme_pcie_ctrlr_create_io_qpair (
ctrlr=0x103ffe93c0, qpair=0x103ff8ec90, qid=1) at nvme_pcie.c:1360
#4 0x00000000004118db in nvme_pcie_ctrlr_create_io_qpair (ctrlr=0x103ffe93c0,
qid=1, opts=0x7f012053ecf0) at nvme_pcie.c:1426
#5 0x000000000040d7d7 in nvme_transport_ctrlr_create_io_qpair (
ctrlr=0x103ffe93c0, qid=1, opts=0x7f012053ecf0) at nvme_transport.c:153
#6 0x000000000040512b in spdk_nvme_ctrlr_alloc_io_qpair (ctrlr=0x103ffe93c0,
user_opts=0x0, opts_size=0) at nvme_ctrlr.c:281
#7 0x000000000040441c in test_io_func1 (arg=0x7fff327fba48) at iostat.c:288
#8 0x00007f01211126ba in start_thread (arg=0x7f012053f700)
at pthread_create.c:333
#9 0x00007f0120e483dd in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S
please take a look and let know any pointers to debug the issue.
I changed the ns_entry as shown here.
struct ns_entry {
struct spdk_nvme_ctrlr *ctrlr;
struct spdk_nvme_ns *ns;
int nsid;
char ctrlr_name[MAX_CTRLR_NAME];
struct ns_entry *next;
struct spdk_nvme_qpair *qpair[5];
int qpair_allocated;
}
Thanks,
Sreeni
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, November 17, 2017 11:02 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Cool, so working as you would expect now?
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Friday, November 17, 2017 11:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Hi Jim,
Thanks for taking a look at the problem. It is a logical bug. I fixed it.
The memory for the qpair was allocated outside the ns_entry loop.
I have two SSD devices and the memory was not allocated for the qpair for second ns_entry.
Here is the problem.
ns_entry = g_namespaces;
ns_entry->qpair_2 = spdk_nvme_ctrlr_alloc_io_qpair(ns_entry->ctrlr, NULL, 0);
if (ns_entry->qpair_2 == NULL) {
printf("The qpair allocation failed.\n");
exit (0);
}
while (ns_entry != NULL) {
cnt = 1;
while (cnt) {
// Fails to be successfully submit
rc = stellus_spdk_nvme_ns_cmd_write(ns_entry, ns_entry->qpair_2, sequence->buf,
0, /* LBA start */
1, /* number of LBAs */
io_complete, sequence, 0)
….
Sreeni
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, November 17, 2017 8:44 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Hi Sreeni,
Can you step through your second call to spdk_nvme_ctrlr_alloc_io_qpair? The callstack clearly shows that qpair=0x0 was passed into stellus_spdk_nvme_ns_cmd_write() at frame #3. So I think we should back up and figure out why no I/O qpair was allocated (or maybe it was allocated but not saved in a structure or something).
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Sreeni (Sreenivasa) Busam (Stellus)" <s.busam(a)stellus.com<mailto:s.busam(a)stellus.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, November 16, 2017 at 4:57 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
I have been trying to test the number of commands that can be given to the device at a time. I verified that a maximum of 254 commands could be issued for a qpair. So I created a 2nd qpair for ns_entry and issued the I/O commands, it was failing in the first command itself. Is it invalid to create 2 qpair for the same ns_entry and send command to device? The qpair is successfully created, but I could not submit command.
I modified the hello_world program to test this and attached the related code.
Please take a look and let me know what is the problem.
0x000000000040bae2 in nvme_allocate_request (qpair=0x0,
payload=0x7fffa4726ba0, payload_size=512, cb_fn=0x4041a4 <write_complete>,
cb_arg=0x7b4270) at nvme.c:85
#1 0x000000000040996c in _nvme_ns_cmd_rw (ns=0x100ff8ee40, qpair=0x0,
payload=0x7fffa4726ba0, payload_offset=0, md_offset=0, lba=0, lba_count=1,
cb_fn=0x4041a4 <write_complete>, cb_arg=0x7b4270, opc=1, io_flags=0,
apptag_mask=0, apptag=0, check_sgl=true) at nvme_ns_cmd.c:440
#2 0x0000000000409fea in spdk_nvme_ns_cmd_write (ns=0x100ff8ee40, qpair=0x0,
buffer=0x10000f7000, lba=0, lba_count=1, cb_fn=0x4041a4 <write_complete>,
cb_arg=0x7b4270, io_flags=0) at nvme_ns_cmd.c:649
#3 0x000000000040439d in stellus_spdk_nvme_ns_cmd_write (ns_entry=0x7b13c0,
qpair=0x0, buffer=0x10000f7000, lba=0, lba_count=1,
cb_fn=0x40420d <io_complete>, cb_arg=0x7b4270, ioflags=0) at iostat.c:233
#4 0x00000000004046b8 in test_io_func1 () at iostat.c:342
#5 0x0000000000404a94 in main (argc=1, argv=0x7fffa4726db8) at iostat.c:503
(gdb) f 3
#3 0x000000000040439d in stellus_spdk_nvme_ns_cmd_write (ns_entry=0x7b13c0,
qpair=0x0, buffer=0x10000f7000, lba=0, lba_count=1,
cb_fn=0x40420d <io_complete>, cb_arg=0x7b4270, ioflags=0) at iostat.c:233
233 rc = spdk_nvme_ns_cmd_write(ns_entry->ns, qpair, buffer,
(gdb) p qpair
$1 = (struct spdk_nvme_qpair *) 0x0
If any of you get time, please look at it. Thank you for your suggestion.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Thursday, November 16, 2017 11:26 AM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] Regarding NVMe driver command queue depth.
Hi Paul,
I was reading about the driver from SPDK site, and interested in understanding the queue depth for a device.
“The specification allows for thousands, but most devices support between 32 and 128. The specification makes no guarantees about the performance available from each queue pair, but in practice the full performance of a device is almost always achievable using just one queue pair. For example, if a device claims to be capable of 450,000 I/O per second at queue depth 128, in practice it does not matter if the driver is using 4 queue pairs each with queue depth 32, or a single queue pair with queue depth 128”
When queue depth is mentioned for device, is it the number of commands that can be issued from application to controller, and outstanding at any time?
Is there NVMe driver API to set the queue depth? Is my understanding correct if I think that the size of queue is at firmware level?
Please give some detail about the parameter.
Thanks,
Sreeni
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 25367 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [SPDK] Program core dumps when I/O is issued in thread.
@ 2017-11-21 21:42 Sreeni Busam
0 siblings, 0 replies; 5+ messages in thread
From: Sreeni Busam @ 2017-11-21 21:42 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 9474 bytes --]
Jim,
Great, Thanks a lot for your suggestion.
I fixed the code by doing pthread_join() and it works now.
Sreeni
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Tuesday, November 21, 2017 12:31 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Program core dumps when I/O is issued in thread.
Hi Sreeni,
I think you need a pthread_join() to make sure the main thread doesn’t start cleaning up immediately after calling pthread_create().
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Sreeni (Sreenivasa) Busam (Stellus)" <s.busam(a)stellus.com<mailto:s.busam(a)stellus.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, November 21, 2017 at 1:14 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Program core dumps when I/O is issued in thread.
Did any of you get a chance to look it this? I have modified the code for hello_world.c to issue I/O on device in a thread. The program crashes in qpair allocation.
Please let me know your thoughts on why it fails. Thanks for your help.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Monday, November 20, 2017 12:50 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Program core dumps when I/O is issued in thread.
Hi Paul/Jim,
I am trying to create some threads and issue I/O write and read by creating qpair in each of the thread. If the I/O commands are run in main thread then the program works fine, but if I create a thread, the programs crashes and fails in qpair allocation spdk_nvme_ctrlr_alloc_io_qpair().
I have attached the complete program. Here is the trace where crash happens.
0x0000000000412844 in nvme_pcie_qpair_process_completions (
qpair=0x103ffe9310, max_completions=127) at nvme_pcie.c:1893
#1 0x000000000040da10 in nvme_transport_qpair_process_completions (
qpair=0x103ffe9310, max_completions=0) at nvme_transport.c:201
#2 0x000000000040b4d4 in spdk_nvme_qpair_process_completions (
qpair=0x103ffe9310, max_completions=0) at nvme_qpair.c:368
#3 0x000000000041166f in _nvme_pcie_ctrlr_create_io_qpair (
ctrlr=0x103ffe93c0, qpair=0x103ff8ec90, qid=1) at nvme_pcie.c:1360
#4 0x00000000004118db in nvme_pcie_ctrlr_create_io_qpair (ctrlr=0x103ffe93c0,
qid=1, opts=0x7f012053ecf0) at nvme_pcie.c:1426
#5 0x000000000040d7d7 in nvme_transport_ctrlr_create_io_qpair (
ctrlr=0x103ffe93c0, qid=1, opts=0x7f012053ecf0) at nvme_transport.c:153
#6 0x000000000040512b in spdk_nvme_ctrlr_alloc_io_qpair (ctrlr=0x103ffe93c0,
user_opts=0x0, opts_size=0) at nvme_ctrlr.c:281
#7 0x000000000040441c in test_io_func1 (arg=0x7fff327fba48) at iostat.c:288
#8 0x00007f01211126ba in start_thread (arg=0x7f012053f700)
at pthread_create.c:333
#9 0x00007f0120e483dd in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S
please take a look and let know any pointers to debug the issue.
I changed the ns_entry as shown here.
struct ns_entry {
struct spdk_nvme_ctrlr *ctrlr;
struct spdk_nvme_ns *ns;
int nsid;
char ctrlr_name[MAX_CTRLR_NAME];
struct ns_entry *next;
struct spdk_nvme_qpair *qpair[5];
int qpair_allocated;
}
Thanks,
Sreeni
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, November 17, 2017 11:02 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Cool, so working as you would expect now?
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Friday, November 17, 2017 11:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Hi Jim,
Thanks for taking a look at the problem. It is a logical bug. I fixed it.
The memory for the qpair was allocated outside the ns_entry loop.
I have two SSD devices and the memory was not allocated for the qpair for second ns_entry.
Here is the problem.
ns_entry = g_namespaces;
ns_entry->qpair_2 = spdk_nvme_ctrlr_alloc_io_qpair(ns_entry->ctrlr, NULL, 0);
if (ns_entry->qpair_2 == NULL) {
printf("The qpair allocation failed.\n");
exit (0);
}
while (ns_entry != NULL) {
cnt = 1;
while (cnt) {
// Fails to be successfully submit
rc = stellus_spdk_nvme_ns_cmd_write(ns_entry, ns_entry->qpair_2, sequence->buf,
0, /* LBA start */
1, /* number of LBAs */
io_complete, sequence, 0)
….
Sreeni
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, November 17, 2017 8:44 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Hi Sreeni,
Can you step through your second call to spdk_nvme_ctrlr_alloc_io_qpair? The callstack clearly shows that qpair=0x0 was passed into stellus_spdk_nvme_ns_cmd_write() at frame #3. So I think we should back up and figure out why no I/O qpair was allocated (or maybe it was allocated but not saved in a structure or something).
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Sreeni (Sreenivasa) Busam (Stellus)" <s.busam(a)stellus.com<mailto:s.busam(a)stellus.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, November 16, 2017 at 4:57 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
I have been trying to test the number of commands that can be given to the device at a time. I verified that a maximum of 254 commands could be issued for a qpair. So I created a 2nd qpair for ns_entry and issued the I/O commands, it was failing in the first command itself. Is it invalid to create 2 qpair for the same ns_entry and send command to device? The qpair is successfully created, but I could not submit command.
I modified the hello_world program to test this and attached the related code.
Please take a look and let me know what is the problem.
0x000000000040bae2 in nvme_allocate_request (qpair=0x0,
payload=0x7fffa4726ba0, payload_size=512, cb_fn=0x4041a4 <write_complete>,
cb_arg=0x7b4270) at nvme.c:85
#1 0x000000000040996c in _nvme_ns_cmd_rw (ns=0x100ff8ee40, qpair=0x0,
payload=0x7fffa4726ba0, payload_offset=0, md_offset=0, lba=0, lba_count=1,
cb_fn=0x4041a4 <write_complete>, cb_arg=0x7b4270, opc=1, io_flags=0,
apptag_mask=0, apptag=0, check_sgl=true) at nvme_ns_cmd.c:440
#2 0x0000000000409fea in spdk_nvme_ns_cmd_write (ns=0x100ff8ee40, qpair=0x0,
buffer=0x10000f7000, lba=0, lba_count=1, cb_fn=0x4041a4 <write_complete>,
cb_arg=0x7b4270, io_flags=0) at nvme_ns_cmd.c:649
#3 0x000000000040439d in stellus_spdk_nvme_ns_cmd_write (ns_entry=0x7b13c0,
qpair=0x0, buffer=0x10000f7000, lba=0, lba_count=1,
cb_fn=0x40420d <io_complete>, cb_arg=0x7b4270, ioflags=0) at iostat.c:233
#4 0x00000000004046b8 in test_io_func1 () at iostat.c:342
#5 0x0000000000404a94 in main (argc=1, argv=0x7fffa4726db8) at iostat.c:503
(gdb) f 3
#3 0x000000000040439d in stellus_spdk_nvme_ns_cmd_write (ns_entry=0x7b13c0,
qpair=0x0, buffer=0x10000f7000, lba=0, lba_count=1,
cb_fn=0x40420d <io_complete>, cb_arg=0x7b4270, ioflags=0) at iostat.c:233
233 rc = spdk_nvme_ns_cmd_write(ns_entry->ns, qpair, buffer,
(gdb) p qpair
$1 = (struct spdk_nvme_qpair *) 0x0
If any of you get time, please look at it. Thank you for your suggestion.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Thursday, November 16, 2017 11:26 AM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] Regarding NVMe driver command queue depth.
Hi Paul,
I was reading about the driver from SPDK site, and interested in understanding the queue depth for a device.
“The specification allows for thousands, but most devices support between 32 and 128. The specification makes no guarantees about the performance available from each queue pair, but in practice the full performance of a device is almost always achievable using just one queue pair. For example, if a device claims to be capable of 450,000 I/O per second at queue depth 128, in practice it does not matter if the driver is using 4 queue pairs each with queue depth 32, or a single queue pair with queue depth 128”
When queue depth is mentioned for device, is it the number of commands that can be issued from application to controller, and outstanding at any time?
Is there NVMe driver API to set the queue depth? Is my understanding correct if I think that the size of queue is at firmware level?
Please give some detail about the parameter.
Thanks,
Sreeni
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 30098 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [SPDK] Program core dumps when I/O is issued in thread.
@ 2017-11-21 21:22 Sreeni Busam
0 siblings, 0 replies; 5+ messages in thread
From: Sreeni Busam @ 2017-11-21 21:22 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 9425 bytes --]
Ok. Thanks Jim. That is the problem.
Let me test it and confirm you.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Tuesday, November 21, 2017 12:31 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Program core dumps when I/O is issued in thread.
Hi Sreeni,
I think you need a pthread_join() to make sure the main thread doesn’t start cleaning up immediately after calling pthread_create().
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Sreeni (Sreenivasa) Busam (Stellus)" <s.busam(a)stellus.com<mailto:s.busam(a)stellus.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Tuesday, November 21, 2017 at 1:14 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Program core dumps when I/O is issued in thread.
Did any of you get a chance to look it this? I have modified the code for hello_world.c to issue I/O on device in a thread. The program crashes in qpair allocation.
Please let me know your thoughts on why it fails. Thanks for your help.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Monday, November 20, 2017 12:50 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Program core dumps when I/O is issued in thread.
Hi Paul/Jim,
I am trying to create some threads and issue I/O write and read by creating qpair in each of the thread. If the I/O commands are run in main thread then the program works fine, but if I create a thread, the programs crashes and fails in qpair allocation spdk_nvme_ctrlr_alloc_io_qpair().
I have attached the complete program. Here is the trace where crash happens.
0x0000000000412844 in nvme_pcie_qpair_process_completions (
qpair=0x103ffe9310, max_completions=127) at nvme_pcie.c:1893
#1 0x000000000040da10 in nvme_transport_qpair_process_completions (
qpair=0x103ffe9310, max_completions=0) at nvme_transport.c:201
#2 0x000000000040b4d4 in spdk_nvme_qpair_process_completions (
qpair=0x103ffe9310, max_completions=0) at nvme_qpair.c:368
#3 0x000000000041166f in _nvme_pcie_ctrlr_create_io_qpair (
ctrlr=0x103ffe93c0, qpair=0x103ff8ec90, qid=1) at nvme_pcie.c:1360
#4 0x00000000004118db in nvme_pcie_ctrlr_create_io_qpair (ctrlr=0x103ffe93c0,
qid=1, opts=0x7f012053ecf0) at nvme_pcie.c:1426
#5 0x000000000040d7d7 in nvme_transport_ctrlr_create_io_qpair (
ctrlr=0x103ffe93c0, qid=1, opts=0x7f012053ecf0) at nvme_transport.c:153
#6 0x000000000040512b in spdk_nvme_ctrlr_alloc_io_qpair (ctrlr=0x103ffe93c0,
user_opts=0x0, opts_size=0) at nvme_ctrlr.c:281
#7 0x000000000040441c in test_io_func1 (arg=0x7fff327fba48) at iostat.c:288
#8 0x00007f01211126ba in start_thread (arg=0x7f012053f700)
at pthread_create.c:333
#9 0x00007f0120e483dd in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S
please take a look and let know any pointers to debug the issue.
I changed the ns_entry as shown here.
struct ns_entry {
struct spdk_nvme_ctrlr *ctrlr;
struct spdk_nvme_ns *ns;
int nsid;
char ctrlr_name[MAX_CTRLR_NAME];
struct ns_entry *next;
struct spdk_nvme_qpair *qpair[5];
int qpair_allocated;
}
Thanks,
Sreeni
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, November 17, 2017 11:02 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Cool, so working as you would expect now?
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Friday, November 17, 2017 11:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Hi Jim,
Thanks for taking a look at the problem. It is a logical bug. I fixed it.
The memory for the qpair was allocated outside the ns_entry loop.
I have two SSD devices and the memory was not allocated for the qpair for second ns_entry.
Here is the problem.
ns_entry = g_namespaces;
ns_entry->qpair_2 = spdk_nvme_ctrlr_alloc_io_qpair(ns_entry->ctrlr, NULL, 0);
if (ns_entry->qpair_2 == NULL) {
printf("The qpair allocation failed.\n");
exit (0);
}
while (ns_entry != NULL) {
cnt = 1;
while (cnt) {
// Fails to be successfully submit
rc = stellus_spdk_nvme_ns_cmd_write(ns_entry, ns_entry->qpair_2, sequence->buf,
0, /* LBA start */
1, /* number of LBAs */
io_complete, sequence, 0)
….
Sreeni
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, November 17, 2017 8:44 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Hi Sreeni,
Can you step through your second call to spdk_nvme_ctrlr_alloc_io_qpair? The callstack clearly shows that qpair=0x0 was passed into stellus_spdk_nvme_ns_cmd_write() at frame #3. So I think we should back up and figure out why no I/O qpair was allocated (or maybe it was allocated but not saved in a structure or something).
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Sreeni (Sreenivasa) Busam (Stellus)" <s.busam(a)stellus.com<mailto:s.busam(a)stellus.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, November 16, 2017 at 4:57 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
I have been trying to test the number of commands that can be given to the device at a time. I verified that a maximum of 254 commands could be issued for a qpair. So I created a 2nd qpair for ns_entry and issued the I/O commands, it was failing in the first command itself. Is it invalid to create 2 qpair for the same ns_entry and send command to device? The qpair is successfully created, but I could not submit command.
I modified the hello_world program to test this and attached the related code.
Please take a look and let me know what is the problem.
0x000000000040bae2 in nvme_allocate_request (qpair=0x0,
payload=0x7fffa4726ba0, payload_size=512, cb_fn=0x4041a4 <write_complete>,
cb_arg=0x7b4270) at nvme.c:85
#1 0x000000000040996c in _nvme_ns_cmd_rw (ns=0x100ff8ee40, qpair=0x0,
payload=0x7fffa4726ba0, payload_offset=0, md_offset=0, lba=0, lba_count=1,
cb_fn=0x4041a4 <write_complete>, cb_arg=0x7b4270, opc=1, io_flags=0,
apptag_mask=0, apptag=0, check_sgl=true) at nvme_ns_cmd.c:440
#2 0x0000000000409fea in spdk_nvme_ns_cmd_write (ns=0x100ff8ee40, qpair=0x0,
buffer=0x10000f7000, lba=0, lba_count=1, cb_fn=0x4041a4 <write_complete>,
cb_arg=0x7b4270, io_flags=0) at nvme_ns_cmd.c:649
#3 0x000000000040439d in stellus_spdk_nvme_ns_cmd_write (ns_entry=0x7b13c0,
qpair=0x0, buffer=0x10000f7000, lba=0, lba_count=1,
cb_fn=0x40420d <io_complete>, cb_arg=0x7b4270, ioflags=0) at iostat.c:233
#4 0x00000000004046b8 in test_io_func1 () at iostat.c:342
#5 0x0000000000404a94 in main (argc=1, argv=0x7fffa4726db8) at iostat.c:503
(gdb) f 3
#3 0x000000000040439d in stellus_spdk_nvme_ns_cmd_write (ns_entry=0x7b13c0,
qpair=0x0, buffer=0x10000f7000, lba=0, lba_count=1,
cb_fn=0x40420d <io_complete>, cb_arg=0x7b4270, ioflags=0) at iostat.c:233
233 rc = spdk_nvme_ns_cmd_write(ns_entry->ns, qpair, buffer,
(gdb) p qpair
$1 = (struct spdk_nvme_qpair *) 0x0
If any of you get time, please look at it. Thank you for your suggestion.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Thursday, November 16, 2017 11:26 AM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] Regarding NVMe driver command queue depth.
Hi Paul,
I was reading about the driver from SPDK site, and interested in understanding the queue depth for a device.
“The specification allows for thousands, but most devices support between 32 and 128. The specification makes no guarantees about the performance available from each queue pair, but in practice the full performance of a device is almost always achievable using just one queue pair. For example, if a device claims to be capable of 450,000 I/O per second at queue depth 128, in practice it does not matter if the driver is using 4 queue pairs each with queue depth 32, or a single queue pair with queue depth 128”
When queue depth is mentioned for device, is it the number of commands that can be issued from application to controller, and outstanding at any time?
Is there NVMe driver API to set the queue depth? Is my understanding correct if I think that the size of queue is at firmware level?
Please give some detail about the parameter.
Thanks,
Sreeni
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 29753 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [SPDK] Program core dumps when I/O is issued in thread.
@ 2017-11-21 20:31 Harris, James R
0 siblings, 0 replies; 5+ messages in thread
From: Harris, James R @ 2017-11-21 20:31 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 8946 bytes --]
Hi Sreeni,
I think you need a pthread_join() to make sure the main thread doesn’t start cleaning up immediately after calling pthread_create().
-Jim
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of "Sreeni (Sreenivasa) Busam (Stellus)" <s.busam(a)stellus.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Tuesday, November 21, 2017 at 1:14 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Program core dumps when I/O is issued in thread.
Did any of you get a chance to look it this? I have modified the code for hello_world.c to issue I/O on device in a thread. The program crashes in qpair allocation.
Please let me know your thoughts on why it fails. Thanks for your help.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Monday, November 20, 2017 12:50 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Program core dumps when I/O is issued in thread.
Hi Paul/Jim,
I am trying to create some threads and issue I/O write and read by creating qpair in each of the thread. If the I/O commands are run in main thread then the program works fine, but if I create a thread, the programs crashes and fails in qpair allocation spdk_nvme_ctrlr_alloc_io_qpair().
I have attached the complete program. Here is the trace where crash happens.
0x0000000000412844 in nvme_pcie_qpair_process_completions (
qpair=0x103ffe9310, max_completions=127) at nvme_pcie.c:1893
#1 0x000000000040da10 in nvme_transport_qpair_process_completions (
qpair=0x103ffe9310, max_completions=0) at nvme_transport.c:201
#2 0x000000000040b4d4 in spdk_nvme_qpair_process_completions (
qpair=0x103ffe9310, max_completions=0) at nvme_qpair.c:368
#3 0x000000000041166f in _nvme_pcie_ctrlr_create_io_qpair (
ctrlr=0x103ffe93c0, qpair=0x103ff8ec90, qid=1) at nvme_pcie.c:1360
#4 0x00000000004118db in nvme_pcie_ctrlr_create_io_qpair (ctrlr=0x103ffe93c0,
qid=1, opts=0x7f012053ecf0) at nvme_pcie.c:1426
#5 0x000000000040d7d7 in nvme_transport_ctrlr_create_io_qpair (
ctrlr=0x103ffe93c0, qid=1, opts=0x7f012053ecf0) at nvme_transport.c:153
#6 0x000000000040512b in spdk_nvme_ctrlr_alloc_io_qpair (ctrlr=0x103ffe93c0,
user_opts=0x0, opts_size=0) at nvme_ctrlr.c:281
#7 0x000000000040441c in test_io_func1 (arg=0x7fff327fba48) at iostat.c:288
#8 0x00007f01211126ba in start_thread (arg=0x7f012053f700)
at pthread_create.c:333
#9 0x00007f0120e483dd in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S
please take a look and let know any pointers to debug the issue.
I changed the ns_entry as shown here.
struct ns_entry {
struct spdk_nvme_ctrlr *ctrlr;
struct spdk_nvme_ns *ns;
int nsid;
char ctrlr_name[MAX_CTRLR_NAME];
struct ns_entry *next;
struct spdk_nvme_qpair *qpair[5];
int qpair_allocated;
}
Thanks,
Sreeni
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, November 17, 2017 11:02 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Cool, so working as you would expect now?
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Friday, November 17, 2017 11:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Hi Jim,
Thanks for taking a look at the problem. It is a logical bug. I fixed it.
The memory for the qpair was allocated outside the ns_entry loop.
I have two SSD devices and the memory was not allocated for the qpair for second ns_entry.
Here is the problem.
ns_entry = g_namespaces;
ns_entry->qpair_2 = spdk_nvme_ctrlr_alloc_io_qpair(ns_entry->ctrlr, NULL, 0);
if (ns_entry->qpair_2 == NULL) {
printf("The qpair allocation failed.\n");
exit (0);
}
while (ns_entry != NULL) {
cnt = 1;
while (cnt) {
// Fails to be successfully submit
rc = stellus_spdk_nvme_ns_cmd_write(ns_entry, ns_entry->qpair_2, sequence->buf,
0, /* LBA start */
1, /* number of LBAs */
io_complete, sequence, 0)
….
Sreeni
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, November 17, 2017 8:44 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Hi Sreeni,
Can you step through your second call to spdk_nvme_ctrlr_alloc_io_qpair? The callstack clearly shows that qpair=0x0 was passed into stellus_spdk_nvme_ns_cmd_write() at frame #3. So I think we should back up and figure out why no I/O qpair was allocated (or maybe it was allocated but not saved in a structure or something).
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Sreeni (Sreenivasa) Busam (Stellus)" <s.busam(a)stellus.com<mailto:s.busam(a)stellus.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, November 16, 2017 at 4:57 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
I have been trying to test the number of commands that can be given to the device at a time. I verified that a maximum of 254 commands could be issued for a qpair. So I created a 2nd qpair for ns_entry and issued the I/O commands, it was failing in the first command itself. Is it invalid to create 2 qpair for the same ns_entry and send command to device? The qpair is successfully created, but I could not submit command.
I modified the hello_world program to test this and attached the related code.
Please take a look and let me know what is the problem.
0x000000000040bae2 in nvme_allocate_request (qpair=0x0,
payload=0x7fffa4726ba0, payload_size=512, cb_fn=0x4041a4 <write_complete>,
cb_arg=0x7b4270) at nvme.c:85
#1 0x000000000040996c in _nvme_ns_cmd_rw (ns=0x100ff8ee40, qpair=0x0,
payload=0x7fffa4726ba0, payload_offset=0, md_offset=0, lba=0, lba_count=1,
cb_fn=0x4041a4 <write_complete>, cb_arg=0x7b4270, opc=1, io_flags=0,
apptag_mask=0, apptag=0, check_sgl=true) at nvme_ns_cmd.c:440
#2 0x0000000000409fea in spdk_nvme_ns_cmd_write (ns=0x100ff8ee40, qpair=0x0,
buffer=0x10000f7000, lba=0, lba_count=1, cb_fn=0x4041a4 <write_complete>,
cb_arg=0x7b4270, io_flags=0) at nvme_ns_cmd.c:649
#3 0x000000000040439d in stellus_spdk_nvme_ns_cmd_write (ns_entry=0x7b13c0,
qpair=0x0, buffer=0x10000f7000, lba=0, lba_count=1,
cb_fn=0x40420d <io_complete>, cb_arg=0x7b4270, ioflags=0) at iostat.c:233
#4 0x00000000004046b8 in test_io_func1 () at iostat.c:342
#5 0x0000000000404a94 in main (argc=1, argv=0x7fffa4726db8) at iostat.c:503
(gdb) f 3
#3 0x000000000040439d in stellus_spdk_nvme_ns_cmd_write (ns_entry=0x7b13c0,
qpair=0x0, buffer=0x10000f7000, lba=0, lba_count=1,
cb_fn=0x40420d <io_complete>, cb_arg=0x7b4270, ioflags=0) at iostat.c:233
233 rc = spdk_nvme_ns_cmd_write(ns_entry->ns, qpair, buffer,
(gdb) p qpair
$1 = (struct spdk_nvme_qpair *) 0x0
If any of you get time, please look at it. Thank you for your suggestion.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Thursday, November 16, 2017 11:26 AM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] Regarding NVMe driver command queue depth.
Hi Paul,
I was reading about the driver from SPDK site, and interested in understanding the queue depth for a device.
“The specification allows for thousands, but most devices support between 32 and 128. The specification makes no guarantees about the performance available from each queue pair, but in practice the full performance of a device is almost always achievable using just one queue pair. For example, if a device claims to be capable of 450,000 I/O per second at queue depth 128, in practice it does not matter if the driver is using 4 queue pairs each with queue depth 32, or a single queue pair with queue depth 128”
When queue depth is mentioned for device, is it the number of commands that can be issued from application to controller, and outstanding at any time?
Is there NVMe driver API to set the queue depth? Is my understanding correct if I think that the size of queue is at firmware level?
Please give some detail about the parameter.
Thanks,
Sreeni
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 28438 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [SPDK] Program core dumps when I/O is issued in thread.
@ 2017-11-20 20:49 Sreeni Busam
0 siblings, 0 replies; 5+ messages in thread
From: Sreeni Busam @ 2017-11-20 20:49 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 7874 bytes --]
Hi Paul/Jim,
I am trying to create some threads and issue I/O write and read by creating qpair in each of the thread. If the I/O commands are run in main thread then the program works fine, but if I create a thread, the programs crashes and fails in qpair allocation spdk_nvme_ctrlr_alloc_io_qpair().
I have attached the complete program. Here is the trace where crash happens.
0x0000000000412844 in nvme_pcie_qpair_process_completions (
qpair=0x103ffe9310, max_completions=127) at nvme_pcie.c:1893
#1 0x000000000040da10 in nvme_transport_qpair_process_completions (
qpair=0x103ffe9310, max_completions=0) at nvme_transport.c:201
#2 0x000000000040b4d4 in spdk_nvme_qpair_process_completions (
qpair=0x103ffe9310, max_completions=0) at nvme_qpair.c:368
#3 0x000000000041166f in _nvme_pcie_ctrlr_create_io_qpair (
ctrlr=0x103ffe93c0, qpair=0x103ff8ec90, qid=1) at nvme_pcie.c:1360
#4 0x00000000004118db in nvme_pcie_ctrlr_create_io_qpair (ctrlr=0x103ffe93c0,
qid=1, opts=0x7f012053ecf0) at nvme_pcie.c:1426
#5 0x000000000040d7d7 in nvme_transport_ctrlr_create_io_qpair (
ctrlr=0x103ffe93c0, qid=1, opts=0x7f012053ecf0) at nvme_transport.c:153
#6 0x000000000040512b in spdk_nvme_ctrlr_alloc_io_qpair (ctrlr=0x103ffe93c0,
user_opts=0x0, opts_size=0) at nvme_ctrlr.c:281
#7 0x000000000040441c in test_io_func1 (arg=0x7fff327fba48) at iostat.c:288
#8 0x00007f01211126ba in start_thread (arg=0x7f012053f700)
at pthread_create.c:333
#9 0x00007f0120e483dd in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S
please take a look and let know any pointers to debug the issue.
I changed the ns_entry as shown here.
struct ns_entry {
struct spdk_nvme_ctrlr *ctrlr;
struct spdk_nvme_ns *ns;
int nsid;
char ctrlr_name[MAX_CTRLR_NAME];
struct ns_entry *next;
struct spdk_nvme_qpair *qpair[5];
int qpair_allocated;
}
Thanks,
Sreeni
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Friday, November 17, 2017 11:02 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Cool, so working as you would expect now?
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Friday, November 17, 2017 11:52 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Hi Jim,
Thanks for taking a look at the problem. It is a logical bug. I fixed it.
The memory for the qpair was allocated outside the ns_entry loop.
I have two SSD devices and the memory was not allocated for the qpair for second ns_entry.
Here is the problem.
ns_entry = g_namespaces;
ns_entry->qpair_2 = spdk_nvme_ctrlr_alloc_io_qpair(ns_entry->ctrlr, NULL, 0);
if (ns_entry->qpair_2 == NULL) {
printf("The qpair allocation failed.\n");
exit (0);
}
while (ns_entry != NULL) {
cnt = 1;
while (cnt) {
// Fails to be successfully submit
rc = stellus_spdk_nvme_ns_cmd_write(ns_entry, ns_entry->qpair_2, sequence->buf,
0, /* LBA start */
1, /* number of LBAs */
io_complete, sequence, 0)
….
Sreeni
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Friday, November 17, 2017 8:44 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
Hi Sreeni,
Can you step through your second call to spdk_nvme_ctrlr_alloc_io_qpair? The callstack clearly shows that qpair=0x0 was passed into stellus_spdk_nvme_ns_cmd_write() at frame #3. So I think we should back up and figure out why no I/O qpair was allocated (or maybe it was allocated but not saved in a structure or something).
-Jim
From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Sreeni (Sreenivasa) Busam (Stellus)" <s.busam(a)stellus.com<mailto:s.busam(a)stellus.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, November 16, 2017 at 4:57 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Regarding NVMe driver command queue depth.
I have been trying to test the number of commands that can be given to the device at a time. I verified that a maximum of 254 commands could be issued for a qpair. So I created a 2nd qpair for ns_entry and issued the I/O commands, it was failing in the first command itself. Is it invalid to create 2 qpair for the same ns_entry and send command to device? The qpair is successfully created, but I could not submit command.
I modified the hello_world program to test this and attached the related code.
Please take a look and let me know what is the problem.
0x000000000040bae2 in nvme_allocate_request (qpair=0x0,
payload=0x7fffa4726ba0, payload_size=512, cb_fn=0x4041a4 <write_complete>,
cb_arg=0x7b4270) at nvme.c:85
#1 0x000000000040996c in _nvme_ns_cmd_rw (ns=0x100ff8ee40, qpair=0x0,
payload=0x7fffa4726ba0, payload_offset=0, md_offset=0, lba=0, lba_count=1,
cb_fn=0x4041a4 <write_complete>, cb_arg=0x7b4270, opc=1, io_flags=0,
apptag_mask=0, apptag=0, check_sgl=true) at nvme_ns_cmd.c:440
#2 0x0000000000409fea in spdk_nvme_ns_cmd_write (ns=0x100ff8ee40, qpair=0x0,
buffer=0x10000f7000, lba=0, lba_count=1, cb_fn=0x4041a4 <write_complete>,
cb_arg=0x7b4270, io_flags=0) at nvme_ns_cmd.c:649
#3 0x000000000040439d in stellus_spdk_nvme_ns_cmd_write (ns_entry=0x7b13c0,
qpair=0x0, buffer=0x10000f7000, lba=0, lba_count=1,
cb_fn=0x40420d <io_complete>, cb_arg=0x7b4270, ioflags=0) at iostat.c:233
#4 0x00000000004046b8 in test_io_func1 () at iostat.c:342
#5 0x0000000000404a94 in main (argc=1, argv=0x7fffa4726db8) at iostat.c:503
(gdb) f 3
#3 0x000000000040439d in stellus_spdk_nvme_ns_cmd_write (ns_entry=0x7b13c0,
qpair=0x0, buffer=0x10000f7000, lba=0, lba_count=1,
cb_fn=0x40420d <io_complete>, cb_arg=0x7b4270, ioflags=0) at iostat.c:233
233 rc = spdk_nvme_ns_cmd_write(ns_entry->ns, qpair, buffer,
(gdb) p qpair
$1 = (struct spdk_nvme_qpair *) 0x0
If any of you get time, please look at it. Thank you for your suggestion.
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Thursday, November 16, 2017 11:26 AM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] Regarding NVMe driver command queue depth.
Hi Paul,
I was reading about the driver from SPDK site, and interested in understanding the queue depth for a device.
“The specification allows for thousands, but most devices support between 32 and 128. The specification makes no guarantees about the performance available from each queue pair, but in practice the full performance of a device is almost always achievable using just one queue pair. For example, if a device claims to be capable of 450,000 I/O per second at queue depth 128, in practice it does not matter if the driver is using 4 queue pairs each with queue depth 32, or a single queue pair with queue depth 128”
When queue depth is mentioned for device, is it the number of commands that can be issued from application to controller, and outstanding at any time?
Is there NVMe driver API to set the queue depth? Is my understanding correct if I think that the size of queue is at firmware level?
Please give some detail about the parameter.
Thanks,
Sreeni
[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 24134 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-11-21 21:42 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-11-21 20:14 [SPDK] Program core dumps when I/O is issued in thread Sreeni Busam
-- strict thread matches above, loose matches on Subject: below --
2017-11-21 21:42 Sreeni Busam
2017-11-21 21:22 Sreeni Busam
2017-11-21 20:31 Harris, James R
2017-11-20 20:49 Sreeni Busam
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.