Re: [SPDK] Regarding NVMe driver command queue depth.

[Sreeni (Sreenivasa) Busam (Stellus) <s.busam at stellus.com>]

[-- Attachment #1: Type: text/plain, Size: 3248 bytes --]

I have been trying to test the number of commands that can be given to the device at a time. I verified that a maximum of 254 commands could be issued for a qpair. So I created a 2nd qpair for ns_entry and issued the I/O commands, it was failing in the first command itself. Is it invalid to create 2 qpair for the same ns_entry and send command to device? The qpair is successfully created, but I could not submit command.
I modified the hello_world program to test this and attached the related code.
Please take a look and let me know what is the problem.

0x000000000040bae2 in nvme_allocate_request (qpair=0x0,
    payload=0x7fffa4726ba0, payload_size=512, cb_fn=0x4041a4 <write_complete>,
    cb_arg=0x7b4270) at nvme.c:85
#1  0x000000000040996c in _nvme_ns_cmd_rw (ns=0x100ff8ee40, qpair=0x0,
    payload=0x7fffa4726ba0, payload_offset=0, md_offset=0, lba=0, lba_count=1,
    cb_fn=0x4041a4 <write_complete>, cb_arg=0x7b4270, opc=1, io_flags=0,
    apptag_mask=0, apptag=0, check_sgl=true) at nvme_ns_cmd.c:440
#2  0x0000000000409fea in spdk_nvme_ns_cmd_write (ns=0x100ff8ee40, qpair=0x0,
    buffer=0x10000f7000, lba=0, lba_count=1, cb_fn=0x4041a4 <write_complete>,
    cb_arg=0x7b4270, io_flags=0) at nvme_ns_cmd.c:649
#3  0x000000000040439d in stellus_spdk_nvme_ns_cmd_write (ns_entry=0x7b13c0,
    qpair=0x0, buffer=0x10000f7000, lba=0, lba_count=1,
    cb_fn=0x40420d <io_complete>, cb_arg=0x7b4270, ioflags=0) at iostat.c:233
#4  0x00000000004046b8 in test_io_func1 () at iostat.c:342
#5  0x0000000000404a94 in main (argc=1, argv=0x7fffa4726db8) at iostat.c:503
(gdb) f 3
#3  0x000000000040439d in stellus_spdk_nvme_ns_cmd_write (ns_entry=0x7b13c0,
    qpair=0x0, buffer=0x10000f7000, lba=0, lba_count=1,
    cb_fn=0x40420d <io_complete>, cb_arg=0x7b4270, ioflags=0) at iostat.c:233
233                     rc = spdk_nvme_ns_cmd_write(ns_entry->ns, qpair, buffer,
(gdb) p qpair
$1 = (struct spdk_nvme_qpair *) 0x0

If any of you get time, please look at it. Thank you for your suggestion.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Thursday, November 16, 2017 11:26 AM
To: spdk(a)lists.01.org
Subject: [SPDK] Regarding NVMe driver command queue depth.

Hi Paul,

I was reading about the driver from SPDK site, and interested in understanding the queue depth for a device.
"The specification allows for thousands, but most devices support between 32 and 128. The specification makes no guarantees about the performance available from each queue pair, but in practice the full performance of a device is almost always achievable using just one queue pair. For example, if a device claims to be capable of 450,000 I/O per second at queue depth 128, in practice it does not matter if the driver is using 4 queue pairs each with queue depth 32, or a single queue pair with queue depth 128"
When queue depth is mentioned for device, is it the number of commands that can be issued from application to controller, and outstanding at any time?
Is there NVMe driver API to set the queue depth? Is my understanding correct if I think that the size of queue is at firmware level?
Please give some detail about the parameter.

Thanks,
Sreeni

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 9248 bytes --]

[-- Attachment #3: hello_test_prog_v2.obj --]
[-- Type: application/octet-stream, Size: 4049 bytes --]

modified ns_entry

struct ns_entry {
	struct spdk_nvme_ctrlr	*ctrlr;
	struct spdk_nvme_ns	*ns;
	struct ns_entry		*next;
	struct spdk_nvme_qpair	*qpair;
	struct spdk_nvme_qpair	*qpair_2;
}
modified program
while (ns_entry != NULL) {
		/*
		 * Allocate an I/O qpair that we can use to submit read/write requests
		 *  to namespaces on the controller.  NVMe controllers typically support
		 *  many qpairs per controller.  Any I/O qpair allocated for a controller
		 *  can submit I/O to any namespace on that controller.
		 *
		 * The SPDK NVMe driver provides no synchronization for qpair accesses -
		 *  the application must ensure only a single thread submits I/O to a
		 *  qpair, and that same thread must also check for completions on that
		 *  qpair.  This enables extremely efficient I/O processing by making all
		 *  I/O operations completely lockless.
		 */
		if (ns_entry->qpair_allocated == 0) {
			ns_entry->qpair = spdk_nvme_ctrlr_alloc_io_qpair(ns_entry->ctrlr, NULL, 0);
			if (ns_entry->qpair == NULL) {
				printf("ERROR: spdk_nvme_ctrlr_alloc_io_qpair() failed\n");
				return;
			}
		}

/*
		 * Use spdk_dma_zmalloc to allocate a 4KB zeroed buffer.  This memory
		 * will be pinned, which is required for data buffers used for SPDK NVMe
		 * I/O operations.
		 */
		sequence = calloc(1, sizeof(struct io_sequence));
			sequence->buf = spdk_dma_zmalloc(0x1000, 0x1000, NULL);
			sequence->is_completed = 0;
			sequence->ns_entry = ns_entry;
			ns_entry->qpair_allocated = 1;

		sequence1 = calloc(1, sizeof(struct io_sequence));
			sequence1->buf = spdk_dma_zmalloc(0x1000, 0x1000, NULL);
			sequence1->is_completed = 0;
			sequence1->ns_entry = ns_entry;
		/*
		 * Print "Hello world!" to sequence.buf.  We will write this data to LBA
		 *  0 on the namespace, and then later read it back into a separate buffer
		 *  to demonstrate the full I/O path.
		 */
		snprintf(sequence->buf, 0x1000, "%s", "Hello world!\n");

		/*
		 * Write the data buffer to LBA 0 of this namespace.  "write_complete" and
		 *  "&sequence" are specified as the completion callback function and
		 *  argument respectively.  write_complete() will be called with the
		 *  value of &sequence as a parameter when the write I/O is completed.
		 *  This allows users to potentially specify different completion
		 *  callback routines for each I/O, as well as pass a unique handle
		 *  as an argument so the application knows which I/O has completed.
		 *
		 * Note that the SPDK NVMe driver will only check for completions
		 *  when the application calls spdk_nvme_qpair_process_completions().
		 *  It is the responsibility of the application to trigger the polling
		 *  process.
		 */
		cnt = 127;
		while (cnt) {
			rc = stellus_spdk_nvme_ns_cmd_write(ns_entry, ns_entry->qpair, sequence->buf,
						    0, /* LBA start */
						    1, /* number of LBAs */
						    io_complete, sequence, 0);
			if (rc != 0) {
				fprintf(stderr, "starting write I/O failed\n");
				exit(1);
			}
			rc = stellus_spdk_nvme_ns_cmd_read(ns_entry, ns_entry->qpair, sequence1->buf,
						    0, /* LBA start */
						    1, /* number of LBAs */
						    io_complete, sequence1, 0);
			if (rc != 0) {
				fprintf(stderr, "starting read I/O failed\n");
				exit(1);
			}
			cnt--;
		}
		spdk_nvme_qpair_process_completions(ns_entry->qpair, 0);
		ns_entry = ns_entry->next;
	}
	ns_entry = g_namespaces;
	ns_entry->qpair_2 = spdk_nvme_ctrlr_alloc_io_qpair(ns_entry->ctrlr, NULL, 0);
	if (ns_entry->qpair_2 == NULL) {
		printf("The qpair allocation failed.\n");
		exit (0);
	}
	while (ns_entry != NULL) {
		cnt = 1;
		while (cnt) {
		// Fails to be successfully submit
			rc = stellus_spdk_nvme_ns_cmd_write(ns_entry, ns_entry->qpair_2, sequence->buf,
						    0, /* LBA start */
						    1, /* number of LBAs */
						    io_complete, sequence, 0);
			if (rc != 0) {
				fprintf(stderr, "starting write I/O failed\n");
				exit(1);
			}
			...
		}
	}