From: Jens Axboe <axboe@kernel.dk>
To: linux-block@vger.kernel.org
Cc: Jens Axboe <axboe@kernel.dk>
Subject: [PATCH 9/9] nvme: wire up completion batching for the IRQ path
Date: Tue, 12 Oct 2021 12:17:42 -0600 [thread overview]
Message-ID: <20211012181742.672391-10-axboe@kernel.dk> (raw)
In-Reply-To: <20211012181742.672391-1-axboe@kernel.dk>
Trivial to do now, just need our own io_batch on the stack and pass that
in to the usual command completion handling.
I pondered making this dependent on how many entries we had to process,
but even for a single entry there's no discernable difference in
performance or latency. Running a sync workload over io_uring:
t/io_uring -b512 -d1 -s1 -c1 -p0 -F1 -B1 -n2 /dev/nvme1n1 /dev/nvme2n1
yields the below performance before the patch:
IOPS=254820, BW=124MiB/s, IOS/call=1/1, inflight=(1 1)
IOPS=251174, BW=122MiB/s, IOS/call=1/1, inflight=(1 1)
IOPS=250806, BW=122MiB/s, IOS/call=1/1, inflight=(1 1)
and the following after:
IOPS=255972, BW=124MiB/s, IOS/call=1/1, inflight=(1 1)
IOPS=251920, BW=123MiB/s, IOS/call=1/1, inflight=(1 1)
IOPS=251794, BW=122MiB/s, IOS/call=1/1, inflight=(1 1)
which definitely isn't slower, about the same if you factor in a bit of
variance. For peak performance workloads, benchmarking shows a 2%
improvement.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
drivers/nvme/host/pci.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 4713da708cd4..fb3de6f68eb1 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1076,8 +1076,10 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq)
static inline int nvme_process_cq(struct nvme_queue *nvmeq)
{
+ struct io_batch ib;
int found = 0;
+ ib.req_list = NULL;
while (nvme_cqe_pending(nvmeq)) {
found++;
/*
@@ -1085,12 +1087,15 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq)
* the cqe requires a full read memory barrier
*/
dma_rmb();
- nvme_handle_cqe(nvmeq, NULL, nvmeq->cq_head);
+ nvme_handle_cqe(nvmeq, &ib, nvmeq->cq_head);
nvme_update_cq_head(nvmeq);
}
- if (found)
+ if (found) {
+ if (ib.req_list)
+ nvme_pci_complete_batch(&ib);
nvme_ring_cq_doorbell(nvmeq);
+ }
return found;
}
--
2.33.0
next prev parent reply other threads:[~2021-10-12 18:17 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-12 18:17 [PATCHSET 0/9] Batched completions Jens Axboe
2021-10-12 18:17 ` [PATCH 1/9] block: add a struct io_batch argument to fops->iopoll() Jens Axboe
2021-10-12 18:25 ` Bart Van Assche
2021-10-12 18:28 ` Jens Axboe
2021-10-12 18:17 ` [PATCH 2/9] sbitmap: add helper to clear a batch of tags Jens Axboe
2021-10-12 18:29 ` Bart Van Assche
2021-10-12 18:34 ` Jens Axboe
2021-10-12 18:17 ` [PATCH 3/9] sbitmap: test bit before calling test_and_set_bit() Jens Axboe
2021-10-12 18:17 ` [PATCH 4/9] block: add support for blk_mq_end_request_batch() Jens Axboe
2021-10-12 18:32 ` Bart Van Assche
2021-10-12 18:55 ` Jens Axboe
2021-10-12 18:17 ` [PATCH 5/9] nvme: move the fast path nvme error and disposition helpers Jens Axboe
2021-10-13 6:57 ` Christoph Hellwig
2021-10-13 14:41 ` Jens Axboe
2021-10-13 15:11 ` Christoph Hellwig
2021-10-12 18:17 ` [PATCH 6/9] nvme: add support for batched completion of polled IO Jens Axboe
2021-10-13 7:08 ` Christoph Hellwig
2021-10-13 15:10 ` Jens Axboe
2021-10-13 15:16 ` Christoph Hellwig
2021-10-13 15:42 ` Jens Axboe
2021-10-13 15:49 ` Jens Axboe
2021-10-13 15:50 ` Christoph Hellwig
2021-10-13 16:04 ` Jens Axboe
2021-10-13 16:13 ` Christoph Hellwig
2021-10-13 16:33 ` Jens Axboe
2021-10-13 16:45 ` Jens Axboe
2021-10-13 9:09 ` John Garry
2021-10-13 15:07 ` Jens Axboe
2021-10-12 18:17 ` [PATCH 7/9] block: assign batch completion handler in blk_poll() Jens Axboe
2021-10-12 18:17 ` [PATCH 8/9] io_uring: utilize the io_batch infrastructure for more efficient polled IO Jens Axboe
2021-10-12 18:17 ` Jens Axboe [this message]
2021-10-13 7:12 ` [PATCH 9/9] nvme: wire up completion batching for the IRQ path Christoph Hellwig
2021-10-13 15:04 ` Jens Axboe
-- strict thread matches above, loose matches on Subject: below --
2021-10-13 16:54 [PATCHSET v2 0/9] Batched completions Jens Axboe
2021-10-13 16:54 ` [PATCH 9/9] nvme: wire up completion batching for the IRQ path Jens Axboe
2021-10-14 7:53 ` Christoph Hellwig
2021-10-14 15:49 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211012181742.672391-10-axboe@kernel.dk \
--to=axboe@kernel.dk \
--cc=linux-block@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).