From: Jens Axboe <axboe@kernel.dk>
To: stable@vger.kernel.org
Subject: [PATCH 6.6.y 17/19] io_uring/sqpoll: Increase task_work submission batch size
Date: Sat, 21 Nov 2026 21:30:47 +0000 [thread overview]
Message-ID: <20261121213049.1788332-18-axboe@kernel.dk> (raw)
In-Reply-To: <20261121213049.1788332-1-axboe@kernel.dk>
From: Gabriel Krisman Bertazi <krisman@suse.de>
Commit 92835cebab120f8a5f023a26a792a2ac3f816c4f upstream.
Our QA team reported a 10%-23%, throughput reduction on an io_uring
sqpoll testcase doing IO to a null_blk, that I traced back to a
reduction of the device submission queue depth utilization. It turns out
that, after commit af5d68f8892f ("io_uring/sqpoll: manage task_work
privately"), we capped the number of task_work entries that can be
completed from a single spin of sqpoll to only 8 entries, before the
sqpoll goes around to (potentially) sleep. While this cap doesn't drive
the submission side directly, it impacts the completion behavior, which
affects the number of IO queued by fio per sqpoll cycle on the
submission side, and io_uring ends up seeing less ios per sqpoll cycle.
As a result, block layer plugging is less effective, and we see more
time spent inside the block layer in profilings charts, and increased
submission latency measured by fio.
There are other places that have increased overhead once sqpoll sleeps
more often, such as the sqpoll utilization calculation. But, in this
microbenchmark, those were not representative enough in perf charts, and
their removal didn't yield measurable changes in throughput. The major
overhead comes from the fact we plug less, and less often, when submitting
to the block layer.
My benchmark is:
fio --ioengine=io_uring --direct=1 --iodepth=128 --runtime=300 --bs=4k \
--invalidate=1 --time_based --ramp_time=10 --group_reporting=1 \
--filename=/dev/nullb0 --name=RandomReads-direct-nullb-sqpoll-4k-1 \
--rw=randread --numjobs=1 --sqthread_poll
In one machine, tested on top of Linux 6.15-rc1, we have the following
baseline:
READ: bw=4994MiB/s (5236MB/s), 4994MiB/s-4994MiB/s (5236MB/s-5236MB/s), io=439GiB (471GB), run=90001-90001msec
With this patch:
READ: bw=5762MiB/s (6042MB/s), 5762MiB/s-5762MiB/s (6042MB/s-6042MB/s), io=506GiB (544GB), run=90001-90001msec
which is a 15% improvement in measured bandwidth. The average
submission latency is noticeably lowered too. As measured by
fio:
Baseline:
lat (usec): min=20, max=241, avg=99.81, stdev=3.38
Patched:
lat (usec): min=26, max=226, avg=86.48, stdev=4.82
If we look at blktrace, we can also see the plugging behavior is
improved. In the baseline, we end up limited to plugging 8 requests in
the block layer regardless of the device queue depth size, while after
patching we can drive more io, and we manage to utilize the full device
queue.
In the baseline, after a stabilization phase, an ordinary submission
looks like:
254,0 1 49942 0.016028795 5977 U N [iou-sqp-5976] 7
After patching, I see consistently more requests per unplug.
254,0 1 4996 0.001432872 3145 U N [iou-sqp-3144] 32
Ideally, the cap size would at least be the deep enough to fill the
device queue, but we can't predict that behavior, or assume all IO goes
to a single device, and thus can't guess the ideal batch size. We also
don't want to let the tw run unbounded, though I'm not sure it would
really be a problem. Instead, let's just give it a more sensible value
that will allow for more efficient batching. I've tested with different
cap values, and initially proposed to increase the cap to 1024. Jens
argued it is too big of a bump and I observed that, with 32, I'm no
longer able to observe this bottleneck in any of my machines.
Fixes: af5d68f8892f ("io_uring/sqpoll: manage task_work privately")
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://lore.kernel.org/r/20250508181203.3785544-1-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/sqpoll.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c
index bde67560ace8..3d0fbae80938 100644
--- a/io_uring/sqpoll.c
+++ b/io_uring/sqpoll.c
@@ -19,7 +19,7 @@
#include "sqpoll.h"
#define IORING_SQPOLL_CAP_ENTRIES_VALUE 8
-#define IORING_TW_CAP_ENTRIES_VALUE 8
+#define IORING_TW_CAP_ENTRIES_VALUE 32
enum {
IO_SQ_THREAD_SHOULD_STOP = 0,
--
2.55.0
next prev parent reply other threads:[~2026-11-21 21:31 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 01/19] io_uring: enable toggle of iowait usage when waiting on CQEs Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 02/19] io_uring: fail soft links on negative IOU_COMPLETE results Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 03/19] io_uring/rw: fail soft links on IOPOLL short or error IO Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 04/19] io_uring/timeout: don't rearm a multishot timeout on cancel Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 05/19] io_uring/rsrc: keep the old file when a file update fails Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 06/19] io_uring: cleanup handle_tw_list() calling convention Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 07/19] io_uring: pass in counter to handle_tw_list() rather than return it Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 08/19] io_uring/sqpoll: manage task_work privately Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 09/19] io_uring: wake SQPOLL task when task_work is added to an empty queue Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 10/19] io_uring/sqpoll: use the correct check for pending task_work Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 11/19] io_uring/sqpoll: early exit thread if task_context wasn't allocated Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 12/19] io_uring/sqpoll: ensure that normal task_work is also run timely Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 13/19] io_uring: signal SQPOLL task_work with TWA_SIGNAL_NO_IPI Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 14/19] io_uring/sqpoll: ensure task state is TASK_RUNNING when running task_work Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 15/19] io_uring: don't touch sqd->thread off tw add Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 16/19] io_uring/sqpoll: zero sqd->thread on tctx errors Jens Axboe
2026-11-21 21:30 ` Jens Axboe [this message]
2026-11-21 21:30 ` [PATCH 6.6.y 18/19] io_uring: fix task_work add use-after-free with SQPOLL Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 19/19] io_uring/sqpoll: do the return-to-userspace work in the SQPOLL thread Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261121213049.1788332-18-axboe@kernel.dk \
--to=axboe@kernel.dk \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox