* [PATCH 6.6.y 01/19] io_uring: enable toggle of iowait usage when waiting on CQEs
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 02/19] io_uring: fail soft links on negative IOU_COMPLETE results Jens Axboe
` (17 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
Commit 07754bfd9aee59063f8549f6e4d455eae636ecc7 upstream.
By default, io_uring marks a waiting task as being in iowait, if it's
sleeping waiting on events and there are pending requests. This isn't
necessarily always useful, and may be confusing on non-storage setups
where iowait isn't expected. It can also cause extra power usage, by
preventing the CPU from entering lower sleep states.
This adds a new enter flag, IORING_ENTER_NO_IOWAIT. If set, then
io_uring will not account the sleeping task as being in iowait. If the
kernel supports this feature, then it will be marked by having the
IORING_FEAT_NO_IOWAIT feature flag set.
As the kernel currently does not support separating the iowait
accounting and CPU frequency boosting, the IORING_ENTER_NO_IOWAIT
controls both of these at the same time. In the future, if those do end
up being split, then it'd be possible to control them separately.
However, it seems more likely that the kernel will decouple iowait and
CPU frequency boosting anyway.
[ 6.6: no struct ext_arg here. Pass the enter flags to io_cqring_wait()
and keep the setting in struct io_wait_queue. Flag bits match
upstream. ]
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
include/uapi/linux/io_uring.h | 2 ++
io_uring/io_uring.c | 16 ++++++++++------
2 files changed, 12 insertions(+), 6 deletions(-)
diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h
index 333769bc6abf..7c8c70859c94 100644
--- a/include/uapi/linux/io_uring.h
+++ b/include/uapi/linux/io_uring.h
@@ -466,6 +466,7 @@ struct io_cqring_offsets {
#define IORING_ENTER_SQ_WAIT (1U << 2)
#define IORING_ENTER_EXT_ARG (1U << 3)
#define IORING_ENTER_REGISTERED_RING (1U << 4)
+#define IORING_ENTER_NO_IOWAIT (1U << 7)
/*
* Passed in for io_uring_setup(2). Copied back with updated info on success
@@ -500,6 +501,7 @@ struct io_uring_params {
#define IORING_FEAT_CQE_SKIP (1U << 11)
#define IORING_FEAT_LINKED_FILE (1U << 12)
#define IORING_FEAT_REG_REG_RING (1U << 13)
+#define IORING_FEAT_NO_IOWAIT (1U << 17)
/*
* io_uring_register(2) opcodes and arguments
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 43860048d2bf..b14651c187d4 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2520,6 +2520,7 @@ struct io_wait_queue {
unsigned cq_tail;
unsigned nr_timeouts;
ktime_t timeout;
+ bool iowait;
};
static inline bool io_has_work(struct io_ring_ctx *ctx)
@@ -2601,7 +2602,7 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx,
* can take into account that the task is waiting for IO - turns out
* to be important for low QD IO.
*/
- if (current_pending_io())
+ if (iowq->iowait && current_pending_io())
current->in_iowait = 1;
ret = 0;
if (iowq->timeout == KTIME_MAX)
@@ -2616,7 +2617,7 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx,
* Wait until events become available, if we don't already have some. The
* application must reap them itself, as they reside on the shared cq ring.
*/
-static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events,
+static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, u32 flags,
const sigset_t __user *sig, size_t sigsz,
struct __kernel_timespec __user *uts)
{
@@ -2638,6 +2639,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events,
iowq.wq.private = current;
INIT_LIST_HEAD(&iowq.wq.entry);
iowq.ctx = ctx;
+ iowq.iowait = !(flags & IORING_ENTER_NO_IOWAIT);
iowq.nr_timeouts = atomic_read(&ctx->cq_timeouts);
iowq.cq_tail = READ_ONCE(ctx->rings->cq.head) + min_events;
iowq.timeout = KTIME_MAX;
@@ -3828,7 +3830,8 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit,
if (unlikely(flags & ~(IORING_ENTER_GETEVENTS | IORING_ENTER_SQ_WAKEUP |
IORING_ENTER_SQ_WAIT | IORING_ENTER_EXT_ARG |
- IORING_ENTER_REGISTERED_RING)))
+ IORING_ENTER_REGISTERED_RING |
+ IORING_ENTER_NO_IOWAIT)))
return -EINVAL;
/*
@@ -3928,8 +3931,8 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit,
if (likely(!ret2)) {
min_complete = min(min_complete,
ctx->cq_entries);
- ret2 = io_cqring_wait(ctx, min_complete, sig,
- argsz, ts);
+ ret2 = io_cqring_wait(ctx, min_complete, flags,
+ sig, argsz, ts);
}
}
@@ -4211,7 +4214,8 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p,
IORING_FEAT_POLL_32BITS | IORING_FEAT_SQPOLL_NONFIXED |
IORING_FEAT_EXT_ARG | IORING_FEAT_NATIVE_WORKERS |
IORING_FEAT_RSRC_TAGS | IORING_FEAT_CQE_SKIP |
- IORING_FEAT_LINKED_FILE | IORING_FEAT_REG_REG_RING;
+ IORING_FEAT_LINKED_FILE | IORING_FEAT_REG_REG_RING |
+ IORING_FEAT_NO_IOWAIT;
if (copy_to_user(params, p, sizeof(*p))) {
ret = -EFAULT;
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 02/19] io_uring: fail soft links on negative IOU_COMPLETE results
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 01/19] io_uring: enable toggle of iowait usage when waiting on CQEs Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 03/19] io_uring/rw: fail soft links on IOPOLL short or error IO Jens Axboe
` (16 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
From: Yang Xiuwei <yangxiuwei@kylinos.cn>
Commit 9ed19503a7b34e1ffb2b81236843204c473b0ae0 upstream.
Handlers such as fsync return IOU_COMPLETE with a negative cqe.res.
The old io_req_complete() path called req_set_fail() for res < 0.
After completions moved into the core, that check was dropped,
so IOSQE_IO_LINK dependents kept running after the head failed.
Fixes: 97b388d70b53 ("io_uring: handle completions in the core")
Signed-off-by: Yang Xiuwei <yangxiuwei@kylinos.cn>
Link: https://patch.msgid.link/20261114041123.109163-1-yangxiuwei@kylinos.cn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
[axboe: 6.6 has IOU_OK rather than IOU_COMPLETE]
---
io_uring/io_uring.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index b14651c187d4..8bbcab4d1d7c 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1921,9 +1921,11 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)
}
if (ret == IOU_OK) {
- /* not all handlers take care of the internal restart codes */
- if (unlikely(req->cqe.res <= -ERESTARTSYS))
- req->cqe.res = io_fixup_restart_res(req->cqe.res);
+ if (unlikely(req->cqe.res < 0)) {
+ if (req->cqe.res <= -ERESTARTSYS)
+ req->cqe.res = io_fixup_restart_res(req->cqe.res);
+ req_set_fail(req);
+ }
if (issue_flags & IO_URING_F_COMPLETE_DEFER)
io_req_complete_defer(req);
else
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 03/19] io_uring/rw: fail soft links on IOPOLL short or error IO
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 01/19] io_uring: enable toggle of iowait usage when waiting on CQEs Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 02/19] io_uring: fail soft links on negative IOU_COMPLETE results Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 04/19] io_uring/timeout: don't rearm a multishot timeout on cancel Jens Axboe
` (15 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
From: Yang Xiuwei <yangxiuwei@kylinos.cn>
Commit 4d3db66c5363dcf42c3d43ebaea330ac2d1888bb upstream.
io_complete_rw_iopoll() updates cqe.res when the transfer does not
match the requested length, but it does not call req_set_fail().
Soft linked requests then still run after a short or failed IOPOLL
read or write.
Fixes: b66ceaf324b3 ("io_uring: move iopoll reissue into regular IO path")
Signed-off-by: Yang Xiuwei <yangxiuwei@kylinos.cn>
Link: https://patch.msgid.link/20261114072246.741950-3-yangxiuwei@kylinos.cn
Signed-off-by: Jens Axboe <axboe@kernel.dk>
[axboe: 6.6 reissues from inside the res != cqe.res branch, fail the
request after the reissue check there]
---
io_uring/rw.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/io_uring/rw.c b/io_uring/rw.c
index 102f92dce672..ec8e719a9386 100644
--- a/io_uring/rw.c
+++ b/io_uring/rw.c
@@ -318,6 +318,7 @@ static void io_complete_rw_iopoll(struct kiocb *kiocb, long res)
req->flags |= REQ_F_REISSUE | REQ_F_PARTIAL_IO;
return;
}
+ req_set_fail(req);
req->cqe.res = res;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 04/19] io_uring/timeout: don't rearm a multishot timeout on cancel
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (2 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 03/19] io_uring/rw: fail soft links on IOPOLL short or error IO Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 05/19] io_uring/rsrc: keep the old file when a file update fails Jens Axboe
` (14 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
From: Cheng Yongkang <teel4res@gmail.com>
Commit 97d4a585db2b91d855f957be0a9f126068c5be93 upstream.
io_timeout_complete() does not check tw.cancel. When it runs from
io_cancel_local_task_work(), a multishot timeout that has not reached
its count still posts an IORING_CQE_F_MORE completion and rearms its
timer. With a zero interval the timer expires again right away and
queues new task work, so the drain keeps running it until posting a
CQE fails.
Normally the CQ fills up quickly and the request then completes. But
if userspace moves the CQ head far ahead of the tail, for example to
cq.tail + 0x7fffffff, posting keeps succeeding and ring teardown is
delayed for a long time.
Don't post another CQE or rearm the timer when tw.cancel is set, so
the request completes through io_req_task_complete(), like the poll and
linked timeout task work does when cancelled.
Fixes: ea97f6c8558e ("io_uring: add support for multishot timeouts")
Signed-off-by: Cheng Yongkang <teel4res@gmail.com>
Link: https://patch.msgid.link/20261116044519.281756-1-teel4res@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
[axboe: 6.6 has no tw.cancel, use io_should_terminate_tw() like
io_req_task_link_timeout() does]
---
io_uring/timeout.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index df9ef01ebf19..4eec4445fafb 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -72,9 +72,11 @@ static void io_timeout_complete(struct io_kiocb *req, struct io_tw_state *ts)
struct io_ring_ctx *ctx = req->ctx;
if (!io_timeout_finish(timeout, data)) {
- bool filled;
- filled = io_fill_cqe_req_aux(req, ts->locked, -ETIME,
- IORING_CQE_F_MORE);
+ bool filled = false;
+
+ if (!io_should_terminate_tw(ctx))
+ filled = io_fill_cqe_req_aux(req, ts->locked, -ETIME,
+ IORING_CQE_F_MORE);
if (filled) {
/* re-arm timer */
spin_lock_irq(&ctx->timeout_lock);
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 05/19] io_uring/rsrc: keep the old file when a file update fails
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (3 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 04/19] io_uring/timeout: don't rearm a multishot timeout on cancel Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 06/19] io_uring: cleanup handle_tw_list() calling convention Jens Axboe
` (13 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
From: Cheng Yongkang <teel4res@gmail.com>
Commit e5a346e4309d720c47fe8331a00068608fad47dd upstream.
__io_sqe_files_update() resets the slot before it calls fget() on the
new fd. If fget() fails, the fd is an io_uring file, or the node
allocation fails, the update stops and reports that entry as not done,
but the old file has already been removed. Later requests that use the
slot fail with -EBADF, and if the old file was registered with a tag,
its tag CQE is posted as if it had been removed on purpose.
Reset the slot only after the new file has been checked and its node
allocated, as io_install_fixed_file() does. Removal with fd == -1 still
resets the slot right away.
Fixes: c3a31e605620 ("io_uring: add support for IORING_REGISTER_FILES_UPDATE")
Cc: stable@vger.kernel.org
Signed-off-by: Cheng Yongkang <teel4res@gmail.com>
Link: https://patch.msgid.link/20261116044531.281978-1-teel4res@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
[axboe: 6.6 has no rsrc nodes for files, the old file is queued for
removal with io_queue_rsrc_removal(), which can fail. Get and check the
new file first, and only remove the old one once that has succeeded]
---
io_uring/rsrc.c | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index ba1dedaaa7c4..0cc498a0673d 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -369,6 +369,7 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx,
return -EINVAL;
for (done = 0; done < nr_args; done++) {
+ struct file *file = NULL;
u64 tag = 0;
if ((tags && copy_from_user(&tag, &tags[done], sizeof(tag))) ||
@@ -386,17 +387,8 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx,
i = array_index_nospec(up->offset + done, ctx->nr_user_files);
file_slot = io_fixed_file_slot(&ctx->file_table, i);
- if (file_slot->file_ptr) {
- err = io_queue_rsrc_removal(data, i,
- io_slot_file(file_slot));
- if (err)
- break;
- file_slot->file_ptr = 0;
- io_file_bitmap_clear(&ctx->file_table, i);
- }
if (fd != -1) {
- struct file *file = fget(fd);
-
+ file = fget(fd);
if (!file) {
err = -EBADF;
break;
@@ -409,6 +401,19 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx,
err = -EBADF;
break;
}
+ }
+ if (file_slot->file_ptr) {
+ err = io_queue_rsrc_removal(data, i,
+ io_slot_file(file_slot));
+ if (err) {
+ if (file)
+ fput(file);
+ break;
+ }
+ file_slot->file_ptr = 0;
+ io_file_bitmap_clear(&ctx->file_table, i);
+ }
+ if (file) {
*io_get_tag_slot(data, i) = tag;
io_fixed_file_set(file_slot, file);
io_file_bitmap_set(&ctx->file_table, i);
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 06/19] io_uring: cleanup handle_tw_list() calling convention
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (4 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 05/19] io_uring/rsrc: keep the old file when a file update fails Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 07/19] io_uring: pass in counter to handle_tw_list() rather than return it Jens Axboe
` (12 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
Commit 42c0905f0cac9a86d2cb8138665a6d62ea607078 upstream.
Now that we don't loop around task_work anymore, there's no point in
maintaining the ring and locked state outside of handle_tw_list(). Get
rid of passing in those pointers (and pointers to pointers) and just do
the management internally in handle_tw_list().
Signed-off-by: Jens Axboe <axboe@kernel.dk>
[axboe: 6.6 runs normal task_work in LIFO order, as it doesn't have
170539bdf109 ("io_uring: handle traditional task_work in FIFO order"),
so call handle_tw_list() without llist_reverse_order()]
---
io_uring/io_uring.c | 29 +++++++++++++----------------
1 file changed, 13 insertions(+), 16 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 8bbcab4d1d7c..96474d9ebef8 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1197,10 +1197,10 @@ static void ctx_flush_and_put(struct io_ring_ctx *ctx, struct io_tw_state *ts)
percpu_ref_put(&ctx->refs);
}
-static unsigned int handle_tw_list(struct llist_node *node,
- struct io_ring_ctx **ctx,
- struct io_tw_state *ts)
+static unsigned int handle_tw_list(struct llist_node *node)
{
+ struct io_ring_ctx *ctx = NULL;
+ struct io_tw_state ts = { };
unsigned int count = 0;
do {
@@ -1210,25 +1210,26 @@ static unsigned int handle_tw_list(struct llist_node *node,
prefetch(container_of(next, struct io_kiocb, io_task_work.node));
- if (req->ctx != *ctx) {
- ctx_flush_and_put(*ctx, ts);
- *ctx = req->ctx;
+ if (req->ctx != ctx) {
+ ctx_flush_and_put(ctx, &ts);
+ ctx = req->ctx;
/* if not contended, grab and improve batching */
- ts->locked = mutex_trylock(&(*ctx)->uring_lock);
- percpu_ref_get(&(*ctx)->refs);
+ ts.locked = mutex_trylock(&ctx->uring_lock);
+ percpu_ref_get(&ctx->refs);
}
INDIRECT_CALL_2(req->io_task_work.func,
io_poll_task_func, io_req_rw_complete,
- req, ts);
+ req, &ts);
node = next;
count++;
if (unlikely(need_resched())) {
- ctx_flush_and_put(*ctx, ts);
- *ctx = NULL;
+ ctx_flush_and_put(ctx, &ts);
+ ctx = NULL;
cond_resched();
}
} while (node);
+ ctx_flush_and_put(ctx, &ts);
return count;
}
@@ -1277,8 +1278,6 @@ static __cold void io_fallback_tw(struct io_uring_task *tctx, bool sync)
void tctx_task_work(struct callback_head *cb)
{
- struct io_tw_state ts = {};
- struct io_ring_ctx *ctx = NULL;
struct io_uring_task *tctx = container_of(cb, struct io_uring_task,
task_work);
struct llist_node *node;
@@ -1291,9 +1290,7 @@ void tctx_task_work(struct callback_head *cb)
node = llist_del_all(&tctx->task_list);
if (node)
- count = handle_tw_list(node, &ctx, &ts);
-
- ctx_flush_and_put(ctx, &ts);
+ count = handle_tw_list(node);
/* relaxed read is enough as only the task itself sets ->in_cancel */
if (unlikely(atomic_read(&tctx->in_cancel)))
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 07/19] io_uring: pass in counter to handle_tw_list() rather than return it
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (5 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 06/19] io_uring: cleanup handle_tw_list() calling convention Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 08/19] io_uring/sqpoll: manage task_work privately Jens Axboe
` (11 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
Commit 2708af1adc11700c6c3ce4109e3b133079a36a78 upstream.
No functional changes in this patch, just in preparation for returning
something other than count from this helper.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
[axboe: 6.6 runs normal task_work in LIFO order, as it doesn't have
170539bdf109 ("io_uring: handle traditional task_work in FIFO order"),
so call handle_tw_list() without llist_reverse_order()]
---
io_uring/io_uring.c | 8 +++-----
1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 96474d9ebef8..ddae460e06c0 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1197,11 +1197,10 @@ static void ctx_flush_and_put(struct io_ring_ctx *ctx, struct io_tw_state *ts)
percpu_ref_put(&ctx->refs);
}
-static unsigned int handle_tw_list(struct llist_node *node)
+static void handle_tw_list(struct llist_node *node, unsigned int *count)
{
struct io_ring_ctx *ctx = NULL;
struct io_tw_state ts = { };
- unsigned int count = 0;
do {
struct llist_node *next = node->next;
@@ -1221,7 +1220,7 @@ static unsigned int handle_tw_list(struct llist_node *node)
io_poll_task_func, io_req_rw_complete,
req, &ts);
node = next;
- count++;
+ (*count)++;
if (unlikely(need_resched())) {
ctx_flush_and_put(ctx, &ts);
ctx = NULL;
@@ -1230,7 +1229,6 @@ static unsigned int handle_tw_list(struct llist_node *node)
} while (node);
ctx_flush_and_put(ctx, &ts);
- return count;
}
/**
@@ -1290,7 +1288,7 @@ void tctx_task_work(struct callback_head *cb)
node = llist_del_all(&tctx->task_list);
if (node)
- count = handle_tw_list(node);
+ handle_tw_list(node, &count);
/* relaxed read is enough as only the task itself sets ->in_cancel */
if (unlikely(atomic_read(&tctx->in_cancel)))
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 08/19] io_uring/sqpoll: manage task_work privately
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (6 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 07/19] io_uring: pass in counter to handle_tw_list() rather than return it Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 09/19] io_uring: wake SQPOLL task when task_work is added to an empty queue Jens Axboe
` (10 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
Commit af5d68f8892f8ee8f137648b79ceb2abc153a19b upstream.
Decouple from task_work running, and cap the number of entries we process
at the time. If we exceed that number, push remaining entries to a retry
list that we'll process first next time.
We cap the number of entries to process at 8, which is fairly random.
We just want to get enough per-ctx batching here, while not processing
endlessly.
Since we manually run PF_IO_WORKER related task_work anyway as the task
never exits to userspace, with this we no longer need to add an actual
task_work item to the per-process list.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
[axboe: 6.6 runs normal task_work in LIFO order, so don't reverse the
list in tctx_task_work_run(), and its trace_io_uring_task_work_run()
still takes the 'loops' argument. Run the retry_list in io_sq_thread()
before the IO_SQ_THREAD_EXITING marking that 6.6 has]
---
io_uring/io_uring.c | 42 +++++++++++++++++++++++++++++++++---------
io_uring/io_uring.h | 24 ++++++++++++++++++------
io_uring/sqpoll.c | 29 ++++++++++++++++++++++++++++-
3 files changed, 79 insertions(+), 16 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index ddae460e06c0..3fd27002f4f7 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1197,7 +1197,14 @@ static void ctx_flush_and_put(struct io_ring_ctx *ctx, struct io_tw_state *ts)
percpu_ref_put(&ctx->refs);
}
-static void handle_tw_list(struct llist_node *node, unsigned int *count)
+/*
+ * Run queued task_work, returning the number of entries processed in *count.
+ * If more entries than max_entries are available, stop processing once this
+ * is reached and return the rest of the list.
+ */
+struct llist_node *io_handle_tw_list(struct llist_node *node,
+ unsigned int *count,
+ unsigned int max_entries)
{
struct io_ring_ctx *ctx = NULL;
struct io_tw_state ts = { };
@@ -1226,9 +1233,10 @@ static void handle_tw_list(struct llist_node *node, unsigned int *count)
ctx = NULL;
cond_resched();
}
- } while (node);
+ } while (node && *count < max_entries);
ctx_flush_and_put(ctx, &ts);
+ return node;
}
/**
@@ -1274,27 +1282,39 @@ static __cold void io_fallback_tw(struct io_uring_task *tctx, bool sync)
}
}
-void tctx_task_work(struct callback_head *cb)
+struct llist_node *tctx_task_work_run(struct io_uring_task *tctx,
+ unsigned int max_entries,
+ unsigned int *count)
{
- struct io_uring_task *tctx = container_of(cb, struct io_uring_task,
- task_work);
struct llist_node *node;
- unsigned int count = 0;
if (unlikely(current->flags & PF_EXITING)) {
io_fallback_tw(tctx, true);
- return;
+ return NULL;
}
node = llist_del_all(&tctx->task_list);
if (node)
- handle_tw_list(node, &count);
+ node = io_handle_tw_list(node, count, max_entries);
/* relaxed read is enough as only the task itself sets ->in_cancel */
if (unlikely(atomic_read(&tctx->in_cancel)))
io_uring_drop_tctx_refs(current);
- trace_io_uring_task_work_run(tctx, count, 1);
+ trace_io_uring_task_work_run(tctx, *count, 1);
+ return node;
+}
+
+void tctx_task_work(struct callback_head *cb)
+{
+ struct io_uring_task *tctx;
+ struct llist_node *ret;
+ unsigned int count = 0;
+
+ tctx = container_of(cb, struct io_uring_task, task_work);
+ ret = tctx_task_work_run(tctx, UINT_MAX, &count);
+ /* can't happen */
+ WARN_ON_ONCE(ret);
}
static void io_poll_wq_wake_rcu(struct rcu_head *rcu)
@@ -1388,6 +1408,10 @@ static void io_req_normal_work_add(struct io_kiocb *req)
if (ctx->flags & IORING_SETUP_TASKRUN_FLAG)
atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags);
+ /* SQPOLL doesn't need the task_work added, it'll run it itself */
+ if (ctx->flags & IORING_SETUP_SQPOLL)
+ return;
+
if (likely(!task_work_add(req->task, &tctx->task_work, ctx->notify_method)))
return;
diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h
index a3c97ffb4d14..0d749588ab21 100644
--- a/io_uring/io_uring.h
+++ b/io_uring/io_uring.h
@@ -77,6 +77,8 @@ void io_req_task_queue(struct io_kiocb *req);
void io_req_task_complete(struct io_kiocb *req, struct io_tw_state *ts);
void io_req_task_queue_fail(struct io_kiocb *req, int ret);
void io_req_task_submit(struct io_kiocb *req, struct io_tw_state *ts);
+struct llist_node *io_handle_tw_list(struct llist_node *node, unsigned int *count, unsigned int max_entries);
+struct llist_node *tctx_task_work_run(struct io_uring_task *tctx, unsigned int max_entries, unsigned int *count);
void tctx_task_work(struct callback_head *cb);
__cold void io_uring_cancel_generic(bool cancel_all, struct io_sq_data *sqd);
int io_uring_alloc_task_context(struct task_struct *task,
@@ -338,6 +340,8 @@ static inline unsigned int io_sqring_entries(struct io_ring_ctx *ctx)
static inline int io_run_task_work(void)
{
+ bool ret = false;
+
/*
* Always check-and-clear the task_work notification signal. With how
* signaling works for task_work, we can find it set with nothing to
@@ -349,18 +353,26 @@ static inline int io_run_task_work(void)
* PF_IO_WORKER never returns to userspace, so check here if we have
* notify work that needs processing.
*/
- if (current->flags & PF_IO_WORKER &&
- test_thread_flag(TIF_NOTIFY_RESUME)) {
- __set_current_state(TASK_RUNNING);
- resume_user_mode_work(NULL);
+ if (current->flags & PF_IO_WORKER) {
+ if (test_thread_flag(TIF_NOTIFY_RESUME)) {
+ __set_current_state(TASK_RUNNING);
+ resume_user_mode_work(NULL);
+ }
+ if (current->io_uring) {
+ unsigned int count = 0;
+
+ tctx_task_work_run(current->io_uring, UINT_MAX, &count);
+ if (count)
+ ret = true;
+ }
}
if (task_work_pending(current)) {
__set_current_state(TASK_RUNNING);
task_work_run();
- return 1;
+ ret = true;
}
- return 0;
+ return ret;
}
static inline bool io_task_work_pending(struct io_ring_ctx *ctx)
diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c
index 75065de8d7d3..716cb3cc7e2c 100644
--- a/io_uring/sqpoll.c
+++ b/io_uring/sqpoll.c
@@ -19,6 +19,7 @@
#include "sqpoll.h"
#define IORING_SQPOLL_CAP_ENTRIES_VALUE 8
+#define IORING_TW_CAP_ENTRIES_VALUE 8
enum {
IO_SQ_THREAD_SHOULD_STOP = 0,
@@ -238,8 +239,31 @@ static bool io_sqd_handle_event(struct io_sq_data *sqd)
return did_sig || test_bit(IO_SQ_THREAD_SHOULD_STOP, &sqd->state);
}
+/*
+ * Run task_work, processing the retry_list first. The retry_list holds
+ * entries that we passed on in the previous run, if we had more task_work
+ * than we were asked to process. Newly queued task_work isn't run until the
+ * retry list has been fully processed.
+ */
+static unsigned int io_sq_tw(struct llist_node **retry_list, int max_entries)
+{
+ struct io_uring_task *tctx = current->io_uring;
+ unsigned int count = 0;
+
+ if (*retry_list) {
+ *retry_list = io_handle_tw_list(*retry_list, &count, max_entries);
+ if (count >= max_entries)
+ return count;
+ max_entries -= count;
+ }
+
+ *retry_list = tctx_task_work_run(tctx, max_entries, &count);
+ return count;
+}
+
static int io_sq_thread(void *data)
{
+ struct llist_node *retry_list = NULL;
struct io_sq_data *sqd = data;
struct io_ring_ctx *ctx;
unsigned long timeout = 0;
@@ -284,7 +308,7 @@ static int io_sq_thread(void *data)
if (!sqt_spin && (ret > 0 || !wq_list_empty(&ctx->iopoll_list)))
sqt_spin = true;
}
- if (io_run_task_work())
+ if (io_sq_tw(&retry_list, IORING_TW_CAP_ENTRIES_VALUE))
sqt_spin = true;
if (sqt_spin || !time_after(jiffies, timeout)) {
@@ -346,6 +370,9 @@ static int io_sq_thread(void *data)
timeout = jiffies + sqd->sq_thread_idle;
}
+ if (retry_list)
+ io_sq_tw(&retry_list, UINT_MAX);
+
/*
* Cancelation drops sqd->lock while it waits, and a ring could attach
* to a thread that is on its way out. A fatal signal doesn't set the
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 09/19] io_uring: wake SQPOLL task when task_work is added to an empty queue
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (7 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 08/19] io_uring/sqpoll: manage task_work privately Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 10/19] io_uring/sqpoll: use the correct check for pending task_work Jens Axboe
` (9 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
Commit 78f9b61bd8e5466f0e90823e64e3d87c41f6258c upstream.
If there's no current work on the list, we still need to potentially
wake the SQPOLL task if it is sleeping. This is ordered with the
wait queue addition in sqpoll, which adds to the wait queue before
checking for pending work items.
Fixes: af5d68f8892f ("io_uring/sqpoll: manage task_work privately")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io_uring.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 3fd27002f4f7..895011a3d8ce 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1409,8 +1409,13 @@ static void io_req_normal_work_add(struct io_kiocb *req)
atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags);
/* SQPOLL doesn't need the task_work added, it'll run it itself */
- if (ctx->flags & IORING_SETUP_SQPOLL)
+ if (ctx->flags & IORING_SETUP_SQPOLL) {
+ struct io_sq_data *sqd = ctx->sq_data;
+
+ if (wq_has_sleeper(&sqd->wait))
+ wake_up(&sqd->wait);
return;
+ }
if (likely(!task_work_add(req->task, &tctx->task_work, ctx->notify_method)))
return;
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 10/19] io_uring/sqpoll: use the correct check for pending task_work
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (8 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 09/19] io_uring: wake SQPOLL task when task_work is added to an empty queue Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 11/19] io_uring/sqpoll: early exit thread if task_context wasn't allocated Jens Axboe
` (8 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
Commit c8d8fc3b2d9d4c81e3d3b23eca504ad713a91219 upstream.
A previous commit moved to using just the private task_work list for
SQPOLL, but it neglected to update the check for whether we have
pending task_work. Normally this is fine as we'll attempt to run it
unconditionally, but if we race with going to sleep AND task_work
being added, then we certainly need the right check here.
Fixes: af5d68f8892f ("io_uring/sqpoll: manage task_work privately")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/sqpoll.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c
index 716cb3cc7e2c..ecd336c373b1 100644
--- a/io_uring/sqpoll.c
+++ b/io_uring/sqpoll.c
@@ -261,6 +261,13 @@ static unsigned int io_sq_tw(struct llist_node **retry_list, int max_entries)
return count;
}
+static bool io_sq_tw_pending(struct llist_node *retry_list)
+{
+ struct io_uring_task *tctx = current->io_uring;
+
+ return retry_list || !llist_empty(&tctx->task_list);
+}
+
static int io_sq_thread(void *data)
{
struct llist_node *retry_list = NULL;
@@ -324,7 +331,7 @@ static int io_sq_thread(void *data)
}
prepare_to_wait(&sqd->wait, &wait, TASK_INTERRUPTIBLE);
- if (!io_sqd_events_pending(sqd) && !task_work_pending(current)) {
+ if (!io_sqd_events_pending(sqd) && !io_sq_tw_pending(retry_list)) {
bool needs_sched = true;
list_for_each_entry(ctx, &sqd->ctx_list, sqd_list) {
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 11/19] io_uring/sqpoll: early exit thread if task_context wasn't allocated
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (9 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 10/19] io_uring/sqpoll: use the correct check for pending task_work Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 12/19] io_uring/sqpoll: ensure that normal task_work is also run timely Jens Axboe
` (7 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
Commit 1251d2025c3e1bcf1f17ec0f3c0dfae5e5bbb146 upstream.
Ideally we'd want to simply kill the task rather than wake it, but for
now let's just add a startup check that causes the thread to exit.
This can only happen if io_uring_alloc_task_context() fails, which
generally requires fault injection.
Reported-by: Ubisectech Sirius <bugreport@ubisectech.com>
Fixes: af5d68f8892f ("io_uring/sqpoll: manage task_work privately")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
[axboe: 6.6 has no sqd->exited, the thread drops its sqd reference with
io_sq_data_release() on exit, so put err_out before that]
---
io_uring/sqpoll.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c
index ecd336c373b1..3849252f6135 100644
--- a/io_uring/sqpoll.c
+++ b/io_uring/sqpoll.c
@@ -277,6 +277,10 @@ static int io_sq_thread(void *data)
char buf[TASK_COMM_LEN];
DEFINE_WAIT(wait);
+ /* offload context creation failed, just exit */
+ if (!current->io_uring)
+ goto err_out;
+
snprintf(buf, sizeof(buf), "iou-sqp-%d", sqd->task_pid);
set_task_comm(current, buf);
@@ -393,7 +397,7 @@ static int io_sq_thread(void *data)
atomic_or(IORING_SQ_NEED_WAKEUP, &ctx->rings->sq_flags);
io_run_task_work();
mutex_unlock(&sqd->lock);
-
+err_out:
io_sq_data_release(sqd);
do_exit(0);
}
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 12/19] io_uring/sqpoll: ensure that normal task_work is also run timely
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (10 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 11/19] io_uring/sqpoll: early exit thread if task_context wasn't allocated Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 13/19] io_uring: signal SQPOLL task_work with TWA_SIGNAL_NO_IPI Jens Axboe
` (6 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
Commit d13ddd9c893f0e8498526bf88c6b5fad01f0edd8 upstream.
With the move to private task_work, SQPOLL neglected to also run the
normal task_work, if any is pending. This will eventually get run, but
we should run it with the private task_work to ensure that things like
a final fput() is processed in a timely fashion.
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/all/313824bc-799d-414f-96b7-e6de57c7e21d@gmail.com/
Reported-by: Andrew Udvare <audvare@gmail.com>
Fixes: af5d68f8892f ("io_uring/sqpoll: manage task_work privately")
Tested-by: Christian Heusel <christian@heusel.eu>
Tested-by: Andrew Udvare <audvare@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/sqpoll.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c
index 3849252f6135..b1428f91b5ee 100644
--- a/io_uring/sqpoll.c
+++ b/io_uring/sqpoll.c
@@ -253,11 +253,13 @@ static unsigned int io_sq_tw(struct llist_node **retry_list, int max_entries)
if (*retry_list) {
*retry_list = io_handle_tw_list(*retry_list, &count, max_entries);
if (count >= max_entries)
- return count;
+ goto out;
max_entries -= count;
}
-
*retry_list = tctx_task_work_run(tctx, max_entries, &count);
+out:
+ if (task_work_pending(current))
+ task_work_run();
return count;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 13/19] io_uring: signal SQPOLL task_work with TWA_SIGNAL_NO_IPI
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (11 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 12/19] io_uring/sqpoll: ensure that normal task_work is also run timely Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 14/19] io_uring/sqpoll: ensure task state is TASK_RUNNING when running task_work Jens Axboe
` (5 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
Commit dbcabac138fdfc730ba458ed2199ff1f29a271fc upstream.
Before SQPOLL was transitioned to managing its own task_work, the core
used TWA_SIGNAL_NO_IPI to ensure that task_work was processed. If not,
we can't be sure that all task_work is processed at SQPOLL thread exit
time.
Fixes: af5d68f8892f ("io_uring/sqpoll: manage task_work privately")
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io_uring.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 895011a3d8ce..4c88bd805b8c 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1412,8 +1412,8 @@ static void io_req_normal_work_add(struct io_kiocb *req)
if (ctx->flags & IORING_SETUP_SQPOLL) {
struct io_sq_data *sqd = ctx->sq_data;
- if (wq_has_sleeper(&sqd->wait))
- wake_up(&sqd->wait);
+ if (sqd->thread)
+ __set_notify_signal(sqd->thread);
return;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 14/19] io_uring/sqpoll: ensure task state is TASK_RUNNING when running task_work
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (12 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 13/19] io_uring: signal SQPOLL task_work with TWA_SIGNAL_NO_IPI Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 15/19] io_uring: don't touch sqd->thread off tw add Jens Axboe
` (4 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
Commit 8f7033aa4089fbaf7a33995f0f2ee6c9d7b9ca1b upstream.
When the sqpoll is exiting and cancels pending work items, it may need
to run task_work. If this happens from within io_uring_cancel_generic(),
then it may be under waiting for the io_uring_task waitqueue. This
results in the below splat from the scheduler, as the ring mutex may be
attempted grabbed while in a TASK_INTERRUPTIBLE state.
Ensure that the task state is set appropriately for that, just like what
is done for the other cases in io_run_task_work().
do not call blocking ops when !TASK_RUNNING; state=1 set at [<0000000029387fd2>] prepare_to_wait+0x88/0x2fc
WARNING: CPU: 6 PID: 59939 at kernel/sched/core.c:8561 __might_sleep+0xf4/0x140
Modules linked in:
CPU: 6 UID: 0 PID: 59939 Comm: iou-sqp-59938 Not tainted 6.12.0-rc3-00113-g8d020023b155 #7456
Hardware name: linux,dummy-virt (DT)
pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
pc : __might_sleep+0xf4/0x140
lr : __might_sleep+0xf4/0x140
sp : ffff80008c5e7830
x29: ffff80008c5e7830 x28: ffff0000d93088c0 x27: ffff60001c2d7230
x26: dfff800000000000 x25: ffff0000e16b9180 x24: ffff80008c5e7a50
x23: 1ffff000118bcf4a x22: ffff0000e16b9180 x21: ffff0000e16b9180
x20: 000000000000011b x19: ffff80008310fac0 x18: 1ffff000118bcd90
x17: 30303c5b20746120 x16: 74657320313d6574 x15: 0720072007200720
x14: 0720072007200720 x13: 0720072007200720 x12: ffff600036c64f0b
x11: 1fffe00036c64f0a x10: ffff600036c64f0a x9 : dfff800000000000
x8 : 00009fffc939b0f6 x7 : ffff0001b6327853 x6 : 0000000000000001
x5 : ffff0001b6327850 x4 : ffff600036c64f0b x3 : ffff8000803c35bc
x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000e16b9180
Call trace:
__might_sleep+0xf4/0x140
mutex_lock+0x84/0x124
io_handle_tw_list+0xf4/0x260
tctx_task_work_run+0x94/0x340
io_run_task_work+0x1ec/0x3c0
io_uring_cancel_generic+0x364/0x524
io_sq_thread+0x820/0x124c
ret_from_fork+0x10/0x20
Cc: stable@vger.kernel.org
Fixes: af5d68f8892f ("io_uring/sqpoll: manage task_work privately")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io_uring.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h
index 0d749588ab21..cdb35cb651da 100644
--- a/io_uring/io_uring.h
+++ b/io_uring/io_uring.h
@@ -361,6 +361,7 @@ static inline int io_run_task_work(void)
if (current->io_uring) {
unsigned int count = 0;
+ __set_current_state(TASK_RUNNING);
tctx_task_work_run(current->io_uring, UINT_MAX, &count);
if (count)
ret = true;
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 15/19] io_uring: don't touch sqd->thread off tw add
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (13 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 14/19] io_uring/sqpoll: ensure task state is TASK_RUNNING when running task_work Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 16/19] io_uring/sqpoll: zero sqd->thread on tctx errors Jens Axboe
` (3 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
From: Pavel Begunkov <asml.silence@gmail.com>
Commit bd2703b42decebdcddf76e277ba76b4c4a142d73 upstream.
With IORING_SETUP_SQPOLL all requests are created by the SQPOLL task,
which means that req->task should always match sqd->thread. Since
accesses to sqd->thread should be separately protected, use req->task
in io_req_normal_work_add() instead.
Note, in the eyes of io_req_normal_work_add(), the SQPOLL task struct
is always pinned and alive, and sqd->thread can either be the task or
NULL. It's only problematic if the compiler decides to reload the value
after the null check, which is not so likely.
Cc: stable@vger.kernel.org
Cc: Bui Quang Minh <minhquangbui99@gmail.com>
Reported-by: lizetao <lizetao1@huawei.com>
Fixes: 78f9b61bd8e54 ("io_uring: wake SQPOLL task when task_work is added to an empty queue")
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/1cbbe72cf32c45a8fee96026463024cd8564a7d7.1736541357.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
[axboe: 6.6 has no tctx->task, use req->task like the commit message
says. With SQPOLL all requests are allocated by the SQPOLL task, so
req->task is that task]
---
io_uring/io_uring.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 4c88bd805b8c..3108af4514eb 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1410,10 +1410,7 @@ static void io_req_normal_work_add(struct io_kiocb *req)
/* SQPOLL doesn't need the task_work added, it'll run it itself */
if (ctx->flags & IORING_SETUP_SQPOLL) {
- struct io_sq_data *sqd = ctx->sq_data;
-
- if (sqd->thread)
- __set_notify_signal(sqd->thread);
+ __set_notify_signal(req->task);
return;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 16/19] io_uring/sqpoll: zero sqd->thread on tctx errors
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (14 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 15/19] io_uring: don't touch sqd->thread off tw add Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 17/19] io_uring/sqpoll: Increase task_work submission batch size Jens Axboe
` (2 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
From: Pavel Begunkov <asml.silence@gmail.com>
Commit 4b7cfa8b6c28a9fa22b86894166a1a34f6d630ba upstream.
Syzkeller reports:
BUG: KASAN: slab-use-after-free in thread_group_cputime+0x409/0x700 kernel/sched/cputime.c:341
Read of size 8 at addr ffff88803578c510 by task syz.2.3223/27552
Call Trace:
<TASK>
...
kasan_report+0x143/0x180 mm/kasan/report.c:602
thread_group_cputime+0x409/0x700 kernel/sched/cputime.c:341
thread_group_cputime_adjusted+0xa6/0x340 kernel/sched/cputime.c:639
getrusage+0x1000/0x1340 kernel/sys.c:1863
io_uring_show_fdinfo+0xdfe/0x1770 io_uring/fdinfo.c:197
seq_show+0x608/0x770 fs/proc/fd.c:68
...
That's due to sqd->task not being cleared properly in cases where
SQPOLL task tctx setup fails, which can essentially only happen with
fault injection to insert allocation errors.
Cc: stable@vger.kernel.org
Fixes: 1251d2025c3e1 ("io_uring/sqpoll: early exit thread if task_context wasn't allocated")
Reported-by: syzbot+3d92cfcfa84070b0a470@syzkaller.appspotmail.com
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/efc7ec7010784463b2e7466d7b5c02c2cb381635.1736519461.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/sqpoll.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c
index b1428f91b5ee..bde67560ace8 100644
--- a/io_uring/sqpoll.c
+++ b/io_uring/sqpoll.c
@@ -280,8 +280,12 @@ static int io_sq_thread(void *data)
DEFINE_WAIT(wait);
/* offload context creation failed, just exit */
- if (!current->io_uring)
+ if (!current->io_uring) {
+ mutex_lock(&sqd->lock);
+ sqd->thread = NULL;
+ mutex_unlock(&sqd->lock);
goto err_out;
+ }
snprintf(buf, sizeof(buf), "iou-sqp-%d", sqd->task_pid);
set_task_comm(current, buf);
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 17/19] io_uring/sqpoll: Increase task_work submission batch size
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (15 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 16/19] io_uring/sqpoll: zero sqd->thread on tctx errors Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 18/19] io_uring: fix task_work add use-after-free with SQPOLL Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 19/19] io_uring/sqpoll: do the return-to-userspace work in the SQPOLL thread Jens Axboe
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
From: Gabriel Krisman Bertazi <krisman@suse.de>
Commit 92835cebab120f8a5f023a26a792a2ac3f816c4f upstream.
Our QA team reported a 10%-23%, throughput reduction on an io_uring
sqpoll testcase doing IO to a null_blk, that I traced back to a
reduction of the device submission queue depth utilization. It turns out
that, after commit af5d68f8892f ("io_uring/sqpoll: manage task_work
privately"), we capped the number of task_work entries that can be
completed from a single spin of sqpoll to only 8 entries, before the
sqpoll goes around to (potentially) sleep. While this cap doesn't drive
the submission side directly, it impacts the completion behavior, which
affects the number of IO queued by fio per sqpoll cycle on the
submission side, and io_uring ends up seeing less ios per sqpoll cycle.
As a result, block layer plugging is less effective, and we see more
time spent inside the block layer in profilings charts, and increased
submission latency measured by fio.
There are other places that have increased overhead once sqpoll sleeps
more often, such as the sqpoll utilization calculation. But, in this
microbenchmark, those were not representative enough in perf charts, and
their removal didn't yield measurable changes in throughput. The major
overhead comes from the fact we plug less, and less often, when submitting
to the block layer.
My benchmark is:
fio --ioengine=io_uring --direct=1 --iodepth=128 --runtime=300 --bs=4k \
--invalidate=1 --time_based --ramp_time=10 --group_reporting=1 \
--filename=/dev/nullb0 --name=RandomReads-direct-nullb-sqpoll-4k-1 \
--rw=randread --numjobs=1 --sqthread_poll
In one machine, tested on top of Linux 6.15-rc1, we have the following
baseline:
READ: bw=4994MiB/s (5236MB/s), 4994MiB/s-4994MiB/s (5236MB/s-5236MB/s), io=439GiB (471GB), run=90001-90001msec
With this patch:
READ: bw=5762MiB/s (6042MB/s), 5762MiB/s-5762MiB/s (6042MB/s-6042MB/s), io=506GiB (544GB), run=90001-90001msec
which is a 15% improvement in measured bandwidth. The average
submission latency is noticeably lowered too. As measured by
fio:
Baseline:
lat (usec): min=20, max=241, avg=99.81, stdev=3.38
Patched:
lat (usec): min=26, max=226, avg=86.48, stdev=4.82
If we look at blktrace, we can also see the plugging behavior is
improved. In the baseline, we end up limited to plugging 8 requests in
the block layer regardless of the device queue depth size, while after
patching we can drive more io, and we manage to utilize the full device
queue.
In the baseline, after a stabilization phase, an ordinary submission
looks like:
254,0 1 49942 0.016028795 5977 U N [iou-sqp-5976] 7
After patching, I see consistently more requests per unplug.
254,0 1 4996 0.001432872 3145 U N [iou-sqp-3144] 32
Ideally, the cap size would at least be the deep enough to fill the
device queue, but we can't predict that behavior, or assume all IO goes
to a single device, and thus can't guess the ideal batch size. We also
don't want to let the tw run unbounded, though I'm not sure it would
really be a problem. Instead, let's just give it a more sensible value
that will allow for more efficient batching. I've tested with different
cap values, and initially proposed to increase the cap to 1024. Jens
argued it is too big of a bump and I observed that, with 32, I'm no
longer able to observe this bottleneck in any of my machines.
Fixes: af5d68f8892f ("io_uring/sqpoll: manage task_work privately")
Signed-off-by: Gabriel Krisman Bertazi <krisman@suse.de>
Link: https://lore.kernel.org/r/20250508181203.3785544-1-krisman@suse.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/sqpoll.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c
index bde67560ace8..3d0fbae80938 100644
--- a/io_uring/sqpoll.c
+++ b/io_uring/sqpoll.c
@@ -19,7 +19,7 @@
#include "sqpoll.h"
#define IORING_SQPOLL_CAP_ENTRIES_VALUE 8
-#define IORING_TW_CAP_ENTRIES_VALUE 8
+#define IORING_TW_CAP_ENTRIES_VALUE 32
enum {
IO_SQ_THREAD_SHOULD_STOP = 0,
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 18/19] io_uring: fix task_work add use-after-free with SQPOLL
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (16 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 17/19] io_uring/sqpoll: Increase task_work submission batch size Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 19/19] io_uring/sqpoll: do the return-to-userspace work in the SQPOLL thread Jens Axboe
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
Commit a92193c91e8839fe5fc209616ea95465ae4b919d upstream.
For SQPOLL, the sqpoll thread runs task_work off its own list and
doesn't need to be told about it, so it can pop and complete a request
as soon as mpscq_push() has published it. If that was the last request
holding the ring alive, io_ring_exit_work() then frees the ctx, stops
the thread, and with it the tctx, while io_req_normal_work_add() is
still looking at both after the push:
BUG: KASAN: slab-use-after-free in io_req_normal_work_add+0x439/0x510
Read of size 4 at addr ff11000000ca0000 by task repro/55
(...)
BUG: KASAN: slab-use-after-free in queue_work_on+0x25/0x70
Write of size 8 at addr ff11000000c3bd00 by task repro/55
Hold an RCU read lock across the add, like io_req_local_work_add() already
does, and have io_ring_exit_work() wait for a grace period for SQPOLL rings
as well before freeing the ctx.
Reported-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Link: https://lore.kernel.org/io-uring/20260924205056.3759980-2-Jeremy.Jean@oss.cyber.gouv.fr/
Fixes: af5d68f8892f ("io_uring/sqpoll: manage task_work privately")
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
[axboe: 6.6 has io_req_normal_work_add() in io_uring.c and adds with
llist_add(), not mpscq_push()]
---
io_uring/io_uring.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 3108af4514eb..fda288405e44 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1401,6 +1401,9 @@ static void io_req_normal_work_add(struct io_kiocb *req)
struct io_uring_task *tctx = req->task->io_uring;
struct io_ring_ctx *ctx = req->ctx;
+ /* SQPOLL can retire the request on push, see io_ring_exit_work() */
+ guard(rcu)();
+
/* task_work already pending, we're done */
if (!llist_add(&req->io_task_work.node, &tctx->task_list))
return;
@@ -3369,8 +3372,8 @@ static __cold void io_ring_exit_work(struct work_struct *work)
spin_lock(&ctx->completion_lock);
spin_unlock(&ctx->completion_lock);
- /* pairs with RCU read section in io_req_local_work_add() */
- if (ctx->flags & IORING_SETUP_DEFER_TASKRUN)
+ /* pairs with the RCU read sections in the task_work add paths */
+ if (ctx->flags & (IORING_SETUP_DEFER_TASKRUN | IORING_SETUP_SQPOLL))
synchronize_rcu();
io_ring_ctx_free(ctx);
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH 6.6.y 19/19] io_uring/sqpoll: do the return-to-userspace work in the SQPOLL thread
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
` (17 preceding siblings ...)
2026-11-21 21:30 ` [PATCH 6.6.y 18/19] io_uring: fix task_work add use-after-free with SQPOLL Jens Axboe
@ 2026-11-21 21:30 ` Jens Axboe
18 siblings, 0 replies; 20+ messages in thread
From: Jens Axboe @ 2026-11-21 21:30 UTC (permalink / raw)
To: stable
From: Shakeel Butt <shakeel.butt@linux.dev>
Commit f013b695506491c5202a412a6568313549c81f50 upstream.
The SQPOLL thread never returns to userspace. io_sq_tw() runs its
task_work, but the rest of resume_user_mode_work(), like memory.high
reclaim and block cgroup throttling, only runs when the thread exits.
So memory.high reclaim only happens inside a charge, with sqd->lock and
ctx->uring_lock held, and not at all if the charges can't block.
This used to work. The loop called io_run_task_work(), which does this
work since commit b5d3ae202fbf ("io_uring: handle TIF_NOTIFY_RESUME
when checking for task_work"). Commit af5d68f8892f ("io_uring/sqpoll:
manage task_work privately") replaced that call with io_sq_tw().
Call resume_user_mode_work() at the top of the loop when
TIF_NOTIFY_RESUME is set, with sqd->lock dropped. It runs once per pass,
so a pass that charges two or more batches still reclaims inside the
charge, as before.
Tested in a VM with memory.high=64M and swap on zram. An SQPOLL ring
reads 256M from /dev/zero into untouched memory, 64K at a time. Before,
the thread made 337 calls to __mem_cgroup_handle_over_high(), all from
inside the charge. After, it makes 704, all from the top of the loop.
memory.peak stays at 64M. With 1M reads, 296 of its 385 calls still come
from inside the charge. Both runs take about 50% longer, and the extra
time is in the handler, which now runs more often. Without memory.high
pressure, the test never calls resume_user_mode_work() and the times
do not change. Lockdep showed no warnings, also with io_uring_register()
racing for sqd->lock.
Fixes: af5d68f8892f ("io_uring/sqpoll: manage task_work privately")
Cc: stable@vger.kernel.org
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Link: https://patch.msgid.link/20261101022146.309201-1-shakeel.butt@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/sqpoll.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c
index 3d0fbae80938..263da37d45ee 100644
--- a/io_uring/sqpoll.c
+++ b/io_uring/sqpoll.c
@@ -318,6 +318,18 @@ static int io_sq_thread(void *data)
timeout = jiffies + sqd->sq_thread_idle;
}
+ /*
+ * The SQPOLL thread never returns to userspace, so do the
+ * return-to-userspace work, like memory.high reclaim, here.
+ * It can sleep for a long time, so drop sqd->lock.
+ */
+ if (test_thread_flag(TIF_NOTIFY_RESUME)) {
+ mutex_unlock(&sqd->lock);
+ resume_user_mode_work(NULL);
+ mutex_lock(&sqd->lock);
+ sqd->sq_cpu = raw_smp_processor_id();
+ }
+
cap_entries = !list_is_singular(&sqd->ctx_list);
list_for_each_entry(ctx, &sqd->ctx_list, sqd_list) {
int ret = __io_sq_thread(ctx, cap_entries);
--
2.55.0
^ permalink raw reply related [flat|nested] 20+ messages in thread