* Queued patches for 6.12-stable
@ 2026-10-28 13:51 Jens Axboe
0 siblings, 0 replies; only message in thread
From: Jens Axboe @ 2026-10-28 13:51 UTC (permalink / raw)
To: Greg Kroah-Hartman; +Cc: stable
[-- Attachment #1: Type: text/plain, Size: 79 bytes --]
Hi,
Like the 6.1 series, but for 6.12-stable.
Please apply!
--
Jens Axboe
[-- Attachment #2: 0038-io_uring-waitid-prevent-cancel-reference-wrap.patch --]
[-- Type: text/x-patch, Size: 2414 bytes --]
From d6a3e0af7898aea4baebc16275eaa8f2aed68b95 Mon Sep 17 00:00:00 2001
From: Junyeong Lee <jylee5498@gmail.com>
Date: Tue, 20 Oct 2026 00:21:00 +0900
Subject: [PATCH 38/38] io_uring/waitid: prevent cancel reference wrap
Commit 98787d3fa4e99564d2b2bc21255b21f670c450d0 upstream.
io_waitid packs the cancellation state into bit 31 of ->refs and uses
bits 30:0 as a 31-bit reference counter:
#define IO_WAITID_CANCEL_FLAG BIT(31)
#define IO_WAITID_REF_MASK GENMASK(30, 0)
__io_waitid_cancel() marks a request canceled, then increments the low
bits of ->refs to claim completion ownership. When another path owns the
request, the increment remains even though cancellation returns false.
The request remains discoverable by cancellation, so repeated failed
attempts can wrap the 31-bit reference field to zero. A later
cancellation can claim ownership and complete the request a second time.
Use atomic_fetch_or() when setting the cancel flag and inspect the old
flag value. The first cancel attempt continues to the existing ownership
increment, preserving its wake and issuer handoff semantics. Later
attempts see the cancel flag and return without incrementing the
reference field.
This makes repeated failed cancellation attempts idempotent while
leaving the first-cancel state transitions unchanged.
Fixes: f31ecf671ddc ("io_uring: add IORING_OP_WAITID support")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Junyeong Lee <jylee5498@gmail.com>
Link: https://patch.msgid.link/20261019152100.63565-1-jylee5498@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/waitid.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/io_uring/waitid.c b/io_uring/waitid.c
index c0997432aadb..8811b5e275cc 100644
--- a/io_uring/waitid.c
+++ b/io_uring/waitid.c
@@ -161,9 +161,12 @@ static bool __io_waitid_cancel(struct io_ring_ctx *ctx, struct io_kiocb *req,
/*
* Mark us canceled regardless of ownership. This will prevent a
- * potential retry from a spurious wakeup.
+ * potential retry from a spurious wakeup. Only the first cancel
+ * attempt should take a reference for the ownership handoff.
*/
- atomic_or(IO_WAITID_CANCEL_FLAG, &iw->refs);
+ if (atomic_fetch_or(IO_WAITID_CANCEL_FLAG, &iw->refs) &
+ IO_WAITID_CANCEL_FLAG)
+ return false;
/* claim ownership */
if (atomic_fetch_inc(&iw->refs) & IO_WAITID_REF_MASK)
--
2.56.0
[-- Attachment #3: 0037-io_uring-wait-wake-real-time-waiters-from-hard-inter.patch --]
[-- Type: text/x-patch, Size: 3850 bytes --]
From 69300c6989f4d69b0db732f77064e7b318806b9a Mon Sep 17 00:00:00 2001
From: Austin Schuh <austin.linux@gmail.com>
Date: Sat, 17 Oct 2026 10:17:26 -0700
Subject: [PATCH 37/38] io_uring/wait: wake real-time waiters from hard
interrupt context
Commit 5b01d5955ac206e38097f777eff6ab2a8b948c41 upstream.
Since commit cebf123c634a ("io_uring: implement our own schedule timeout
handling"), a CQ wait with a timeout sleeps on io_uring's own hrtimer
instead of an hrtimer sleeper. On PREEMPT_RT, an hrtimer that isn't
marked for hard interrupt expiry expires in the per-CPU ktimers thread,
which runs at SCHED_FIFO 1. A real-time task waiting in io_uring_enter()
is therefore not woken at its deadline while any other real-time task is
runnable on its CPU: the wakeup waits until every one of them has blocked.
hrtimer sleepers, which schedule_hrtimeout_range_clock() used to give
io_uring, don't have this problem: __hrtimer_setup_sleeper() marks the
timer hard when the sleeping task is in a real-time or deadline class.
Both of io_uring's callbacks are fine in hard interrupt context:
io_cqring_timer_wakeup() sets a flag and calls wake_up_process(), and
io_cqring_min_timer_wakeup() only reads ring state, under RCU, before
doing the same or re-arming the timer. Apply the sleeper rule to them.
Measured on a Jetson Orin Nano running 6.18.39-rt with this change
backported: a SCHED_FIFO 40 task waits in io_uring_enter() for 40,000
deadlines 500us apart, while a SCHED_FIFO 20 task runs bursts of up to
1ms on the same CPU. Before, 6,647 wakeups were more than 100us late,
the latest by 1,040us, and 2,758 deadlines passed without a wakeup at
all. After, none were more than 100us late and the latest was 18us.
This mirrors epoll: epoll_pwait2() waits on an hrtimer sleeper, so its
timeout is already hard for real-time tasks, and on the same setup it
was never more than 17us late.
[ 6.12: the wait code is still in io_uring.c, hrtimer_init_on_stack() API ]
Fixes: cebf123c634a ("io_uring: implement our own schedule timeout handling")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Austin Schuh <austin.linux@gmail.com>
Link: https://patch.msgid.link/20261017171726.601443-1-austin.linux@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io_uring.c | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index a12cbc61b463..67868b00ecba 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -48,6 +48,7 @@
#include <linux/uio.h>
#include <linux/bits.h>
+#include <linux/sched/rt.h>
#include <linux/sched/signal.h>
#include <linux/fs.h>
#include <linux/file.h>
@@ -2495,12 +2496,21 @@ static enum hrtimer_restart io_cqring_min_timer_wakeup(struct hrtimer *timer)
return io_cqring_timer_wakeup(timer);
}
+/* Same policy as for hrtimer sleepers, see __hrtimer_setup_sleeper(). */
+static enum hrtimer_mode io_cqring_timer_mode(void)
+{
+ if (IS_ENABLED(CONFIG_PREEMPT_RT) && rt_or_dl_task_policy(current))
+ return HRTIMER_MODE_ABS_HARD;
+ return HRTIMER_MODE_ABS;
+}
+
static int io_cqring_schedule_timeout(struct io_wait_queue *iowq,
clockid_t clock_id, ktime_t start_time)
{
+ enum hrtimer_mode mode = io_cqring_timer_mode();
ktime_t timeout;
- hrtimer_init_on_stack(&iowq->t, clock_id, HRTIMER_MODE_ABS);
+ hrtimer_init_on_stack(&iowq->t, clock_id, mode);
if (iowq->min_timeout) {
timeout = ktime_add_ns(iowq->min_timeout, start_time);
iowq->t.function = io_cqring_min_timer_wakeup;
@@ -2510,7 +2520,7 @@ static int io_cqring_schedule_timeout(struct io_wait_queue *iowq,
}
hrtimer_set_expires_range_ns(&iowq->t, timeout, 0);
- hrtimer_start_expires(&iowq->t, HRTIMER_MODE_ABS);
+ hrtimer_start_expires(&iowq->t, mode);
if (!READ_ONCE(iowq->hit_timeout))
schedule();
--
2.56.0
[-- Attachment #4: 0036-io_uring-net-keep-recv-buffer-select-length-within-M.patch --]
[-- Type: text/x-patch, Size: 1671 bytes --]
From 8b1f75b91aebac6eb49d9ef8f89fca19f5a65bc8 Mon Sep 17 00:00:00 2001
From: Tamaki Yanagawa <ty@000ty.net>
Date: Sun, 18 Oct 2026 04:22:32 +0000
Subject: [PATCH 36/38] io_uring/net: keep recv buffer select length within
MAX_RW_COUNT
Commit 96262cfab554cf11412de7a6247f0e82c94e130f upstream.
Since commit dcbd1c054848 ("io_uring/net: let io_recv_buf_select return
the length of the buffer region"), io_recv_buf_select() returns the
selected length as an int. For a non-bundle recv with sqe->len == 0 the
length comes from the ring provided buffer, which can be up to 4G, so a
buffer of 2G or more makes io_recv() fail the request with a bogus
negative result (e.g. -2147483648) instead of receiving into it. Clamp
the length to MAX_RW_COUNT, which import_ubuf() limits the transfer to
anyway.
Fixes: dcbd1c054848 ("io_uring/net: let io_recv_buf_select return the length of the buffer region")
Cc: stable@vger.kernel.org
Signed-off-by: Tamaki Yanagawa <ty@000ty.net>
Assisted-by: Claude:claude-opus-5-5
Link: https://patch.msgid.link/20261018042227.1105-1-ty@000ty.net
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/net.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/io_uring/net.c b/io_uring/net.c
index 2c4ba1d833d4..b7f7c5fa0bf8 100644
--- a/io_uring/net.c
+++ b/io_uring/net.c
@@ -1188,6 +1188,8 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg
*sel = io_buffer_select(req, &len, issue_flags);
if (!sel->addr)
return -ENOBUFS;
+ /* ring provided buffers can be up to 4G, keep len an int */
+ len = min_t(size_t, len, MAX_RW_COUNT);
sr->buf = sel->addr;
sr->len = len;
map_ubuf:
--
2.56.0
[-- Attachment #5: 0035-io_uring-poll-don-t-loop-on-HUP-with-multiple-refere.patch --]
[-- Type: text/x-patch, Size: 2675 bytes --]
From 808e376bfc4a691ce07ee485d4d34813228d08be Mon Sep 17 00:00:00 2001
From: Jens Axboe <axboe@kernel.dk>
Date: Fri, 16 Oct 2026 09:15:13 -0600
Subject: [PATCH 35/38] io_uring/poll: don't loop on HUP with multiple
references
Commit cf1d5c9d335e2c673811da641531da82cb01393a upstream.
io_poll_check_events() drops down to one reference when a multishot
request sees HUP with more than one, so that the loop goes around once
more. It did so by subtracting one less than it had. With the cmpxchg
that only takes the local count down: the exchange fails, the next pass
reads the unchanged count, and goes through the same motions again. The
loop then only ends once io_poll_issue() stops the request.
Drop the extra references from the counter instead, and loop once more
without trying the exchange. The next pass sees one reference, like it
did before, and ends unless a wake up came in.
Fixes: 937d38dcacd6 ("io_uring/poll: avoid retry wakeup loss when dropping ownership")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/poll.c | 16 ++++++++++++----
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/io_uring/poll.c b/io_uring/poll.c
index 355a4467f427..31454c27072b 100644
--- a/io_uring/poll.c
+++ b/io_uring/poll.c
@@ -264,12 +264,14 @@ static inline void io_poll_execute(struct io_kiocb *req, int res,
*/
static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts)
{
+ bool again;
int v;
if (unlikely(io_should_terminate_tw(req->ctx)))
return -ECANCELED;
do {
+ again = false;
v = atomic_read(&req->poll_refs);
if (unlikely(v != 1)) {
@@ -331,9 +333,15 @@ static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts)
} else {
int ret;
- /* multiple refs and HUP, ensure we loop once more */
- if ((req->cqe.res & (POLLHUP | POLLRDHUP)) && v != 1)
- v--;
+ /*
+ * Multiple refs and HUP, ensure we loop once more. Drop
+ * down to one ref now, so the next pass ends unless a
+ * wake up came in.
+ */
+ if ((req->cqe.res & (POLLHUP | POLLRDHUP)) && v != 1) {
+ atomic_sub(v - 1, &req->poll_refs);
+ again = true;
+ }
ret = io_poll_issue(req, ts);
if (ret == IOU_STOP_MULTISHOT)
return IOU_POLL_REMOVE_POLL_USE_RES;
@@ -351,7 +359,7 @@ static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts)
* task_work while we were executing it.
*/
v &= IO_POLL_REF_MASK;
- } while (atomic_cmpxchg(&req->poll_refs, v, 0) != v);
+ } while (again || atomic_cmpxchg(&req->poll_refs, v, 0) != v);
io_napi_add(req);
return IOU_POLL_NO_ACTION;
--
2.56.0
[-- Attachment #6: 0034-io_uring-timeout-don-t-lose-a-cancel-as-io-wq-arms-a.patch --]
[-- Type: text/x-patch, Size: 3884 bytes --]
From b89aa93c4baa4cc4ca056d68224627f9730d33d3 Mon Sep 17 00:00:00 2001
From: Jens Axboe <axboe@kernel.dk>
Date: Thu, 15 Oct 2026 16:38:45 -0600
Subject: [PATCH 34/38] io_uring/timeout: don't lose a cancel as io-wq arms a
timeout
Commit 89f193dc3ef462059387e9691bf6af4b1808379a upstream.
An IORING_OP_TIMEOUT run by io-wq, because of IOSQE_ASYNC, has the same
problem as poll, futex and waitid. io_wq_submit_work() looks at
IO_WQ_WORK_CANCEL once, before it issues the request. A cancel in between
flags the work, gets -EALREADY, and searches ->timeout_list in vain,
while the worker then adds the request and starts the timer:
io-wq worker io_try_cancel()
io_wq_submit_work()
IO_WQ_WORK_CANCEL isn't set
io_async_cancel_one()
sets IO_WQ_WORK_CANCEL
= -EALREADY
io_timeout_cancel() = -ENOENT
io_timeout()
adds the request to ->timeout_list
hrtimer_start()
= IOU_ISSUE_SKIP_COMPLETE
Nobody looks at the flag again, and the timeout runs its course. At ring
exit, io_uring_try_cancel_requests() cancels io-wq work and then kills
the timeouts the same way, so an exiting task can get stuck for the full
duration of such a timeout.
Look at the flag under ->timeout_lock, which the search takes as well,
and fail the request if it is set. If the search came first, the flag is
visible. If it comes later, it finds the request on the list.
[ 6.12: ->timeout_lock is a spinlock here ]
Fixes: ce35a47a3a02 ("io_uring: add IOSQE_ASYNC")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/io_uring.h | 14 ++++++++++----
io_uring/timeout.c | 7 +++++++
2 files changed, 17 insertions(+), 4 deletions(-)
diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h
index 2ca1910c19c2..628ced4b44b2 100644
--- a/io_uring/io_uring.h
+++ b/io_uring/io_uring.h
@@ -274,16 +274,22 @@ static inline void io_ring_submit_lock(struct io_ring_ctx *ctx,
* cancellations find it, rather than completing it. Whoever tried to cancel
* the request before that found it running. It has flagged the work item,
* has searched the list in vain, and won't come back. To be called with
- * ->uring_lock held, which the search needs as well, once the request is on
- * the list: if the search came first, the flag is visible.
+ * the lock the search takes held: if the search came first, the flag is
+ * visible. For most lists that is ->uring_lock, timeouts use ->timeout_lock.
*/
+static inline bool __io_wq_cancel_seen(struct io_kiocb *req,
+ unsigned int issue_flags)
+{
+ return (issue_flags & IO_URING_F_IOWQ) &&
+ (atomic_read(&req->work.flags) & IO_WQ_WORK_CANCEL);
+}
+
static inline bool io_wq_cancel_seen(struct io_kiocb *req,
unsigned int issue_flags)
{
lockdep_assert_held(&req->ctx->uring_lock);
- return (issue_flags & IO_URING_F_IOWQ) &&
- (atomic_read(&req->work.flags) & IO_WQ_WORK_CANCEL);
+ return __io_wq_cancel_seen(req, issue_flags);
}
static inline void io_commit_cqring(struct io_ring_ctx *ctx)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index 4c792dda3042..ef215accd284 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -607,6 +607,13 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
spin_lock_irq(&ctx->timeout_lock);
+ /* the list is searched under ->timeout_lock, see io_wq_cancel_seen() */
+ if (__io_wq_cancel_seen(req, issue_flags)) {
+ spin_unlock_irq(&ctx->timeout_lock);
+ io_req_task_queue_fail(req, -ECANCELED);
+ return IOU_ISSUE_SKIP_COMPLETE;
+ }
+
/*
* sqe->off holds how many events that need to occur for this
* timeout event to be satisfied. If it isn't set, then this is
--
2.56.0
[-- Attachment #7: 0033-io_uring-timeout-don-t-flush-with-the-sequence-behin.patch --]
[-- Type: text/x-patch, Size: 2178 bytes --]
From c76aa4138154a7eedaf502991c773ccf82f24ba1 Mon Sep 17 00:00:00 2001
From: Jens Axboe <axboe@kernel.dk>
Date: Thu, 15 Oct 2026 14:25:00 -0600
Subject: [PATCH 33/38] io_uring/timeout: don't flush with the sequence behind
the last flush
Commit 85f85d573ffa7772e1ad4504d0cec5113a5c5cf1 upstream.
io_flush_timeouts() measures the events since the last flush as the
distance of the current sequence from ->cq_last_tm_flush. A fired
timeout bumps ->cq_timeouts before its CQE is posted, so until it is
the sequence can be behind the last flush, and the distance wraps:
last flush: seq 100, cq_last_tm_flush = 100
Y: IORING_OP_TIMEOUT, off 50, target_seq 150
pure timeout A fires, cq_timeouts = 1
task_work posts A's CQE, cached_cq_tail = 101
pure timeout B fires before the flush, cq_timeouts = 2
io_flush_timeouts(): seq = 99, events_got = 0xffffffff
Y, and every sequence timeout behind it, is ended with -ETIME although
no event happened. Two timeouts expiring close together is all it takes,
a completion with IOSQE_CQE_SKIP_SUCCESS after a single one does it too.
Nothing but those in flight CQEs separates the two, so count from the
last flush when the sequence is behind it, as io_timeout() does.
Fixes: f010505b78a4 ("io_uring: flush timeouts that should already have expired")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/timeout.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index 5657ef75a79a..4c792dda3042 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -113,6 +113,14 @@ __cold void io_flush_timeouts(struct io_ring_ctx *ctx)
spin_lock_irq(&ctx->timeout_lock);
seq = ctx->cached_cq_tail - atomic_read(&ctx->cq_timeouts);
+ /*
+ * A fired timeout bumps ->cq_timeouts before its CQE is posted, which
+ * leaves the sequence behind the last flush until it is. No event has
+ * happened then, see io_timeout().
+ */
+ if ((s32)(seq - ctx->cq_last_tm_flush) < 0)
+ seq = ctx->cq_last_tm_flush;
+
list_for_each_entry_safe(timeout, tmp, &ctx->timeout_list, list) {
struct io_kiocb *req = cmd_to_io_kiocb(timeout);
u32 events_needed, events_got;
--
2.56.0
[-- Attachment #8: 0032-io_uring-timeout-don-t-put-a-sequence-target-behind-.patch --]
[-- Type: text/x-patch, Size: 2205 bytes --]
From 9f2cbef70ddfe51f978a0c02557ec545e01da51a Mon Sep 17 00:00:00 2001
From: Jens Axboe <axboe@kernel.dk>
Date: Thu, 15 Oct 2026 14:18:22 -0600
Subject: [PATCH 32/38] io_uring/timeout: don't put a sequence target behind
the last flush
Commit fdfce905a58edea053ee7b68b5bd4348dd1b36e0 upstream.
io_timeout_fn() and io_kill_timeout() bump ->cq_timeouts when a timeout
fires, but its CQE is only posted later from task_work. Until then the
sequence io_timeout() computes is one below the last flush per fired
timeout. Now that ->cq_last_tm_flush is kept while sequence timeouts are
pending, a timeout armed in that window with a small offset gets a target
behind the base. Its distance from the base wraps, and it's never
completed by events, only by its timer. With DEFER_TASKRUN that window
is as wide as the application makes it, as submission runs before the
local task_work.
Nothing but those in flight CQEs separates the two, so use the last
flush as the base for the target when the sequence is behind it.
Fixes: c6c0b3eba88c ("io_uring/timeout: don't skip over reached sequences")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/timeout.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index 88af7abcf682..5657ef75a79a 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -610,7 +610,6 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
}
tail = data_race(ctx->cached_cq_tail) - atomic_read(&ctx->cq_timeouts);
- timeout->target_seq = tail + off;
/*
* Update the last seq here in case io_flush_timeouts() hasn't, it only
@@ -624,6 +623,15 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
ctx->cq_last_tm_flush = tail;
base = ctx->cq_last_tm_flush;
+ /*
+ * A fired timeout bumps ->cq_timeouts before its CQE is posted, which
+ * leaves the sequence behind the last flush until it is. Nothing else
+ * separates the two then, so count from the flush.
+ */
+ if ((s32)(tail - base) < 0)
+ tail = base;
+ timeout->target_seq = tail + off;
+
/*
* Insertion sort, ensuring the first entry in the list is always
* the one we need first.
--
2.56.0
[-- Attachment #9: 0031-io_uring-poll-avoid-retry-wakeup-loss-when-dropping-.patch --]
[-- Type: text/x-patch, Size: 1635 bytes --]
From 26dea51dd474d62549efe80b80e3f12e2edd2a5a Mon Sep 17 00:00:00 2001
From: Juanlu Herrero <juanlu@fastmail.com>
Date: Tue, 13 Oct 2026 15:37:20 -0700
Subject: [PATCH 31/38] io_uring/poll: avoid retry wakeup loss when dropping
ownership
Commit 937d38dcacd675dc0572d6c9b6743960df3128ed upstream.
If a wakeup sets IO_POLL_RETRY_FLAG between the owning task consuming
the existing instance of IO_POLL_RETRY_FLAG and the loop check
of io_poll_check_events(), then we stall on processing the request
until an unrelated wakeup comes along.
Using atomic_cmpxchg instead of atomic_sub_return to guarantee both the
count v and poll_refs are 0 and no wake-ups are missed.
Cc: stable@vger.kernel.org
Fixes: a26a35e9019f ("io_uring: make poll refs more robust")
Signed-off-by: Juanlu Herrero <juanlu@fastmail.com>
Assisted-by: LLM
Suggested-by: Anand Natarajan <anand@meta.com>
Link: https://patch.msgid.link/20261013-pu-cmpxchg-v1-1-4ddc10fd8a03@fastmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/poll.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/io_uring/poll.c b/io_uring/poll.c
index 2ad12b39f245..355a4467f427 100644
--- a/io_uring/poll.c
+++ b/io_uring/poll.c
@@ -350,7 +350,8 @@ static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts)
* Release all references, retry if someone tried to restart
* task_work while we were executing it.
*/
- } while (atomic_sub_return(v, &req->poll_refs) & IO_POLL_REF_MASK);
+ v &= IO_POLL_REF_MASK;
+ } while (atomic_cmpxchg(&req->poll_refs, v, 0) != v);
io_napi_add(req);
return IOU_POLL_NO_ACTION;
--
2.56.0
[-- Attachment #10: 0030-io_uring-look-up-the-file-before-queueing-to-io-wq.patch --]
[-- Type: text/x-patch, Size: 5521 bytes --]
From b6e99a2631b8fdddc01f51d2d8d7e611e649c49b Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:07 +0000
Subject: [PATCH 30/38] io_uring: look up the file before queueing to io-wq
Commit 98bc044d5a5374f8d3605fe55a8e2df6d6fc3f82 upstream.
io_prep_async_work() decides by the file of a request what kind of work
it is for io-wq. I/O on regular files and block devices is bounded work,
limited to min(SQ entries, 4 * CPUs) workers, for anything else that may
block indefinitely the limit is RLIMIT_NPROC. Writes to a regular file
are hashed by the inode, so that they are run one after the other instead
of having a bunch of workers fight for the inode lock.
Since the file is only assigned when a request is issued, a request that
is queued without an attempt to issue it, REQ_F_FORCE_ASYNC, doesn't have
one at that point. io_prep_async_work() takes that for "not a regular
file or block device":
- Reads and writes end up as unbound work. 32 reads with IOSQE_ASYNC from
a slow block device on a ring with 4 SQ entries get 32 workers, not 4.
- Writes aren't hashed. Four buffered writes to one file get four
workers, three of which do nothing but wait for the lock.
The late assignment is for the benefit of links, where a request may use
a fixed file that one of its predecessors installs. By the time a request
gets to io_queue_iowq() these are done, it is as good as issued. Assign
the file there.
io_assign_file() moves up and is forced inline. With a third caller the
compiler may stop inlining it, and every request that is issued would
pay for a call.
IORING_OP_SPLICE is left as it is, unbound and not hashed. The file of
the request is where it writes to, which says nothing about how long it
takes: a splice from an empty pipe or a socket into a regular file waits
for its input. As bounded work, four of these on a ring with 4 SQ entries
keep all other bounded work of the task from running, and as hashed work
one of them blocks all writes to that file.
REQ_F_FORCE_ASYNC is not only set for IOSQE_ASYNC. What changes, and
brings back what was the case before the file assignment was deferred:
- Requests with IOSQE_ASYNC or IOSQE_IO_DRAIN, and all requests that are
submitted while a drain is active: reads and writes on regular files
and block devices count against the limit for bounded workers, including
the one set with IORING_REGISTER_IOWQ_MAX_WORKERS, so fewer of them run
at the same time.
- Of these, writes to one regular file are run one at a time, unless they
are O_DIRECT and the filesystem has FOP_DIO_PARALLEL_WRITE.
- IORING_OP_FALLOCATE and IORING_OP_FTRUNCATE, which always go to io-wq,
are run one at a time per file, and in turn with these writes.
- The descriptor or the fixed file slot is looked up when the request is
queued, not when a worker gets around to it, so closing or replacing it
in between no longer has an effect on the request.
Fixes: 6bf9c47a3989 ("io_uring: defer file assignment")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-43-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io_uring.c | 43 +++++++++++++++++++++++++++++--------------
1 file changed, 29 insertions(+), 14 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index e24046a37bb6..a12cbc61b463 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -481,17 +481,46 @@ static void io_prep_async_link(struct io_kiocb *req)
}
}
+/* Called for every request that is issued: keep it inline */
+static __always_inline bool io_assign_file(struct io_kiocb *req,
+ const struct io_issue_def *def,
+ unsigned int issue_flags)
+{
+ if (req->file || !def->needs_file)
+ return true;
+
+ if (req->flags & REQ_F_FIXED_FILE)
+ req->file = io_file_get_fixed(req, req->cqe.fd, issue_flags);
+ else
+ req->file = io_file_get_normal(req, req->cqe.fd);
+
+ return !!req->file;
+}
+
static void io_queue_iowq(struct io_kiocb *req)
{
struct io_uring_task *tctx = req->task->io_uring;
BUG_ON(!tctx);
+ lockdep_assert_held(&req->ctx->uring_lock);
if ((current->flags & PF_KTHREAD) || !tctx->io_wq) {
io_req_task_queue_fail(req, -ECANCELED);
return;
}
+ /*
+ * The file is looked up as late as possible, as it may be the result
+ * of an earlier request of the link. This one is up now, and what kind
+ * of work it is depends on the file. If that fails, leave the error
+ * handling to the worker.
+ *
+ * Not for a splice: whatever it writes to, it may wait for its input
+ * for ever. It has to remain unbound work that isn't hashed.
+ */
+ if (req->opcode != IORING_OP_SPLICE)
+ io_assign_file(req, &io_issue_defs[req->opcode], 0);
+
/* init ->work of the whole link before punting */
io_prep_async_link(req);
@@ -1724,20 +1753,6 @@ static __cold void io_drain_req(struct io_kiocb *req)
spin_unlock(&ctx->completion_lock);
}
-static bool io_assign_file(struct io_kiocb *req, const struct io_issue_def *def,
- unsigned int issue_flags)
-{
- if (req->file || !def->needs_file)
- return true;
-
- if (req->flags & REQ_F_FIXED_FILE)
- req->file = io_file_get_fixed(req, req->cqe.fd, issue_flags);
- else
- req->file = io_file_get_normal(req, req->cqe.fd);
-
- return !!req->file;
-}
-
#define REQ_ISSUE_SLOW_FLAGS (REQ_F_CREDS | REQ_F_ARM_LTIMEOUT)
static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)
--
2.56.0
[-- Attachment #11: 0029-io_uring-timeout-don-t-end-a-linked-timeout-with-1.patch --]
[-- Type: text/x-patch, Size: 1911 bytes --]
From c4f5759c5895b01365a03237c5bbd33088f2581c Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:06 +0000
Subject: [PATCH 29/38] io_uring/timeout: don't end a linked timeout with 1
Commit c74193430febdd120e35936dda3e5aeec3a54405 upstream.
A linked timeout that fires cancels the request it is linked to, and
completes with -ETIME if that worked out:
ret = io_try_cancel(req->tctx, &cd, 0);
...
io_req_set_res(req, ret ?: -ETIME, 0);
io_try_cancel() returns 0 or an error, except for requests that are parked
on ->futex_list or ->waitid_list. io_futex_cancel() and io_waitid_cancel()
return the number of requests that they have cancelled, and
io_try_cancel() hands it on.
So the linked timeout of an IORING_OP_FUTEX_WAIT, IORING_OP_FUTEX_WAITV or
IORING_OP_WAITID that it has cancelled completes with 1, each time.
The waitid test of liburing checks for that 1, and has to be changed to
expect -ETIME, which is what the linked timeout of any other request
completes with.
Fixes: f31ecf671ddc ("io_uring: add IORING_OP_WAITID support")
Fixes: 194bb58c6090 ("io_uring: add support for futex wake and wait")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-42-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/timeout.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index 65e3677a8688..88af7abcf682 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -317,6 +317,9 @@ static void io_req_task_link_timeout(struct io_kiocb *req, struct io_tw_state *t
};
ret = io_try_cancel(req->task->io_uring, &cd, 0);
+ /* a number of requests for futex and waitid */
+ if (ret > 0)
+ ret = 0;
}
io_req_set_res(req, ret ?: -ETIME, 0);
io_req_task_complete(req, ts);
--
2.56.0
[-- Attachment #12: 0028-io_uring-timeout-no-ABS-update-of-multishot-timeouts.patch --]
[-- Type: text/x-patch, Size: 3314 bytes --]
From 49e6e77f0c69731fcecac6a88750ea41c1ad0ab3 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:05 +0000
Subject: [PATCH 28/38] io_uring/timeout: no ABS update of multishot timeouts
Commit b088ae34d4da730dc647261b893a6eb622fe5e1f upstream.
IORING_TIMEOUT_MULTISHOT can't be combined with IORING_TIMEOUT_ABS, prep
rejects that, as the request is armed again with the same value each time
it fires. IORING_TIMEOUT_UPDATE doesn't check and takes an absolute time
for a multishot timeout. The first expiry is as asked for, but
io_timeout_complete() then arms the timer with ->mode of the original
request, i.e. it uses the absolute time as the interval. With
CLOCK_MONOTONIC the next expiry is an uptime away, with CLOCK_REALTIME
more than fifty years.
Fail such an update with -EINVAL, and do so before the timeout is
touched.
Fixes: ea97f6c8558e ("io_uring: add support for multishot timeouts")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-41-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/timeout.c | 38 ++++++++++++++++++++++++++------------
1 file changed, 26 insertions(+), 12 deletions(-)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index b81ba01aa2fb..65e3677a8688 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -254,22 +254,29 @@ static enum hrtimer_restart io_timeout_fn(struct hrtimer *timer)
return HRTIMER_NORESTART;
}
+static struct io_kiocb *io_timeout_find(struct io_ring_ctx *ctx,
+ struct io_cancel_data *cd)
+ __must_hold(&ctx->timeout_lock)
+{
+ struct io_timeout *timeout;
+
+ list_for_each_entry(timeout, &ctx->timeout_list, list) {
+ struct io_kiocb *req = cmd_to_io_kiocb(timeout);
+
+ if (io_cancel_req_match(req, cd))
+ return req;
+ }
+ return NULL;
+}
+
static struct io_kiocb *io_timeout_extract(struct io_ring_ctx *ctx,
struct io_cancel_data *cd)
__must_hold(&ctx->timeout_lock)
{
+ struct io_kiocb *req = io_timeout_find(ctx, cd);
struct io_timeout *timeout;
struct io_timeout_data *io;
- struct io_kiocb *req = NULL;
- list_for_each_entry(timeout, &ctx->timeout_list, list) {
- struct io_kiocb *tmp = cmd_to_io_kiocb(timeout);
-
- if (io_cancel_req_match(tmp, cd)) {
- req = tmp;
- break;
- }
- }
if (!req)
return ERR_PTR(-ENOENT);
@@ -400,14 +407,21 @@ static int io_timeout_update(struct io_ring_ctx *ctx, __u64 user_data,
__must_hold(&ctx->timeout_lock)
{
struct io_cancel_data cd = { .ctx = ctx, .data = user_data, };
- struct io_kiocb *req = io_timeout_extract(ctx, &cd);
+ struct io_kiocb *req = io_timeout_find(ctx, &cd);
struct io_timeout *timeout = io_kiocb_to_cmd(req, struct io_timeout);
struct io_timeout_data *data;
- if (IS_ERR(req))
- return PTR_ERR(req);
+ if (!req)
+ return -ENOENT;
data = req->async_data;
+ /* multishot requests only make sense with rel values */
+ if ((data->flags & IORING_TIMEOUT_MULTISHOT) && mode == HRTIMER_MODE_ABS)
+ return -EINVAL;
+
+ req = io_timeout_extract(ctx, &cd);
+ if (IS_ERR(req))
+ return PTR_ERR(req);
/* for a multishot request this isn't a sequence, keep the repeats */
if (!(data->flags & IORING_TIMEOUT_MULTISHOT))
timeout->off = 0; /* noseq */
--
2.56.0
[-- Attachment #13: 0027-io_uring-net-don-t-retry-recv-on-uncommitted-buffers.patch --]
[-- Type: text/x-patch, Size: 4154 bytes --]
From db18fcf0bfdba8d672dd86ee3e71d73a49803e98 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:04 +0000
Subject: [PATCH 27/38] io_uring/net: don't retry recv on uncommitted buffers
Commit 8444de681a179e41a4eb92d5bbe770a0677074a6 upstream.
Commit 41b70df5b38b ("io_uring/net: commit partial buffers on retry")
commits ring provided buffers when a receive is going to be retried after
a partial transfer, so that another request using the same buffer group
can't be handed the memory the first one keeps receiving into. It commits
the number of bytes received so far. That does the job for a single
buffer which is consumed as a whole, but the request hangs on to all it
has mapped:
- For a buffer ring with IOU_PBUF_RING_INC only these bytes are consumed.
The rest of the buffer stays at the head of the ring.
- For a bundle only the buffers which hold these bytes are consumed, the
others that were peeked stay in the ring.
In both cases the next receive with that buffer group gets memory that
the retried request is still going to write to, and whoever comes last
overwrites the data of the other. What the retry fills isn't committed at
all, as the buffer list isn't known any more at that point. So the kernel
hands it out again later on, while the CQE tells the application that it
was used, and for an incremental buffer that it is done with.
The remainder can't be committed in advance: if the request ends up
receiving less, there is no way to tell the application that more was
consumed than the CQE accounts for. Complete the request with what it has
got instead of retrying it. MSG_WAITALL is no longer honoured for
receives into incremental buffers or multi buffer bundles, they can
return a short count, with the request marked as failed like for other
short MSG_WAITALL transfers.
Fixes: 41b70df5b38b ("io_uring/net: commit partial buffers on retry")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-40-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/net.c | 24 ++++++++++++++++++++++--
1 file changed, 22 insertions(+), 2 deletions(-)
diff --git a/io_uring/net.c b/io_uring/net.c
index f738f0da87a5..2c4ba1d833d4 100644
--- a/io_uring/net.c
+++ b/io_uring/net.c
@@ -500,6 +500,24 @@ static int io_bundle_nbufs(struct io_async_msghdr *kmsg, int ret)
return nbufs;
}
+/*
+ * A receive that is retried after a partial transfer keeps writing to what it
+ * has mapped. io_net_kbuf_recyle() only commits what was filled so far, which
+ * takes a buffer out of circulation if it's a single one that is consumed as a
+ * whole. The tail of an incrementally consumed buffer, or the yet unused
+ * buffers of a bundle, would be up for grabs. They can't be committed in
+ * advance either, the CQE has no way to tell if they end up not being used.
+ */
+static bool io_net_kbuf_can_retry(struct io_kiocb *req,
+ struct io_buffer_list *bl,
+ struct io_async_msghdr *kmsg)
+{
+ /* not ring provided, or committed at selection time */
+ if (!(req->flags & REQ_F_BUFFERS_COMMIT))
+ return true;
+ return !(bl->flags & IOBL_INC) && iter_is_ubuf(&kmsg->msg.msg_iter);
+}
+
static int io_net_kbuf_recyle(struct io_kiocb *req, struct io_buffer_list *bl,
struct io_async_msghdr *kmsg, int len)
{
@@ -1086,7 +1104,8 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags)
return IOU_ISSUE_SKIP_COMPLETE;
return -EAGAIN;
}
- if (ret > 0 && io_net_retry(sock, flags)) {
+ if (ret > 0 && io_net_retry(sock, flags) &&
+ io_net_kbuf_can_retry(req, sel.buf_list, kmsg)) {
sr->done_io += ret;
return io_net_kbuf_recyle(req, sel.buf_list, kmsg, ret);
}
@@ -1233,7 +1252,8 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags)
return IOU_ISSUE_SKIP_COMPLETE;
return -EAGAIN;
}
- if (ret > 0 && io_net_retry(sock, flags)) {
+ if (ret > 0 && io_net_retry(sock, flags) &&
+ io_net_kbuf_can_retry(req, sel.buf_list, kmsg)) {
sr->len -= ret;
sr->buf += ret;
sr->done_io += ret;
--
2.56.0
[-- Attachment #14: 0026-io_uring-don-t-pass-restart-error-codes-to-userspace.patch --]
[-- Type: text/x-patch, Size: 4162 bytes --]
From c6c6dac11a4112e2f5791aeeafcd8a101bc1e60d Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:03 +0000
Subject: [PATCH 26/38] io_uring: don't pass restart error codes to userspace
Commit e1a64bc64897f4c2fe460922c345f2ed8f889b5f upstream.
-ERESTARTSYS and friends are instructions for the system call exit code,
which either restarts the call or turns them into -EINTR. They are not
supposed to be seen by applications. A request isn't a system call,
nobody looks at what it returns before that is put into a CQE.
The read/write and the networking opcodes convert these codes to -EINTR
by hand. The others don't. An io-wq worker that is blocked in the open of
a FIFO, a tty or some other device, or in a splice from an empty pipe,
and that gets interrupted because the request is cancelled, completes it
with -512:
IORING_OP_OPENAT, IOSQE_ASYNC // FIFO without a writer
IORING_OP_ASYNC_CANCEL // -EALREADY, worker is signalled
CQE of the open: res == -512
There are more candidates, like whatever a file system or driver returns
from ->flush(), ->fsync(), ->fallocate() or ->uring_cmd(), or breaking a
lease or delegation.
Take care of it in io_issue_sqe(), which all of them go through. The
helper is moved from rw.c as is. The cost is a compare and a branch that
is not taken per request.
Fixes: 15b71abe7b52 ("io_uring: add support for IORING_OP_OPENAT")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-39-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io_uring.c | 5 +++++
io_uring/io_uring.h | 18 ++++++++++++++++++
io_uring/rw.c | 18 ------------------
3 files changed, 23 insertions(+), 18 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 8f064995137c..e24046a37bb6 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1773,6 +1773,9 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)
}
if (ret == IOU_OK) {
+ /* not all handlers take care of the internal restart codes */
+ if (unlikely(req->cqe.res <= -ERESTARTSYS))
+ req->cqe.res = io_fixup_restart_res(req->cqe.res);
if (issue_flags & IO_URING_F_COMPLETE_DEFER)
io_req_complete_defer(req);
else
@@ -1788,6 +1791,8 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)
if ((req->ctx->flags & IORING_SETUP_IOPOLL) && def->iopoll_queue)
io_iopoll_req_issued(req, issue_flags);
}
+ if (unlikely(ret <= -ERESTARTSYS))
+ ret = io_fixup_restart_res(ret);
return ret;
}
diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h
index 91838d9b85f2..2ca1910c19c2 100644
--- a/io_uring/io_uring.h
+++ b/io_uring/io_uring.h
@@ -203,6 +203,24 @@ static __always_inline bool io_fill_cqe_req(struct io_ring_ctx *ctx,
return true;
}
+static inline ssize_t io_fixup_restart_res(ssize_t ret)
+{
+ switch (ret) {
+ case -ERESTARTSYS:
+ case -ERESTARTNOINTR:
+ case -ERESTARTNOHAND:
+ case -ERESTART_RESTARTBLOCK:
+ /*
+ * We can't just restart the syscall, since previously
+ * submitted sqes may already be in progress. Just fail
+ * this IO with EINTR.
+ */
+ return -EINTR;
+ default:
+ return ret;
+ }
+}
+
static inline void req_set_fail(struct io_kiocb *req)
{
req->flags |= REQ_F_FAIL;
diff --git a/io_uring/rw.c b/io_uring/rw.c
index c8a536ce2fe3..b18b10cdbb52 100644
--- a/io_uring/rw.c
+++ b/io_uring/rw.c
@@ -560,24 +560,6 @@ static void io_complete_rw_iopoll(struct kiocb *kiocb, long res)
smp_store_release(&req->iopoll_completed, 1);
}
-static inline ssize_t io_fixup_restart_res(ssize_t ret)
-{
- switch (ret) {
- case -ERESTARTSYS:
- case -ERESTARTNOINTR:
- case -ERESTARTNOHAND:
- case -ERESTART_RESTARTBLOCK:
- /*
- * We can't just restart the syscall, since previously
- * submitted sqes may already be in progress. Just fail
- * this IO with EINTR.
- */
- return -EINTR;
- default:
- return ret;
- }
-}
-
static inline void io_rw_done(struct kiocb *kiocb, ssize_t ret)
{
/* IO was queued async, completion will happen later */
--
2.56.0
[-- Attachment #15: 0025-io_uring-sqpoll-let-the-thread-sleep-on-disabled-rin.patch --]
[-- Type: text/x-patch, Size: 1794 bytes --]
From 99a4feb60c88384122490dc20c64f30684a077dc Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:01 +0000
Subject: [PATCH 25/38] io_uring/sqpoll: let the thread sleep on disabled rings
Commit d492dcbb3b81c1c2d0aabef41fa1307093f3a556 upstream.
__io_sq_thread() doesn't submit anything as long as the ring has
IORING_SETUP_R_DISABLED set. The check that keeps the SQPOLL thread from
going to sleep while the SQ ring holds entries doesn't know about that.
If an application fills in SQEs first and enables the ring later, the
thread finds that there is work each time it is about to sleep, goes
around, doesn't submit it, and so on. It uses up a CPU until the ring
gets enabled.
Entries in the SQ ring of a disabled ring are no reason to stay awake.
io_register_enable_rings() clears the flag and then wakes the thread, and
the flag is looked at after prepare_to_wait(), so the thread can't miss
that.
Fixes: 0298ef969a11 ("io_uring: clean R_DISABLED startup mess")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-37-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/sqpoll.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c
index b39067a049fd..1ea2e35e4ce8 100644
--- a/io_uring/sqpoll.c
+++ b/io_uring/sqpoll.c
@@ -387,6 +387,13 @@ static int io_sq_thread(void *data)
*/
smp_mb__after_atomic();
+ /*
+ * Nothing gets submitted before the ring is
+ * enabled, which comes with a wake up.
+ */
+ if (READ_ONCE(ctx->flags) & IORING_SETUP_R_DISABLED)
+ continue;
+
if (io_sqring_entries(ctx)) {
needs_sched = false;
break;
--
2.56.0
[-- Attachment #16: 0024-io_uring-kbuf-handle-index-wrap-in-last-buffer-check.patch --]
[-- Type: text/x-patch, Size: 1646 bytes --]
From ca8099c994623f2372383d737865bfdfdb7e059c Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:31:00 +0000
Subject: [PATCH 24/38] io_uring/kbuf: handle index wrap in last buffer check
Commit 6e773b5161506a4d0e739eb3aa4667d28f20219c upstream.
io_ring_buffer_select() flags the request with REQ_F_BL_EMPTY if it takes
the last buffer of the ring. Head and tail are 16-bit counters, but
"head + 1" is computed as an int. If head is 65535 and tail has wrapped
to 0, that compares 65536 to 0 and the flag isn't set.
A bundle receive relies on the flag to tell that there is no point in
trying to get more data. Without it, it goes for another round, finds no
buffer, and the request, which has received data, is marked as failed on
account of the -ENOBUFS. That severs a link, and ends a multishot.
Fixes: c3f9109dbc9e ("io_uring/kbuf: flag request if buffer pool is empty after buffer pick")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-36-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/kbuf.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c
index df5ba0d7fbb3..b1c5e80b94fa 100644
--- a/io_uring/kbuf.c
+++ b/io_uring/kbuf.c
@@ -181,7 +181,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len,
if (unlikely(tail == head))
return sel;
- if (head + 1 == tail)
+ if ((__u16)(head + 1) == tail)
req->flags |= REQ_F_BL_EMPTY;
buf = io_ring_head_to_buf(br, head, bl->mask);
--
2.56.0
[-- Attachment #17: 0023-io_uring-napi-don-t-busy-poll-past-an-expired-timeou.patch --]
[-- Type: text/x-patch, Size: 1557 bytes --]
From 9710eff40504c88a22b9d5ad80feae3d553bb837 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:58 +0000
Subject: [PATCH 23/38] io_uring/napi: don't busy poll past an expired timeout
Commit 21ed976c12f6b77d7ea028a4fe9a08e69a56179f upstream.
__io_napi_busy_loop() caps the busy poll time to what is left until the
timeout of the wait. The difference is negative if the timeout has passed
already, which is what an application gets if it uses
IORING_ENTER_ABS_TIMER with a time that isn't in the future any more. The
min_t(u64, ...) turns that into a huge value, so the cap has no effect
and the wait, which should return right away, spins for the full busy
poll time first.
Fixes: d5cce407e4f5 ("io_uring/napi: postpone napi timeout adjustment")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-34-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/napi.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/io_uring/napi.c b/io_uring/napi.c
index a4b1aedf07c8..bdae13367e0f 100644
--- a/io_uring/napi.c
+++ b/io_uring/napi.c
@@ -288,6 +288,8 @@ void __io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq)
if (iowq->timeout != KTIME_MAX) {
ktime_t dt = ktime_sub(iowq->timeout, io_get_time(ctx));
+ /* an absolute timeout may have passed already */
+ dt = max_t(ktime_t, dt, 0);
iowq->napi_busy_poll_dt = min_t(u64, iowq->napi_busy_poll_dt, dt);
}
--
2.56.0
[-- Attachment #18: 0022-io_uring-napi-make-stale-entries-time-out-after-60s.patch --]
[-- Type: text/x-patch, Size: 1576 bytes --]
From 0599d6e46c8bc9bb8f00d7cc7f7daada2ea7e5d9 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:57 +0000
Subject: [PATCH 22/38] io_uring/napi: make stale entries time out after 60s
Commit 9c151648780d89e5446e5f89f3fa2e416c3bb911 upstream.
With dynamic tracking a NAPI id that hasn't been seen for a minute is
meant to be dropped from the list the ring busy polls. The timeout is
added to jiffies, but it is defined as 60 * SEC_CONVERSION, and
SEC_CONVERSION isn't the number of jiffies in a second. It is the
internal factor of __timespec64_to_jiffies(), HZ scaled up by
SEC_JIFFIE_SC bits. For HZ=1000 that is 1000 << 22, which puts the
timeout at about eight years. On 32-bit the multiplication overflows.
So entries are never removed, and every id that a socket of the ring
ever had keeps being polled on each wait.
Fixes: 8d0c12a80cde ("io-uring: add napi busy poll support")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-33-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/napi.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/io_uring/napi.c b/io_uring/napi.c
index 5542d78a4bd8..a4b1aedf07c8 100644
--- a/io_uring/napi.c
+++ b/io_uring/napi.c
@@ -6,7 +6,7 @@
#ifdef CONFIG_NET_RX_BUSY_POLL
/* Timeout for cleanout of stale entries. */
-#define NAPI_TIMEOUT (60 * SEC_CONVERSION)
+#define NAPI_TIMEOUT (60 * HZ)
struct io_napi_entry {
unsigned int napi_id;
--
2.56.0
[-- Attachment #19: 0021-io_uring-wait-no-early-ETIME-if-min_wait-finds-work.patch --]
[-- Type: text/x-patch, Size: 2652 bytes --]
From 8a7a1b4791f3b4265c3155ff889d23253f2015a1 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:56 +0000
Subject: [PATCH 21/38] io_uring/wait: no early -ETIME if min_wait finds work
Commit 83cce67298ab1f4d2e9205a49f56975475cdb154 upstream.
When the min wait time is over and there is work to run, which for
DEFER_TASKRUN includes task_work that was queued lazily and didn't wake
the waiter, io_cqring_min_timer_wakeup() ends the wait via
io_cqring_timer_wakeup(). That flags it as timed out. io_cqring_wait()
runs the work, leaves the loop, and returns 0 if the CQ ring holds
something.
Running task_work doesn't have to result in a CQE, though. A poll wake up
can turn out to be spurious, for instance if somebody else has read the
data by the time the request is retried. The application then gets -ETIME
once the min wait time has passed, no matter how far away the timeout it
asked for is.
Wake the task without declaring a timeout. From now on a single CQE
satisfies the wait, like in the case where the timer is switched over to
the normal timeout. If running the work doesn't post one, the loop in
io_cqring_wait() goes back to sleep with the normal timeout.
[ 6.12: io_cqring_min_timer_wakeup() is in io_uring.c ]
Fixes: 1100c4a2656d ("io_uring: add support for batch wait timeout")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-32-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/io_uring.c | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 92f731b7423c..8f064995137c 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2435,9 +2435,16 @@ static enum hrtimer_restart io_cqring_min_timer_wakeup(struct hrtimer *timer)
if (iowq->timeout == KTIME_MAX ||
ktime_compare(iowq->min_timeout, iowq->timeout) >= 0)
goto out_wake;
- /* work we may need to run, wake function will see if we need to wake */
- if (io_has_work(ctx))
- goto out_wake;
+ /*
+ * Work we need to run. It isn't known if that is going to post a CQE,
+ * so this isn't a timeout. Carry on with the normal one if it doesn't.
+ */
+ if (io_has_work(ctx)) {
+ iowq->min_timeout = 0;
+ iowq->cq_tail = iowq->cq_min_tail + 1;
+ wake_up_process(iowq->wq.private);
+ return HRTIMER_NORESTART;
+ }
/* got events since we started waiting, min timeout is done */
if (iowq->cq_min_tail != READ_ONCE(ctx->rings->cq.tail))
goto out_wake;
--
2.56.0
[-- Attachment #20: 0020-io_uring-timeout-keep-repeat-count-on-multishot-upda.patch --]
[-- Type: text/x-patch, Size: 1714 bytes --]
From 3e4f0388cdf6f3eca80c6c080a7455776b03ab6e Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:54 +0000
Subject: [PATCH 20/38] io_uring/timeout: keep repeat count on multishot update
Commit 3dc2a32cd356ee382aa4a8c7d844c53b2c0ad83b upstream.
IORING_TIMEOUT_UPDATE clears ->off of the timeout it updates, to turn a
timeout that waits for a number of completions into a pure one. For a
multishot timeout ->off isn't a sequence though, it tells whether the
number of repeats is limited: io_timeout_finish() only looks at
->repeats if ->off is set.
A multishot timeout that was set up to fire N times hence fires forever
once its interval has been updated. Leave ->off alone for those, they are
never treated as sequence timeouts anyway.
Fixes: ea97f6c8558e ("io_uring: add support for multishot timeouts")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-30-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/timeout.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index 296329f06242..b81ba01aa2fb 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -407,8 +407,10 @@ static int io_timeout_update(struct io_ring_ctx *ctx, __u64 user_data,
if (IS_ERR(req))
return PTR_ERR(req);
- timeout->off = 0; /* noseq */
data = req->async_data;
+ /* for a multishot request this isn't a sequence, keep the repeats */
+ if (!(data->flags & IORING_TIMEOUT_MULTISHOT))
+ timeout->off = 0; /* noseq */
data->ts = *ts;
list_add_tail(&timeout->list, &ctx->timeout_list);
--
2.56.0
[-- Attachment #21: 0019-io_uring-timeout-don-t-skip-over-reached-sequences.patch --]
[-- Type: text/x-patch, Size: 4214 bytes --]
From 89114341b6942a954a21cb4dc62581eff695e2bf Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:53 +0000
Subject: [PATCH 19/38] io_uring/timeout: don't skip over reached sequences
Commit c6c0b3eba88c3b13ca11ea112404982e86d58f7a upstream.
io_flush_timeouts() copes with the wrap of the CQ sequence by looking at
the distance from ->cq_last_tm_flush. That relies on the target of each
pending timeout being at or ahead of the last flush.
io_timeout() breaks that. It sets ->cq_last_tm_flush to the current
sequence, on the assumption that nothing has been posted since the last
flush. That isn't true. A multishot request posts its CQEs right when it
is issued or retried, but they are committed, and the timeouts flushed,
at the end of the submission or task_work run:
X: IORING_OP_TIMEOUT, off == 1 target_seq == N + 1
io_submit_sqes()
multishot recv, posts two CQEs sequence == N + 2
Y: IORING_OP_TIMEOUT, off == 100 ->cq_last_tm_flush = N + 2
io_flush_timeouts()
X: events_needed = N + 1 - (N + 2) // 0xffffffff
events_got = 0 // not yet
X isn't completed although the completions it was waiting for have
happened, only its timer can end it now. And as the list is sorted and
the walk ends at the first timeout that isn't due, it gets in the way of
the ones behind it. The insertion sort has the same problem, it measures
the distance of the pending targets from the current sequence.
->cq_last_tm_flush only needs to be brought up to date if no timeout is
waiting for a sequence, as io_flush_timeouts() isn't called before the
first one is set up. Otherwise it is current enough. Timeouts with a
sequence come first in the list, so looking at the first entry will do.
Use the last flush as the base for the insertion sort as well.
Fixes: f010505b78a4 ("io_uring: flush timeouts that should already have expired")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-29-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/timeout.c | 21 ++++++++++++++-------
1 file changed, 14 insertions(+), 7 deletions(-)
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index 9cc5776103fe..296329f06242 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -574,8 +574,9 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
struct io_timeout *timeout = io_kiocb_to_cmd(req, struct io_timeout);
struct io_ring_ctx *ctx = req->ctx;
struct io_timeout_data *data = req->async_data;
+ struct io_timeout *first;
struct list_head *entry;
- u32 tail, off = timeout->off;
+ u32 tail, base, off = timeout->off;
spin_lock_irq(&ctx->timeout_lock);
@@ -592,11 +593,17 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
tail = data_race(ctx->cached_cq_tail) - atomic_read(&ctx->cq_timeouts);
timeout->target_seq = tail + off;
- /* Update the last seq here in case io_flush_timeouts() hasn't.
- * This is safe because ->completion_lock is held, and submissions
- * and completions are never mixed in the same ->completion_lock section.
+ /*
+ * Update the last seq here in case io_flush_timeouts() hasn't, it only
+ * gets called once sequences are in use. Leave it alone if there are
+ * timeouts waiting for one. CQEs may have been posted since the last
+ * flush, and a target that has been reached must not fall behind it.
*/
- ctx->cq_last_tm_flush = tail;
+ first = list_first_entry_or_null(&ctx->timeout_list, struct io_timeout,
+ list);
+ if (!first || io_is_timeout_noseq(cmd_to_io_kiocb(first)))
+ ctx->cq_last_tm_flush = tail;
+ base = ctx->cq_last_tm_flush;
/*
* Insertion sort, ensuring the first entry in the list is always
@@ -608,8 +615,8 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
if (io_is_timeout_noseq(nxt))
continue;
- /* nxt.seq is behind @tail, otherwise would've been completed */
- if (off >= nextt->target_seq - tail)
+ /* no target is behind the last flush, see io_flush_timeouts() */
+ if (timeout->target_seq - base >= nextt->target_seq - base)
break;
}
add:
--
2.56.0
[-- Attachment #22: 0018-io_uring-poll-drop-stale-double-poll-entry-on-rearm.patch --]
[-- Type: text/x-patch, Size: 2671 bytes --]
From e89d5686cf9d64b267a3f84b75e771063173a579 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:45 +0000
Subject: [PATCH 18/38] io_uring/poll: drop stale double poll entry on rearm
Commit f2dbe87968a3aef74a11c025776d293b80964d3f upstream.
A file that uses two wait queues for poll, e.g. a tty, gets a second
io_poll entry, which IORING_OP_POLL_ADD keeps in ->async_data until the
request is freed. io_poll_add() can run more than once for a request
though: IORING_OP_POLL_REMOVE with one of the update flags disarms and
rearms it, and io_poll_task_func() reissues it if a wake up turns out to
be spurious. The second poll_wait() of the new round then finds
*poll_ptr set, with ->head cleared by the disarm, and __io_queue_proc()
fails what it takes for a third wait queue with -EINVAL.
For an update that means that the poll request is completed with
-ECANCELED, while the update itself reports success. For a spurious wake
up, e.g. tcsetattr() on the polled tty, the poll request fails with
-EINVAL.
Free the old entry and clear the poll flags before arming, which is what
io_arm_apoll() does for internal polling. All entries have been removed
from their wait queues at that point.
[ 6.12: no io_req_async_data_free(), open-coded ]
Fixes: b69de288e913 ("io_uring: allow events and user_data update of running poll requests")
Fixes: 6e5aedb9324a ("io_uring/poll: attempt request issue after racy poll wakeup")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-21-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/poll.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/io_uring/poll.c b/io_uring/poll.c
index 140422c70528..2ad12b39f245 100644
--- a/io_uring/poll.c
+++ b/io_uring/poll.c
@@ -996,6 +996,18 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags)
if (req->ctx->flags & (IORING_SETUP_SQPOLL|IORING_SETUP_SINGLE_ISSUER))
req->flags |= REQ_F_HASH_LOCKED;
+ /*
+ * A poll update or a spurious wake up arms the request again. Forget
+ * the entries of the previous round like io_arm_apoll() does, a stale
+ * second entry would be taken for a third wait queue.
+ */
+ if (req_has_async_data(req)) {
+ kfree(req->async_data);
+ req->async_data = NULL;
+ req->flags &= ~REQ_F_ASYNC_DATA;
+ }
+ req->flags &= ~(REQ_F_SINGLE_POLL | REQ_F_DOUBLE_POLL);
+
ret = __io_arm_poll_handler(req, poll, &ipt, poll->events, issue_flags);
if (ret > 0) {
io_req_set_res(req, ipt.result_mask, 0);
--
2.56.0
[-- Attachment #23: 0017-io_uring-wait-look-at-all-CQ-waiters-on-wake-up.patch --]
[-- Type: text/x-patch, Size: 2011 bytes --]
From 666de8e02eb36bc3394be4fe73ffe2e86a7ba6fc Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:44 +0000
Subject: [PATCH 17/38] io_uring/wait: look at all CQ waiters on wake up
Commit 356f92f2412686f822383b784fa102924407457a upstream.
io_wake_function() returns -1 if the waiter it is called for doesn't have
enough events yet, which makes __wake_up_common() stop walking the wait
queue. That assumes that the waiters behind it can't be satisfied either.
Each waiter has its own target though, computed from the min_complete it
passed in:
thread A: io_uring_enter(GETEVENTS, min_complete = 10) // queued first
thread B: io_uring_enter(GETEVENTS, min_complete = 1)
one CQE is posted: A isn't woken, and B isn't even considered
B keeps sleeping with the event it asked for sitting in the CQ ring, until
A's demand is met or a timeout or signal ends the wait.
Return 0 to carry on with the next waiter. Only waiters whose condition
is met are woken, like before.
[ 6.12: io_wake_function() is in io_uring.c ]
Fixes: bda521624e75 ("io_uring: make CQ ring wakeups be more efficient")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-20-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/io_uring.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 00afd2ce4806..92f731b7423c 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2384,7 +2384,8 @@ static int io_wake_function(struct wait_queue_entry *curr, unsigned int mode,
*/
if (io_should_wake(iowq) || io_has_work(iowq->ctx))
return autoremove_wake_function(curr, mode, wake_flags, key);
- return -1;
+ /* keep going, a waiter further down may be content with less */
+ return 0;
}
int io_run_task_work_sig(struct io_ring_ctx *ctx)
--
2.56.0
[-- Attachment #24: 0016-io_uring-wake-ring-pollers-for-local-task_work.patch --]
[-- Type: text/x-patch, Size: 4367 bytes --]
From 186a050c9c5aa32ea6d44bdaa04ed6f5bdd38ccb Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:43 +0000
Subject: [PATCH 16/38] io_uring: wake ring pollers for local task_work
Commit b88902d8745a701d24275d012f0830e3628e2a22 upstream.
With IORING_SETUP_DEFER_TASKRUN completions are queued as local work, and
only become CQEs when the submitter enters the kernel to wait for events.
For an application that finds out via poll() or epoll on the ring fd when
it is time to do that, io_uring_poll() reports EPOLLIN if local work is
pending.
That only helps if the work is there at the time of the call. When
DEFER_TASKRUN was added, io_req_local_work_add() did io_cqring_wake(),
which took care of tasks sleeping in poll() as well. Then polling got a
wait queue of its own, ->poll_wq, and the wake up in
io_req_local_work_add() was replaced by waking the submitter directly.
Since then ->poll_wq is only woken when CQEs are committed, which for
such a ring is done by the very task that is waiting to be told to:
submitter completion
poll(ring_fd)
io_uring_poll() // nothing yet
schedule()
io_req_local_work_add()
// ->cq_wait_nr: nobody waits
// sleeps on
Wake ->poll_wq when the first entry is put on the work list. Like for the
eventfd, that must not be done right away if the work is queued from a
wake up handler, as the chain of wake ups could get back to the wait
queue whose lock is held. Use an RCU callback in that case, with a
reference to the ring.
[ 6.12: the code is in io_uring.c, ->poll_activated is a bitfield, and
there is no IOU_F_TWQ_IN_WAKE to tell a wake up handler apart, so the
wake up always goes through the RCU callback ]
Fixes: 130bd686d9be ("io_uring: waitqueue-less cq waiting")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-19-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
include/linux/io_uring_types.h | 3 +++
io_uring/io_uring.c | 28 ++++++++++++++++++++++++++++
2 files changed, 31 insertions(+)
diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h
index 7711d9920141..15da758064b0 100644
--- a/include/linux/io_uring_types.h
+++ b/include/linux/io_uring_types.h
@@ -403,6 +403,9 @@ struct io_ring_ctx {
u32 iowq_limits[2];
struct callback_head poll_wq_task_work;
+ /* deferred ->poll_wq wake up for local task_work */
+ struct rcu_head poll_wq_rcu;
+ atomic_t poll_wq_rcu_pending;
struct list_head defer_list;
#ifdef CONFIG_NET_RX_BUSY_POLL
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index ba6175bac757..00afd2ce4806 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1143,6 +1143,31 @@ void tctx_task_work(struct callback_head *cb)
WARN_ON_ONCE(ret);
}
+static void io_poll_wq_wake_rcu(struct rcu_head *rcu)
+{
+ struct io_ring_ctx *ctx = container_of(rcu, struct io_ring_ctx,
+ poll_wq_rcu);
+
+ atomic_set(&ctx->poll_wq_rcu_pending, 0);
+ io_poll_wq_wake(ctx);
+ percpu_ref_put(&ctx->refs);
+}
+
+/*
+ * io_uring_poll() reports pending local work as EPOLLIN, as it takes a trip to
+ * the kernel to turn it into CQEs. Let those know who are asleep already.
+ * This may be called from a wake up handler with some wait queue lock held,
+ * and this wake up could lead back to it. Do it from a clean context.
+ */
+static void io_local_work_wake_poll(struct io_ring_ctx *ctx)
+{
+ if (!wq_has_sleeper(&ctx->poll_wq) ||
+ atomic_xchg(&ctx->poll_wq_rcu_pending, 1))
+ return;
+ percpu_ref_get(&ctx->refs);
+ call_rcu_hurry(&ctx->poll_wq_rcu, io_poll_wq_wake_rcu);
+}
+
static inline void io_req_local_work_add(struct io_kiocb *req,
struct io_ring_ctx *ctx,
unsigned flags)
@@ -1202,6 +1227,9 @@ static inline void io_req_local_work_add(struct io_kiocb *req,
atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags);
if (ctx->has_evfd)
io_eventfd_signal(ctx, flags & IOU_F_TWQ_IN_WAKE);
+ /* whoever sets the flag wakes ->poll_wq afterwards */
+ if (ctx->poll_activated)
+ io_local_work_wake_poll(ctx);
}
nr_wait = atomic_read(&ctx->cq_wait_nr);
--
2.56.0
[-- Attachment #25: 0015-io_uring-cancel-let-CANCEL_ALL-find-parked-requests.patch --]
[-- Type: text/x-patch, Size: 3827 bytes --]
From ea9a69f0e7dae16373ac59f89c661c232a9137c0 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:42 +0000
Subject: [PATCH 15/38] io_uring/cancel: let CANCEL_ALL find parked requests
Commit fe6ca13c98b1223f6005f32913ebd32e4df64b77 upstream.
A cancellation with IORING_ASYNC_CANCEL_ALL or _ANY goes on until it finds
nothing more. A request may stay where it was found for a while after it
has been cancelled, so each one that matches is stamped with the sequence
number of the cancellation, and is skipped from then on.
io_cancel_cb() does so for io-wq work as well. There is no need to:
pending work is taken off the list, and the workers are looked at once per
call. And it does harm. A request that is run by io-wq may be parked
while it still is the current work of the worker: poll gets armed for it,
or it is put on ->futex_list or ->waitid_list.
io-wq worker io_try_cancel()
io_wq_submit_work()
io_arm_poll_handler()
io_poll_add_hash()
io_async_cancel_one()
io_cancel_cb()
stamps the request
sets IO_WQ_WORK_CANCEL
= -EALREADY
io_poll_cancel()
skips the request
= -ENOENT
io_assign_current_work(NULL)
The flag comes too late for the worker to see it, and the search that
would have found the request takes it for done. The cancellation returns,
and the request stays.
With a multishot poll, IOSQE_ASYNC, and IORING_OP_ASYNC_CANCEL with
IORING_ASYNC_CANCEL_ALL up to 63us later, that happens to 4 of 1,000
requests (4 CPUs, KVM).
Take the stamp off again if the work matches. If the request is found
on one of the lists it gets a new one, and io_cancel_cb() doesn't match it
a second time.
__io_async_cancel() tries the io-wq of the caller twice. So far the stamp
has kept it from counting work that is running there the second time.
Skip it instead, so that the number that is returned stays the same.
Fixes: 8e29da69fead ("io_uring: add support for IORING_ASYNC_CANCEL_ALL")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-18-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/cancel.c | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/io_uring/cancel.c b/io_uring/cancel.c
index eed9d5c72f31..13d5d4f2924f 100644
--- a/io_uring/cancel.c
+++ b/io_uring/cancel.c
@@ -70,7 +70,16 @@ static bool io_cancel_cb(struct io_wq_work *work, void *data)
struct io_kiocb *req = container_of(work, struct io_kiocb, work);
struct io_cancel_data *cd = data;
- return io_cancel_req_match(req, cd);
+ if (!io_cancel_req_match(req, cd))
+ return false;
+ /*
+ * Pending work is gone once it has matched. Work that is running may
+ * park the request where the same cancellation looks next. Don't make
+ * it skip the request as one that it has dealt with.
+ */
+ if (cd->flags & (IORING_ASYNC_CANCEL_ALL | IORING_ASYNC_CANCEL_ANY))
+ req->cancel_seq_set = false;
+ return true;
}
static int io_async_cancel_one(struct io_uring_task *tctx,
@@ -187,6 +196,9 @@ static int __io_async_cancel(struct io_cancel_data *cd,
io_ring_submit_lock(ctx, issue_flags);
ret = -ENOENT;
list_for_each_entry(node, &ctx->tctx_list, ctx_node) {
+ /* has been done above, don't count what is running there */
+ if (all && node->task->io_uring == tctx)
+ continue;
ret = io_async_cancel_one(node->task->io_uring, cd);
if (ret != -ENOENT) {
if (!all)
--
2.56.0
[-- Attachment #26: 0014-io_uring-cancel-recognize-no-timeout-on-32-bit.patch --]
[-- Type: text/x-patch, Size: 1903 bytes --]
From 17a5f32a8d270b1bc7cc57ae29d9da262a0b7cb3 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:41 +0000
Subject: [PATCH 14/38] io_uring/cancel: recognize "no timeout" on 32-bit
Commit 3774e4d4c8533da2caae95556c4d635c7c4f45b8 upstream.
IORING_REGISTER_SYNC_CANCEL waits without a limit if both fields of
io_uring_sync_cancel_reg::timeout are -1. io_sync_cancel() compares them
with -1UL. They are of type long long. With a long of 32 bits -1UL is
4294967295, which is what it stays when it is converted for the
comparison, and the fields are never equal to it.
So { -1, -1 } is taken for a timeout. ktime_set() takes the nanoseconds as
an unsigned long, so what becomes of it is -1s + 4294967295ns = 3.29s.
If the request is being run by io-wq and can't be interrupted, the call
returns -ETIME after that time, rather than waiting for the request to be
gone. On i386, with a buffered read from a null_blk device that takes 4s
to complete a request, IOSQE_ASYNC, that is what each call does.
Fixes: 78a861b94959 ("io_uring: add sync cancelation API through io_uring_register()")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-17-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/cancel.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/io_uring/cancel.c b/io_uring/cancel.c
index 38123aac4f86..eed9d5c72f31 100644
--- a/io_uring/cancel.c
+++ b/io_uring/cancel.c
@@ -315,7 +315,7 @@ int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg)
if (ret != -EALREADY)
goto out;
- if (sc.timeout.tv_sec != -1UL || sc.timeout.tv_nsec != -1UL) {
+ if (sc.timeout.tv_sec != -1 || sc.timeout.tv_nsec != -1) {
struct timespec64 ts = {
.tv_sec = sc.timeout.tv_sec,
.tv_nsec = sc.timeout.tv_nsec
--
2.56.0
[-- Attachment #27: 0013-io_uring-cancel-don-t-forget-that-the-work-was-runni.patch --]
[-- Type: text/x-patch, Size: 3859 bytes --]
From 5b3d8fa94e1e7ddee15e76165c799fc565acf57c Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:40 +0000
Subject: [PATCH 13/38] io_uring/cancel: don't forget that the work was running
Commit 8aa0e5db4845fb5ac11abe40d7b1cac0656c3ee3 upstream.
If the request to be cancelled is being run by io-wq,
io_async_cancel_one() flags the work, signals the worker and returns
-EALREADY. Since commit ccbf726171b7 ("io_uring: perform poll removal even
if async work removal is successful") io_try_cancel() goes on regardless,
as poll may be armed for the request. If it isn't, and nothing else turns
up, the result is the -ENOENT of the last search.
__io_async_cancel() makes up for that. On -ENOENT it tries the io-wq of
every task that uses the ring, and that of the caller is one of them. So
the work is found a second time, and the result is -EALREADY after all.
Unless the worker is done with it by then. If the work is flagged before
io_wq_submit_work() looks at IO_WQ_WORK_CANCEL, all the worker does is to
queue the request to be failed by task work. Then it isn't anywhere to be
found until the task work has run, and
- IORING_REGISTER_SYNC_CANCEL returns -ENOENT,
- IORING_OP_ASYNC_CANCEL completes with -ENOENT,
for a request that they have just cancelled, and that completes with
-ECANCELED.
With a read of an empty pipe, IOSQE_ASYNC, and a sync cancel right behind
it, 1 to 3 of 1,000,000 calls return -ENOENT, and 50 to 110 with
IORING_SETUP_DEFER_TASKRUN (4 CPUs, KVM). Nothing is written to the pipe,
and all of these reads end with -ECANCELED.
Remember that the work was found. IORING_OP_ASYNC_CANCEL completes with
-EALREADY, as it does if the worker is a bit slower. For
IORING_REGISTER_SYNC_CANCEL it means to try again, which ends with 0.
The result of a linked timeout, which calls io_try_cancel() by itself,
and the number that is returned for IORING_ASYNC_CANCEL_ALL and _ANY don't
change.
Fixes: ccbf726171b7 ("io_uring: perform poll removal even if async work removal is successful")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-16-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/cancel.c | 13 ++++++++++++-
io_uring/cancel.h | 2 ++
2 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/io_uring/cancel.c b/io_uring/cancel.c
index a6e58a20efdd..38123aac4f86 100644
--- a/io_uring/cancel.c
+++ b/io_uring/cancel.c
@@ -115,6 +115,8 @@ int io_try_cancel(struct io_uring_task *tctx, struct io_cancel_data *cd,
*/
if (!ret)
return 0;
+ if (ret == -EALREADY)
+ cd->running = true;
ret = io_poll_cancel(ctx, cd, issue_flags);
if (ret != -ENOENT)
@@ -171,6 +173,7 @@ static int __io_async_cancel(struct io_cancel_data *cd,
struct io_tctx_node *node;
int ret, nr = 0;
+ cd->running = false;
do {
ret = io_try_cancel(tctx, cd, issue_flags);
if (ret == -ENOENT)
@@ -192,7 +195,15 @@ static int __io_async_cancel(struct io_cancel_data *cd,
}
}
io_ring_submit_unlock(ctx, issue_flags);
- return all ? nr : ret;
+ if (all)
+ return nr;
+ /*
+ * The worker may be done with the work that was found running, and
+ * flagged, above. That isn't the same as not having found it.
+ */
+ if (ret == -ENOENT && cd->running)
+ ret = -EALREADY;
+ return ret;
}
int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags)
diff --git a/io_uring/cancel.h b/io_uring/cancel.h
index da13a1d82062..3f54b040efe4 100644
--- a/io_uring/cancel.h
+++ b/io_uring/cancel.h
@@ -11,6 +11,8 @@ struct io_cancel_data {
u8 opcode;
u32 flags;
int seq;
+ /* io_try_cancel() has found it being run by io-wq, and flagged it */
+ bool running;
};
int io_async_cancel_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);
--
2.56.0
[-- Attachment #28: 0012-io_uring-don-t-lose-a-cancel-as-io-wq-parks-a-reques.patch --]
[-- Type: text/x-patch, Size: 5283 bytes --]
From cb9959d6eb5513fb4082586a65a4737e50bf5c6f Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:39 +0000
Subject: [PATCH 12/38] io_uring: don't lose a cancel as io-wq parks a request
Commit 5cb049d2203d7a81a1c1a737c8df4575e71078e3 upstream.
IORING_OP_FUTEX_WAIT, IORING_OP_FUTEX_WAITV and IORING_OP_WAITID don't
block. They put the request on a list of the ring, where a wake up or a
cancellation finds it, and return IOU_ISSUE_SKIP_COMPLETE. They do so as
well if they are run by io-wq, e.g. because of IOSQE_ASYNC.
io_wq_submit_work() looks at IO_WQ_WORK_CANCEL once, before it issues the
request. Somebody who wants to cancel the request after that, and before
it is on the list, finds it running:
io-wq worker io_try_cancel()
io_wq_submit_work()
IO_WQ_WORK_CANCEL isn't set
io_async_cancel_one()
sets IO_WQ_WORK_CANCEL
signals the worker
= -EALREADY
...
io_futex_cancel() = -ENOENT
...
io_futex_wait()
adds the request to ->futex_list
= IOU_ISSUE_SKIP_COMPLETE
Nobody looks at the flag again. IORING_OP_ASYNC_CANCEL returns -EALREADY
and the request stays.
It is the problem that "io_uring/poll: don't lose a cancel as io-wq arms
poll" deals with for poll.
Look at the flag once the request is on the list, and cancel it if it is
set. The lists are searched and the requests are added with ->uring_lock
held, and the flag is set before the search. So either the search finds
the request, or it came first and the flag is seen.
What the canceller is told doesn't change, see the other patch, also for
IORING_ASYNC_CANCEL_ALL and _ANY.
[ 6.12: __io_futex_cancel() and __io_waitid_cancel() take the ctx as well.
futex_wait_setup() returns with the hash bucket locked and the futex_q
not queued yet, so io_futex_wait() queues it before the check, with
->uring_lock still held ]
Fixes: 194bb58c6090 ("io_uring: add support for futex wake and wait")
Fixes: 8f350194d5cf ("io_uring: add support for vectored futex waits")
Fixes: f31ecf671ddc ("io_uring: add IORING_OP_WAITID support")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-15-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/futex.c | 7 +++++--
io_uring/io_uring.h | 17 +++++++++++++++++
io_uring/waitid.c | 2 ++
3 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/io_uring/futex.c b/io_uring/futex.c
index 589769761cea..990568baf393 100644
--- a/io_uring/futex.c
+++ b/io_uring/futex.c
@@ -307,6 +307,8 @@ int io_futexv_wait(struct io_kiocb *req, unsigned int issue_flags)
*/
__set_current_state(TASK_RUNNING);
hlist_add_head(&req->hash_node, &ctx->futex_list);
+ if (io_wq_cancel_seen(req, issue_flags))
+ __io_futex_cancel(ctx, req);
} else {
iof->futexv_unqueued = 1;
if (woken != -1)
@@ -348,9 +350,10 @@ int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags)
&ifd->q, &hb);
if (!ret) {
hlist_add_head(&req->hash_node, &ctx->futex_list);
- io_ring_submit_unlock(ctx, issue_flags);
-
futex_queue(&ifd->q, hb, NULL);
+ if (io_wq_cancel_seen(req, issue_flags))
+ __io_futex_cancel(ctx, req);
+ io_ring_submit_unlock(ctx, issue_flags);
return IOU_ISSUE_SKIP_COMPLETE;
}
diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h
index 3aec115c71aa..91838d9b85f2 100644
--- a/io_uring/io_uring.h
+++ b/io_uring/io_uring.h
@@ -251,6 +251,23 @@ static inline void io_ring_submit_lock(struct io_ring_ctx *ctx,
lockdep_assert_held(&ctx->uring_lock);
}
+/*
+ * For a handler that is run by io-wq and parks the request on a list where
+ * cancellations find it, rather than completing it. Whoever tried to cancel
+ * the request before that found it running. It has flagged the work item,
+ * has searched the list in vain, and won't come back. To be called with
+ * ->uring_lock held, which the search needs as well, once the request is on
+ * the list: if the search came first, the flag is visible.
+ */
+static inline bool io_wq_cancel_seen(struct io_kiocb *req,
+ unsigned int issue_flags)
+{
+ lockdep_assert_held(&req->ctx->uring_lock);
+
+ return (issue_flags & IO_URING_F_IOWQ) &&
+ (atomic_read(&req->work.flags) & IO_WQ_WORK_CANCEL);
+}
+
static inline void io_commit_cqring(struct io_ring_ctx *ctx)
{
/* order cqe stores with ring update */
diff --git a/io_uring/waitid.c b/io_uring/waitid.c
index 34b428fb4273..c0997432aadb 100644
--- a/io_uring/waitid.c
+++ b/io_uring/waitid.c
@@ -375,6 +375,8 @@ int io_waitid(struct io_kiocb *req, unsigned int issue_flags)
* a waitqueue callback, or if someone cancels it.
*/
if (!io_waitid_drop_issue_ref(req)) {
+ if (io_wq_cancel_seen(req, issue_flags))
+ __io_waitid_cancel(ctx, req, true);
io_ring_submit_unlock(ctx, issue_flags);
return IOU_ISSUE_SKIP_COMPLETE;
}
--
2.56.0
[-- Attachment #29: 0011-io_uring-poll-don-t-lose-a-cancel-as-io-wq-arms-poll.patch --]
[-- Type: text/x-patch, Size: 4592 bytes --]
From 06f52c410d125fdedeb9bdc16d2fdf3aa92838f2 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:38 +0000
Subject: [PATCH 11/38] io_uring/poll: don't lose a cancel as io-wq arms poll
Commit 6d2d4c2530e95ff6ddeb98195d49d99dbca14369 upstream.
A request on a pollable file that was forced to io-wq is issued without
blocking there. If that ends in -EAGAIN, io_wq_submit_work() arms poll
for it and returns. It looks at IO_WQ_WORK_CANCEL once, before it issues
the request.
Somebody who wants to cancel the request in between finds it running:
io-wq worker io_try_cancel()
io_wq_submit_work()
IO_WQ_WORK_CANCEL isn't set
io_issue_sqe() = -EAGAIN
io_async_cancel_one()
sets IO_WQ_WORK_CANCEL
signals the worker
= -EALREADY
io_poll_cancel() = -ENOENT
...
= -ENOENT
io_arm_poll_handler() = IO_APOLL_OK
return
Nobody looks at the flag again, and the signal doesn't matter to a
worker that doesn't sleep. The request is armed as if nothing had
happened:
- a linked timeout completes with -ENOENT, and the request it was
meant to limit waits for as long as it takes.
- IORING_OP_ASYNC_CANCEL returns -EALREADY, and the request stays.
- IORING_REGISTER_SYNC_CANCEL waits for the request to complete if it
gets -EALREADY. Since commit 5623eb1ed035 ("io_uring/tctx: add
separate lock for list of tctx's in ctx") it doesn't sleep but spins,
so its next attempt finds the armed poll.
The same goes for a multishot poll, recv or accept that is armed from
io-wq.
With a read of an empty pipe, IOSQE_ASYNC, and a linked timeout of 5ms
or a cancel in the same submission, in a guest that is slow enough (TCG,
KASAN), the request is lost in 21 of 24 rounds of 40 or 100 attempts.
While it arms poll from io-wq, __io_arm_poll_handler() owns the request.
Look at the flag once the request is in the hash table. The table is
searched and the request is added with ->uring_lock held, and the flag is
set before the search. So either the search finds the request, or it
came first and the flag is seen. With the request marked as cancelled, the
task work that is queued, right away or because ownership can't be
released, fails it with -ECANCELED.
What the canceller is told doesn't change. In this window a linked timeout
still completes with -ENOENT rather than -ETIME, a cancel request with
-EALREADY, and one with IORING_ASYNC_CANCEL_ALL or _ANY with a count that
doesn't include the request. But the request is cancelled, so what is new
is a request that completes with -ECANCELED after such a result.
This doesn't help in two cases:
- IORING_ASYNC_CANCEL_ALL and _ANY, if the work is flagged after the
test added here, while io-wq still has it as the current work of the
worker. io_cancel_cb() stamps it with the sequence of the cancel, so the
search of the hash table on behalf of the same cancel takes it for
done and skips it.
- requests that are parked elsewhere by io-wq, like IORING_OP_FUTEX_WAIT
and IORING_OP_WAITID. They have the same problem.
Fixes: 90fa02883f06 ("io_uring: implement async hybrid mode for pollable requests")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-14-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/poll.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/io_uring/poll.c b/io_uring/poll.c
index 1a6179ac491b..140422c70528 100644
--- a/io_uring/poll.c
+++ b/io_uring/poll.c
@@ -656,6 +656,18 @@ static int __io_arm_poll_handler(struct io_kiocb *req,
io_poll_add_hash(req);
+ /*
+ * Whoever tried to cancel the request while io-wq was running it has
+ * flagged the work item. If it searched the hash table before the
+ * request got there it found nothing, and it won't come back. Both
+ * the search and the insertion are done with ->uring_lock held, and
+ * the flag is set before the search. If the search came first,
+ * the flag is visible by now.
+ */
+ if (ipt->owning &&
+ (atomic_read(&req->work.flags) & IO_WQ_WORK_CANCEL))
+ io_poll_mark_cancelled(req);
+
if (mask && (poll->events & EPOLLET) &&
io_poll_can_finish_inline(req, ipt)) {
__io_poll_execute(req, mask, 0);
--
2.56.0
[-- Attachment #30: 0010-io_uring-io-wq-let-cancel-find-picked-up-hashed-work.patch --]
[-- Type: text/x-patch, Size: 4395 bytes --]
From 108080b5dacb5a14eac3de294ed357790972d16a Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:37 +0000
Subject: [PATCH 10/38] io_uring/io-wq: let cancel find picked up hashed work
Commit e9efeb4fc97b0879512a3c3b995ed448f2487e12 upstream.
io_get_next_work() takes all the work items that share a hash off the
work list in one go, and the worker runs them one after the other. Only
the one it is working on is visible as ->cur_work. The others are neither
on the list nor current, so io_wq_cancel_cb() can't find them.
IORING_OP_ASYNC_CANCEL and IORING_REGISTER_SYNC_CANCEL return -ENOENT for
a request that has not even been started, e.g. a buffered write queued
behind another write to the same file that is blocked, and the request is
carried out. The cancellation done for a task or ring that goes away gets
there in the end, as it keeps trying, but it has to wait for each item to
become current.
Keep a pointer to the remainder of the chain next to ->cur_work, and have
io_wq_worker_cancel() look at it. The items on it belong to the worker,
which needs ->lock to move on to the next one. Matching work is flagged
with IO_WQ_WORK_CANCEL and fails with -ECANCELED when its turn comes. The
worker isn't signalled for it, that would interrupt the unrelated work it
is busy with. As the request isn't gone yet, it is reported like running
work, with -EALREADY.
Fixes: 86f3cd1b589a ("io-wq: handle hashed writes in chains")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-13-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io-wq.c | 35 +++++++++++++++++++++++++++++------
1 file changed, 29 insertions(+), 6 deletions(-)
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index 496d2155c94d..965dc487f240 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -54,6 +54,8 @@ struct io_worker {
struct io_wq *wq;
struct io_wq_work *cur_work;
+ /* rest of the hashed chain ->cur_work was taken off the list with */
+ struct io_wq_work *next_work;
raw_spinlock_t lock;
struct completion ref_done;
@@ -535,6 +537,17 @@ static struct io_wq_work *io_get_next_work(struct io_wq_acct *acct,
return NULL;
}
+static void __io_assign_current_work(struct io_worker *worker,
+ struct io_wq_work *work)
+{
+ raw_spin_lock(&worker->lock);
+ worker->cur_work = work;
+ /* stays valid while in between two items of a chain */
+ if (work)
+ worker->next_work = wq_next_work(work);
+ raw_spin_unlock(&worker->lock);
+}
+
static void io_assign_current_work(struct io_worker *worker,
struct io_wq_work *work)
{
@@ -543,9 +556,7 @@ static void io_assign_current_work(struct io_worker *worker,
cond_resched();
}
- raw_spin_lock(&worker->lock);
- worker->cur_work = work;
- raw_spin_unlock(&worker->lock);
+ __io_assign_current_work(worker, work);
}
/*
@@ -576,9 +587,7 @@ static void io_worker_handle_work(struct io_wq_acct *acct,
* work list, but isn't yet discoverable as the
* current work item for this worker.
*/
- raw_spin_lock(&worker->lock);
- worker->cur_work = work;
- raw_spin_unlock(&worker->lock);
+ __io_assign_current_work(worker, work);
}
raw_spin_unlock(&acct->lock);
@@ -1083,6 +1092,7 @@ static bool __io_wq_worker_cancel(struct io_worker *worker,
static bool io_wq_worker_cancel(struct io_worker *worker, void *data)
{
struct io_cb_cancel_data *match = data;
+ struct io_wq_work *work;
/*
* Hold the lock to avoid ->cur_work going out of scope, caller
@@ -1091,6 +1101,19 @@ static bool io_wq_worker_cancel(struct io_worker *worker, void *data)
raw_spin_lock(&worker->lock);
if (__io_wq_worker_cancel(worker, match, worker->cur_work))
match->nr_running++;
+ /*
+ * Hashed work is taken off the work list a chain at a time, what the
+ * worker hasn't started yet can only be found here. It doesn't move on
+ * without the lock. Just mark the work, it fails once it is its turn.
+ */
+ for (work = worker->next_work; work; work = wq_next_work(work)) {
+ if (match->nr_running && !match->cancel_all)
+ break;
+ if (match->fn(work, match->data)) {
+ atomic_or(IO_WQ_WORK_CANCEL, &work->flags);
+ match->nr_running++;
+ }
+ }
raw_spin_unlock(&worker->lock);
return match->nr_running && !match->cancel_all;
--
2.56.0
[-- Attachment #31: 0009-io_uring-io-wq-don-t-exit-with-a-creation-pending.patch --]
[-- Type: text/x-patch, Size: 3191 bytes --]
From f9ac3ad7deb52ab2ab8121b93b4193b4789f41c8 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:36 +0000
Subject: [PATCH 09/38] io_uring/io-wq: don't exit with a creation pending
Commit e231d2e8f80d3649ecd3851b7f2abdfbe606c04b upstream.
A worker that is the last one to go to sleep while work is pending asks
for another worker. io_wq_dec_running() bumps ->nr_running on behalf of
the new worker and queues create_worker_cb() as task_work for the owner
of the io-wq. Until that has run, everybody else takes it that a worker
is running which is going to look after the work list: io_wq_enqueue()
doesn't create one, and other workers that go to sleep don't find
themselves to be the last.
If the worker that made the request exits, io_worker_exit() cancels the
task_work, and io_worker_cancel_cb() drops ->nr_running without a look at
the work list:
worker A worker B io_wq_enqueue()
blocks, work is pending
nr_running++
queues create_worker_cb()
wakes, handles the work
idle timeout, decides to exit
picks up work 1
queues work 2
B is on the free list
still: wakes B
work 1 blocks
nr_running 2 -> 1
io_worker_exit()
io_worker_cancel_cb()
nr_running 1 -> 0
Work 2 has to wait for work 1, although there is room for more workers.
It takes an owner that doesn't get around to run its task_work for the
time of the idle timeout, less if the worker exits because its affinity
was changed or the io-wq is marked exit-on-idle.
Let the worker stay around until its request has been dealt with. Only
the worker itself sets ->create_state, so it can't be set behind its
back once it has decided to leave.
[ 6.12: no exit-on-idle state ]
Fixes: 1d5f5ea7cb7d ("io-wq: remove worker to owner tw dependency")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-12-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/io-wq.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index fcb54dd78fac..496d2155c94d 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -708,8 +708,12 @@ static int io_wq_worker(void *data)
/*
* Last sleep timed out. Exit if we're not the last worker,
* or if someone modified our affinity.
+ * Not as long as a request of ours for another worker is
+ * pending, though. It is accounted as a running worker, which
+ * others rely on, and io_worker_exit() would cancel it.
*/
- if (last_timeout && (exit_mask || acct->nr_workers > 1)) {
+ if (last_timeout && (exit_mask || acct->nr_workers > 1) &&
+ !test_bit(0, &worker->create_state)) {
acct->nr_workers--;
if (test_and_clear_bit(IO_WORKER_F_FREE, &worker->flags))
hlist_nulls_del_init_rcu(&worker->nulls_node);
--
2.56.0
[-- Attachment #32: 0008-io_uring-io-wq-don-t-strand-work-when-a-worker-exits.patch --]
[-- Type: text/x-patch, Size: 4283 bytes --]
From 5fb36ff2b74603bcf3fd535e9885bee49114e199 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:35 +0000
Subject: [PATCH 08/38] io_uring/io-wq: don't strand work when a worker exits
Commit ad41f325be1271b22d99353819c90f0d5e45acc8 upstream.
A worker exits if its idle timeout fired and it isn't the last one, if
its affinity was changed, or if the wq is marked exit-on-idle. It decides
so after it found the work list empty, and until io_worker_exit() is
through it stays on the free list and in ->nr_running:
worker io_wq_enqueue()
io_acct_run_queue() == false
queues work
io_acct_activate_free_worker()
wake_up_process(worker)
// found one, nothing to create
acct->nr_workers--
break
io_worker_exit()
Nobody is going to look at the work. If other workers exist, it has to
wait until one of them finishes what it is blocked on or hits its idle
timeout. If not, which is possible for the affinity and the exit-on-idle
case, it sits there until something else is queued. For exit-on-idle it
takes a task that closes its last ring and sets up a new one while the
workers are on their way out.
Make the worker take itself off the free list when it gives up its
->nr_workers slot, then leave ->nr_running, and only then take the final
look at the work list. That look and queueing work are both done under
acct->lock, so either the worker sees the work and stays, or the enqueue
side sees that the worker is gone and wakes or creates another one. If it
wants to stay and finds that its slot has been taken, a worker has been
created in its place, which is going to look at the work list, and it can
leave after all.
[ 6.12: ->nr_workers and the free list are under wq->lock, there is no
acct->workers_lock ]
Fixes: 38aa434ab933 ("io_uring/io-wq: add exit-on-idle state")
Fixes: 05c5f4ee4da7 ("io-wq: get rid of FIXED worker flag")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-11-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/io-wq.c | 38 +++++++++++++++++++++++++++++++++++++-
1 file changed, 37 insertions(+), 1 deletion(-)
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index d64259d06e6f..fcb54dd78fac 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -647,6 +647,37 @@ static void io_worker_handle_work(struct io_wq_acct *acct,
} while (1);
}
+/*
+ * The worker has decided to exit, and has given up its ->nr_workers slot and
+ * its place on the free list. io_wq_enqueue() may have picked it from that
+ * list, or seen it running, after it last looked at the work list. Leave
+ * ->nr_running as well and check again. Returns false if the worker has to
+ * stay.
+ */
+static bool io_worker_retire(struct io_wq_acct *acct, struct io_worker *worker)
+{
+ struct io_wq *wq = worker->wq;
+
+ /* turns io_wq_dec_running() and the scheduler hooks into no-ops */
+ clear_bit(IO_WORKER_F_UP, &worker->flags);
+ atomic_dec(&acct->nr_running);
+ if (!io_acct_run_queue(acct))
+ return true;
+ raw_spin_unlock(&acct->lock);
+
+ raw_spin_lock(&wq->lock);
+ if (acct->nr_workers >= acct->max_workers) {
+ /* a worker was created in our place, the work is its business */
+ raw_spin_unlock(&wq->lock);
+ return true;
+ }
+ acct->nr_workers++;
+ raw_spin_unlock(&wq->lock);
+ atomic_inc(&acct->nr_running);
+ set_bit(IO_WORKER_F_UP, &worker->flags);
+ return false;
+}
+
static int io_wq_worker(void *data)
{
struct io_worker *worker = data;
@@ -680,9 +711,14 @@ static int io_wq_worker(void *data)
*/
if (last_timeout && (exit_mask || acct->nr_workers > 1)) {
acct->nr_workers--;
+ if (test_and_clear_bit(IO_WORKER_F_FREE, &worker->flags))
+ hlist_nulls_del_init_rcu(&worker->nulls_node);
raw_spin_unlock(&wq->lock);
__set_current_state(TASK_RUNNING);
- break;
+ if (io_worker_retire(acct, worker))
+ break;
+ last_timeout = false;
+ continue;
}
last_timeout = false;
__io_worker_idle(wq, worker);
--
2.56.0
[-- Attachment #33: 0007-io_uring-don-t-flood-a-cancelling-task-with-task_wor.patch --]
[-- Type: text/x-patch, Size: 3560 bytes --]
From c3cb98bfbc4a57f7dd827a93fe18ac1d376cadae Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:30 +0000
Subject: [PATCH 07/38] io_uring: don't flood a cancelling task with task_work
Commit 22e643e61b9b6299b42ff7d88bab48bccada8a5d upstream.
io_ring_exit_work() asks each task which has used the ring to remove its
tctx node, by task_work, and repeats that until the list is empty.
io_tctx_exit_cb() doesn't touch the node while the task is in
io_uring_cancel_generic(), which removes all the nodes at its end. So the
work item queues the callback again as soon as it has completed, for
nothing.
That is more than a waste of time. task_work_run() only returns when it
finds the list empty. If the rings are released from inside the
cancellation, because requests of other rings held the last references to
their files, up to 64 work items keep sending callbacks to the one task
which is running them. It never finds the list empty, never gets back to
the cancellation, the nodes stay, and so it goes on: a task with 60 pairs
of rings which poll each other ran 280000 callbacks in 5 seconds, 20
seconds after exit_group(). It ends when all the senders happen to be late
at the same time. Until then the task keeps a CPU and the work items busy.
An exiting task can't be killed meanwhile. execve() cancels the requests in
the same way, so a live task can get stuck there as well, for minutes. If
it takes more than five, the work items run into the
WARN_ON_ONCE(time_after(jiffies, timeout)) of io_ring_exit_work(). No
privilege is needed for any of this.
If the callback has met a task in cancellation, wait a little before it is
sent again. Nobody has to wake the work item up for that: either the node
is gone by then, or it is the turn of the next round.
Fixes: d56d938b4bef ("io_uring: do ctx initiated file note removal")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-6-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io_uring.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 75265ebc3630..ba6175bac757 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2873,6 +2873,7 @@ struct io_tctx_exit {
struct callback_head task_work;
struct completion completion;
struct io_ring_ctx *ctx;
+ bool in_cancel;
};
static __cold void io_tctx_exit_cb(struct callback_head *cb)
@@ -2887,7 +2888,8 @@ static __cold void io_tctx_exit_cb(struct callback_head *cb)
* tctx can be NULL if the queueing of this task_work raced with
* work cancelation off the exec path.
*/
- if (tctx && !atomic_read(&tctx->in_cancel))
+ work->in_cancel = tctx && atomic_read(&tctx->in_cancel);
+ if (tctx && !work->in_cancel)
io_uring_del_tctx_node((unsigned long)work->ctx);
complete(&work->completion);
}
@@ -2981,6 +2983,14 @@ static __cold void io_ring_exit_work(struct work_struct *work)
* wait is marked as interruptible.
*/
wait_for_completion_interruptible(&exit.completion);
+ /*
+ * The task removes the node when it is done with its
+ * cancellation. Give it the time for that: it doesn't get back
+ * to it as long as it finds new task_work each time it has run
+ * the old one, and we may be one of many who send it.
+ */
+ if (exit.in_cancel)
+ schedule_timeout_idle(HZ / 20);
mutex_lock(&ctx->uring_lock);
}
mutex_unlock(&ctx->uring_lock);
--
2.56.0
[-- Attachment #34: 0006-io_uring-io-wq-don-t-hide-a-link-from-cancellations.patch --]
[-- Type: text/x-patch, Size: 8775 bytes --]
From b823501f67778da3cbf107b781509341659d2ca2 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:29 +0000
Subject: [PATCH 06/38] io_uring/io-wq: don't hide a link from cancellations
Commit 436d48a2ed13112d2d27085325838e5bce6b4507 upstream.
A task that exits cancels its tracked requests and waits until they are
gone. io_uring_cancel_generic() looks everywhere a request can be, and
goes to sleep if it found nothing to cancel. It is woken up each time
one of the requests of the task is completed by task work or freed.
When io-wq is done with a request that has a link, there is a time in which
the next request of the link can't be found:
io_worker_handle_work()
io_assign_current_work(worker, NULL)
io_wq_free_work(work)
io_req_find_next() // takes the link off the request
io_free_req() // task work for the task
io_assign_current_work(worker, linked)
io_run_task_work()
cond_resched()
worker->cur_work = linked
It is not pending, it is not the current work of a worker, and it is not
yet on the list that it is about to be parked on. If the pass that the
exiting task makes because of io_free_req() falls into that time, it finds
nothing and the task goes to sleep. The worker then runs the link, which
IO_WQ_BIT_EXIT doesn't prevent for bounded work. If that parks the request
nothing is completed or freed, and nobody wakes up the task. If it is the
process that exits it can't be killed any more either.
NOP, IOSQE_ASYNC | IOSQE_IO_LINK
FUTEX_WAIT, FUTEX2_PRIVATE
exit of the thread, or of the process, up to 63us later
With four of these per round 5 to 8 of 10,000 rounds hang (4 CPUs, KVM):
io_uring_cancel_generic
do_exit
Commit f8b632e89a10 ("io_uring: tighten task exit cancellations") is about
the same gap, and makes the task look again for whatever happens before
io_free_req().
Free the request once the link is the current work of the worker, or
pending. Whoever is woken up by that can find the link, until it is parked
where cancellations look as well. To do so split io_wq_free_work() in two.
[ 6.12: io-wq still calls back through io_wq_data, so a ->put_work callback
is added next to ->free_work. io_wq_put_work() keeps the link check of
6.12, without the REQ_F_CQE_SKIP condition that commit 5530a5102e3c adds
in 7.3 ]
Fixes: 768134d4f481 ("io_uring: don't do flush cancel under inflight_lock")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-5-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/io-wq.c | 22 ++++++++++++++++++----
io_uring/io-wq.h | 4 +++-
io_uring/io_uring.c | 29 +++++++++++++++++++++--------
io_uring/io_uring.h | 3 ++-
io_uring/tctx.c | 1 +
5 files changed, 45 insertions(+), 14 deletions(-)
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index 4a0d7300c275..d64259d06e6f 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -98,6 +98,7 @@ enum {
struct io_wq {
unsigned long state;
+ put_work_fn *put_work;
free_work_fn *free_work;
io_wq_work_fn *do_work;
@@ -593,8 +594,9 @@ static void io_worker_handle_work(struct io_wq_acct *acct,
/* handle a whole dependent link */
do {
bool do_kill = test_bit(IO_WQ_BIT_EXIT, &wq->state);
- struct io_wq_work *next_hashed, *linked;
+ struct io_wq_work *next_hashed, *linked, *done = work;
unsigned int hash = io_get_work_hash(work);
+ bool last;
next_hashed = wq_next_work(work);
@@ -611,7 +613,7 @@ static void io_worker_handle_work(struct io_wq_acct *acct,
wq->do_work(work);
io_assign_current_work(worker, NULL);
- linked = wq->free_work(work);
+ last = wq->put_work(work, &linked);
work = next_hashed;
if (!work && linked && !io_wq_is_hashed(linked)) {
work = linked;
@@ -620,6 +622,13 @@ static void io_worker_handle_work(struct io_wq_acct *acct,
io_assign_current_work(worker, work);
if (linked)
io_wq_enqueue(wq, linked);
+ /*
+ * Not before the link is current or pending work again.
+ * This may be the last time that a task is woken up
+ * which exits and looks for requests to cancel.
+ */
+ if (last)
+ wq->free_work(done);
if (hash != -1U && !next_hashed) {
/* serialize hash clear with wake_up() */
@@ -920,9 +929,13 @@ static bool io_wq_worker_wake(struct io_worker *worker, void *data)
static void io_run_cancel(struct io_wq_work *work, struct io_wq *wq)
{
do {
+ struct io_wq_work *linked;
+
atomic_or(IO_WQ_WORK_CANCEL, &work->flags);
wq->do_work(work);
- work = wq->free_work(work);
+ if (wq->put_work(work, &linked))
+ wq->free_work(work);
+ work = linked;
} while (work);
}
@@ -1174,7 +1187,7 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data)
int ret, i;
struct io_wq *wq;
- if (WARN_ON_ONCE(!data->free_work || !data->do_work))
+ if (WARN_ON_ONCE(!data->put_work || !data->free_work || !data->do_work))
return ERR_PTR(-EINVAL);
if (WARN_ON_ONCE(!bounded))
return ERR_PTR(-EINVAL);
@@ -1185,6 +1198,7 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data)
refcount_inc(&data->hash->refs);
wq->hash = data->hash;
+ wq->put_work = data->put_work;
wq->free_work = data->free_work;
wq->do_work = data->do_work;
diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h
index b3b004a7b625..aeddc5b2b867 100644
--- a/io_uring/io-wq.h
+++ b/io_uring/io-wq.h
@@ -21,7 +21,8 @@ enum io_wq_cancel {
IO_WQ_CANCEL_NOTFOUND, /* work not found */
};
-typedef struct io_wq_work *(free_work_fn)(struct io_wq_work *);
+typedef bool (put_work_fn)(struct io_wq_work *, struct io_wq_work **);
+typedef void (free_work_fn)(struct io_wq_work *);
typedef void (io_wq_work_fn)(struct io_wq_work *);
struct io_wq_hash {
@@ -40,6 +41,7 @@ struct io_wq_data {
struct io_wq_hash *hash;
struct task_struct *task;
io_wq_work_fn *do_work;
+ put_work_fn *put_work;
free_work_fn *free_work;
};
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index d7b2284ae034..75265ebc3630 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1770,17 +1770,30 @@ int io_poll_issue(struct io_kiocb *req, struct io_tw_state *ts)
IO_URING_F_COMPLETE_DEFER);
}
-struct io_wq_work *io_wq_free_work(struct io_wq_work *work)
+/*
+ * io-wq is done with @work. Returns true if it held the last reference, and
+ * hands out the next request of the link then. The caller has to make sure
+ * that cancellations can find the latter before it calls io_wq_free_work().
+ */
+bool io_wq_put_work(struct io_wq_work *work, struct io_wq_work **linked)
{
struct io_kiocb *req = container_of(work, struct io_kiocb, work);
struct io_kiocb *nxt = NULL;
- if (req_ref_put_and_test_atomic(req)) {
- if (req->flags & IO_REQ_LINK_FLAGS)
- nxt = io_req_find_next(req);
- io_free_req(req);
- }
- return nxt ? &nxt->work : NULL;
+ *linked = NULL;
+ if (!req_ref_put_and_test_atomic(req))
+ return false;
+
+ if (req->flags & IO_REQ_LINK_FLAGS)
+ nxt = io_req_find_next(req);
+ if (nxt)
+ *linked = &nxt->work;
+ return true;
+}
+
+void io_wq_free_work(struct io_wq_work *work)
+{
+ io_free_req(container_of(work, struct io_kiocb, work));
}
void io_wq_submit_work(struct io_wq_work *work)
@@ -1791,7 +1804,7 @@ void io_wq_submit_work(struct io_wq_work *work)
bool needs_poll = false;
int ret = 0, err = -ECANCELED;
- /* one will be dropped by ->io_wq_free_work() after returning to io-wq */
+ /* one will be dropped by ->put_work() after returning to io-wq */
if (!(req->flags & REQ_F_REFCOUNT))
__io_req_set_refcount(req, 2);
else
diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h
index f1d3b4928997..3aec115c71aa 100644
--- a/io_uring/io_uring.h
+++ b/io_uring/io_uring.h
@@ -101,7 +101,8 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr);
int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin);
void __io_submit_flush_completions(struct io_ring_ctx *ctx);
-struct io_wq_work *io_wq_free_work(struct io_wq_work *work);
+bool io_wq_put_work(struct io_wq_work *work, struct io_wq_work **linked);
+void io_wq_free_work(struct io_wq_work *work);
void io_wq_submit_work(struct io_wq_work *work);
void io_free_req(struct io_kiocb *req);
diff --git a/io_uring/tctx.c b/io_uring/tctx.c
index 84f6a8385720..8597bcb903b2 100644
--- a/io_uring/tctx.c
+++ b/io_uring/tctx.c
@@ -35,6 +35,7 @@ static struct io_wq *io_init_wq_offload(struct io_ring_ctx *ctx,
data.hash = hash;
data.task = task;
+ data.put_work = io_wq_put_work;
data.free_work = io_wq_free_work;
data.do_work = io_wq_submit_work;
--
2.56.0
[-- Attachment #35: 0005-io_uring-io-wq-don-t-lose-the-signal-that-ends-work.patch --]
[-- Type: text/x-patch, Size: 3093 bytes --]
From 4ba6604ae71427a19a51d170b69124133968ff4f Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:28 +0000
Subject: [PATCH 05/38] io_uring/io-wq: don't lose the signal that ends work
Commit 63c103279a95e82280092f2f4232e91e8117a252 upstream.
When a thread exits, only its tracked requests are cancelled. Then its
io-wq is torn down. io_wq_exit_workers() signals every worker, to get it
out of an interruptible sleep, and waits for all of them to exit. Before
they do, they go through the work that is left. Unbounded work is
cancelled. Bounded work is run, as it is expected to end by itself.
Not all of it does. An open has no file yet, so it counts as bounded, and
the open of a FIFO waits for the other end. That is what the signal is for.
But the workers are signalled once, and a worker that hasn't started the
work by then loses the signal:
io_wq_exit_workers() io_worker_handle_work()
io_wq_worker_wake()
set TIF_NOTIFY_SIGNAL
io_assign_current_work()
io_run_task_work()
clear TIF_NOTIFY_SIGNAL
io_wq_submit_work()
fifo_open()
wait_for_partner()
wait_for_completion_timeout()
The thread never gets out of do_exit():
IORING_OP_OPENAT of a FIFO without a writer, IOSQE_ASYNC
exit of the thread that has submitted it
hangs in more than 9 of 10 attempts (4 CPUs, KVM), until the process is
killed.
Set TIF_NOTIFY_SIGNAL for bounded work that is started with IO_WQ_BIT_EXIT
set. The bit is set before the workers are signalled, and it is tested
after the flag has been cleared, with clear_notify_signal() providing the
barrier. So either the flag stays, or the bit is seen.
Work that is started while the io-wq exits is treated like work that is
being run at that time: it isn't interrupted unless it waits
interruptibly.
Fixes: afcc4015d1bf ("io-wq: provide an io_wq_put_and_exit() helper")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-4-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/io-wq.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index bd308709ba8b..4a0d7300c275 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -598,9 +598,16 @@ static void io_worker_handle_work(struct io_wq_acct *acct,
next_hashed = wq_next_work(work);
+ /*
+ * io_wq_exit_workers() signals us once, so that we
+ * don't sleep for ever. If that was before this work
+ * became the current one, the signal is gone.
+ */
if (do_kill &&
(atomic_read(&work->flags) & IO_WQ_WORK_UNBOUND))
atomic_or(IO_WQ_WORK_CANCEL, &work->flags);
+ else if (do_kill)
+ set_notify_signal(current);
wq->do_work(work);
io_assign_current_work(worker, NULL);
--
2.56.0
[-- Attachment #36: 0004-io_uring-wait-don-t-spin-on-a-stuck-overflow-list.patch --]
[-- Type: text/x-patch, Size: 2813 bytes --]
From feab0542a37f3c2fd36f8229d038b9b9a35a08c3 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:27 +0000
Subject: [PATCH 04/38] io_uring/wait: don't spin on a stuck overflow list
Commit 34f5a664c34d3cb62f6bec7f4156754594a7d25b upstream.
As long as IO_CHECK_CQ_OVERFLOW_BIT is set, io_cqring_wait_schedule()
returns right away, before it gets to look at signals or arms the timeout,
for io_cqring_wait() to flush the overflow list. If that doesn't empty the
list, and the CQ ring doesn't hold the number of events the caller wants,
the loop starts over, with the same result. The task spins in the kernel,
and can't be killed.
The assumption is that this can't happen: if CQEs can't be flushed, the
ring is full, and min_events is capped to the size of the ring. But:
- With IORING_SETUP_CQE_MIXED a 32b CQE needs two contiguous entries. It
is stuck on the list if a single one is free, and then the ring holds
less than cq_entries events. A single threaded application that asks
for as many events as the ring has entries never gets out of
io_uring_enter().
- Whether the ring is full is up to the CQ head, which the application
can write to. While one thread waits for events on an empty ring,
another one can move the head back so that the ring looks full and
have a CQE overflow. The waiter can't be stopped any more, not even by
SIGKILL.
Only the application can make room in the CQ ring. Return to it if the
list could not be flushed.
[ 6.12: io_cqring_wait() is in io_uring.c ]
Fixes: 3fcf19d592d5 ("io_uring: parse check_cq out of wq waiting")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-3-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---
io_uring/io_uring.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 0076c7599c94..d7b2284ae034 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2608,9 +2608,18 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, u32 flags,
check_cq = READ_ONCE(ctx->check_cq);
if (unlikely(check_cq)) {
- /* let the caller flush overflows, retry */
- if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT))
+ if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) {
io_cqring_do_overflow_flush(ctx);
+ /*
+ * No room for what has overflown. It takes the
+ * application to change that, so return to it.
+ */
+ if (test_bit(IO_CHECK_CQ_OVERFLOW_BIT,
+ &ctx->check_cq)) {
+ ret = 0;
+ break;
+ }
+ }
if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) {
ret = -EBADR;
break;
--
2.56.0
[-- Attachment #37: 0003-io_uring-cmd-don-t-leave-list-poison-in-hash_node.patch --]
[-- Type: text/x-patch, Size: 2180 bytes --]
From f93922ae1be32c550b3ea0210794abc7e5f95c9d Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:26 +0000
Subject: [PATCH 03/38] io_uring/cmd: don't leave list poison in ->hash_node
Commit 4c83e39f4d1944d7146fd06094f2c8053238d55c upstream.
io_uring_cmd_del_cancelable() uses hlist_del() to take a command off
->cancelable_uring_cmd, which leaves LIST_POISON1/2 behind in
->hash_node. Requests are recycled through the request cache without
being reinitialised, and not every user of ->hash_node sets it up before
it is used:
If a vectored futex wait is woken while futex_wait_multiple_setup() is
still queueing the futexes, io_futexv_wait() doesn't add the request to
->futex_list, and leaves the completion to the task_work queued by the
wake up. io_futexv_complete() ends in __io_futex_complete(), whose
hlist_del_init() is fine for a request that is unhashed, which is how all
other users of the node leave it. If the request was a cancelable
uring_cmd in its previous life, e.g. a ublk FETCH_REQ, it writes through
LIST_POISON2:
Oops: general protection fault, probably for non-canonical address
KASAN: maybe wild-memory-access in range [0xdead000000000120-...]
RIP: 0010:io_futexv_complete+0x219/0x420
Call Trace:
tctx_task_work_run+0x804/0x950
tctx_task_work+0x71/0xa0
task_work_run+0x118/0x200
Leave the node unhashed.
Fixes: 93b8cc60c37b ("io_uring: cancelable uring_cmd")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-2-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/uring_cmd.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c
index e5d0cc8a8a56..08d7e93c63cf 100644
--- a/io_uring/uring_cmd.c
+++ b/io_uring/uring_cmd.c
@@ -85,7 +85,7 @@ static void io_uring_cmd_del_cancelable(struct io_uring_cmd *cmd,
cmd->flags &= ~IORING_URING_CMD_CANCELABLE;
io_ring_submit_lock(ctx, issue_flags);
- hlist_del(&req->hash_node);
+ hlist_del_init(&req->hash_node);
io_ring_submit_unlock(ctx, issue_flags);
}
--
2.56.0
[-- Attachment #38: 0002-io_uring-waitid-own-the-request-before-clearing-head.patch --]
[-- Type: text/x-patch, Size: 3288 bytes --]
From e25a96eb5ca3b7eb5791986120a45852f5f8a6c3 Mon Sep 17 00:00:00 2001
From: Josef Bacik <josef@toxicpanda.com>
Date: Thu, 15 Oct 2026 19:30:25 +0000
Subject: [PATCH 02/38] io_uring/waitid: own the request before clearing ->head
Commit c254eec4364b9f4b4cbdce43be3fb4df5a651a70 upstream.
io_waitid_remove_wq() only takes the wait queue lock if ->head is set.
io_waitid_wait() clears ->head first thing, before it tries to become the
owner of the request by bumping ->refs. If somebody else owns the request
at that point, nothing keeps it from completing the request while the
wake up handler is still going to touch it:
CPU0 (io_waitid_wait) CPU1 (io_waitid, holds a ref)
list_del_init(&wait->entry)
smp_store_release(&iw->head, NULL)
__do_wait() finds another child
io_waitid_remove_wq()
// ->head == NULL, no locking
io_waitid_finish(), IOU_COMPLETE
// request freed and reused
atomic_fetch_inc(&iw->refs)
The same goes for __io_waitid_cancel() if it claims the request in that
window. ->refs is part of the per opcode data of the request, so this
increments whatever the next user of the request keeps there. If that
happens to be zero, the handler concludes that it owns the request, and
queues io_waitid_cb() for it. With the window held open by a delay, and
the request reused for a receive that is waiting for data:
BUG: KASAN: slab-use-after-free in __wake_up_common+0x1ca/0x1f0
__wake_up_sync_key+0x36/0x50
sock_def_readable+0xf1/0x420
unix_stream_sendmsg+0x60b/0x1110
Allocated by task 87:
io_arm_apoll+0x1ed/0x940
Freed by task 87:
__io_submit_flush_completions+0x118d/0x1a10
tctx_task_work_run+0x327/0x950
Only clear ->head once the handler owns the request. If it doesn't, the
owner finds ->head set and has to wait for the lock, which is held by the
caller of the handler. Deleting the already deleted entry again is
harmless.
Fixes: ab673c1bcaf2 ("io_uring/waitid: use io_waitid_remove_wq() consistently")
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Link: https://patch.msgid.link/20261015-b4-io-uring-fixes-v1-1-c1556f5d7bf5@toxicpanda.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
io_uring/waitid.c | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/io_uring/waitid.c b/io_uring/waitid.c
index 4475b02bafba..34b428fb4273 100644
--- a/io_uring/waitid.c
+++ b/io_uring/waitid.c
@@ -291,12 +291,18 @@ static int io_waitid_wait(struct wait_queue_entry *wait, unsigned mode,
return 0;
list_del_init(&wait->entry);
- smp_store_release(&iw->head, NULL);
- /* cancel is in progress */
+ /*
+ * Cancel or issue is in progress. Leave ->head alone, so that whoever
+ * owns the request has to take the wait queue lock, which we hold,
+ * before it can complete and free it.
+ */
if (atomic_fetch_inc(&iw->refs) & IO_WAITID_REF_MASK)
return 1;
+ /* pairs with the smp_load_acquire() in io_waitid_remove_wq() */
+ smp_store_release(&iw->head, NULL);
+
req->io_task_work.func = io_waitid_cb;
__io_req_task_work_add(req, IOU_F_TWQ_IN_WAKE);
return 1;
--
2.56.0
[-- Attachment #39: 0001-io-wq-order-the-exit-bit-against-worker-creation-tas.patch --]
[-- Type: text/x-patch, Size: 2127 bytes --]
From c5b78944dab510f32d90d0204c23cca998cbe3c2 Mon Sep 17 00:00:00 2001
From: Christian Brauner <brauner@kernel.org>
Date: Mon, 21 Sep 2026 15:44:53 +0200
Subject: [PATCH 01/38] io-wq: order the exit bit against worker creation task
work
Commit 0e5663856b69219ef2f02e04bac9f5c5be19718a upstream.
io_wq_exit_workers() cancels task work for queued worker creation with
task_work_cancel_match(). That has a plain load of task->task_works.
io_queue_worker_create() adds the new entry and tests IO_WQ_BIT_EXIT to
cancel it if the workqueue is already on its way out.
That must be ordered. task_work_add() has a full barrier via cmpxchg().
But io_wq_exit_start() sets the bit with set_bit() which doesn't have
any memory ordering.
So afaict, on weakly ordered architectures the exiting task may load
task_works before the store of the bit is visible. The other side tests
the bit before the store is visible as well. So the entry remains queued
with worker_refs and the exiting task waits on worker_done indefinitely.
If the task still has rings then io_uring_del_tctx_node() provides the
barrier via test_and_set_bit() in io_wq_set_exit_on_idle(). When it
has closed all rings though that barrier is gone.
Add the barrier after the set_bit().
Fixes: 71a85387546e ("io-wq: check for wq exit after adding new worker task_work")
Cc: stable@vger.kernel.org
Reviewed-by: Jens Axboe <axboe@kernel.dk>
Acked-by: Oleg Nesterov <oleg@redhat.com>
Link: https://patch.msgid.link/20260921-work-coredump-fixes-v3-4-8e4adb1619e6@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
---
io_uring/io-wq.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index 3c1818e8feaa..bd308709ba8b 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -1234,6 +1234,8 @@ static bool io_task_work_match(struct callback_head *cb, void *data)
void io_wq_exit_start(struct io_wq *wq)
{
set_bit(IO_WQ_BIT_EXIT, &wq->state);
+ /* Pairs with task_work_add() in io_queue_worker_create(). */
+ smp_mb__after_atomic();
}
static void io_wq_cancel_tw_create(struct io_wq *wq)
--
2.56.0
^ permalink raw reply related [flat|nested] only message in thread
only message in thread, other threads:[~2026-10-28 13:51 UTC | newest]
Thread overview: (only message) (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-28 13:51 Queued patches for 6.12-stable Jens Axboe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox