Linux kernel -stable discussions
 help / color / mirror / Atom feed
From: Jens Axboe <axboe@kernel.dk>
To: stable@vger.kernel.org
Subject: [PATCH 6.6.y 19/19] io_uring/sqpoll: do the return-to-userspace work in the SQPOLL thread
Date: Sat, 21 Nov 2026 21:30:49 +0000	[thread overview]
Message-ID: <20261121213049.1788332-20-axboe@kernel.dk> (raw)
In-Reply-To: <20261121213049.1788332-1-axboe@kernel.dk>

From: Shakeel Butt <shakeel.butt@linux.dev>

Commit f013b695506491c5202a412a6568313549c81f50 upstream.

The SQPOLL thread never returns to userspace. io_sq_tw() runs its
task_work, but the rest of resume_user_mode_work(), like memory.high
reclaim and block cgroup throttling, only runs when the thread exits.
So memory.high reclaim only happens inside a charge, with sqd->lock and
ctx->uring_lock held, and not at all if the charges can't block.

This used to work. The loop called io_run_task_work(), which does this
work since commit b5d3ae202fbf ("io_uring: handle TIF_NOTIFY_RESUME
when checking for task_work"). Commit af5d68f8892f ("io_uring/sqpoll:
manage task_work privately") replaced that call with io_sq_tw().

Call resume_user_mode_work() at the top of the loop when
TIF_NOTIFY_RESUME is set, with sqd->lock dropped. It runs once per pass,
so a pass that charges two or more batches still reclaims inside the
charge, as before.

Tested in a VM with memory.high=64M and swap on zram. An SQPOLL ring
reads 256M from /dev/zero into untouched memory, 64K at a time. Before,
the thread made 337 calls to __mem_cgroup_handle_over_high(), all from
inside the charge. After, it makes 704, all from the top of the loop.
memory.peak stays at 64M. With 1M reads, 296 of its 385 calls still come
from inside the charge. Both runs take about 50% longer, and the extra
time is in the handler, which now runs more often. Without memory.high
pressure, the test never calls resume_user_mode_work() and the times
do not change. Lockdep showed no warnings, also with io_uring_register()
racing for sqd->lock.

Fixes: af5d68f8892f ("io_uring/sqpoll: manage task_work privately")
Cc: stable@vger.kernel.org
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Link: https://patch.msgid.link/20261101022146.309201-1-shakeel.butt@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 io_uring/sqpoll.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c
index 3d0fbae80938..263da37d45ee 100644
--- a/io_uring/sqpoll.c
+++ b/io_uring/sqpoll.c
@@ -318,6 +318,18 @@ static int io_sq_thread(void *data)
 			timeout = jiffies + sqd->sq_thread_idle;
 		}
 
+		/*
+		 * The SQPOLL thread never returns to userspace, so do the
+		 * return-to-userspace work, like memory.high reclaim, here.
+		 * It can sleep for a long time, so drop sqd->lock.
+		 */
+		if (test_thread_flag(TIF_NOTIFY_RESUME)) {
+			mutex_unlock(&sqd->lock);
+			resume_user_mode_work(NULL);
+			mutex_lock(&sqd->lock);
+			sqd->sq_cpu = raw_smp_processor_id();
+		}
+
 		cap_entries = !list_is_singular(&sqd->ctx_list);
 		list_for_each_entry(ctx, &sqd->ctx_list, sqd_list) {
 			int ret = __io_sq_thread(ctx, cap_entries);
-- 
2.55.0


      parent reply	other threads:[~2026-11-21 21:31 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-21 21:30 [PATCH 6.6.y 00/19] io_uring fixes for 6.6-stable Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 01/19] io_uring: enable toggle of iowait usage when waiting on CQEs Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 02/19] io_uring: fail soft links on negative IOU_COMPLETE results Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 03/19] io_uring/rw: fail soft links on IOPOLL short or error IO Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 04/19] io_uring/timeout: don't rearm a multishot timeout on cancel Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 05/19] io_uring/rsrc: keep the old file when a file update fails Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 06/19] io_uring: cleanup handle_tw_list() calling convention Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 07/19] io_uring: pass in counter to handle_tw_list() rather than return it Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 08/19] io_uring/sqpoll: manage task_work privately Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 09/19] io_uring: wake SQPOLL task when task_work is added to an empty queue Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 10/19] io_uring/sqpoll: use the correct check for pending task_work Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 11/19] io_uring/sqpoll: early exit thread if task_context wasn't allocated Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 12/19] io_uring/sqpoll: ensure that normal task_work is also run timely Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 13/19] io_uring: signal SQPOLL task_work with TWA_SIGNAL_NO_IPI Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 14/19] io_uring/sqpoll: ensure task state is TASK_RUNNING when running task_work Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 15/19] io_uring: don't touch sqd->thread off tw add Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 16/19] io_uring/sqpoll: zero sqd->thread on tctx errors Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 17/19] io_uring/sqpoll: Increase task_work submission batch size Jens Axboe
2026-11-21 21:30 ` [PATCH 6.6.y 18/19] io_uring: fix task_work add use-after-free with SQPOLL Jens Axboe
2026-11-21 21:30 ` Jens Axboe [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261121213049.1788332-20-axboe@kernel.dk \
    --to=axboe@kernel.dk \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox