[PATCH 7.3.y 12/16] io_uring: free the task context after a grace period

[Jens Axboe <axboe@kernel.dk>]

From: Chris Mason <mason@kernel.org>

Commit 9f5315d3f4368590c9fe06d4313b7247470bcf15 upstream.

A task other than the owner can free a request, io_ring_exit_work()
running the local task_work of a DEFER_TASKRUN ring for example, or
io_do_iopoll() reaping completions. io_put_task() then takes the else
branch:

    percpu_counter_sub(&tctx->inflight, 1);
    if (unlikely(atomic_read(&tctx->in_cancel)))
            wake_up(&tctx->wait);
    put_task_struct(tctx->task);

The first line is what lets the owner leave io_uring_cancel_generic(),
and on exec it frees the tctx right after. Nothing holds that off for
the remaining three lines, the task reference only covers a tctx freed
along with the task_struct. It used to work out because
io_uring_del_tctx_node() took the uring_lock that the freeing side holds
across io_put_task(), until that moved to tctx_lock.

Free the tctx with kfree_rcu(), and enter an RCU read section in
io_put_task() before the count is dropped.

Fixes: 5623eb1ed035 ("io_uring/tctx: add separate lock for list of tctx's in ctx")
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 include/linux/io_uring_types.h | 3 +++
 io_uring/io_uring.c            | 2 ++
 io_uring/tctx.c                | 2 +-
 3 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h
index c5edf6246a4b..1c85ce0e471d 100644
--- a/include/linux/io_uring_types.h
+++ b/include/linux/io_uring_types.h
@@ -161,6 +161,9 @@ struct io_uring_task {
 	/* drains ->task_list once the task can no longer run task_work */
 	struct work_struct		fallback_work;
 
+	/* the tctx is freed after a grace period, see io_put_task() */
+	struct rcu_head			rcu;
+
 	struct { /* task_work */
 		struct mpscq		task_list;
 		struct callback_head	task_work;
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 880be1fed3a6..c8aa6ca4a7cc 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -634,6 +634,8 @@ static inline void io_put_task(struct io_kiocb *req)
 	if (likely(tctx->task == current)) {
 		tctx->cached_refs++;
 	} else {
+		guard(rcu)();
+
 		percpu_counter_sub(&tctx->inflight, 1);
 		if (unlikely(atomic_read(&tctx->in_cancel)))
 			wake_up(&tctx->wait);
diff --git a/io_uring/tctx.c b/io_uring/tctx.c
index 291e30e00e45..b4fecea3ea47 100644
--- a/io_uring/tctx.c
+++ b/io_uring/tctx.c
@@ -64,7 +64,7 @@ void io_uring_free_tctx(struct task_struct *tsk)
 		WARN_ON_ONCE(tctx->cached_refs);
 
 		percpu_counter_destroy(&tctx->inflight);
-		kfree(tctx);
+		kfree_rcu(tctx, rcu);
 		tsk->io_uring = NULL;
 	}
 }
-- 
2.56.0