Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed

[K Prateek Nayak <kprateek.nayak@amd.com>]

Hello John,

On 4/28/2026 12:08 AM, John Stultz wrote:
> Vineeth reported seeing a KVM related deadlock connected to work
> queue lockups using the android17-6.18 tree, which has
> Proxy Execution enabled (using the full patch stack), but I've
> subsequently reproduced it on v7.1-rc1.
> 
> On further debugging he found:
> - kvm-irqfd-cleanup workqueue and rcu_gp lands in a per-cpu
>   pwq(work queue pool)
> - one of kvm-irqfd-cleanup worker(say A) takes a mutex and then
>   calls synchronize_srcu_expedited()
> - one other kvm-irqfd-cleanup worker worker(Say B) tries to
>   acquire the lock and then gets blocked
> - On the way to blocking, this cpu gets an IPI and on return
>   from IPI, it calls __schedule() and did not get to complete
>   workqueue accounting(worker->sleeping = 0 and decrementing
>   pool->nr_running). This is done in sched_submit_work() ->
>   wq_worker_sleeping() called from schedule() and we got
>   preempted before that.
> - proxy execution doesn't immediately take it off run queue as
>   p->blocked_on is set during __mutex_lock
> - Next time when B is picked for running, it notices A(mutex
>   holder) is not on a runqueue and then blocks B.
>   find_proxy_task() -> proxy_deactivate() -> block_task()
> - And things are then stuck. A is waiting for the workqueue to
>   be run, but B can't run the workqueue as it is blocked on A.
> 
> The trouble is that with Proxy Execution, in
> __mutex_lock_common() we set the task state to
> TASK_UNINTERRUPTIBLE, and set blocked_on before calling into
> schedule(), where sched_submit_work() will be called.

Geez! That is an interesting race.

> 
> But if an IPI comes in before we call schedule() the interrupt
> will call __schedule(SM_PREEMPT) directly. This causes the
> scheduler to see the current task as blocked_on, and deactivate
> it (because the owner is off the runqueue).
> 
> Since its deactivated, it wont' be run, and it won't get to
> call sched_submit_work().
> 
> Without proxy-execution, the SM_PREEMPT case will prevent the
> task from being dequeued, and it can be reselected again and
> run, which will allow it to finish calling into schedule()
> and calling sched_submit_work() before actually blocking.
> 
> So we need to make sure on the SM_PREEMPT case, if current is
> marked as blocked_on, we should clear the blocked_on state and
> mark the task RUNNABLE so the task can be selected to complete
> its call to schedule() -> sched_submit_work().
> 
> Now because we cleared BLOCKED_ON and set the task RUNNABLE,
> the task will be able to be selected and run again and loop back
> in __mutex_lock_common() where it can re-set the blocked_on
> state and call back into schedule() in order to properly be
> chosen as a donor.
> 
> Many thanks to Vineeth for figuring this very obscure race out
> and for implementing a test tool to make it easily reproducible!
> 
> Reported-by: Vineeth Pillai <vineethrp@google.com>
> Tested-by: Vineeth Pillai <vineethrp@google.com>
> Signed-off-by: John Stultz <jstultz@google.com>

I guess it is missing a:

Fixes: be41bde4c3a8 ("sched: Add an initial sketch of the find_proxy_task() function")

since that is where we began blocking a task on task_is_blocked(). I
really wish there was a better way to have detected this but I cannot
think of any better way at the moment so feel free to include:

Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>

> ---
> Cc: Vineeth Pillai <vineethrp@google.com>
> Cc: Sonam Sanju <sonam.sanju@intel.com>
> Cc: Sean Christopherson <seanjc@google.com>
> Cc: Kunwu Chan <kunwu.chan@linux.dev>
> Cc: Tejun Heo <tj@kernel.org>
> Cc: Joel Fernandes <joelagnelf@nvidia.com>
> Cc: Qais Yousef <qyousef@layalina.io>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Juri Lelli <juri.lelli@redhat.com>
> Cc: Vincent Guittot <vincent.guittot@linaro.org>
> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
> Cc: Valentin Schneider <vschneid@redhat.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Will Deacon <will@kernel.org>
> Cc: Waiman Long <longman@redhat.com>
> Cc: Boqun Feng <boqun.feng@gmail.com>
> Cc: "Paul E. McKenney" <paulmck@kernel.org>
> Cc: Metin Kaya <Metin.Kaya@arm.com>
> Cc: Xuewen Yan <xuewen.yan94@gmail.com>
> Cc: K Prateek Nayak <kprateek.nayak@amd.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
> Cc: Suleiman Souhlal <suleiman@google.com>
> Cc: kuyo chang <kuyo.chang@mediatek.com>
> Cc: hupu <hupu.gm@gmail.com>
> Cc: kernel-team@android.com
> ---
>  kernel/sched/core.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index da20fb6ea25ae..5f684caefd8b2 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -7097,6 +7097,17 @@ static void __sched notrace __schedule(int sched_mode)
>  		try_to_block_task(rq, prev, &prev_state,
>  				  !task_is_blocked(prev));
>  		switch_count = &prev->nvcsw;
> +	} else if (preempt && prev->blocked_on) {
> +		/*
> +		 * If we are SM_PREEMPT, we may have interrupted
> +		 * after blocked_on was set, before schedule()
> +		 * was run, preventing workques from running. So
> +		 * clear blocked_on and mark task RUNNING so it
> +		 * can be reselected to run and complete its
> +		 * logic
> +		 */
> +		WRITE_ONCE(prev->__state, TASK_RUNNING);

nit.

You probably need to update "prev_state" too for trace_sched_switch() to
capture the right state down below.

Since this is on the way to schedule(), I wonder if it possible to just
do a "next = prev" and goto picked ... but that adds more latency on
PREEMPT_RT so that is a no go I presume.

> +		clear_task_blocked_on(prev, NULL);
>  	}
>  
>  pick_again:

-- 
Thanks and Regards,
Prateek