Re: [PATCH] rcu: Avoid invalid wakeup for rcuc kthreads in RCU_KTHREAD_OFFCPU status

[Frederic Weisbecker <frederic@kernel.org>]

On Tue, Nov 15, 2022 at 09:19:26PM +0800, Zqiang wrote:
> For CONFIG_PREEMPT_RT=y kernel, the "use_softirq=0" will be set, the
> RCU_SOFTIRQ processing is moved to per-CPU rcuc kthreads which created
> by smpboot_register_percpu_thread(). when CPU is going offline, the
> corresponding rcu_data.rcu_cpu_kthread_status is set RCU_KTHREAD_OFFCPU,
> and the rcuc kthreads enter TASK_PARKED state, kthreads in TASK_PARKED
> state only accept kthread_unpark() to wakeup.
> 
> Therefore, This commit avoid invoke wake_up_process() to rcuc kthreads
> in TASK_PARKED state.
> 
> Signed-off-by: Zqiang <qiang1.zhang@intel.com>
> ---
>  kernel/rcu/tree.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 3ccad468887e..49dd87356851 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -2375,7 +2375,8 @@ static void rcu_wake_cond(struct task_struct *t, int status)
>  	 * If the thread is yielding, only wake it when this
>  	 * is invoked from idle
>  	 */
> -	if (t && (status != RCU_KTHREAD_YIELDING || is_idle_task(current)))
> +	if (t && (status != RCU_KTHREAD_YIELDING || is_idle_task(current)) &&
> +				status != RCU_KTHREAD_OFFCPU)
>  		wake_up_process(t);

There is a tiny window where this can happen (between CPUHP_TEARDOWN_CPU
and CPUHP_AP_SMPBOOT_THREADS) and it can't cause a spurious unpark because
wake_up_process() only wakes up from TASK_[UN]INTERRUPTIBLE states. And even
if it did, the KTHREAD_SHOULD_PARK bit would still be on.

And more important! On unpark time RCU_KTHREAD_OFFCPU isn't cleared. Only the
rcuc kthread does it, and after your patch it couldn't be awaken to perform
that, unless rcuc is lucky enough to have rcu_data.rcu_cpu_has_work = 1
by the time it unparks and that isn't guaranteed. So rcuc may sleep forever.

OTOH one cleanup that could be done is to make rcu_cpu_kthread_park() to use
__this_cpu_write as it's guaranteed that cpu == smp_processor_id().

Thanks.