Re: sched/isolation: tick_take_do_timer_from_boot() calls smp_call_function_single() with irqs disabled

Oleg Nesterov
2024-05-23T13:25:32Z

On 05/22, Oleg Nesterov wrote:
>
> After the recent comment 5097cbcb38e6 ("sched/isolation: Prevent boot crash
> when the boot CPU is nohz_full") the kernel no longer crashes, but there is
> another problem.
>
> In this case tick_setup_device() does tick_take_do_timer_from_boot() to
> update tick_do_timer_cpu and this triggers WARN_ON_ONCE(irqs_disabled())
> in smp_call_function_single().
>
> I don't understand this code even remotely, I failed to find the fix.
>
> Perhaps we can use smp_call_function_single_async() as a workaround ?
>
> But I don't even understand why exactly we need smp_call_function()...

...

> Race with tick_nohz_stop_tick() on boot CPU which can set
> tick_do_timer_cpu = TICK_DO_TIMER_NONE? Is it really bad?

And is it supposed to happen if tick_nohz_full_running ?

tick_sched_do_timer() and can_stop_idle_tick() claim that
TICK_DO_TIMER_NONE is not possible in this case...

So, once again, could you explain why the patch below is wrong?

Oleg.
---

diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c
index d88b13076b79..907b44d8cf1f 100644
--- a/kernel/time/tick-common.c
+++ b/kernel/time/tick-common.c
@@ -178,26 +178,6 @@ void tick_setup_periodic(struct clock_event_device *dev, int broadcast)
}
}

-#ifdef CONFIG_NO_HZ_FULL
-static void giveup_do_timer(void *info)
-{
- int cpu = *(unsigned int *)info;
-
- WARN_ON(tick_do_timer_cpu != smp_processor_id());
-
- tick_do_timer_cpu = cpu;
-}
-
-static void tick_take_do_timer_from_boot(void)
-{
- int cpu = smp_processor_id();
- int from = tick_do_timer_boot_cpu;
-
- if (from >= 0 && from != cpu)
- smp_call_function_single(from, giveup_do_timer, &cpu, 1);
-}
-#endif
-
/*
* Setup the tick device
*/
@@ -231,9 +211,8 @@ static void tick_setup_device(struct tick_device *td,

} else if (tick_do_timer_boot_cpu != -1 &&
!tick_nohz_full_cpu(cpu)) {
- tick_take_do_timer_from_boot();
tick_do_timer_boot_cpu = -1;
- WARN_ON(READ_ONCE(tick_do_timer_cpu) != cpu);
+ WRITE_ONCE(tick_do_timer_cpu, cpu);
#endif
}

diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 71a792cd8936..3b1d011d45e1 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -1014,6 +1014,9 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
*/
tick_cpu = READ_ONCE(tick_do_timer_cpu);
if (tick_cpu == cpu) {
+#ifdef CONFIG_NO_HZ_FULL
+ WARN_ON_ONCE(tick_nohz_full_running);
+#endif
WRITE_ONCE(tick_do_timer_cpu, TICK_DO_TIMER_NONE);
tick_sched_flag_set(ts, TS_FLAG_DO_TIMER_LAST);
} else if (tick_cpu != TICK_DO_TIMER_NONE) {