sched/isolation: tick_take_do_timer_from_boot() calls smp_call_function_single() with irqs disabled

Oleg Nesterov
2024-05-22T15:19:19Z

After the recent comment 5097cbcb38e6 ("sched/isolation: Prevent boot crash
when the boot CPU is nohz_full") the kernel no longer crashes, but there is
another problem.

In this case tick_setup_device() does tick_take_do_timer_from_boot() to
update tick_do_timer_cpu and this triggers WARN_ON_ONCE(irqs_disabled())
in smp_call_function_single().

I don't understand this code even remotely, I failed to find the fix.

Perhaps we can use smp_call_function_single_async() as a workaround ?

But I don't even understand why exactly we need smp_call_function()...
Lets suppose we change

--- a/kernel/time/tick-common.c
+++ b/kernel/time/tick-common.c
@@ -231,9 +231,9 @@ static void tick_setup_device(struct tick_device *td,

} else if (tick_do_timer_boot_cpu != -1 &&
!tick_nohz_full_cpu(cpu)) {
- tick_take_do_timer_from_boot();
+// tick_take_do_timer_from_boot();
tick_do_timer_boot_cpu = -1;
- WARN_ON(READ_ONCE(tick_do_timer_cpu) != cpu);
+ WRITE_ONCE(tick_do_timer_cpu), cpu);
#endif
}

Can anyone explain what exactly can go wrong?

Race with tick_nohz_stop_tick() on boot CPU which can set
tick_do_timer_cpu = TICK_DO_TIMER_NONE? Is it really bad?

Something else?

Please help, I don't think I can find a sane solution :/

Oleg.