Re: sched/isolation: tick_take_do_timer_from_boot() calls smp_call_function_single() with irqs disabled

[Oleg Nesterov <oleg@redhat.com>]

On 05/22, Oleg Nesterov wrote:
>
> After the recent comment 5097cbcb38e6 ("sched/isolation: Prevent boot crash
> when the boot CPU is nohz_full") the kernel no longer crashes, but there is
> another problem.
>
> In this case tick_setup_device() does tick_take_do_timer_from_boot() to
> update tick_do_timer_cpu and this triggers WARN_ON_ONCE(irqs_disabled())
> in smp_call_function_single().
>
> I don't understand this code even remotely, I failed to find the fix.
>
> Perhaps we can use smp_call_function_single_async() as a workaround ?
>
> But I don't even understand why exactly we need smp_call_function()...

...

> Race with tick_nohz_stop_tick() on boot CPU which can set
> tick_do_timer_cpu = TICK_DO_TIMER_NONE? Is it really bad?

And is it supposed to happen if tick_nohz_full_running ?

tick_sched_do_timer() and can_stop_idle_tick() claim that
TICK_DO_TIMER_NONE is not possible in this case...

So, once again, could you explain why the patch below is wrong?

Oleg.
---

diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c
index d88b13076b79..907b44d8cf1f 100644
--- a/kernel/time/tick-common.c
+++ b/kernel/time/tick-common.c
@@ -178,26 +178,6 @@ void tick_setup_periodic(struct clock_event_device *dev, int broadcast)
 	}
 }
 
-#ifdef CONFIG_NO_HZ_FULL
-static void giveup_do_timer(void *info)
-{
-	int cpu = *(unsigned int *)info;
-
-	WARN_ON(tick_do_timer_cpu != smp_processor_id());
-
-	tick_do_timer_cpu = cpu;
-}
-
-static void tick_take_do_timer_from_boot(void)
-{
-	int cpu = smp_processor_id();
-	int from = tick_do_timer_boot_cpu;
-
-	if (from >= 0 && from != cpu)
-		smp_call_function_single(from, giveup_do_timer, &cpu, 1);
-}
-#endif
-
 /*
  * Setup the tick device
  */
@@ -231,9 +211,8 @@ static void tick_setup_device(struct tick_device *td,
 
 		} else if (tick_do_timer_boot_cpu != -1 &&
 						!tick_nohz_full_cpu(cpu)) {
-			tick_take_do_timer_from_boot();
 			tick_do_timer_boot_cpu = -1;
-			WARN_ON(READ_ONCE(tick_do_timer_cpu) != cpu);
+			WRITE_ONCE(tick_do_timer_cpu, cpu);
 #endif
 		}
 
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 71a792cd8936..3b1d011d45e1 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -1014,6 +1014,9 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
 	 */
 	tick_cpu = READ_ONCE(tick_do_timer_cpu);
 	if (tick_cpu == cpu) {
+#ifdef CONFIG_NO_HZ_FULL
+		WARN_ON_ONCE(tick_nohz_full_running);
+#endif
 		WRITE_ONCE(tick_do_timer_cpu, TICK_DO_TIMER_NONE);
 		tick_sched_flag_set(ts, TS_FLAG_DO_TIMER_LAST);
 	} else if (tick_cpu != TICK_DO_TIMER_NONE) {