BUG-RT: scheduling while in atomic in the watchdog's hrtimer

[Daniel Bristot de Oliveira <bristot@redhat.com>]

Hi,

In the 4.19/5.0-rt configured !SMP, the following chain of events can
take place:

------------------------------ %< -------------------------------------
smp_apic_timer_interrupt(){
	hrtimer_interrupt() {
		__hrtimer_run_queues() {
			watchdog_timer_fn() {
				stop_one_cpu_nowait() {
					#ifdef !CONFIG_SMP
						schedule_work() {
							queue_work() {
								queue_work_on() {
								/* phew, long, ah!?.... */	
									local_lock_irqsave() {
										__local_lock_irqsave() {
											__local_lock_irq() {
												spin_lock_irqsave() {
													rt_spin_lock()...
														/* from here on you already know...*/
														/* a lot of } */
------------------------------ >% -------------------------------------

And this can cause a scheduling while in atomic.

Tracking down, the commit that introduced the stop_one_cpu_nowait()->
queue_work_on() to the path was:

------------------------------ %< -------------------------------------
commit 9cf57731b63e37ed995b46690adc604891a9a28f
Author: Peter Zijlstra 
Date:   Thu Jun 7 10:52:03 2018 +0200

    watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work
    
    Oleg suggested to replace the "watchdog/%u" threads with
    cpu_stop_work. That removes one thread per CPU while at the same time
    fixes softlockup vs SCHED_DEADLINE.
    
    But more importantly, it does away with the single
    smpboot_update_cpumask_percpu_thread() user, which allows
    cleanups/shrinkage of the smpboot interface.
    
    Suggested-by: Oleg Nesterov 
    Signed-off-by: Peter Zijlstra (Intel) 
    Cc: Linus Torvalds 
    Cc: Peter Zijlstra 
    Cc: Thomas Gleixner 
    Cc: linux-kernel@vger.kernel.org
    Signed-off-by: Ingo Molnar 
------------------------------ >% -------------------------------------

Later modified by:

------------------------------ %< -------------------------------------
commit be45bf5395e0886a93fc816bbe41a008ec2e42e2
Author: Peter Zijlstra 
Date:   Fri Jul 13 12:42:08 2018 +0200

    watchdog/softlockup: Fix cpu_stop_queue_work() double-queue bug
    
    When scheduling is delayed for longer than the softlockup interrupt
    period it is possible to double-queue the cpu_stop_work, causing list
    corruption.
    
    Cure this by adding a completion to track the cpu_stop_work's
    progress.
    
    Reported-by: kernel test robot 
    Tested-by: Rong Chen 
    Signed-off-by: Peter Zijlstra (Intel) 
    Cc: Linus Torvalds 
    Cc: Peter Zijlstra 
    Cc: Thomas Gleixner 
    Fixes: 9cf57731b63e ("watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work")
    Link: http://lkml.kernel.org/r/20180713104208.GW2494@hirez.programming.kicks-ass.net
    Signed-off-by: Ingo Molnar 
------------------------------ >% -------------------------------------

I hit it while trying the automata model against the 4.19-rt/5.0-rt 
kernels.

You can find more about it here:
http://bristot.me/another-bug-found-with-the-model/

Thanks
-- Daniel