Re: [PATCH 1/2] powerpc/watchdog: prevent printk and send IPI while holding the wd lock

[Nicholas Piggin <npiggin@gmail.com>]

Excerpts from Laurent Dufour's message of October 27, 2021 6:14 pm:
> Le 27/10/2021 à 05:29, Nicholas Piggin a écrit :
>> Excerpts from Laurent Dufour's message of October 27, 2021 2:27 am:
>>> When handling the Watchdog interrupt, long processing should not be done
>>> while holding the __wd_smp_lock. This prevents the other CPUs to grab it
>>> and to process Watchdog timer interrupts. Furhtermore, this could lead to
>>> the following situation:
>>>
>>> CPU x detect lockup on CPU y and grab the __wd_smp_lock
>>>        in watchdog_smp_panic()
>>> CPU y caught the watchdog interrupt and try to grab the __wd_smp_lock
>>>        in soft_nmi_interrupt()
>>> CPU x wait for CPU y to catch the IPI for 1s in __smp_send_nmi_ipi()
>> 
>> CPU y should get the IPI here if it's a NMI IPI (which will be true for
>>> = POWER9 64s).
>> 
>> That said, not all platforms support it and the console lock problem
>> seems real, so okay.
>> 
>>> CPU x will timeout and so has spent 1s waiting while holding the
>>>        __wd_smp_lock.
>>>
>>> A deadlock may also happen between the __wd_smp_lock and the console_owner
>>> 'lock' this way:
>>> CPU x grab the console_owner
>>> CPU y grab the __wd_smp_lock
>>> CPU x catch the watchdog timer interrupt and needs to grab __wd_smp_lock
>>> CPU y wants to print something and wait for console_owner
>>> -> deadlock
>>>
>>> Doing all the long processing without holding the _wd_smp_lock prevents
>>> these situations.
>> 
>> The intention was to avoid logs getting garbled e.g., if multiple
>> different CPUs fire at once.
>> 
>> I wonder if instead we could deal with that by protecting the IPI
>> sending and printing stuff with a trylock, and if you don't get the
>> trylock then just return, and you'll come back with the next timer
>> interrupt.
> 
> That sounds a bit risky to me, especially on large system when system goes 
> wrong, all the CPU may try lock here.

That should be okay though, one will get through and the others will 
skip.

> Furthermore, now operation done under the lock protection are quite fast, there 
> is no more spinning like the delay loop done when sending an IPI.
> 
> Protecting the IPI sending is a nightmare, if the target CPU is later play with 
> the lock we are taking during the IPI processing, furthermore, if the target CPU 
> is not responding the sending CPU is waiting for 1s, which slows all the system 
> due to the lock held.
> Since I do a copy of the pending CPU mask and clear it under the lock 
> protection, the IPI sending is safe while done without holding the lock.

Protecting IPI sending basically has all the same issues in the NMI
IPI layer.

> 
> Regarding the interleaved traces, I don't think this has to be managed down 
> here, but rather in the printk/console path.

It can't necessarily be because some of the problem is actually that a 
NMI handler can be interrupted by another NMI IPI because the caller
can return only after handlers start running rather than complete.

I don't think it would be an additional nightmare to trylock.

Thanks,
Nick