[PATCH] panic: Fix a possible deadlock in panic()

[PATCH] panic: Fix a possible deadlock in panic()

[Vikram Mulukutla]

panic_lock is meant to ensure that panic processing takes
place only on one cpu; if any of the other cpus encounter
a panic, they will spin waiting to be shut down.

However, this causes a regression in this scenario:

1. Cpu 0 encounters a panic and acquires the panic_lock
   and proceeds with the panic processing.
2. There is an interrupt on cpu 0 that also encounters
   an error condition and invokes panic.
3. This second invocation fails to acquire the panic_lock
   and enters the infinite while loop in panic_smp_self_stop.

Thus all panic processing is stopped, and the cpu is stuck
for eternity in the while(1) inside panic_smp_self_stop.

To address this, disable local interrupts with
local_irq_disable before acquiring the panic_lock. This will
prevent interrupt handlers from executing during the panic
processing, thus avoiding this particular problem.

Signed-off-by: Vikram Mulukutla <markivx@codeaurora.org>
Reviewed-by: Stephen Boyd <sboyd@codeaurora.org>
---
 kernel/panic.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/kernel/panic.c b/kernel/panic.c
index d2a5f4e..e1b2822 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -75,6 +75,14 @@ void panic(const char *fmt, ...)
 	int state = 0;
 
 	/*
+	 * Disable local interrupts. This will prevent panic_smp_self_stop
+	 * from deadlocking the first cpu that invokes the panic, since
+	 * there is nothing to prevent an interrupt handler (that runs
+	 * after the panic_lock is acquired) from invoking panic again.
+	 */
+	local_irq_disable();
+
+	/*
 	 * It's possible to come here directly from a panic-assertion and
 	 * not have preempt disabled. Some functions called from here want
 	 * preempt to be disabled. No point enabling it later though...
-- 
1.7.8.3

Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: [PATCH] panic: Fix a possible deadlock in panic()

[Michael Holzheu]

Hello Vikram,

Putting "linux-arch" on cc...

On Thu, 28 Jun 2012 16:43:05 -0700
Vikram Mulukutla <markivx@codeaurora.org> wrote:
> panic_lock is meant to ensure that panic processing takes
> place only on one cpu; if any of the other cpus encounter
> a panic, they will spin waiting to be shut down.
> 
> However, this causes a regression in this scenario:
> 
> 1. Cpu 0 encounters a panic and acquires the panic_lock
>    and proceeds with the panic processing.
> 2. There is an interrupt on cpu 0 that also encounters
>    an error condition and invokes panic.
> 3. This second invocation fails to acquire the panic_lock
>    and enters the infinite while loop in panic_smp_self_stop.
> 
> Thus all panic processing is stopped, and the cpu is stuck
> for eternity in the while(1) inside panic_smp_self_stop.
> 
> To address this, disable local interrupts with
> local_irq_disable before acquiring the panic_lock. This will
> prevent interrupt handlers from executing during the panic
> processing, thus avoiding this particular problem.

Looks good to me.

I re-read the panic lock discussion and in fact one version of my patch
also disabled interrupts:

http://lists.infradead.org/pipermail/kexec/2011-October/005695.html

I think the reason why we later took a version with irqs enabled was
that we did not think about the scenario you described above and
we wanted to make the change as less intrusive as possible. But I am
not really sure about that.

Regarding you patch: Perhaps we could use spin_trylock_irq() instead of
local_irq_disable() and spin_lock().

Michael

Re: [PATCH] panic: Fix a possible deadlock in panic()

[Andrew Morton]

On Thu, 28 Jun 2012 16:43:05 -0700
Vikram Mulukutla <markivx@codeaurora.org> wrote:

> panic_lock is meant to ensure that panic processing takes
> place only on one cpu; if any of the other cpus encounter
> a panic, they will spin waiting to be shut down.
> 
> However, this causes a regression in this scenario:
> 
> 1. Cpu 0 encounters a panic and acquires the panic_lock
>    and proceeds with the panic processing.
> 2. There is an interrupt on cpu 0 that also encounters
>    an error condition and invokes panic.
> 3. This second invocation fails to acquire the panic_lock
>    and enters the infinite while loop in panic_smp_self_stop.
> 
> Thus all panic processing is stopped, and the cpu is stuck
> for eternity in the while(1) inside panic_smp_self_stop.
> 
> To address this, disable local interrupts with
> local_irq_disable before acquiring the panic_lock. This will
> prevent interrupt handlers from executing during the panic
> processing, thus avoiding this particular problem.
> 
> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -75,6 +75,14 @@ void panic(const char *fmt, ...)
>  	int state = 0;
>  
>  	/*
> +	 * Disable local interrupts. This will prevent panic_smp_self_stop
> +	 * from deadlocking the first cpu that invokes the panic, since
> +	 * there is nothing to prevent an interrupt handler (that runs
> +	 * after the panic_lock is acquired) from invoking panic again.
> +	 */
> +	local_irq_disable();
> +
> +	/*
>  	 * It's possible to come here directly from a panic-assertion and
>  	 * not have preempt disabled. Some functions called from here want
>  	 * preempt to be disabled. No point enabling it later though...

Seems sane.  panic() *should* work correctly when called with
interrupts disabled, so there be no bad effects from internally
disabling interrupts.  If there are bad effects, we should fix them up.