[PATCH][KDUMP] Ignore spurious IPI

[PATCH][KDUMP] Ignore spurious IPI

[Takao Indoh]

Hi all,

I found a problem that kdump(2nd kernel) sometimes hangs up. It seems
that system panic occurs as follows.

(1)
2nd kernel boot up

(2)
A pending IPI from 1st kernel comes after unmasking interrupts at the
following point.

asmlinkage void __init start_kernel(void)
{
(snip)
    time_init();
    profile_init();
    if (!irqs_disabled())
            printk(KERN_CRIT "start_kernel(): bug: interrupts were "
                             "enabled early\n");
    early_boot_irqs_disabled = false;
    local_irq_enable(); <=======================================HERE

(3)
Kernel tries to handle the interrupt, but some data structures are not
initialized yet at this point. As a result, in the
generic_smp_call_function_single_interrupt(), NULL pointer dereference
occurs when list_replace_init() tries to access &q->list.next.

I took a look at local_apic_timer_interrupt() and found a few lines to
handle such a pending LAPIC interrupt(in this case, timer interrupt).
Therefore I made a patch to ignore spurious IPI in the same manner. I
confirmed this problem does not occur with this patch.

Any comments?

Signed-off-by: Takao Indoh <indou.takao@jp.fujitsu.com>
---
 kernel/smp.c |    6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/kernel/smp.c b/kernel/smp.c
index 9910744..f2f561b 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -260,6 +260,12 @@ void generic_smp_call_function_single_interrupt(void)
 	 */
 	WARN_ON_ONCE(!cpu_online(smp_processor_id()));
 
+	if (unlikely(!q->list.next)) {
+		/* Pending interrupt from previous kernel(e.g. kdump), just ignore */
+		pr_warning("Spurious IPI on cpu %d\n", smp_processor_id());
+		return;
+	}
+
 	raw_spin_lock(&q->lock);
 	list_replace_init(&q->list, &list);
 	raw_spin_unlock(&q->lock);

Re: [KDUMP] Ignore spurious IPI

[Milton Miller]

On Wed, 23 Mar 2011 about 18:40:12 -0000, Takao Indoh wrote:
> Hi all,
> 
> I found a problem that kdump(2nd kernel) sometimes hangs up. It seems
> that system panic occurs as follows.
..
> (2)
> A pending IPI from 1st kernel comes after unmasking interrupts at the
> following point.
> 
> asmlinkage void __init start_kernel(void)
> {
> (snip)
>     time_init();
>     profile_init();
>     if (!irqs_disabled())
>             printk(KERN_CRIT "start_kernel(): bug: interrupts were "
>                              "enabled early\n");
>     early_boot_irqs_disabled = false;
>     local_irq_enable(); <=======================================HERE
> 
> (3)
> Kernel tries to handle the interrupt, but some data structures are not
> initialized yet at this point. As a result, in the
> generic_smp_call_function_single_interrupt(), NULL pointer dereference
> occurs when list_replace_init() tries to access &q->list.next.
> 
[tried to match lapic timer interrupt]
> Any comments?

So this occurs because unlike device interrupts, this vector has the action
defined statically and no per-interrupt disable on your architecture?


If so, just initialize the data structure earlier -- change
init_call_single_data from early_initcall to an explict call after the
per-cpu areas are initialized.

milton

Re: [KDUMP] Ignore spurious IPI

[Takao Indoh]

On Thu, 24 Mar 2011 08:20:32 -0600, Milton Miller wrote:

>On Wed, 23 Mar 2011 about 18:40:12 -0000, Takao Indoh wrote:
>> Hi all,
>> 
>> I found a problem that kdump(2nd kernel) sometimes hangs up. It seems
>> that system panic occurs as follows.
>..
>> (2)
>> A pending IPI from 1st kernel comes after unmasking interrupts at the
>> following point.
>> 
>> asmlinkage void __init start_kernel(void)
>> {
>> (snip)
>>     time_init();
>>     profile_init();
>>     if (!irqs_disabled())
>>             printk(KERN_CRIT "start_kernel(): bug: interrupts were "
>>                              "enabled early\n");
>>     early_boot_irqs_disabled = false;
>>     local_irq_enable(); <=======================================HERE
>> 
>> (3)
>> Kernel tries to handle the interrupt, but some data structures are not
>> initialized yet at this point. As a result, in the
>> generic_smp_call_function_single_interrupt(), NULL pointer dereference
>> occurs when list_replace_init() tries to access &q->list.next.
>> 
>[tried to match lapic timer interrupt]
>> Any comments?
>
>So this occurs because unlike device interrupts, this vector has the action
>defined statically and no per-interrupt disable on your architecture?

I think there is not per-interrupt disable for IPI.

>If so, just initialize the data structure earlier -- change
>init_call_single_data from early_initcall to an explict call after the
>per-cpu areas are initialized.

That makes sense. I'll do this, thanks.

Thanks,
Takao Indoh


>
>milton

Re: [KDUMP] Ignore spurious IPI

[WANG Cong]

On Thu, 24 Mar 2011 08:20:32 -0600, Milton Miller wrote:
> If so, just initialize the data structure earlier -- change
> init_call_single_data from early_initcall to an explict call after the
> per-cpu areas are initialized.
> 

Yeah, this is a good idea, IPI related data structures should be
ready before interrupts are enabled. We should not assume that
there is no IPI pending before init_call_single_data gets initialized,
altough this is true in non-kdump kernel.

Thanks.