Re: Re: [PATCH] [BUGFIX] crash/ioapic: Prevent crash_kexec() from deadlocking of ioapic_lock

[Yoshihiro YUNOMAE <yoshihiro.yunomae.ez@hitachi.com>]

Hi Ingo,

Thank you for fixing typos!
OK, I'll fix them and rename to ioapic_zap_locks().

Thank you again!

Yoshihiro YUNOMAE

(2013/08/19 18:46), Ingo Molnar wrote:
>
> * Yoshihiro YUNOMAE <yoshihiro.yunomae.ez@hitachi.com> wrote:
>
>> Prevent crash_kexec() from deadlocking of ioapic_lock.
>
> s/of/on
>
>> When crash_kexec() is executed on a cpu, the cpu will get
>> ioapic_lock in disable_IO_APIC(). So if the cpu gets NMI
>> while locking ioapic_lock, a deadlock wiil happen. In
>
> s/will
>
>> this patch, ioapic_lock is initialized before
>> disable_IO_APIC().
>>
>> To confirm this deadlocking, you'll set up as follows:
>
> s/deadlocking/deadlock
>
>> 1. Add mdelay(1000) after raw_spin_lock_irqsave() in
>>     native_ioapic_set_affinity()@arch/x86/kernel/apic/io_apic.c
>>
>>     Although the deadlocking can occur without this modification, it will
>>    increase the potential of the deadlocking problem.
>
> s/deadlocking/deadlock
>
>>
>> 2. Build and install the kernel
>>
>> 3. Set up the OS which will run panic() and kexec when NMI is injected
>>      # echo "kernel.unknown_nmi_panic=1" >> /etc/sysctl.conf
>>      # vim /etc/default/grub
>>        add "nmi_watchdog=0 crashkernel=256M" in GRUB_CMDLINE_LINUX line
>>      # grub2-mkconfig
>>
>> 4. Reboot the OS
>>
>> 5. Run following command for each vcpu on the guest
>>      # while true; do echo <CPU num> > /proc/irq/<IO-APIC-edge or IO-APIC-fasteoi>/smp_affinitity; done;
>>     By running this command, cpus will get ioapic_lock for setting affinity.
>>
>> 6. Inject NMI (push a dump button or execute 'virsh inject-nmi <domain>' if you
>>     use VM)
>>     After injecting NMI, panic() is called in an nmi-handler context.
>>     Then, kexec will normally run in panic(), but the operation will be stopped
>>     by deadlock of ioapic_lock in crash_kexec()->machine_crash_shutdown()->
>
> s/of/on
>
>>     native_machine_crash_shutdown()->disable_IO_APIC()->clear_IO_APIC()->
>>     clear_IO_APIC_pin()->ioapic_read_entry().
>
> I suppose we could do this patch if it's a common
> occurance.
>
> A few minor details need fixing:
>
>>
>> Signed-off-by: Yoshihiro YUNOMAE <yoshihiro.yunomae.ez@hitachi.com>
>> Cc: Thomas Gleixner <tglx@linutronix.de>
>> Cc: Ingo Molnar <mingo@redhat.com>
>> Cc: "H. Peter Anvin" <hpa@zytor.com>
>> Cc: x86@kernel.org
>> Cc: Andrew Morton <akpm@linux-foundation.org>
>> Cc: Andi Kleen <ak@linux.intel.com>
>> Cc: Seiji Aguchi <seiji.aguchi@hds.com>
>> Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
>> Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
>> Cc: Joerg Roedel <joro@8bytes.org>
>> Cc: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
>> Cc: "Eric W. Biederman" <ebiederm@xmission.com>
>> Cc: Gleb Natapov <gleb@redhat.com>
>> Cc: Marcelo Tosatti <mtosatti@redhat.com>
>> Cc: linux-kernel@vger.kernel.org
>> Cc: stable@vger.kernel.org
>> ---
>>   arch/x86/include/asm/apic.h    |    2 ++
>>   arch/x86/kernel/apic/io_apic.c |    5 +++++
>>   arch/x86/kernel/crash.c        |    4 ++++
>>   3 files changed, 11 insertions(+)
>>
>> diff --git a/arch/x86/include/asm/apic.h b/arch/x86/include/asm/apic.h
>> index f8119b5..ddb06af 100644
>> --- a/arch/x86/include/asm/apic.h
>> +++ b/arch/x86/include/asm/apic.h
>> @@ -715,4 +715,6 @@ static inline void exiting_ack_irq(void)
>>   	ack_APIC_irq();
>>   }
>>
>> +extern void ioapic_lock_init(void);
>> +
>>   #endif /* _ASM_X86_APIC_H */
>> diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
>> index 9ed796c..2816c07 100644
>> --- a/arch/x86/kernel/apic/io_apic.c
>> +++ b/arch/x86/kernel/apic/io_apic.c
>> @@ -1534,6 +1534,11 @@ void intel_ir_io_apic_print_entries(unsigned int apic,
>>   	}
>>   }
>>
>> +void ioapic_lock_init(void)
>> +{
>> +	raw_spin_lock_init(&ioapic_lock);
>> +}
>
> Please name this ioapic_zap_locks() to make clear that this
> is crash handling related.
>
>> +
>>   __apicdebuginit(void) print_IO_APIC(int ioapic_idx)
>>   {
>>   	union IO_APIC_reg_00 reg_00;
>> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
>> index 74467fe..ea039d5 100644
>> --- a/arch/x86/kernel/crash.c
>> +++ b/arch/x86/kernel/crash.c
>> @@ -129,6 +129,10 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
>>
>>   	lapic_shutdown();
>>   #if defined(CONFIG_X86_IO_APIC)
>
> Please enhance this #ifdef while at it.
>
>> +	/*
>> +	 * Prevent crash_kexec() from deadlocking of ioapic_lock.
>> +	 */
>
> s/of/on.
>
> Also, single-line comment can go /* here */.
>
>> +	ioapic_lock_init();
>>   	disable_IO_APIC();
>>   #endif
>>   #ifdef CONFIG_HPET_TIMER
>>
>
> Thanks,
>
> 	Ingo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

-- 
Yoshihiro YUNOMAE
Software Platform Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: yoshihiro.yunomae.ez@hitachi.com