Re: NMI problems with Dell SMP Xeons

[Keith Owens <kaos@sgi.com>]

Andi Kleen (on Wed, 7 Jun 2006 09:20:23 +0200) wrote:
>On Wednesday 07 June 2006 06:49, Keith Owens wrote:
>> Following a suggestion by Brendan Trotter, I ran some more tests to
>> track down the problem with sending NMI IPI on Dell Xeons.
>>
>> BIOS Logical    OS ACPI     Cpus    IPI 2             NMI IPI
>>  Processor                BIOS  OS                 (APIC_DM_NMI)
>>
>> Enabled         Enabled    4    4  Not delivered   Delivered as NMI
>> Enabled         Disabled   4    2  Machine reset   Machine reset
>> Disabled        Enabled    2    2  Not delivered   Delivered as NMI
>> Disabled        Disabled   2    2  Not delivered   Delivered as NMI
>>
>> So the killer combination with this motherboard is when the BIOS knows
>> about logical processors but the OS does not.  Sending IPI 2 or NMI IPI
>> with that combination kills the machine.  Brendan suggested that the
>> BIOS is seeing the broadcast NMI on the logical processors which are
>> not under OS control and that the BIOS cannot cope.
>
>How did you manage that? Normally the OS should use all CPUs
>known to BIOS. Or did you boot with special boot options to limit it?

Two ways:

(1) Boot with a kernel with CONFIG_ACPI=n, so the OS only finds 2 cpus
    in the MPT instead of the 4 listed by ACPI.

(2) The kernel has ACPI=y, but is booted with maxcpus=2.

In both cases, send_IPI_allbutself() with IPI 2 or an NMI will result
in a hard reset.

>> Should we change the x86_64 send_IPI_allbutself() so it is only
>> delivered to cpus that the OS knows about, instead of doing a general
>> broadcast. 
>
>Hmm, we should be doing that already to avoid races for CPU hotplug.  But 
>maybe it's not working correctly for KDB.

This problem is not KDB specific, although that is where it was first
noticed.  Any code that sends a broadcast IPI 2 or an NMI IPI will
crash these Dell boxes when there is a mismatch between the cpus known
to the BIOS and the cpus known to the OS.

>Does it go away when you
>enable CPU hotplug?

HOTPLUG_CPU was already on in all of my test kernels.

>Anyways, should be a SMOP to force it. I wouldn't
>have a problem to use sequence ipis  always and get rid of the broadcasts.
>There were benchmarks at some point and there wasn't a noticeable
>difference. 

I will try forcing send_IPI_allbutself() to use the mask version rather
than the broadcast shortcut.  Later tonight ...