Re: NMI problems with Dell SMP Xeons

Keith Owens
2006-06-07T07:44:14Z

Andi Kleen (on Wed, 7 Jun 2006 09:20:23 +0200) wrote:
>On Wednesday 07 June 2006 06:49, Keith Owens wrote:
>> Following a suggestion by Brendan Trotter, I ran some more tests to
>> track down the problem with sending NMI IPI on Dell Xeons.
>>
>> BIOS Logical OS ACPI Cpus IPI 2 NMI IPI
>> Processor BIOS OS (APIC_DM_NMI)
>>
>> Enabled Enabled 4 4 Not delivered Delivered as NMI
>> Enabled Disabled 4 2 Machine reset Machine reset
>> Disabled Enabled 2 2 Not delivered Delivered as NMI
>> Disabled Disabled 2 2 Not delivered Delivered as NMI
>>
>> So the killer combination with this motherboard is when the BIOS knows
>> about logical processors but the OS does not. Sending IPI 2 or NMI IPI
>> with that combination kills the machine. Brendan suggested that the
>> BIOS is seeing the broadcast NMI on the logical processors which are
>> not under OS control and that the BIOS cannot cope.
>
>How did you manage that? Normally the OS should use all CPUs
>known to BIOS. Or did you boot with special boot options to limit it?

Two ways:

(1) Boot with a kernel with CONFIG_ACPI=n, so the OS only finds 2 cpus
in the MPT instead of the 4 listed by ACPI.

(2) The kernel has ACPI=y, but is booted with maxcpus=2.

In both cases, send_IPI_allbutself() with IPI 2 or an NMI will result
in a hard reset.

>> Should we change the x86_64 send_IPI_allbutself() so it is only
>> delivered to cpus that the OS knows about, instead of doing a general
>> broadcast.
>
>Hmm, we should be doing that already to avoid races for CPU hotplug. But
>maybe it's not working correctly for KDB.

This problem is not KDB specific, although that is where it was first
noticed. Any code that sends a broadcast IPI 2 or an NMI IPI will
crash these Dell boxes when there is a mismatch between the cpus known
to the BIOS and the cpus known to the OS.

>Does it go away when you
>enable CPU hotplug?

HOTPLUG_CPU was already on in all of my test kernels.

>Anyways, should be a SMOP to force it. I wouldn't
>have a problem to use sequence ipis always and get rid of the broadcasts.
>There were benchmarks at some point and there wasn't a noticeable
>difference.

I will try forcing send_IPI_allbutself() to use the mask version rather
than the broadcast shortcut. Later tonight ...