Re: [PATCH] x86/nmi: lower initial watchdog frequency to avoid boot hangs

[Igor Druzhinin <igor.druzhinin@citrix.com>]

On 06/02/18 17:08, Alexey G wrote:
> On Tue, 6 Feb 2018 14:21:12 +0000
> Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> 
>> On 06/02/18 03:10, Alexey G wrote:
>>> I/O port 61h normally is not emulated by SMI legacy kbd handling code
>>> in BIOS, only ports like 60h, 64h, etc.
>>> Contrary to USB legacy emulation, it has to intercept port 61h via a
>>> different approach -- generic SMI I/O trap, which is not common (at
>>> least it was) to use by BIOSes... although it is possible as EFI
>>> interface and code for this is available. The assumption about port
>>> 61h being trapped by the SMI handler must be explicitly confirmed by
>>> checking I/O Trap control regs in the RCBA region.
>>>
>>> If I/O trap regs won't show an active I/O trap on I/O port 61h -- the
>>> root cause might be different (might even be related to stuff like
>>> NMI2SMI logic).
>>>
>>> If the problem is actually due to NMI handler being preempted by
>>> another NMI which occurred after (a long) execution of triggered SMI
>>> handler, it might be better to do all sensitive stuff before
>>> re-enabling NMIs by IRET in the NMI handler.  
>>
>> The problem is that the SMI handler executes enough instructions to
>> trigger another NMI (which is based on the retired instruction count),
>> which gets delivered once the SMI handler returns, and servicing the
>> NMI triggers a new SMI, which triggers a new NMI.  This is why the
>> system stalls.
>>
>> I'll leave the how/why port 0x61 is trapping to SMI to Igor, but it is
>> only a secondary concern here.  We cannot reasonably have the watchdog
>> able to trip because of exclusively SMI activity, or we'll potentially
>> livelock anywhere.
> 
> The major concern here is the possiblity of SMI being triggered _not_
> by some specific I/O port access. Primarily, if it actually was a
> periodic SMI.
> 
> If the actual SMI source is not related to some place in the NMI
> handler code but was eg. due to some SMI timer, lowering NMI watchdog
> frequency might not fix the issue completely, but lower its
> reproducibility (perhaps to some very rare occurrences). So it's better
> be sure what was the real source of SMI.
> 

This *is* related to this instruction - it was confirmed empirically.
Removing this instruction stops SMIs from occurring and effectively
removes the issue leaving the frequency unchanged.

> There are 2 weak spots in the analysis:
> 
> 1. Triggering SMI by reading I/O port 61h -- intercepting the port 61h
> is non-typical behavior for BIOS (unlike 60h/64h)
> 

Apparently, it is now.

> 2. According to the code, it looks like NMI status reading happens while
> NMIs are still blocked -- this means that SMI handler must exec IRET
> by itself to reset NMI blocking state -- again, this is possible (eg.
> in unreal->protmode switching code), but not likely.
> 

According to SDM one NMI might be pending while taken in SMI mode (see
ch. 34.8). This is actually even true if NMI comes while servicing
another NMI. So when we return to the NMI handler from SMI and finish it
properly the next one appears immediately.

Igor

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel