Re: Reentrant NMIs, MCEs and interrupt stack tables.

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 21/11/12 21:17, Tim Deegan wrote:
> At 21:06 +0000 on 21 Nov (1353532004), Andrew Cooper wrote:
>> Hello,
>>
>> While working on a fix for the rare-but-possible problem of reentrant
>> NMIs and MCEs, I have discovered that it is sadly possible to generate
>> fake NMIs and MCEs which will run the relevant handlers on the relevant
>> stacks, without invoking any of the other CPU logic for these special
>> interrupts.
>>
>> A fake NMI can be generated by a processor in PIC mode as opposed to
>> Virtual wire mode, with a delivery of vector 2.  This setup is certainly
>> possible on a 64bit CPU, but I doubt there are many 64bit CPUs running
>> with only PIC.
>>
>> A fake MCE is easy to generate.  A mal-programmed IO-APIC, IOMMU or
>> MSI/MSI-X entry which deliveres vector 0x18 is sufficient.  The LAPIC
>> will reject vectors 0 thru 0xf, but will deliver vectors 0x10 thru 0x1f,
>> despite them being architecturally reserved for exceptions.
> You're not suggesting these could be caused by guest activity?

No.  This would be buggy hardware or buggy Xen.  Perhaps I should have
said "A fake MCE is easy to generate (if you are hacking Xen to try and
deliberately make it happen)" (Although 'easy' is just speculation based
on the description of behaviour of the LAPIC in the Intel SDM Volume 3)

>
>> The possibility of these fake interrupts (however unlikely) means that
>> there is necessarily a race condition between receiving a fake interrupt
>> and a genuine interrupt during which the handler cannot fixup the stack
>> sufficiently to be able to safely get back out.  If this race condition
>> were to occur, the real interrupt will corrupt the exception frame of
>> the fake interrupt, meaning that we cannot possibly resume the original
>> context.  This situation can be detected, but cannot be corrected, and
>> the only course of action is to crash gracefully.
> If once of these could only be casued by a bug in Xen, then I don't think
> we need to handle it at all.  If it's trivial to detect it and crash
> cleanly, that would be nice.

With all the other gubbins in to work around the stack problem, it
becomes two extra conditionals, so for all intents and purposes trivial.

>
>> The above problem made me wonder why we use separate stacks for NMIs and
>> MCEs.  I completely accept that the double fault handler should be on a
>> separate stack, but as we guarentee never to return from it, these
>> problems disappear.
>>
>> Is there any particular reason to have separate stacks for NMIs and
>> MCEs, other than perhaps that it is good/common practice? 
> It's to avoid a race where we take an NMI or MCE after swicthhing to the
> user/guest stack but before SYSRET.
>
> Tim.

Ah yes - I forgot to consider that case.

Thanks,

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com