Reentrant NMIs, MCEs and interrupt stack tables.

[Andrew Cooper <andrew.cooper3@citrix.com>]

Hello,

While working on a fix for the rare-but-possible problem of reentrant
NMIs and MCEs, I have discovered that it is sadly possible to generate
fake NMIs and MCEs which will run the relevant handlers on the relevant
stacks, without invoking any of the other CPU logic for these special
interrupts.

A fake NMI can be generated by a processor in PIC mode as opposed to
Virtual wire mode, with a delivery of vector 2.  This setup is certainly
possible on a 64bit CPU, but I doubt there are many 64bit CPUs running
with only PIC.

A fake MCE is easy to generate.  A mal-programmed IO-APIC, IOMMU or
MSI/MSI-X entry which deliveres vector 0x18 is sufficient.  The LAPIC
will reject vectors 0 thru 0xf, but will deliver vectors 0x10 thru 0x1f,
despite them being architecturally reserved for exceptions.


The possibility of these fake interrupts (however unlikely) means that
there is necessarily a race condition between receiving a fake interrupt
and a genuine interrupt during which the handler cannot fixup the stack
sufficiently to be able to safely get back out.  If this race condition
were to occur, the real interrupt will corrupt the exception frame of
the fake interrupt, meaning that we cannot possibly resume the original
context.  This situation can be detected, but cannot be corrected, and
the only course of action is to crash gracefully.

The above problem made me wonder why we use separate stacks for NMIs and
MCEs.  I completely accept that the double fault handler should be on a
separate stack, but as we guarentee never to return from it, these
problems disappear.

Is there any particular reason to have separate stacks for NMIs and
MCEs, other than perhaps that it is good/common practice?  I can't think
of any other reasons offhand. (I am not necessarily advocating that we
combine NMIs and MCEs back into the regular Xen stack because, while it
would remove the above race condition, it would make other aspects of
the problem harder to solve.)

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com