Re: Interrupt injection with ISR set on Intel hardware

[Chao Gao <chao.gao@intel.com>]

On Mon, Oct 15, 2018 at 01:06:12PM +0100, Andrew Cooper wrote:
>On 15/10/18 11:30, Roger Pau Monné wrote:
>> Hello,
>>
>> Wei recently discovered an issue when running a Linux PVH Dom0 on a
>> box with a Intel Family 6 (0x6), Model 158 (0x9e), Stepping 9 (raw
>> 000906e9) CPU, we are not sure whether the issue is limited to a PVH
>> Dom0, or it just happens to be easier to trigger in this scenario.
>
>This issue has been seen very occasionally for years.  My debugging
>patch dates back to 2013, and it has been observed on Haswell systems as
>well.  There have also been a handful of reports on xen-devel over the
>years.
>
>Wei is the first person to get a reliable enough repro to debug.  It is
>not exclusive to PVH Dom0, but that appears to be the easiest way to
>tickle the problem.
>
>> The issue is caused by what seems to be an interrupt injection while
>> Xen is still servicing a previous interrupt (ie: the interrupt hasn't
>> been EOI'ed and ISR for the vector is set) with the same or lower
>> priority than the interrupt currently being serviced. This injection
>> always happen when returning from idle from a state ACPI_STATE_C3 or
>> lower.
>
>As a bit of background, for some guest irqs, we need to inject the
>interrupt into the guest and wait for an explicit ack.
>
>If the irq source doesn't have a mask bit which Xen can use, the only
>option we have is to avoid repeated interruption is to leave the irq in
>service at the LAPIC.  The purpose of the Pending EOI stack is to manage
>these as acks arrive back from guest context.
>
>For reasons which aren't clear, guest-bound MSI vectors which don't have
>a mask bit also use this PEOI stack mechanism.  I think this is probably
>a Xen bug, but it also relevant to the issue.
>
>In Wei's case, the interrupt in question is an MSI non-maskable
>interrupt from the USB controller.
>
>> Note that I haven't been able to reproduce this issue when using
>> mwait-idle=0 or max_cstate=2 on the Xen command line, but again
>> without knowing the underlying issue it's impossible to tell whether
>> it's relevant.
>>
>> Andrew provided a debug patch which I've expanded to also log power
>> state transition, and is attached to this email.
>>
>> Here is a trace of a crash, together with the debug info.
>>
>> (XEN) *** Pending EOI error ***
>> (XEN)   cpu #1, irq 30, vector 0x21, sp 1
>> (XEN) Peoi stack: sp 1
>> (XEN)   [ 0] irq  30, vec 0x21, ready 0, ISR 1, TMR 0, IRR 0
>> (XEN) Peoi stack trace records:
>> (XEN)   [22619] POP      {sp  1, irq  30, vec 0x21}
>> (XEN)   [22620] POWER    TYPE 4
>> (XEN)   [22621] IDLE     PPR 0x00000010
>> (XEN)                    IRR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN)   [22622] WAKE     PPR 0x00000010
>> (XEN)                    IRR 0000000000000000000000000000000000000000000000000000000000000004
>> (XEN)                    ISR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN)   [22623] ACK_PRE  PPR 0x000000f0
>> (XEN)                    IRR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000000000000000000000000000000000000000000000000000000000004
>> (XEN)   [22624] ACK_POST PPR 0x00000010
>> (XEN)                    IRR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN)   [22625] POWER    TYPE 5
>> (XEN)   [22626] IDLE     PPR 0x00000010
>> (XEN)                    IRR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN)   [22627] WAKE     PPR 0x00000010
>> (XEN)                    IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN)   [22628] PUSH     {sp  0, irq  30, vec 0x21}
>> (XEN)   [22629] POWER    TYPE 5
>> (XEN)   [22630] IDLE     PPR 0x00000020
>> (XEN)                    IRR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)   [22631] WAKE     PPR 0x00000020
>> (XEN)                    IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)   [22632] POWER    TYPE 5
>> (XEN)   [22633] IDLE     PPR 0x00000020
>> (XEN)                    IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)   [22634] WAKE     PPR 0x00000020
>> (XEN)                    IRR 0000000002000000000000000000000000000000000000000000000000000004
>> (XEN)                    ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)   [22635] ACK_PRE  PPR 0x000000f0
>> (XEN)                    IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000002000000000000000000000000000000000000000000000000000004
>> (XEN)   [22636] ACK_POST PPR 0x00000020
>> (XEN)                    IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)   [22637] READY    {sp  1, irq  30, vec 0x21}
>> (XEN)   [22638] ACK_PRE  PPR 0x00000020
>> (XEN)                    IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)   [22639] ACK_POST PPR 0x00000010
>> (XEN)                    IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN)   [22640] POP      {sp  1, irq  30, vec 0x21}
>> (XEN)   [22641] PUSH     {sp  0, irq  30, vec 0x21}
>> (XEN)   [22642] POWER    TYPE 4
>> (XEN)   [22643] IDLE     PPR 0x00000020
>> (XEN)                    IRR 0000000000000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)   [22644] WAKE     PPR 0x00000020
>> (XEN)                    IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)   [22645] POWER    TYPE 3
>> (XEN)   [22646] IDLE     PPR 0x00000020
>> (XEN)                    IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)   [22647] WAKE     PPR 0x00000020
>> (XEN)                    IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)   [22648] POWER    TYPE 3
>> (XEN)   [22649] IDLE     PPR 0x00000020
>> (XEN)                    IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)   [22650] WAKE     PPR 0x00000020
>> (XEN)                    IRR 0000000002000000000000000000000000000000000000000000000000000000
>> (XEN)                    ISR 0000000002000000000000000000000000000000000000000000000000000000
>
>What has happened here is that, despite vector 0x21 being in service
>(starting at the PUSH), we see it injected a second time.  The ASSERT()
>fires because we find this vector still on the pending EOI stack.
>
>After that, we go idle a few times, but never haven't yet acked the
>vector (i.e. whatever we're waiting for the guest to acknowledge hasn't
>happened yet, and Xen has nothing else to do on this CPU).
>
>From the debugging, we see that PPR/IRR/ISR appear to retain their state
>across the mwait, and there is nothing in the manual which I can see
>discussing the interaction of LAPIC state and C states.
>
>However, from the behaviour seen here, we occasionally get woken from
>mwait by an interrupt which already pending.  I can only conclude that
>there is some issue with priority calculations for edge triggered
>interrupts when idle, which allows another one to slip in.  The fact

Hi, Roger, Andrew and Wei,

Jan's patch
(https://lists.xen.org/archives/html/xen-devel/2018-10/msg01031.html)
fixs an issue in handling SVI. Currently, when dealing with EOI from guest, the
SVI was cleared. But the correct way is clearing the corresponding bit in VISR
and then setting SVI to the highest index of bit set in VISR (please refer to
SDM 29.1.4). If SVI is set to a value lower than the vector of the highest
priority interrupt that is in service, the PPR virtualization (29.1.3) might
set the VPPR to a lower value on VMEntry too. Thus an interrupt with same or
lower priority, which should be blocked by VPPR, slips in.

Could you apply Jan's patch and try to reproduce it again?

Thanks
Chao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel