From mboxrd@z Thu Jan  1 00:00:00 1970
From: "Thimo E." <abc@digithi.de>
Subject: Re: cpuidle and un-eoid interrupts at the local apic
Date: Sat, 03 Aug 2013 00:50:17 +0200
Message-ID: <51FC37A9.9090809@digithi.de>
References: <51A908CA.7050604@citrix.com> <51F8CB15.1070608@digithi.de>
	<51F8DD40.2090207@citrix.com>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary="------------070602000608000906060500"
Return-path: <xen-devel-bounces@lists.xen.org>
In-Reply-To: <51F8DD40.2090207@citrix.com>
List-Unsubscribe: <http://lists.xen.org/cgi-bin/mailman/options/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xen.org>
List-Help: <mailto:xen-devel-request@lists.xen.org?subject=help>
List-Subscribe: <http://lists.xen.org/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=subscribe>
Sender: xen-devel-bounces@lists.xen.org
Errors-To: xen-devel-bounces@lists.xen.org
To: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Keir Fraser <keir@xen.org>, Jan Beulich <JBeulich@suse.com>, Xen-devel List <xen-devel@lists.xen.org>
List-Id: xen-devel@lists.xenproject.org

This is a multi-part message in MIME format.
--------------070602000608000906060500
Content-Type: multipart/alternative;
 boundary="------------040704050907080007090701"


--------------040704050907080007090701
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

I've postet it already in the forum thread, but to keep all of you up to 
date for this issue I am copying the logfile into this thread, too:

XenServer crash again, attached you'll find the output with the verbose 
messages Andrew inserted into the code.

Best regards
   Thimo


Am 31.07.2013 11:47, schrieb Andrew Cooper:
> On 31/07/13 09:30, Thimo E. wrote:
>> Hello all,
>>
>> I have also a Haswell system. I am running XenServer 6.2 (with Xen
>> 4.1.5) on it and I am experiencing the same issue. Do you already have
>> a solution for this problem ?
>>
>> Best regards
>>    Thimo
> Hi,
>
> We are still none the wiser on this issue.  I have a debugging patch to
> get more information, but the problem hasn't reoccurred since.  This is
> now 2 crashes on Xen 4.1 and a single crash on Xen 4.2 that I have seen.
>
> For the benefit of anyone else who runs over this issue in the meantime,
> the patch (against Xen-4.3) is attached.
>
> Thimo: I shall put a new version of the XenServer 6.2 Xen with the
> debugging patch on the forum thread.
>
> ~Andrew
>
>> (XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at
>> irq.c:1027^M
>> (XEN) ----[ Xen-4.1.5.debug  x86_64  debug=y  Not tainted ]----^M
>> (XEN) CPU:    1^M
>> (XEN) RIP:    e008:[<ffff82c480169662>] do_IRQ+0x3ba/0x6d9^M
>> (XEN) RFLAGS: 0000000000010002   CONTEXT: hypervisor^M
>> (XEN) rax: 0000000000000001   rbx: ffff83081f080f00   rcx:
>> ffff83081f05b340^M
>> (XEN) rdx: 0000000000000001   rsi: 000000000000002b   rdi:
>> 0000000000000001^M
>> (XEN) rbp: ffff83081f057d88   rsp: ffff83081f057d18   r8:
>> ffff83081f05b63c^M
>> (XEN) r9:  000070044fb97100   r10: ffff8300b858c060   r11:
>> 000020f3f5a4dea5^M
>> (XEN) r12: 000000000000002b   r13: ffff83081f004e80   r14:
>> 000000000000001d^M
>> (XEN) r15: 0000000000000002   cr0: 000000008005003b   cr4:
>> 00000000001026f0^M
>> (XEN) cr3: 000000045915f000   cr2: 0000000000150008^M
>> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008^M
>> (XEN) Xen stack trace from rsp=ffff83081f057d18:^M
>> (XEN)    000000000000001d 000000000000001d ffff83081f080f00
>> 0000000000000000^M
>> (XEN)    00000000ffffffea ffff83081f080f00 0000000000000000
>> 0000000000000000^M
>> (XEN)    ffffffffffffffff ffff83081f057f18 ffff83081f06bb00
>> ffff83081f06bb90^M
>> (XEN)    ffff8300b858c000 0000000000000002 00007cf7e0fa8247
>> ffff82c480161a66^M
>> (XEN)    0000000000000002 ffff8300b858c000 ffff83081f06bb90
>> ffff83081f06bb00^M
>> (XEN)    ffff83081f057ef0 ffff83081f057f18 000020f3f5a4dea5
>> ffff8300b858c060^M
>> (XEN)    000070044fb97100 ffff83081f05bb80 0000000000007f40
>> 0000000000000001^M
>> (XEN)    0000000000000000 000020f3c755a972 ffff83081f06bb90
>> 0000002b00000000^M
>> (XEN)    ffff82c4801a21f0 000000000000e008 0000000000000246
>> ffff83081f057e48^M
>> (XEN)    000000000000e010 ffff83081f057ef0 ffff82c4801a3dc4
>> 000020f3f595c09c^M
>> (XEN)    000020f3f596987e ffff8306383e3010 ffff83081f05b100
>> ffffffffffffffff^M
>> (XEN)    0000000000000001 0000000000000001 ffffffffffffffff
>> ffff83081f057f18^M
>> (XEN)    00000000802d4680 0000000000000000 0000000000000000
>> ffff82c4802d4680^M
>> (XEN)    000002a80000024b ffff8300b8586000 ffff83081f057f18
>> ffff8300b8586000^M
>> (XEN)    ffff8300b858c000 ffff8300b858c000 0000000000000002
>> ffff83081f057f10^M
>> (XEN)    ffff82c48015a261 ffff82c480126ccd 0000000000000001
>> ffff83081f057d18^M
>> (XEN)    0000000000000000 0000000000000000 0000000000000000
>> 0000000000000000^M
>> (XEN)    0000000000000000 0000000000000000 0000000000000246
>> ffff88001a8093a0^M
>> (XEN)    0000000100885e0f 000000000000000f 0000000000000000
>> ffffffff802063aa^M
>> (XEN)    0000000000000001 00000000deadbeef 00000000deadbeef
>> 0000010000000000^M
>> (XEN) Xen call trace:^M
>> (XEN)    [<ffff82c480169662>] do_IRQ+0x3ba/0x6d9^M
>> (XEN)    [<ffff82c480161a66>] common_interrupt+0x26/0x30^M
>> (XEN)    [<ffff82c4801a21f0>] lapic_timer_nop+0x0/0x6^M
>> (XEN)    [<ffff82c48015a261>] idle_loop+0x48/0x59^M
>> (XEN)    ^M
>> (XEN) ^M
>> (XEN) ****************************************^M
>> (XEN) Panic on CPU 1:^M
>> (XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at
>> irq.c:1027^M
>> (XEN) ****************************************^M
>> (XEN) ^M
>> (XEN) Reboot in five seconds...^M
>>
>> Am 31.05.2013 22:32, schrieb Andrew Cooper:
>>> Recently our automated testing system has caught a curious assertion
>>> while testing Xen 4.1.5 on a HaswellDT system.
>>>
>>> (XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at
>>> irq.c:1030
>>> (XEN) ----[ Xen-4.1.5  x86_64  debug=n  Not tainted ]----
>>> (XEN) CPU:    0
>>> (XEN) RIP:    e008:[<ffff82c48016b2b4>] do_IRQ+0x514/0x750
>>> (XEN) RFLAGS: 0000000000010093   CONTEXT: hypervisor
>>> (XEN) rax: 000000000000002f   rbx: ffff830249841e80   rcx:
>>> ffff82c4803127c0
>>> (XEN) rdx: 0000000000000004   rsi: 0000000000000027   rdi:
>>> 0000000000000001
>>> (XEN) rbp: 0000000000001e00   rsp: ffff82c4802bfd48   r8:
>>> ffff82c480312abc
>>> (XEN) r9:  ffff8302498a5948   r10: 0000000000000009   r11:
>>> ffff8302498c6c80
>>> (XEN) r12: ffff830243b07f50   r13: ffff8300a24f8000   r14:
>>> 00000af8373788e3
>>> (XEN) r15: ffff830249841e80   cr0: 000000008005003b   cr4:
>>> 00000000001026f0
>>> (XEN) cr3: 00000002479e6000   cr2: 00000000e6d3c090
>>> (XEN) ds: 007b   es: 007b   fs: 00d8   gs: 0000   ss: 0000   cs: e008
>>> (XEN) Xen stack trace from rsp=ffff82c4802bfd48:
>>> (XEN)    ffff830249841eb4 ffff82c480312ec0 000000000000001e
>>> 0000001e00000000
>>> (XEN)    0000000000000000 00000000498a5670 ffff830249841d80
>>> ffff830249840080
>>> (XEN)    ffff830249841db4 0000000000000000 ffff8302498a55e0
>>> ffff8302498a5670
>>> (XEN)    ffff8300a24f8000 00000af8373788e3 00000af83736b8ed
>>> ffff82c480162ca0
>>> (XEN)    00000af83736b8ed 00000af8373788e3 ffff8300a24f8000
>>> ffff8302498a5670
>>> (XEN)    ffff8302498a55e0 0000000000000000 ffff8302498c6c80
>>> 0000000000000009
>>> (XEN)    ffff8302498a5948 ffff82c480313000 0000000000007f40
>>> 0000000000000001
>>> (XEN)    0000000000000000 0000000000000000 00000af80db652fd
>>> 0000002700000000
>>> (XEN)    ffff82c4801a50a0 000000000000e008 0000000000000246
>>> ffff82c4802bfe78
>>> (XEN)    0000000000000000 ffff8302498a5670 ffff82c4801a6a56
>>> ffffffffffffffff
>>> (XEN)    ffff830249818000 0000000000000000 ffff8300a24f8000
>>> ffff82c480122c11
>>> (XEN)    00000af839021119 0000000000000000 0000000000000000
>>> 00000000802bff18
>>> (XEN)    0000025c0000013b ffff82c4802e7580 ffff82c4802bff18
>>> ffff8300a2838000
>>> (XEN)    ffff82c4802f61a0 ffff8300a24f8000 0000000000000002
>>> 00000af837304b45
>>> (XEN)    ffff82c48015b67a 0000000000000000 0000000000000000
>>> 0000000000000000
>>> (XEN)    0000000000000000 0000000000000000 00000000ee8a3f8c
>>> 0000000000000001
>>> (XEN)    0000000000000000 0000000000000000 0000000000000000
>>> 0000000000000000
>>> (XEN)    0000000000000000 0000000000000000 00000000ee8a3f74
>>> 0000000000000af8
>>> (XEN)    0000000000000001 0000010000000000 00000000c01013a7
>>> 0000000000000061
>>> (XEN)    0000000000000246 00000000ee8a3f70 0000000000000069
>>> 0000000000000000
>>> (XEN) Xen call trace:
>>> (XEN)       [<ffff82c48016b2b4>] do_IRQ+0x514/0x750
>>> (XEN)     15[<ffff82c480162ca0>] common_interrupt+0x20/0x30
>>> (XEN)     32[<ffff82c4801a50a0>] lapic_timer_nop+0x0/0x10
>>> (XEN)     38[<ffff82c4801a6a56>] acpi_processor_idle+0x376/0x740
>>> (XEN)     43[<ffff82c480122c11>] do_block+0x71/0xd0
>>> (XEN)     56[<ffff82c48015b67a>] idle_loop+0x1a/0x50
>>> (XEN)
>>> (XEN)
>>> (XEN) ****************************************
>>> (XEN) Panic on CPU 0:
>>> (XEN) Assertion '(sp == 0) || (peoi[sp-1].vector < vector)' failed at
>>> irq.c:1030
>>> (XEN) ****************************************
>>>
>>> And the disassembly before the assertion:
>>>
>>> ffff82c48016b29f:       48 8d 14 85 00 00 00    lea    0x0(,%rax,4),%rdx
>>> ffff82c48016b2a6:       00
>>> ffff82c48016b2a7:       0f b6 44 11 ff          movzbl
>>> -0x1(%rcx,%rdx,1),%eax
>>> ffff82c48016b2ac:       39 c6                   cmp    %eax,%esi
>>> ffff82c48016b2ae:       0f 8f 5c ff ff ff       jg
>>> ffff82c48016b210 <do_IRQ+0x470>
>>> ffff82c48016b2b4:       0f 0b                   ud2
>>>
>>>
>>> Xen has been woken up by an interrupt of vector 0x27, but has a vector
>>> 0x2f on the top of the pending EOI stack for the local APIC.
>>>
>>> I have put in more debugging to dump the LAPIC state of the two
>>> interesting vectors and the IOAPIC state, but I have no idea if/when the
>>> problem might reoccur.
>>>
>>> My understanding of LAPIC priority leads me to think that Xen really
>>> shouldn't be woken up by a lower priority vector if a higher priority
>>> one is still un-eoi'd.  There is not yet sufficient information to tell
>>> whether this is truely the case, or that Xen has simply gotten confused
>>> about which vectors it eoi'd.
>>>
>>> Having said that, we do keep line level interrupts un-eoi'd for extended
>>> periods while guests service the interrupt.  Given that vectors are
>>> chosen at random, we could get into a situation where a line interrupt
>>> has a vector 0xdf and stays pending for 150ms (which I measured as a
>>> not-overly-uncommon mean-time-till-eoi for line level interrupt).  This
>>> would starve any other guest interrupts for an extended period.
>>>
>>> Given directed-eoi support in the past few generations of processor, the
>>> requirement for the pending EOI stack has disappeared as far as I am
>>> aware.  Would it be sensible idea in general to make use of the pending
>>> eoi stack conditional on not having/using directed EOI support?
>>>
>>> ~Andrew
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xen.org
>>> http://lists.xen.org/xen-devel
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


--------------040704050907080007090701
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">Hi,<br>
      <br>
      I've postet it already in the forum thread, but to keep all of you
      up to date for this issue I am copying the logfile into this
      thread, too:<br>
      <br>
      XenServer crash again, attached you'll find the output with the
      verbose messages Andrew inserted into the code.<br>
      <br>
      Best regards<br>
      &nbsp; Thimo<br>
      <br>
      <br>
      Am 31.07.2013 11:47, schrieb Andrew Cooper:<br>
    </div>
    <blockquote cite="mid:51F8DD40.2090207@citrix.com" type="cite">
      <pre wrap="">On 31/07/13 09:30, Thimo E. wrote:
</pre>
      <blockquote type="cite">
        <pre wrap="">Hello all,

I have also a Haswell system. I am running XenServer 6.2 (with Xen
4.1.5) on it and I am experiencing the same issue. Do you already have
a solution for this problem ?

Best regards
  Thimo
</pre>
      </blockquote>
      <pre wrap="">
Hi,

We are still none the wiser on this issue.  I have a debugging patch to
get more information, but the problem hasn't reoccurred since.  This is
now 2 crashes on Xen 4.1 and a single crash on Xen 4.2 that I have seen.

For the benefit of anyone else who runs over this issue in the meantime,
the patch (against Xen-4.3) is attached.

Thimo: I shall put a new version of the XenServer 6.2 Xen with the
debugging patch on the forum thread.

~Andrew

</pre>
      <blockquote type="cite">
        <pre wrap="">
(XEN) Assertion '(sp == 0) || (peoi[sp-1].vector &lt; vector)' failed at
irq.c:1027^M
(XEN) ----[ Xen-4.1.5.debug  x86_64  debug=y  Not tainted ]----^M
(XEN) CPU:    1^M
(XEN) RIP:    e008:[&lt;ffff82c480169662&gt;] do_IRQ+0x3ba/0x6d9^M
(XEN) RFLAGS: 0000000000010002   CONTEXT: hypervisor^M
(XEN) rax: 0000000000000001   rbx: ffff83081f080f00   rcx:
ffff83081f05b340^M
(XEN) rdx: 0000000000000001   rsi: 000000000000002b   rdi:
0000000000000001^M
(XEN) rbp: ffff83081f057d88   rsp: ffff83081f057d18   r8:
ffff83081f05b63c^M
(XEN) r9:  000070044fb97100   r10: ffff8300b858c060   r11:
000020f3f5a4dea5^M
(XEN) r12: 000000000000002b   r13: ffff83081f004e80   r14:
000000000000001d^M
(XEN) r15: 0000000000000002   cr0: 000000008005003b   cr4:
00000000001026f0^M
(XEN) cr3: 000000045915f000   cr2: 0000000000150008^M
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008^M
(XEN) Xen stack trace from rsp=ffff83081f057d18:^M
(XEN)    000000000000001d 000000000000001d ffff83081f080f00
0000000000000000^M
(XEN)    00000000ffffffea ffff83081f080f00 0000000000000000
0000000000000000^M
(XEN)    ffffffffffffffff ffff83081f057f18 ffff83081f06bb00
ffff83081f06bb90^M
(XEN)    ffff8300b858c000 0000000000000002 00007cf7e0fa8247
ffff82c480161a66^M
(XEN)    0000000000000002 ffff8300b858c000 ffff83081f06bb90
ffff83081f06bb00^M
(XEN)    ffff83081f057ef0 ffff83081f057f18 000020f3f5a4dea5
ffff8300b858c060^M
(XEN)    000070044fb97100 ffff83081f05bb80 0000000000007f40
0000000000000001^M
(XEN)    0000000000000000 000020f3c755a972 ffff83081f06bb90
0000002b00000000^M
(XEN)    ffff82c4801a21f0 000000000000e008 0000000000000246
ffff83081f057e48^M
(XEN)    000000000000e010 ffff83081f057ef0 ffff82c4801a3dc4
000020f3f595c09c^M
(XEN)    000020f3f596987e ffff8306383e3010 ffff83081f05b100
ffffffffffffffff^M
(XEN)    0000000000000001 0000000000000001 ffffffffffffffff
ffff83081f057f18^M
(XEN)    00000000802d4680 0000000000000000 0000000000000000
ffff82c4802d4680^M
(XEN)    000002a80000024b ffff8300b8586000 ffff83081f057f18
ffff8300b8586000^M
(XEN)    ffff8300b858c000 ffff8300b858c000 0000000000000002
ffff83081f057f10^M
(XEN)    ffff82c48015a261 ffff82c480126ccd 0000000000000001
ffff83081f057d18^M
(XEN)    0000000000000000 0000000000000000 0000000000000000
0000000000000000^M
(XEN)    0000000000000000 0000000000000000 0000000000000246
ffff88001a8093a0^M
(XEN)    0000000100885e0f 000000000000000f 0000000000000000
ffffffff802063aa^M
(XEN)    0000000000000001 00000000deadbeef 00000000deadbeef
0000010000000000^M
(XEN) Xen call trace:^M
(XEN)    [&lt;ffff82c480169662&gt;] do_IRQ+0x3ba/0x6d9^M
(XEN)    [&lt;ffff82c480161a66&gt;] common_interrupt+0x26/0x30^M
(XEN)    [&lt;ffff82c4801a21f0&gt;] lapic_timer_nop+0x0/0x6^M
(XEN)    [&lt;ffff82c48015a261&gt;] idle_loop+0x48/0x59^M
(XEN)    ^M
(XEN) ^M
(XEN) ****************************************^M
(XEN) Panic on CPU 1:^M
(XEN) Assertion '(sp == 0) || (peoi[sp-1].vector &lt; vector)' failed at
irq.c:1027^M
(XEN) ****************************************^M
(XEN) ^M
(XEN) Reboot in five seconds...^M

Am 31.05.2013 22:32, schrieb Andrew Cooper:
</pre>
        <blockquote type="cite">
          <pre wrap="">Recently our automated testing system has caught a curious assertion
while testing Xen 4.1.5 on a HaswellDT system.

(XEN) Assertion '(sp == 0) || (peoi[sp-1].vector &lt; vector)' failed at
irq.c:1030
(XEN) ----[ Xen-4.1.5  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[&lt;ffff82c48016b2b4&gt;] do_IRQ+0x514/0x750
(XEN) RFLAGS: 0000000000010093   CONTEXT: hypervisor
(XEN) rax: 000000000000002f   rbx: ffff830249841e80   rcx:
ffff82c4803127c0
(XEN) rdx: 0000000000000004   rsi: 0000000000000027   rdi:
0000000000000001
(XEN) rbp: 0000000000001e00   rsp: ffff82c4802bfd48   r8: 
ffff82c480312abc
(XEN) r9:  ffff8302498a5948   r10: 0000000000000009   r11:
ffff8302498c6c80
(XEN) r12: ffff830243b07f50   r13: ffff8300a24f8000   r14:
00000af8373788e3
(XEN) r15: ffff830249841e80   cr0: 000000008005003b   cr4:
00000000001026f0
(XEN) cr3: 00000002479e6000   cr2: 00000000e6d3c090
(XEN) ds: 007b   es: 007b   fs: 00d8   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff82c4802bfd48:
(XEN)    ffff830249841eb4 ffff82c480312ec0 000000000000001e
0000001e00000000
(XEN)    0000000000000000 00000000498a5670 ffff830249841d80
ffff830249840080
(XEN)    ffff830249841db4 0000000000000000 ffff8302498a55e0
ffff8302498a5670
(XEN)    ffff8300a24f8000 00000af8373788e3 00000af83736b8ed
ffff82c480162ca0
(XEN)    00000af83736b8ed 00000af8373788e3 ffff8300a24f8000
ffff8302498a5670
(XEN)    ffff8302498a55e0 0000000000000000 ffff8302498c6c80
0000000000000009
(XEN)    ffff8302498a5948 ffff82c480313000 0000000000007f40
0000000000000001
(XEN)    0000000000000000 0000000000000000 00000af80db652fd
0000002700000000
(XEN)    ffff82c4801a50a0 000000000000e008 0000000000000246
ffff82c4802bfe78
(XEN)    0000000000000000 ffff8302498a5670 ffff82c4801a6a56
ffffffffffffffff
(XEN)    ffff830249818000 0000000000000000 ffff8300a24f8000
ffff82c480122c11
(XEN)    00000af839021119 0000000000000000 0000000000000000
00000000802bff18
(XEN)    0000025c0000013b ffff82c4802e7580 ffff82c4802bff18
ffff8300a2838000
(XEN)    ffff82c4802f61a0 ffff8300a24f8000 0000000000000002
00000af837304b45
(XEN)    ffff82c48015b67a 0000000000000000 0000000000000000
0000000000000000
(XEN)    0000000000000000 0000000000000000 00000000ee8a3f8c
0000000000000001
(XEN)    0000000000000000 0000000000000000 0000000000000000
0000000000000000
(XEN)    0000000000000000 0000000000000000 00000000ee8a3f74
0000000000000af8
(XEN)    0000000000000001 0000010000000000 00000000c01013a7
0000000000000061
(XEN)    0000000000000246 00000000ee8a3f70 0000000000000069
0000000000000000
(XEN) Xen call trace:
(XEN)       [&lt;ffff82c48016b2b4&gt;] do_IRQ+0x514/0x750
(XEN)     15[&lt;ffff82c480162ca0&gt;] common_interrupt+0x20/0x30
(XEN)     32[&lt;ffff82c4801a50a0&gt;] lapic_timer_nop+0x0/0x10
(XEN)     38[&lt;ffff82c4801a6a56&gt;] acpi_processor_idle+0x376/0x740
(XEN)     43[&lt;ffff82c480122c11&gt;] do_block+0x71/0xd0
(XEN)     56[&lt;ffff82c48015b67a&gt;] idle_loop+0x1a/0x50
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Assertion '(sp == 0) || (peoi[sp-1].vector &lt; vector)' failed at
irq.c:1030
(XEN) ****************************************

And the disassembly before the assertion:

ffff82c48016b29f:       48 8d 14 85 00 00 00    lea    0x0(,%rax,4),%rdx
ffff82c48016b2a6:       00
ffff82c48016b2a7:       0f b6 44 11 ff          movzbl
-0x1(%rcx,%rdx,1),%eax
ffff82c48016b2ac:       39 c6                   cmp    %eax,%esi
ffff82c48016b2ae:       0f 8f 5c ff ff ff       jg    
ffff82c48016b210 &lt;do_IRQ+0x470&gt;
ffff82c48016b2b4:       0f 0b                   ud2


Xen has been woken up by an interrupt of vector 0x27, but has a vector
0x2f on the top of the pending EOI stack for the local APIC.

I have put in more debugging to dump the LAPIC state of the two
interesting vectors and the IOAPIC state, but I have no idea if/when the
problem might reoccur.

My understanding of LAPIC priority leads me to think that Xen really
shouldn't be woken up by a lower priority vector if a higher priority
one is still un-eoi'd.  There is not yet sufficient information to tell
whether this is truely the case, or that Xen has simply gotten confused
about which vectors it eoi'd.

Having said that, we do keep line level interrupts un-eoi'd for extended
periods while guests service the interrupt.  Given that vectors are
chosen at random, we could get into a situation where a line interrupt
has a vector 0xdf and stays pending for 150ms (which I measured as a
not-overly-uncommon mean-time-till-eoi for line level interrupt).  This
would starve any other guest interrupts for an extended period.

Given directed-eoi support in the past few generations of processor, the
requirement for the pending EOI stack has disappeared as far as I am
aware.  Would it be sensible idea in general to make use of the pending
eoi stack conditional on not having/using directed EOI support?

~Andrew

_______________________________________________
Xen-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Xen-devel@lists.xen.org">Xen-devel@lists.xen.org</a>
<a class="moz-txt-link-freetext" href="http://lists.xen.org/xen-devel">http://lists.xen.org/xen-devel</a>
</pre>
        </blockquote>
        <pre wrap="">
</pre>
      </blockquote>
      <pre wrap="">
</pre>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
Xen-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Xen-devel@lists.xen.org">Xen-devel@lists.xen.org</a>
<a class="moz-txt-link-freetext" href="http://lists.xen.org/xen-devel">http://lists.xen.org/xen-devel</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>

--------------040704050907080007090701--

--------------070602000608000906060500
Content-Type: text/plain; charset=windows-1252;
 name="20130803-crash.log"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="20130803-crash.log"

(XEN) HVM4: 182520].
(XEN) HVM4: [bind=FFFFFA8002A98240] Binding reference count-- = 1.
(XEN) HVM4: [FFFFFA8002155170] WskKnrCompleteRequest: complete irp with IO status = 000000
(XEN) HVM4: 00.
(XEN) HVM4: [addr=FFFFF8A007182520] WskProAPIFreeAddressInfo freed addrinfo.
(XEN) **Pending EOI error
(XEN)   irq 29, vector 0x2e
(XEN)   s[0] irq 29, vec 0x2e, ready 0, ISR 1, TMR 0, IRR 0
(XEN) All LAPIC state:
(XEN) [vector]      ISR      TMR      IRR
(XEN) [1f:02x] 00000000 00000000 00000000
(XEN) [3f:202x] 00016384 4095716568 00000000
(XEN) [5f:402x] 00000000 4041382474 00000000
(XEN) [7f:602x] 00000000 3967325758 00000000
(XEN) [9f:802x] 00000000 2123395250 00000000
(XEN) [bf:a02x] 00000000 1502837374 00000000
(XEN) [df:c02x] 00000000 4270415335 00000000
(XEN) [ff:e02x] 00000000 00000000 00000000
(XEN) Guest interrupt information:
(XEN)    IRQ:   0 affinity:1 vec:f0 type=IO-APIC-edge    status=00000000 mapped, unbound
(XEN)    IRQ:   1 affinity:1 vec:38 type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  1(----),
(XEN)    IRQ:   2 affinity:f vec:00 type=XT-PIC          status=00000000 mapped, unbound
(XEN)    IRQ:   3 affinity:1 vec:40 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   4 affinity:1 vec:48 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   5 affinity:1 vec:50 type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  5(----),
(XEN)    IRQ:   6 affinity:1 vec:58 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   7 affinity:1 vec:60 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   8 affinity:1 vec:68 type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  8(----),
(XEN)    IRQ:   9 affinity:1 vec:70 type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0:  9(----),
(XEN)    IRQ:  10 affinity:1 vec:78 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  11 affinity:1 vec:88 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  12 affinity:1 vec:90 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  13 affinity:1 vec:98 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  14 affinity:1 vec:a0 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  15 affinity:1 vec:a8 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  16 affinity:4 vec:a4 type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0: 16(----),
(XEN)    IRQ:  18 affinity:2 vec:36 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 18(----),
(XEN)    IRQ:  19 affinity:1 vec:39 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  20 affinity:1 vec:29 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  22 affinity:2 vec:3e type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 22(----),
(XEN)    IRQ:  23 affinity:1 vec:d8 type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0: 23(----),
(XEN)    IRQ:  24 affinity:1 vec:28 type=DMA_MSI         status=00000000 mapped, unbound
(XEN)    IRQ:  25 affinity:1 vec:30 type=DMA_MSI         status=00000000 mapped, unbound
(XEN)    IRQ:  26 affinity:f vec:c0 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  27 affinity:f vec:c8 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  28 affinity:f vec:d0 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  29 affinity:4 vec:2e type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:276(----),
(XEN)    IRQ:  30 affinity:4 vec:d5 type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:275(----),
(XEN)    IRQ:  31 affinity:2 vec:dc type=PCI-MSI         status=00000054 in-flight=0 domain-list=0:274(----),
(XEN)    IRQ:  32 affinity:4 vec:44 type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:273(----),
(XEN)    IRQ:  33 affinity:8 vec:a7 type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:272(----),
(XEN)    IRQ:  34 affinity:1 vec:59 type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:271(----),
(XEN) IO-APIC interrupt information:
(XEN)     IRQ  0 Vec240:
(XEN)       Apic 0x00, Pin  2: vec=f0 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  1 Vec 56:
(XEN)       Apic 0x00, Pin  1: vec=38 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  3 Vec 64:
(XEN)       Apic 0x00, Pin  3: vec=40 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  4 Vec 72:
(XEN)       Apic 0x00, Pin  4: vec=48 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  5 Vec 80:
(XEN)       Apic 0x00, Pin  5: vec=50 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  6 Vec 88:
(XEN)       Apic 0x00, Pin  6: vec=58 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  7 Vec 96:
(XEN)       Apic 0x00, Pin  7: vec=60 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  8 Vec104:
(XEN)       Apic 0x00, Pin  8: vec=68 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  9 Vec112:
(XEN)       Apic 0x00, Pin  9: vec=70 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 10 Vec120:
(XEN)       Apic 0x00, Pin 10: vec=78 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 11 Vec136:
(XEN)       Apic 0x00, Pin 11: vec=88 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 12 Vec144:
(XEN)       Apic 0x00, Pin 12: vec=90 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 13 Vec152:
(XEN)       Apic 0x00, Pin 13: vec=98 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 14 Vec160:
(XEN)       Apic 0x00, Pin 14: vec=a0 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 15 Vec168:
(XEN)       Apic 0x00, Pin 15: vec=a8 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 16 Vec164:
(XEN)       Apic 0x00, Pin 16: vec=a4 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 18 Vec 54:
(XEN)       Apic 0x00, Pin 18: vec=36 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 19 Vec 57:
(XEN)       Apic 0x00, Pin 19: vec=39 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:0
(XEN)     IRQ 20 Vec 41:
(XEN)       Apic 0x00, Pin 20: vec=29 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:0
(XEN)     IRQ 22 Vec 62:
(XEN)       Apic 0x00, Pin 22: vec=3e delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 23 Vec216:
(XEN)       Apic 0x00, Pin 23: vec=d8 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0
(XEN) number of MP IRQ sources: 15.
(XEN) number of IO-APIC #2 registers: 24.
(XEN) testing the IO APIC.......................
(XEN) IO APIC #2......
(XEN) .... register #00: 02000000
(XEN) .......    : physical APIC id: 02
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00170020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 0
(XEN) .......     : IO APIC version: 0020
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  0    0    0   0   0    1    1    38
(XEN)  02 000 00  0    0    0   0   0    1    1    F0
(XEN)  03 000 00  0    0    0   0   0    1    1    40
(XEN)  04 000 00  0    0    0   0   0    1    1    48
(XEN)  05 000 00  0    0    0   0   0    1    1    50
(XEN)  06 000 00  0    0    0   0   0    1    1    58
(XEN)  07 000 00  0    0    0   0   0    1    1    60
(XEN)  08 000 00  0    0    0   0   0    1    1    68
(XEN)  09 000 00  0    1    0   0   0    1    1    70
(XEN)  0a 000 00  0    0    0   0   0    1    1    78
(XEN)  0b 000 00  0    0    0   0   0    1    1    88
(XEN)  0c 000 00  0    0    0   0   0    1    1    90
(XEN)  0d 000 00  0    0    0   0   0    1    1    98
(XEN)  0e 000 00  0    0    0   0   0    1    1    A0
(XEN)  0f 000 00  0    0    0   0   0    1    1    A8
(XEN)  10 000 00  0    1    0   1   0    1    1    A4
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  0    1    0   1   0    1    1    36
(XEN)  13 000 00  1    1    0   1   0    1    1    39
(XEN)  14 000 00  1    1    0   1   0    1    1    29
(XEN)  15 07A 0A  1    0    0   0   0    0    2    B4
(XEN)  16 000 00  0    1    0   1   0    1    1    3E
(XEN)  17 000 00  0    1    0   1   0    1    1    D8
(XEN) Using vector-based indexing
(XEN) IRQ to pin mappings:
(XEN) IRQ240 -> 0:2
(XEN) IRQ56 -> 0:1
(XEN) IRQ64 -> 0:3
(XEN) IRQ72 -> 0:4
(XEN) IRQ80 -> 0:5
(XEN) IRQ88 -> 0:6
(XEN) IRQ96 -> 0:7
(XEN) IRQ104 -> 0:8
(XEN) IRQ112 -> 0:9
(XEN) IRQ120 -> 0:10
(XEN) IRQ136 -> 0:11
(XEN) IRQ144 -> 0:12
(XEN) IRQ152 -> 0:13
(XEN) IRQ160 -> 0:14
(XEN) IRQ168 -> 0:15
(XEN) IRQ164 -> 0:16
(XEN) IRQ54 -> 0:18
(XEN) IRQ57 -> 0:19
(XEN) IRQ41 -> 0:20
(XEN) IRQ62 -> 0:22
(XEN) IRQ216 -> 0:23
(XEN) .................................... done.
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 2:
(XEN) CA-107844****************************************
(XEN)
(XEN) Reboot in five seconds...
(XEN) Executing crash image

--------------070602000608000906060500
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

--------------070602000608000906060500--