From: Andrew Cooper <andrew.cooper3@citrix.com>
To: "Thimo E." <abc@digithi.de>
Cc: Keir Fraser <keir@xen.org>, Jan Beulich <JBeulich@suse.com>,
Eddie Dong <eddie.dong@intel.com>,
Xen-develList <xen-devel@lists.xen.org>,
Jun Nakajima <jun.nakajima@intel.com>,
Xiantao Zhang <xiantao.zhang@intel.com>
Subject: Re: cpuidle and un-eoid interrupts at the local apic
Date: Fri, 9 Aug 2013 22:44:40 +0100 [thread overview]
Message-ID: <520562C8.8080703@citrix.com> (raw)
In-Reply-To: <520561E1.8020809@citrix.com>
[-- Attachment #1.1: Type: text/plain, Size: 13052 bytes --]
On 09/08/13 22:40, Andrew Cooper wrote:
> On 09/08/13 22:27, Thimo E. wrote:
>> Next crash occured, debugging output included.
>>
>> One Remark: Over the last days (besides many linux PV guests) 1
>> Windows Guest (with PV drivers) was running, today I've started
>> another Windows guest and during 3 hours two crashed occured,
>> coincidence ?
>>
>> Best regards
>> Thimo
>
> So according to my debugging, we really have just pushed the same irq
> which we have subsequently seen again unexpectedly.
>
> This bug has only ever been seen on Haswell hardware, and appears
> linked to running HVM guests.
>
> So either there is an erroneous ACK the LAPIC which is clearing the
> ISR before the PEOI stack is expecting (which I
"can't"
Apologies for the confusion.
~Andrew
> obviously see, looking at the code), or something more funky is going
> on with the hardware.
>
> CC'ing in the Intel maintainers: Do you have any ideas? Could this
> be related to APICv?
>
> ~Andrew
>
>>
>> (XEN) **Pending EOI error
>> (XEN) irq 29, vector 0x24
>> (XEN) s[0] irq 29, vec 0x24, ready 0, ISR 00000001, TMR 00000000,
>> IRR 00000000
>> (XEN) All LAPIC state:
>> (XEN) [vector] ISR TMR IRR
>> (XEN) [1f:00] 00000000 00000000 00000000
>> (XEN) [3f:20] 00000010 76efa12e 00000000
>> (XEN) [5f:40] 00000000 e6f0f2fc 00000000
>> (XEN) [7f:60] 00000000 32d096ca 00000000
>> (XEN) [9f:80] 00000000 78fcf87a 00000000
>> (XEN) [bf:a0] 00000000 f9b9fe4e 00000000
>> (XEN) [df:c0] 00000000 ffdfe7ab 00000000
>> (XEN) [ff:e0] 00000000 00000000 00000000
>> (XEN) Peoi stack trace records:
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Marked {sp 0, irq 29, vec 0x24} ready
>> (XEN) Pushed {sp 0, irq 29, vec 0x24}
>> (XEN) Poped entry {sp 1, irq 29, vec 0x24}
>> (XEN) Guest interrupt information:
>> (XEN) IRQ: 0 affinity:1 vec:f0 type=IO-APIC-edge
>> status=00000000 mapped, unbound
>> (XEN) IRQ: 1 affinity:1 vec:38 type=IO-APIC-edge
>> status=00000050 in-flight=0 domain-list=0: 1(----),
>> (XEN) IRQ: 2 affinity:f vec:00 type=XT-PIC
>> status=00000000 mapped, unbound
>> (XEN) IRQ: 3 affinity:1 vec:40 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 4 affinity:1 vec:48 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 5 affinity:1 vec:50 type=IO-APIC-edge
>> status=00000050 in-flight=0 domain-list=0: 5(----),
>> (XEN) IRQ: 6 affinity:1 vec:58 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 7 affinity:1 vec:60 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 8 affinity:1 vec:68 type=IO-APIC-edge
>> status=00000050 in-flight=0 domain-list=0: 8(----),
>> (XEN) IRQ: 9 affinity:1 vec:70 type=IO-APIC-level
>> status=00000050 in-flight=0 domain-list=0: 9(----),
>> (XEN) IRQ: 10 affinity:1 vec:78 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 11 affinity:1 vec:88 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 12 affinity:1 vec:90 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 13 affinity:1 vec:98 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 14 affinity:1 vec:a0 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 15 affinity:1 vec:a8 type=IO-APIC-edge
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 16 affinity:1 vec:db type=IO-APIC-level
>> status=00000010 in-flight=0 domain-list=0: 16(----),
>> (XEN) IRQ: 18 affinity:1 vec:2c type=IO-APIC-level
>> status=00000010 in-flight=0 domain-list=0: 18(----),
>> (XEN) IRQ: 19 affinity:1 vec:51 type=IO-APIC-level
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 20 affinity:1 vec:29 type=IO-APIC-level
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 22 affinity:1 vec:bb type=IO-APIC-level
>> status=00000050 in-flight=0 domain-list=0: 22(----),
>> (XEN) IRQ: 23 affinity:8 vec:c2 type=IO-APIC-level
>> status=00000050 in-flight=0 domain-list=0: 23(----),
>> (XEN) IRQ: 24 affinity:1 vec:28 type=DMA_MSI
>> status=00000000 mapped, unbound
>> (XEN) IRQ: 25 affinity:1 vec:30 type=DMA_MSI
>> status=00000000 mapped, unbound
>> (XEN) IRQ: 26 affinity:f vec:c0 type=PCI-MSI
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 27 affinity:f vec:c8 type=PCI-MSI
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 28 affinity:f vec:d0 type=PCI-MSI
>> status=00000002 mapped, unbound
>> (XEN) IRQ: 29 affinity:2 vec:24 type=PCI-MSI
>> status=00000010 in-flight=0 domain-list=0:276(----),
>> (XEN) IRQ: 30 affinity:4 vec:93 type=PCI-MSI
>> status=00000050 in-flight=0 domain-list=0:275(----),
>> (XEN) IRQ: 31 affinity:2 vec:4a type=PCI-MSI
>> status=00000050 in-flight=0 domain-list=0:274(----),
>> (XEN) IRQ: 32 affinity:2 vec:73 type=PCI-MSI
>> status=00000050 in-flight=0 domain-list=0:273(----),
>> (XEN) IRQ: 33 affinity:1 vec:49 type=PCI-MSI
>> status=00000050 in-flight=0 domain-list=0:272(----),
>> (XEN) IRQ: 34 affinity:8 vec:5f type=PCI-MSI
>> status=00000050 in-flight=0 domain-list=0:271(----),
>> (XEN) IO-APIC interrupt information:
>> (XEN) IRQ 0 Vec240:
>> (XEN) Apic 0x00, Pin 2: vec=f0 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 1 Vec 56:
>> (XEN) Apic 0x00, Pin 1: vec=38 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 3 Vec 64:
>> (XEN) Apic 0x00, Pin 3: vec=40 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 4 Vec 72:
>> (XEN) Apic 0x00, Pin 4: vec=48 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 5 Vec 80:
>> (XEN) Apic 0x00, Pin 5: vec=50 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 6 Vec 88:
>> (XEN) Apic 0x00, Pin 6: vec=58 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 7 Vec 96:
>> (XEN) Apic 0x00, Pin 7: vec=60 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 8 Vec104:
>> (XEN) Apic 0x00, Pin 8: vec=68 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 9 Vec112:
>> (XEN) Apic 0x00, Pin 9: vec=70 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=L mask=0 dest_id:0
>> (XEN) IRQ 10 Vec120:
>> (XEN) Apic 0x00, Pin 10: vec=78 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 11 Vec136:
>> (XEN) Apic 0x00, Pin 11: vec=88 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 12 Vec144:
>> (XEN) Apic 0x00, Pin 12: vec=90 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 13 Vec152:
>> (XEN) Apic 0x00, Pin 13: vec=98 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 14 Vec160:
>> (XEN) Apic 0x00, Pin 14: vec=a0 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 15 Vec168:
>> (XEN) Apic 0x00, Pin 15: vec=a8 delivery=LoPri dest=L status=0
>> polarity=0 irr=0 trig=E mask=0 dest_id:0
>> (XEN) IRQ 16 Vec219:
>> (XEN) Apic 0x00, Pin 16: vec=db delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=0 dest_id:0
>> (XEN) IRQ 18 Vec 44:
>> (XEN) Apic 0x00, Pin 18: vec=2c delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=0 dest_id:0
>> (XEN) IRQ 19 Vec 81:
>> (XEN) Apic 0x00, Pin 19: vec=51 delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=1 dest_id:0
>> (XEN) IRQ 20 Vec 41:
>> (XEN) Apic 0x00, Pin 20: vec=29 delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=1 dest_id:0
>> (XEN) IRQ 22 Vec187:
>> (XEN) Apic 0x00, Pin 22: vec=bb delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=0 dest_id:0
>> (XEN) IRQ 23 Vec194:
>> (XEN) Apic 0x00, Pin 23: vec=c2 delivery=LoPri dest=L status=0
>> polarity=1 irr=0 trig=L mask=0 dest_id:0
>> (XEN) number of MP IRQ sources: 15.
>> (XEN) number of IO-APIC #2 registers: 24.
>> (XEN) testing the IO APIC.......................
>> (XEN) IO APIC #2......
>> (XEN) .... register #00: 02000000
>> (XEN) ....... : physical APIC id: 02
>> (XEN) ....... : Delivery Type: 0
>> (XEN) ....... : LTS : 0
>> (XEN) .... register #01: 00170020
>> (XEN) ....... : max redirection entries: 0017
>> (XEN) ....... : PRQ implemented: 0
>> (XEN) ....... : IO APIC version: 0020
>> (XEN) .... IRQ redirection table:
>> (XEN) NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
>> (XEN) 00 000 00 1 0 0 0 0 0 0 00
>> (XEN) 01 000 00 0 0 0 0 0 1 1 38
>> (XEN) 02 000 00 0 0 0 0 0 1 1 F0
>> (XEN) 03 000 00 0 0 0 0 0 1 1 40
>> (XEN) 04 000 00 0 0 0 0 0 1 1 48
>> (XEN) 05 000 00 0 0 0 0 0 1 1 50
>> (XEN) 06 000 00 0 0 0 0 0 1 1 58
>> (XEN) 07 000 00 0 0 0 0 0 1 1 60
>> (XEN) 08 000 00 0 0 0 0 0 1 1 68
>> (XEN) 09 000 00 0 1 0 0 0 1 1 70
>> (XEN) 0a 000 00 0 0 0 0 0 1 1 78
>> (XEN) 0b 000 00 0 0 0 0 0 1 1 88
>> (XEN) 0c 000 00 0 0 0 0 0 1 1 90
>> (XEN) 0d 000 00 0 0 0 0 0 1 1 98
>> (XEN) 0e 000 00 0 0 0 0 0 1 1 A0
>> (XEN) 0f 000 00 0 0 0 0 0 1 1 A8
>> (XEN) 10 000 00 0 1 0 1 0 1 1 DB
>> (XEN) 11 000 00 1 0 0 0 0 0 0 00
>> (XEN) 12 000 00 0 1 0 1 0 1 1 2C
>> (XEN) 13 000 00 1 1 0 1 0 1 1 51
>> (XEN) 14 000 00 1 1 0 1 0 1 1 29
>> (XEN) 15 07A 0A 1 0 0 0 0 0 2 B4
>> (XEN) 16 000 00 0 1 0 1 0 1 1 BB
>> (XEN) 17 000 00 0 1 0 1 0 1 1 C2
>> (XEN) Using vector-based indexing
>> (XEN) IRQ to pin mappings:
>> (XEN) IRQ240 -> 0:2
>> (XEN) IRQ56 -> 0:1
>> (XEN) IRQ64 -> 0:3
>> (XEN) IRQ72 -> 0:4
>> (XEN) IRQ80 -> 0:5
>> (XEN) IRQ88 -> 0:6
>> (XEN) IRQ96 -> 0:7
>> (XEN) IRQ104 -> 0:8
>> (XEN) IRQ112 -> 0:9
>> (XEN) IRQ120 -> 0:10
>> (XEN) IRQ136 -> 0:11
>> (XEN) IRQ144 -> 0:12
>> (XEN) IRQ152 -> 0:13
>> (XEN) IRQ160 -> 0:14
>> (XEN) IRQ168 -> 0:15
>> (XEN) IRQ219 -> 0:16
>> (XEN) IRQ44 -> 0:18
>> (XEN) IRQ81 -> 0:19
>> (XEN) IRQ41 -> 0:20
>> (XEN) IRQ187 -> 0:22
>> (XEN) IRQ194 -> 0:23
>> (XEN) .................................... done.
>> (XEN)
>> (XEN) ****************************************
>> (XEN) Panic on CPU 1:
>> (XEN) CA-107844****************************************
>> (XEN)
>> (XEN) Reboot in five seconds...
>> (XEN) Executing crash image
>>
>>
>> Am 05.08.2013 16:51, schrieb Andrew Cooper:
>>> All of these crashes are coming out of mwait_idle, so the cpu in
>>> question has literally just been in an lower power state.
>>>
>>> I am wondering whether there is some caching issue where an update to
>>> the Pending EOI stack pointer got "lost", but this seems like a little
>>> too specific to be reasonably explained as a caching issue.
>>>
>>> A new debugging patch is on its way (Sorry - it has been a very busy few
>>> days)
>>>
>>> ~Andrew
>>>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
[-- Attachment #1.2: Type: text/html, Size: 22925 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
next prev parent reply other threads:[~2013-08-09 21:44 UTC|newest]
Thread overview: 63+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-05-31 20:32 cpuidle and un-eoid interrupts at the local apic Andrew Cooper
2013-06-03 14:30 ` Jan Beulich
2013-07-31 8:30 ` Thimo E.
2013-07-31 9:47 ` Andrew Cooper
2013-08-02 22:50 ` Thimo E.
2013-08-02 23:32 ` Andrew Cooper
2013-08-05 12:45 ` Jan Beulich
2013-08-05 14:51 ` Andrew Cooper
2013-08-09 21:27 ` Thimo E.
2013-08-09 21:40 ` Andrew Cooper
2013-08-09 21:44 ` Andrew Cooper [this message]
2013-08-11 17:46 ` Thimo E.
2013-08-12 6:02 ` Zhang, Yang Z
2013-08-12 8:49 ` Zhang, Yang Z
2013-08-12 8:57 ` Jan Beulich
2013-08-12 11:52 ` Thimo E
2013-08-12 12:04 ` Andrew Cooper
2013-08-19 15:14 ` Thimo E.
2013-08-20 5:43 ` Thimo Eichstädt
2013-08-20 8:40 ` Jan Beulich
2013-08-20 8:50 ` Zhang, Yang Z
2013-08-23 7:22 ` Thimo Eichstädt
2013-08-23 7:30 ` Zhang, Yang Z
2013-08-27 1:03 ` Zhang, Yang Z
2013-09-04 18:32 ` Thimo E.
2013-09-04 18:55 ` Andrew Cooper
2013-09-04 19:56 ` Thimo E.
2013-09-04 20:54 ` Andrew Cooper
2013-09-05 1:45 ` Zhang, Yang Z
2013-09-05 7:20 ` Thimo E.
2013-09-05 1:15 ` Zhang, Yang Z
2013-09-17 2:09 ` Zhang, Yang Z
2013-09-17 7:39 ` Thimo E.
2013-09-17 7:43 ` Zhang, Yang Z
2013-09-17 21:04 ` Thimo E.
2013-09-18 1:18 ` Zhang, Xiantao
2013-09-18 17:24 ` Thimo E.
2013-09-18 12:06 ` Andrew Cooper
2013-08-12 13:54 ` Thimo E
2013-08-12 14:06 ` Andrew Cooper
2013-08-13 1:43 ` Zhang, Yang Z
2013-08-13 6:39 ` Thimo E.
2013-08-13 11:39 ` Wu, Feng
2013-08-13 12:46 ` Andrew Cooper
2013-08-12 9:10 ` Andrew Cooper
2013-08-12 5:50 ` Zhang, Yang Z
2013-08-12 8:20 ` Jan Beulich
2013-08-12 9:28 ` Andrew Cooper
2013-08-12 10:05 ` Jan Beulich
2013-08-12 10:27 ` Andrew Cooper
2013-08-14 2:53 ` Zhang, Yang Z
2013-08-14 7:51 ` Thimo E.
2013-08-14 9:52 ` Andrew Cooper
2013-09-07 13:27 ` Thimo E.
2013-09-07 17:02 ` Andrew Cooper
2013-09-07 23:37 ` Thimo E.
2013-09-08 9:53 ` Andrew Cooper
2013-09-08 10:24 ` Thimo E.
2013-09-09 13:16 ` Andrew Cooper
2013-09-09 14:48 ` Thimo Eichstädt
2013-09-09 15:12 ` Andrew Cooper
2013-09-09 7:59 ` Jan Beulich
2013-09-09 12:53 ` Andrew Cooper
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=520562C8.8080703@citrix.com \
--to=andrew.cooper3@citrix.com \
--cc=JBeulich@suse.com \
--cc=abc@digithi.de \
--cc=eddie.dong@intel.com \
--cc=jun.nakajima@intel.com \
--cc=keir@xen.org \
--cc=xen-devel@lists.xen.org \
--cc=xiantao.zhang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).