From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marek Marczykowski Subject: Re: High CPU temp, suspend problem - xen 4.1.5-pre, linux 3.7.x Date: Thu, 28 Mar 2013 15:43:19 +0100 Message-ID: <51545707.8050701@invisiblethingslab.com> References: <5140E69F.9090803@invisiblethingslab.com> <20130315130240.GA8582@phenom.dumpdata.com> <514C79F3.5050504@invisiblethingslab.com> <20130322165651.GA4827@phenom.dumpdata.com> <515036BF.10105@invisiblethingslab.com> <20130325141701.GI11546@phenom.dumpdata.com> <515191CC.6060609@invisiblethingslab.com> <5151AC8C02000078000C88B9@nat28.tlf.novell.com> <5151A788.809@invisiblethingslab.com> <5151D4CC02000078000C8A1C@nat28.tlf.novell.com> <5151D0A9.7070100@invisiblethingslab.com> <5151D49C.2000809@citrix.com> <5151DE1C.1020307@invisiblethingslab.com> <5151E0D5.3050707@citrix.com> <5151E72D.30205@invisiblethingslab.com> <5151EE0B.9030605@citrix.com> <5152C16E02000078000C8CB8@nat28.tlf.novell.com> <515302C3.3000607@invisiblethingslab.com> <5153063C.8020307@citrix.com> <51530709.3050206@invisiblethingslab.com> <51531593.5040701@invisiblethingslab.com> <51531E0B.1030806@citrix.com> <51533771.3080808@invisiblethingslab.com> <515340DD.7020102@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2636314894680947860==" Return-path: In-Reply-To: <515340DD.7020102@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Andrew Cooper Cc: Konrad Rzeszutek Wilk , Jan Beulich , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --===============2636314894680947860== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigA7880512B9B6057D436AB22F" This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigA7880512B9B6057D436AB22F Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 27.03.2013 19:56, Andrew Cooper wrote: > On 27/03/2013 18:16, Marek Marczykowski wrote: >> On 27.03.2013 17:27, Andrew Cooper wrote: >>> On 27/03/2013 15:51, Marek Marczykowski wrote: >>>> On 27.03.2013 15:49, Marek Marczykowski wrote: >>>>> On 27.03.2013 15:46, Andrew Cooper wrote: >>>>>> As for locating the cause of the legacy vectors, it might be a goo= d idea >>>>>> to stick a printk at the top of do_IRQ() which indicates an interr= upt >>>>>> with vector between 0xe0 and 0xef. This might at least indicate w= hether >>>>>> legacy vectors are genuinely being delivered, or whether we have s= ome >>>>>> memory corruption causing these effects. >>>>> Ok, will try something like this. >>>> Nothing interesting here... >>>> Only vector 0xf1 for irq 4 and 0xf0 for irq 0 (which match irq dump = information). >>>> >>> Even in the case where we hit the original assertion? >> Yes, even then. >> >>> If so, then all I can thing is that the move_pending flag for that >>> specific GSI has been corrupted in memory somehow. >> I guest this isn't the case, see below. >> >>> I wonder if hexdumping irq_desc[9] after setup, before sleep, on resu= me >>> and in the case of the assertion failure might give some hints. >> I've tried something like this. Detailed log here: >> http://duch.mimuw.edu.pl/~marmarek/qubes/xen-4.1-suspend-irq9-dump.log= >=20 > This is concerning, unless I am getting utterly confused. Jan: Do you > mind double checking my reasoning? >=20 > irq 0 through 15 should be the PIC irqs, set up in init_IRQ() in > arch/x86/i8259.c >=20 > irq9 should be the irq for the PIC vector which is set up as 0xe9, and > its vector should never change. >=20 > Could you put in extra checks for the sanity of per_cpu(vector_irq, > cpu)[0xe0 thru 0xef] ? Ok, got something here: http://duch.mimuw.edu.pl/~marmarek/qubes/xen-4.1-suspend-irq9-dump2.log Now bug triggered after some time after resume (about 15s). But only CPU0= by scheduler immediately after resume. Interesting part - note vector_irq(e1= ): (XEN) irq_cfg of IRQ 9: (XEN) vector: 188 (XEN) cpu_mask: 00000000,00000000,00000000,00000001 (XEN) old_cpu_mask: 00000000,00000000,00000000,00000002 (XEN) move_cleanup_count: 0x0 (XEN) used_vectors: 49,64,72,74,80-81,88,98,112,120,144,148,152,156,160,164,168,172,178,188,1= 92,196,200,207-208 (XEN) move_in_progress: 0x0 (XEN) irq_desc of IRQ 9: (XEN) status: 16 (XEN) handler: ffff82c480252660 (XEN) msi_desc: 0000000000000000 (XEN) action: ffff83041d9f1ed0 (XEN) depth: 0 (XEN) chip_data: ffff830421080250 (XEN) irq: 9 (XEN) affinity: 00000000,00000000,00000000,00000001 (XEN) pending_mask: 00000000,00000000,00000000,00000000 (XEN) (...) (XEN) vector_irq(e0): 0 (XEN) vector_irq(e1): -1 (XEN) vector_irq(e2): 2 (XEN) vector_irq(e3): 3 (XEN) vector_irq(e4): 4 (XEN) vector_irq(e5): 5 (XEN) vector_irq(e6): 6 (XEN) vector_irq(e7): 7 (XEN) vector_irq(e8): 8 (XEN) vector_irq(e9): 9 (XEN) vector_irq(ea): 10 (XEN) vector_irq(eb): 11 (XEN) vector_irq(ec): 12 (XEN) vector_irq(ed): 13 (XEN) vector_irq(ee): 14 (XEN) vector_irq(ef): 15 (XEN) Xen WARN at io_apic.c:639 (XEN) ----[ Xen-4.1.5-rc1 x86_64 debug=3Dy Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e008:[] smp_irq_move_cleanup_interrupt+0x= 246/0x2c6 (XEN) RFLAGS: 0000000000010046 CONTEXT: hypervisor (XEN) rax: 0000000000000000 rbx: 00000000000000e1 rcx: 00000000000000= 00 (XEN) rdx: 0000000000000000 rsi: 000000000000000a rdi: ffff82c4802592= e0 (XEN) rbp: ffff82c48029fda8 rsp: ffff82c48029fd58 r8: 00000000000000= 04 (XEN) r9: 0000000000000001 r10: 000000000000000f r11: 00000000000000= 02 (XEN) r12: ffff830421080050 r13: ffff830421060134 r14: ffff82c48029ff= 18 (XEN) r15: ffff82c4802dd9e0 cr0: 000000008005003b cr4: 00000000000026= f0 (XEN) cr3: 0000000273d3c000 cr2: ffff88000c360318 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen stack trace from rsp=3Dffff82c48029fd58: (XEN) 0000000000000000 000000008029fd70 ffff82c48029ff18 ffff82c4802dd= 9e0 (XEN) ffff82c480153f55 ffff830421043260 ffff830421043320 0000006f207ab= 134 (XEN) 0000006f207c3b14 ffff82c4802dd600 00007d3b7fd60227 ffff82c48014d= e60 (XEN) ffff82c4802dd600 0000006f207c3b14 0000006f207ab134 ffff830421043= 320 (XEN) ffff82c48029fef0 ffff830421043260 0000ffff0000ffff 0000006f416da= b2e (XEN) ffff830007ef4060 0000006f1fad2570 0000000000003f40 0000000000000= 001 (XEN) 0000000000000000 ffff82c4802de200 0000000002048cac 0000002000000= 000 (XEN) ffff82c480197940 000000000000e008 0000000000000246 ffff82c48029f= e68 (XEN) 000000000000e010 ffff82c48029fef0 ffff82c4801987b7 ffff880402105= d30 (XEN) 00000000ca9a4000 ffffffffffffffff aaaaaaaaaaaaaa00 aaaaaaaaaaaaa= aaa (XEN) 0000006f21136437 0000000000000000 0000000000000000 fffffffffffff= fff (XEN) 000004c200000542 0000000000000000 ffff82c48029ff18 ffff82c48029f= f18 (XEN) 00000000ffffffff 0000000000000002 ffff82c4802dd600 ffff82c48029f= f10 (XEN) ffff82c4801549ce ffff8300ca9a4000 ffff8300ca666000 ffff82c48029f= dc8 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000= 001 (XEN) ffff880402105f00 ffff880402105fd8 0000000000000246 0000000000000= 001 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff81001= 3aa (XEN) ffffffff81a2a858 00000000deadbeef 00000000deadbeef 0000010000000= 000 (XEN) ffffffff810013aa 000000000000e033 0000000000000246 ffff880402105= ee8 (XEN) 000000000000e02b 0000000000000000 0000000000000000 0000000000000= 000 (XEN) Xen call trace: (XEN) [] smp_irq_move_cleanup_interrupt+0x246/0x2c6 (XEN) [] irq_move_cleanup_interrupt+0x30/0x40 (XEN) [] lapic_timer_nop+0x0/0x6 (XEN) [] idle_loop+0x4b/0x59 Ignore rest of comments from my previous mail - I clearly don't understan= d IRQ handling code. --=20 Best Regards / Pozdrawiam, Marek Marczykowski Invisible Things Lab --------------enigA7880512B9B6057D436AB22F Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQEcBAEBAgAGBQJRVFcIAAoJENuP0xzK19csZe4H/1QD/hxhpO/sVGTQ0FxB68ow nhYPUCi4DuBAPAWjm4wn3RKhaxlwBI9fNxGWEILkwsqRZFoQ5bhcAVJfvC3eY3fF 9Bwj1yBUW/pXA07nEKBUqT9RrUjZPKO/BnTrUoW/ZfLdKDbRXtqcRfbnDgFNFCoG WYM06SjxDSE2IMcoSjNc/rSAfTwQ/WFc85OPIxsBwumdfQyZlZ9nvr6LNgdxyUD7 BrkLB8+/2nql0YCjOgeSbkcgQSStzaD+iOihZ9RK8JvOWUwCN67INTcuEHC2q4YJ ACiGFqT/DldAwlNj1d7YAaO5SQqGO4v719QGJty8F7X9if2WmpaPdPbgkW6I/GE= =N8b3 -----END PGP SIGNATURE----- --------------enigA7880512B9B6057D436AB22F-- --===============2636314894680947860== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============2636314894680947860==--