From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: [xen-unstable test] 100518: regressions - FAIL Date: Thu, 18 Aug 2016 15:32:27 +0200 Message-ID: <1471527147.6806.96.camel@citrix.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4032367460187362334==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: osstest service owner , xen-devel@lists.xensource.com Cc: George Dunlap , Andrew Cooper , Jan Beulich List-Id: xen-devel@lists.xenproject.org --===============4032367460187362334== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="=-3a7yijpPDoE/V6ypiscb" --=-3a7yijpPDoE/V6ypiscb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, 2016-08-17 at 13:04 +0000, osstest service owner wrote: > flight 100518 xen-unstable real [real] > http://logs.test-lab.xenproject.org/osstest/logs/100518/ >=20 > Regressions :-( >=20 > Tests which did not succeed and are blocking, > including tests which could not be run: > =C2=A0test-amd64-amd64-xl-qemut-winxpsp3 17 guest-start/win.repeat fail > REGR. vs. 100488 >=20 So, FTR, Jan pointed out to me and George that this has failed like this: Aug 17 04:51:03.173959 (XEN) ----[ Xen-4.8-unstable=C2=A0=C2=A0x86_64=C2=A0= =C2=A0debug=3Dy=C2=A0=C2=A0Not tainted ]---- Aug 17 04:51:23.770049 (XEN) CPU:=C2=A0=C2=A0=C2=A0=C2=A00 Aug 17 04:51:23.770111 (XEN) RIP:=C2=A0=C2=A0=C2=A0=C2=A0e008:[] sched_credit.c#_csched_cpu_pick+0x1be/0x62a Aug 17 04:51:23.778034 (XEN) RFLAGS: 0000000000010287=C2=A0=C2=A0=C2=A0CONT= EXT: hypervisor (d0v0) Aug 17 04:51:23.786025 (XEN) rax: 0200200200200200=C2=A0=C2=A0=C2=A0rbx: 00= 00000000000003=C2=A0=C2=A0=C2=A0rcx: ffff82d08032e280 Aug 17 04:51:23.794013 (XEN) rdx: 00000031dd4d8200=C2=A0=C2=A0=C2=A0rsi: ff= ff82d0802ecd20=C2=A0=C2=A0=C2=A0rdi: ffff83009550fca8 Aug 17 04:51:23.801996 (XEN) rbp: ffff83009550fd18=C2=A0=C2=A0=C2=A0rsp: ff= ff83009550fc28=C2=A0=C2=A0=C2=A0r8:=C2=A0=C2=A00000000000000004 Aug 17 04:51:23.810025 (XEN) r9:=C2=A0=C2=A00000000000000000=C2=A0=C2=A0=C2= =A0r10: 0000000000000000=C2=A0=C2=A0=C2=A0r11: 00000000ffffffff Aug 17 04:51:23.817966 (XEN) r12: 0000000000000003=C2=A0=C2=A0=C2=A0r13: ff= ff8300952f9000=C2=A0=C2=A0=C2=A0r14: ffff83024e88cf50 Aug 17 04:51:23.825975 (XEN) r15: ffff82d080344ec0=C2=A0=C2=A0=C2=A0cr0: 00= 00000080050033=C2=A0=C2=A0=C2=A0cr4: 00000000001526e0 Aug 17 04:51:23.833993 (XEN) cr3: 0000000240f83000=C2=A0=C2=A0=C2=A0cr2: 00= 007f0d36e51b0d Aug 17 04:51:23.834030 (XEN) ds: 0000=C2=A0=C2=A0=C2=A0es: 0000=C2=A0=C2=A0= =C2=A0fs: 0000=C2=A0=C2=A0=C2=A0gs: 0000=C2=A0=C2=A0=C2=A0ss: e010=C2=A0=C2= =A0=C2=A0cs: e008 Aug 17 04:51:23.841976 (XEN) Xen code around (sched_cred= it.c#_csched_cpu_pick+0x1be/0x62a): Aug 17 04:51:23.849972 (XEN)=C2=A0=C2=A018 48 8b 00 48 8b 40 28 <48> 8b 40 = 10 66 81 38 ff 7f 75 0f 44 3b 25 7d 12 Aug 17 04:51:23.857973 (XEN) Xen stack trace from rsp=3Dffff83009550fc28: Aug 17 04:51:23.865960 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0ffff83025d853700 000000= 020000001f ffff82d080344ec8 ffff82d080344ec0 Aug 17 04:51:23.873956 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0ffff82d0802e96c0 ffff83= 0257af5570 0000000100000001 0000000000000000 Aug 17 04:51:23.881954 (XEN)=C2=A0=C2=A0=C2=A0=C2=A00000000000000297 ffff83= 009550fc88 ffff82d080130a3e 0000000000000000 Aug 17 04:51:23.889950 (XEN)=C2=A0=C2=A0=C2=A0=C2=A00000000000000206 ffff83= 009550fca8 ffff82d080130a3e ffff83025d844000 Aug 17 04:51:23.897962 (XEN)=C2=A0=C2=A0=C2=A0=C2=A00000000000000006 000000= 0000000000 0000000000000000 0000000000000000 Aug 17 04:51:23.906002 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0000000000000000f 000000= 0000000000 0000000000000000 0000000000000000 Aug 17 04:51:23.913973 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0ffff83009550fd38 ffff83= 00952f9000 ffff830247c96000 0000000000000000 Aug 17 04:51:23.914015 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0ffff83024e88cf50 000000= 0000000000 ffff83009550fd28 ffff82d0801251f8 Aug 17 04:51:23.921976 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0ffff83009550fd78 ffff82= d080125227 ffff83009550fd58 ffff82d08013c7a9 Aug 17 04:51:23.929974 (XEN)=C2=A0=C2=A0=C2=A0=C2=A00000000000000003 ffff83= 00952f9000 ffff830247c96000 0000000000000000 Aug 17 04:51:23.937961 (XEN)=C2=A0=C2=A0=C2=A0=C2=A00000000000000003 000000= 0000000000 ffff83009550fd98 ffff82d08012d3ba Aug 17 04:51:23.945951 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0ffff8300952f9000 ffff83= 0247c96000 ffff83009550fdd8 ffff82d080105d70 Aug 17 04:51:23.953969 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0ffff830257af5570 000000= 0000000000 ffff830247c96000 00007f0d38354004 Aug 17 04:51:23.961962 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0ffff830257af5570 000000= 0000000002 ffff83009550fef8 ffff82d0801039df Aug 17 04:51:23.969963 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0ffff83009550fe18 ffff82= d000000000 0000000000000000 0000000000000003 Aug 17 04:51:23.977950 (XEN)=C2=A0=C2=A0=C2=A0=C2=A00000000000000000 000000= 0242214025 ffff830000000001 ffff83009550fea8 Aug 17 04:51:23.985962 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0ffff83009550fea4 ffff82= e004818900 0000000c0000000f 00007f0d38140013 Aug 17 04:51:23.993958 (XEN)=C2=A0=C2=A0=C2=A0=C2=A00000000000000002 000000= 0000000000 00007ffe23e9a7c0 00007f0d36e456c8 Aug 17 04:51:24.001946 (XEN)=C2=A0=C2=A0=C2=A0=C2=A00000000000000000 00007f= fe23e9a7f0 0000000000407f60 00007f0d3814b2e5 Aug 17 04:51:24.009952 (XEN)=C2=A0=C2=A0=C2=A0=C2=A00000000000a399f0 000000= 0000000013 0000000000000002 0000000000000013 Aug 17 04:51:24.017959 (XEN) Xen call trace: Aug 17 04:51:24.017992 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0[] sc= hed_credit.c#_csched_cpu_pick+0x1be/0x62a Aug 17 04:51:24.025957 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0[] sc= hed_credit.c#csched_cpu_pick+0x1b/0x1d Aug 17 04:51:24.033963 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0[] sc= hed_credit.c#csched_vcpu_insert+0x2d/0x157 Aug 17 04:51:24.041949 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0[] sc= hed_init_vcpu+0x1b8/0x1fe Aug 17 04:51:24.041984 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0[] al= loc_vcpu+0x1c8/0x2c7 Aug 17 04:51:24.049995 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0[] do= _domctl+0x7ea/0x1b24 Aug 17 04:51:24.057972 (XEN)=C2=A0=C2=A0=C2=A0=C2=A0[] ls= tar_enter+0xdd/0x137 Aug 17 04:51:24.058011 (XEN)=C2=A0 Aug 17 04:51:24.065961 (XEN)=C2=A0 Aug 17 04:51:24.065989 (XEN) **************************************** Aug 17 04:51:24.066018 (XEN) Panic on CPU 0: Aug 17 04:51:24.073936 (XEN) GENERAL PROTECTION FAULT Aug 17 04:51:24.073967 (XEN) [error_code=3D0000] Aug 17 04:51:24.073994 (XEN) **************************************** This looks to me the same thing reported by Andrew a couple of days ago, and then covered in the following emails/threads: https://lists.xen.org/archives/html/xen-devel/2016-08/msg01677.html https://lists.xen.org/archives/html/xen-devel/2016-08/msg01679.html https://lists.xen.org/archives/html/xen-devel/2016-08/msg01752.html The fact that, AFAICT, subsequent flights passed (i.e.,=C2=A0100533) is als= o consistent with that, as it's basically a race, and hence it is indeed supposed to manifests itself intermittently. Patch 1 of this series is my fix: https://lists.xen.org/archives/html/xen-devel/2016-08/msg02253.html https://lists.xen.org/archives/html/xen-devel/2016-08/msg02254.html Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-3a7yijpPDoE/V6ypiscb Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJXtbjsAAoJEBZCeImluHPuDesP/08y2AiQZkJOYoLfdDM+4Faf 0JNJlsHFv/gG8cHYvtIssrqcozNvLZg4rzOzqZxV0b21wAne2Iq6v1GBLULEqTpE TTCB8qbprezGWEDbkpMtirmIPNsqTeboT5rjoDxMznX/Z4KyMNS54elopbOB5pKv xVbTvkyZwi4LYZ0Ha16tyd5Eo6RjZE92dIF/v97lL0A2QgOxbl4RFnep8ptDX9kw 9MPtSGsQG5mBji72yJY+zRjs0Vae0A9B4UWWj5W2mPK6nN/W+vJqpkjwAVy/DqTk rMpljJaadYnKVWHY0CvfvclV/indPSayTI2X5yEYy0QDIBeTOnU6cPrHd8nfD/KW oN/h8CWTabu4G2/m2KpYmoJi76l7NbqTNBnY1+yfiN3iqsLXHOvjKk5JRJDuqMrP N0t0Kk2v9H5LyitDENIwDkrLaksXDixrAHrpYMqbfJfhm5Bl9iNIvXq+WCKtRj7G P4zahLOny0K091VWRxSHycqEl/abS4GgGHhCO0wMuA+n7/d2wpe78S6R6A9A+uqv UJBeaXLIMId1+xAfoDbKBhYS6vovPNKQULbXMqC0nK7ZSk84fr7Khea9Tgv22DuZ V5wY1D/yL8MwDTCoXr1S5YkXoc6Hz1Ykpd1yIVjporAJfjjug04mVXOqa0Fer+6I tEXXhnqRPD7yIoyLgL3V =+Mqj -----END PGP SIGNATURE----- --=-3a7yijpPDoE/V6ypiscb-- --===============4032367460187362334== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============4032367460187362334==--