* [linux-4.1 test] 79008: regressions - FAIL
@ 2016-01-26 13:11 osstest service owner
2016-01-27 11:18 ` Ian Campbell
0 siblings, 1 reply; 5+ messages in thread
From: osstest service owner @ 2016-01-26 13:11 UTC (permalink / raw)
To: xen-devel, osstest-admin
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 13345 bytes --]
flight 79008 linux-4.1 real [real]
http://logs.test-lab.xenproject.org/osstest/logs/79008/
Regressions :-(
Tests which did not succeed and are blocking,
including tests which could not be run:
test-armhf-armhf-xl-credit2 15 guest-start/debian.repeat fail REGR. vs. 66399
test-armhf-armhf-xl-xsm 15 guest-start/debian.repeat fail REGR. vs. 66399
test-armhf-armhf-xl-cubietruck 15 guest-start/debian.repeat fail in 78925 REGR. vs. 66399
Tests which are failing intermittently (not blocking):
test-amd64-amd64-xl-qemuu-debianhvm-amd64 3 host-install(3) broken in 78925 pass in 79008
test-amd64-i386-qemuu-rhel6hvm-amd 3 host-install(3) broken in 78925 pass in 79008
test-amd64-i386-xl-qemut-debianhvm-amd64 9 debian-hvm-install fail in 78925 pass in 79008
test-armhf-armhf-xl-credit2 11 guest-start fail in 78925 pass in 79008
test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm 15 guest-localmigrate/x10 fail in 78925 pass in 79008
test-armhf-armhf-xl-cubietruck 11 guest-start fail pass in 78925
Regressions which are regarded as allowable (not blocking):
test-amd64-i386-rumpuserxen-i386 10 guest-start fail like 66399
test-amd64-amd64-xl-qemut-win7-amd64 16 guest-stop fail like 66399
test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stop fail like 66399
test-armhf-armhf-xl-rtds 15 guest-start/debian.repeat fail like 66399
Tests which did not succeed, but are not blocking:
test-armhf-armhf-xl-cubietruck 12 migrate-support-check fail in 78925 never pass
test-armhf-armhf-xl-cubietruck 13 saverestore-support-check fail in 78925 never pass
test-amd64-amd64-xl-pvh-amd 11 guest-start fail never pass
test-amd64-amd64-xl-pvh-intel 14 guest-saverestore fail never pass
test-armhf-armhf-libvirt-xsm 12 migrate-support-check fail never pass
test-armhf-armhf-libvirt-xsm 14 guest-saverestore fail never pass
test-amd64-amd64-libvirt 12 migrate-support-check fail never pass
test-amd64-amd64-libvirt-xsm 12 migrate-support-check fail never pass
test-armhf-armhf-libvirt 14 guest-saverestore fail never pass
test-armhf-armhf-libvirt 12 migrate-support-check fail never pass
test-armhf-armhf-libvirt-qcow2 11 migrate-support-check fail never pass
test-armhf-armhf-libvirt-qcow2 13 guest-saverestore fail never pass
test-amd64-i386-libvirt-xsm 12 migrate-support-check fail never pass
test-amd64-amd64-qemuu-nested-amd 16 debian-hvm-install/l1/l2 fail never pass
test-armhf-armhf-xl-credit2 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-credit2 12 migrate-support-check fail never pass
test-armhf-armhf-xl-arndale 12 migrate-support-check fail never pass
test-armhf-armhf-xl-arndale 13 saverestore-support-check fail never pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check fail never pass
test-armhf-armhf-xl 12 migrate-support-check fail never pass
test-armhf-armhf-xl 13 saverestore-support-check fail never pass
test-amd64-i386-libvirt 12 migrate-support-check fail never pass
test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop fail never pass
test-amd64-amd64-libvirt-vhd 11 migrate-support-check fail never pass
test-armhf-armhf-xl-xsm 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-xsm 12 migrate-support-check fail never pass
test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop fail never pass
test-armhf-armhf-xl-rtds 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-rtds 12 migrate-support-check fail never pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check fail never pass
test-armhf-armhf-xl-multivcpu 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-multivcpu 12 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 11 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 12 saverestore-support-check fail never pass
test-armhf-armhf-libvirt-raw 9 debian-di-install fail never pass
version targeted for testing:
linux 99c5a856dcee7658ec7e250aa477a9afaab8cfc6
baseline version:
linux 07cc49f66973f49a391c91bf4b158fa0f2562ca8
Last test of basis 66399 2015-12-15 18:20:39 Z 41 days
Testing same since 78925 2016-01-24 13:50:39 Z 1 days 2 attempts
------------------------------------------------------------
People who touched revisions under test:
Alan Stern <stern@rowland.harvard.edu>
Alexey Khoroshilov <khoroshilov@ispras.ru>
Arnd Bergmann <arnd@arndb.de>
Azael Avalos <coproscefalo@gmail.com>
Ben Hutchings <ben@decadent.org.uk>
Bjørn Mork <bjorn@mork.no>
Cong Wang <xiyou.wangcong@gmail.com>
Daniele Palmas <dnlplm@gmail.com>
Darren Hart <dvhart@linux.intel.com>
David Howells <dhowells@redhat.com>
David J. Wilder <dwilder@us.ibm.com>
David S. Miller <davem@davemloft.net>
Dmitry Katsubo <dmitry.katsubo@gmail.com>
Dmitry Vyukov <dvyukov@google.com>
Don Zickus <dzickus@redhat.com>
Eric Dumazet <edumazet@google.com>
Felipe Balbi <balbi@ti.com>
Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Guillaume Nault <g.nault@alphalink.fr>
Hannes Frederic Sowa <hannes@stressinduktion.org>
Hans Yang <hansy@nvidia.com>
Hayes Wang <hayeswang@realtek.com>
Herbert Xu <herbert@gondor.apana.org.au>
James Morris <james.l.morris@oracle.com>
Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Jarod Wilson <jarod@redhat.com>
Jiri Benc <jbenc@redhat.com>
Johan Hovold <johan@kernel.org>
Jonas Jonsson <jonas@ludd.ltu.se>
Konstantin Shkolnyy <konstantin.shkolnyy@gmail.com>
Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
Michael Marley <michael@michaelmarley.com>
Michal Hocko <mhocko@suse.com>
Mimi Zohar <zohar@linux.vnet.ibm.com> (on TPM 1.2)
Nicolas Dichtel <nicolas.dichtel@6wind.com>
Pavel Machek <pavel@ucw.cz>
Peter Huewe <peterhuewe@gmx.de>
Peter Wu <peter@lekensteyn.nl>
Prarit Bhargava <prarit@redhat.com>
Rainer Weikusat <rweikusat@mobileactivedefense.com>
Robert Jarzmik <robert.jarzmik@free.fr>
Sergei Shtylyov <sergei.shtylyov@cogentembedded.com>
Stas Nichiporovich <stasn77@gmail.com>
Stefan Wahren <stefan.wahren@i2se.com>
Tadeusz Struk <tadeusz.struk@intel.com>
tadeusz.struk@intel.com <tadeusz.struk@intel.com>
Tom Herbert <tom@herbertland.com>
Vlad Yasevich <vyasevich@gmail.com>
Vladislav Yasevich <vyasevic@redhat.com>
Vladislav Yasevich <vyasevich@gmail.com>
WANG Cong <xiyou.wangcong@gmail.com>
Willem de Bruijn <willemb@google.com>
Yevgeny Pats <yevgeny@perception-point.io>
Yuchung Cheng <ycheng@google.com>
jobs:
build-amd64-xsm pass
build-armhf-xsm pass
build-i386-xsm pass
build-amd64 pass
build-armhf pass
build-i386 pass
build-amd64-libvirt pass
build-armhf-libvirt pass
build-i386-libvirt pass
build-amd64-pvops pass
build-armhf-pvops pass
build-i386-pvops pass
build-amd64-rumpuserxen pass
build-i386-rumpuserxen pass
test-amd64-amd64-xl pass
test-armhf-armhf-xl pass
test-amd64-i386-xl pass
test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-amd64-libvirt-xsm pass
test-armhf-armhf-libvirt-xsm fail
test-amd64-i386-libvirt-xsm pass
test-amd64-amd64-xl-xsm pass
test-armhf-armhf-xl-xsm fail
test-amd64-i386-xl-xsm pass
test-amd64-amd64-qemuu-nested-amd fail
test-amd64-amd64-xl-pvh-amd fail
test-amd64-i386-qemut-rhel6hvm-amd pass
test-amd64-i386-qemuu-rhel6hvm-amd pass
test-amd64-amd64-xl-qemut-debianhvm-amd64 pass
test-amd64-i386-xl-qemut-debianhvm-amd64 pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-freebsd10-amd64 pass
test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
test-amd64-i386-xl-qemuu-ovmf-amd64 pass
test-amd64-amd64-rumpuserxen-amd64 pass
test-amd64-amd64-xl-qemut-win7-amd64 fail
test-amd64-i386-xl-qemut-win7-amd64 fail
test-amd64-amd64-xl-qemuu-win7-amd64 fail
test-amd64-i386-xl-qemuu-win7-amd64 fail
test-armhf-armhf-xl-arndale pass
test-amd64-amd64-xl-credit2 pass
test-armhf-armhf-xl-credit2 fail
test-armhf-armhf-xl-cubietruck fail
test-amd64-i386-freebsd10-i386 pass
test-amd64-i386-rumpuserxen-i386 fail
test-amd64-amd64-qemuu-nested-intel pass
test-amd64-amd64-xl-pvh-intel fail
test-amd64-i386-qemut-rhel6hvm-intel pass
test-amd64-i386-qemuu-rhel6hvm-intel pass
test-amd64-amd64-libvirt pass
test-armhf-armhf-libvirt fail
test-amd64-i386-libvirt pass
test-amd64-amd64-xl-multivcpu pass
test-armhf-armhf-xl-multivcpu pass
test-amd64-amd64-pair pass
test-amd64-i386-pair pass
test-amd64-amd64-libvirt-pair pass
test-amd64-i386-libvirt-pair pass
test-amd64-amd64-amd64-pvgrub pass
test-amd64-amd64-i386-pvgrub pass
test-amd64-amd64-pygrub pass
test-armhf-armhf-libvirt-qcow2 fail
test-amd64-amd64-xl-qcow2 pass
test-armhf-armhf-libvirt-raw fail
test-amd64-i386-xl-raw pass
test-amd64-amd64-xl-rtds pass
test-armhf-armhf-xl-rtds fail
test-amd64-i386-xl-qemut-winxpsp3-vcpus1 pass
test-amd64-i386-xl-qemuu-winxpsp3-vcpus1 pass
test-amd64-amd64-libvirt-vhd pass
test-armhf-armhf-xl-vhd pass
test-amd64-amd64-xl-qemut-winxpsp3 pass
test-amd64-i386-xl-qemut-winxpsp3 pass
test-amd64-amd64-xl-qemuu-winxpsp3 pass
test-amd64-i386-xl-qemuu-winxpsp3 pass
------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images
Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs
Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
Not pushing.
(No revision log; it would be 1259 lines long.)
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [linux-4.1 test] 79008: regressions - FAIL 2016-01-26 13:11 [linux-4.1 test] 79008: regressions - FAIL osstest service owner @ 2016-01-27 11:18 ` Ian Campbell 2016-01-27 12:05 ` Ian Campbell 0 siblings, 1 reply; 5+ messages in thread From: Ian Campbell @ 2016-01-27 11:18 UTC (permalink / raw) To: osstest service owner, xen-devel, Ian Jackson On Tue, 2016-01-26 at 13:11 +0000, osstest service owner wrote: > flight 79008 linux-4.1 real [real] > http://logs.test-lab.xenproject.org/osstest/logs/79008/ > > Regressions :-( > > Tests which did not succeed and are blocking, > including tests which could not be run: > test-armhf-armhf-xl-credit2 15 guest-start/debian.repeat fail REGR. vs. 66399 > test-armhf-armhf-xl-xsm 15 guest-start/debian.repeat fail REGR. vs. 66399 These were both: 2016-01-26 01:20:33 Z executing ssh ... root@172.16.147.101 echo guest debian.guest.osstest: ok Warning: Permanently added '172.16.147.101' (ECDSA) to the list of known hosts. key_verify failed for server_host_key Searching for that message online shows a variety of results for the last 15 years or so, although it's not clear that any of them are actually the same, many of them also involve a "hash mismatch" message, which is not present here. As far as our own tests go: $ echo ~osstest/pub/logs/*/*/[0-9]*.log | xargs grep -l "key_verify failed for server_host_key" /home/osstest/pub/logs/78033/test-armhf-armhf-xl-cubietruck/15.ts-repeat-test.log /home/osstest/pub/logs/78925/test-armhf-armhf-xl-credit2/11.ts-guest-start.log /home/osstest/pub/logs/78925/test-armhf-armhf-xl-cubietruck/15.ts-repeat-test.log /home/osstest/pub/logs/78925/test-armhf-armhf-xl-xsm/15.ts-repeat-test.log /home/osstest/pub/logs/79008/test-armhf-armhf-xl-credit2/15.ts-repeat-test.log /home/osstest/pub/logs/79008/test-armhf-armhf-xl-cubietruck/11.ts-guest-start.log /home/osstest/pub/logs/79008/test-armhf-armhf-xl-xsm/15.ts-repeat-test.log /home/osstest/pub/logs/79088/test-armhf-armhf-xl-credit2/15.ts-repeat-test.log /home/osstest/pub/logs/79090/test-armhf-armhf-xl-credit2/11.ts-guest-start.log /home/osstest/pub/logs/79090/test-armhf-armhf-xl-cubietruck/11.ts-guest-start.log /home/osstest/pub/logs/79090/test-armhf-armhf-xl-xsm/11.ts-guest-start.log /home/osstest/pub/logs/79120/test-armhf-armhf-xl-xsm/15.ts-repeat-test.log From flights (machines in same order as tests above): [linux-linus bisection] 78033: tested test-armhf-armhf-xl-cubietruck cubietruck-braque [linux-4.1 test] 78925: regressions - trouble: broken/fail/pass cubietruck-metzinger cubietruck-gleizes cubietruck-gleizes [linux-4.1 test] 79008: regressions - FAIL cubietruck-metzinger [linux-4.1 bisection] 79088: testing test-armhf-armhf-xl-credit2 cubietruck-metzinger 79090 is still in progress (nearly complete) but is on linux-4.1 cubietruck-metzinger cubietruck-gleizes cubietruck-gleizes [linux-4.1 bisection] 79120: testing test-armhf-armhf-xl-xsm cubietruck-gleizes Neither of the bisects look likely to make any progress: http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-4.1/test-armhf-armhf-xl-xsm.guest-start--debian.repeat.html http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-4.1/test-armhf-armhf-xl-credit2.guest-start--debian.repeat.html We only have logs back as far as 20 Jan, so it's possible there isn't enough data here to decide anything, but the span does include some logs from linux-3.18 which seems to not be showing this. OTOH linux-linus has only hit it once, maybe this was fixed already upstream. I suspect linux-4.1 has a cubietruck specific hardware issue (perhaps in the network driver). I'm not sure how best to approach it though, especially given that it doesn't seem to be reproducible by the bisector. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [linux-4.1 test] 79008: regressions - FAIL 2016-01-27 11:18 ` Ian Campbell @ 2016-01-27 12:05 ` Ian Campbell 2016-02-02 14:10 ` Ian Campbell 0 siblings, 1 reply; 5+ messages in thread From: Ian Campbell @ 2016-01-27 12:05 UTC (permalink / raw) To: osstest service owner, xen-devel, Ian Jackson On Wed, 2016-01-27 at 11:18 +0000, Ian Campbell wrote: > On Tue, 2016-01-26 at 13:11 +0000, osstest service owner wrote: > > flight 79008 linux-4.1 real [real] > > http://logs.test-lab.xenproject.org/osstest/logs/79008/ > > > > Regressions :-( > > > > Tests which did not succeed and are blocking, > > including tests which could not be run: > > test-armhf-armhf-xl-credit2 15 guest-start/debian.repeat fail REGR. vs. 66399 > > test-armhf-armhf-xl-xsm 15 guest-start/debian.repeat fail REGR. vs. 66399 > > These were both: > > 2016-01-26 01:20:33 Z executing ssh ... root@172.16.147.101 echo guest debian.guest.osstest: ok > Warning: Permanently added '172.16.147.101' (ECDSA) to the list of known hosts. > key_verify failed for server_host_key While poking through the test histories I also spotted: http://logs.test-lab.xenproject.org/osstest/logs/78799/test-armhf-armhf-xl-cubietruck/info.html http://logs.test-lab.xenproject.org/osstest/logs/78997/test-armhf-armhf-xl-cubietruck/info.html Which are: 2016-01-26 01:03:34 Z executing ssh ... root@172.16.144.48 ps -wwef Corrupted MAC on input. Disconnecting: Packet corrupt (this is dom0 rather than the "server_host_key" thing which was guest) But it seems like it could be a different symptom of the same underlying cause. Data-mining the logs a bit I find: 78033 test-armhf-armhf-xl-cubietruck 16.ts-logs-capture.log host cubietruck-braque 78799 test-armhf-armhf-libvirt-raw 8.ts-leak-check.log host~ cubietruck-braque 78799 test-armhf-armhf-libvirt-raw 9.ts-logs-capture.log host~ cubietruck-braque 78799 test-armhf-armhf-libvirt-xsm 8.ts-leak-check.log host~ cubietruck-braque 78799 test-armhf-armhf-libvirt-xsm 9.ts-logs-capture.log host~ cubietruck-braque 78799 test-armhf-armhf-xl 8.ts-leak-check.log host~ cubietruck-picasso 78799 test-armhf-armhf-xl 9.ts-logs-capture.log host~ cubietruck-picasso 78799 test-armhf-armhf-xl-credit2 8.ts-leak-check.log host~ cubietruck-picasso 78799 test-armhf-armhf-xl-credit2 9.ts-logs-capture.log host~ cubietruck-picasso 78799 test-armhf-armhf-xl-cubietruck 8.ts-leak-check.log host~ cubietruck-braque 78799 test-armhf-armhf-xl-cubietruck 9.ts-logs-capture.log host~ cubietruck-braque 78799 test-armhf-armhf-xl-multivcpu 8.ts-leak-check.log host~ cubietruck-gleizes 78799 test-armhf-armhf-xl-multivcpu 9.ts-logs-capture.log host~ cubietruck-gleizes 78799 test-armhf-armhf-xl-vhd 10.ts-logs-capture.log host~ cubietruck-metzinger 78799 test-armhf-armhf-xl-xsm 8.ts-leak-check.log host~ cubietruck-metzinger 78799 test-armhf-armhf-xl-xsm 9.ts-logs-capture.log host~ cubietruck-metzinger 78925 test-armhf-armhf-libvirt-raw 10.ts-logs-capture.log host~ cubietruck-picasso 78925 test-armhf-armhf-xl-credit2 12.ts-logs-capture.log host~ cubietruck-metzinger 78925 test-armhf-armhf-xl-cubietruck 16.ts-logs-capture.log host~ cubietruck-gleizes 78925 test-armhf-armhf-xl-multivcpu 19.ts-logs-capture.log host~ cubietruck-metzinger 78925 test-armhf-armhf-xl-xsm 16.ts-logs-capture.log host~ cubietruck-gleizes 78997 test-armhf-armhf-libvirt-raw 10.ts-logs-capture.log host~ cubietruck-braque 78997 test-armhf-armhf-libvirt-xsm 8.ts-leak-check.log host~ cubietruck-braque 78997 test-armhf-armhf-libvirt-xsm 9.ts-logs-capture.log host~ cubietruck-braque 78997 test-armhf-armhf-xl 8.ts-leak-check.log host~ cubietruck-gleizes 78997 test-armhf-armhf-xl 9.ts-logs-capture.log host~ cubietruck-gleizes 78997 test-armhf-armhf-xl-credit2 8.ts-leak-check.log host~ cubietruck-picasso 78997 test-armhf-armhf-xl-credit2 9.ts-logs-capture.log host~ cubietruck-picasso 78997 test-armhf-armhf-xl-cubietruck 8.ts-leak-check.log host~ cubietruck-braque 78997 test-armhf-armhf-xl-cubietruck 9.ts-logs-capture.log host~ cubietruck-braque 78997 test-armhf-armhf-xl-multivcpu 8.ts-leak-check.log host~ cubietruck-metzinger 78997 test-armhf-armhf-xl-multivcpu 9.ts-logs-capture.log host~ cubietruck-metzinger 78997 test-armhf-armhf-xl-vhd 10.ts-logs-capture.log host~ cubietruck-metzinger 78997 test-armhf-armhf-xl-xsm 8.ts-leak-check.log host~ cubietruck-gleizes 78997 test-armhf-armhf-xl-xsm 9.ts-logs-capture.log host~ cubietruck-gleizes 79008 test-armhf-armhf-libvirt-raw 10.ts-logs-capture.log host~ cubietruck-picasso 79008 test-armhf-armhf-xl-credit2 16.ts-logs-capture.log host~ cubietruck-metzinger 79008 test-armhf-armhf-xl-cubietruck 12.ts-logs-capture.log host~ cubietruck-gleizes 79008 test-armhf-armhf-xl-multivcpu 19.ts-logs-capture.log host~ cubietruck-picasso 79008 test-armhf-armhf-xl-xsm 16.ts-logs-capture.log host~ cubietruck-gleizes 79088 test-armhf-armhf-xl-credit2 16.ts-logs-capture.log host cubietruck-metzinger 79090 test-armhf-armhf-libvirt-raw 10.ts-logs-capture.log host~ cubietruck-picasso 79090 test-armhf-armhf-xl-credit2 12.ts-logs-capture.log host~ cubietruck-metzinger 79090 test-armhf-armhf-xl-cubietruck 12.ts-logs-capture.log host~ cubietruck-gleizes 79090 test-armhf-armhf-xl-vhd 10.ts-logs-capture.log host~ cubietruck-braque 79090 test-armhf-armhf-xl-xsm 12.ts-logs-capture.log host~ cubietruck-gleizes 79120 test-armhf-armhf-xl-xsm 16.ts-logs-capture.log host cubietruck-gleizes and: 78033|linux-linus 78799|linux-next 78925|linux-4.1 78997|linux-next 79008|linux-4.1 79088|linux-4.1 79090|linux-4.1 79120|linux-4.1 Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [linux-4.1 test] 79008: regressions - FAIL 2016-01-27 12:05 ` Ian Campbell @ 2016-02-02 14:10 ` Ian Campbell 2016-02-05 14:51 ` Ian Campbell 0 siblings, 1 reply; 5+ messages in thread From: Ian Campbell @ 2016-02-02 14:10 UTC (permalink / raw) To: osstest service owner, xen-devel, Ian Jackson [-- Attachment #1: Type: text/plain, Size: 2077 bytes --] On Wed, 2016-01-27 at 12:05 +0000, Ian Campbell wrote: > On Wed, 2016-01-27 at 11:18 +0000, Ian Campbell wrote: > > On Tue, 2016-01-26 at 13:11 +0000, osstest service owner wrote: > > > flight 79008 linux-4.1 real [real] > > > http://logs.test-lab.xenproject.org/osstest/logs/79008/ > > > > > > Regressions :-( > > > > > > Tests which did not succeed and are blocking, > > > including tests which could not be run: > > > test-armhf-armhf-xl-credit2 15 guest-start/debian.repeat fail REGR. > > > vs. 66399 > > > test-armhf-armhf-xl-xsm 15 guest-start/debian.repeat fail REGR. > > > vs. 66399 > > > > These were both: > > > > 2016-01-26 01:20:33 Z executing ssh ... root@172.16.147.101 echo guest > > debian.guest.osstest: ok > > Warning: Permanently added '172.16.147.101' (ECDSA) to the list of > > known hosts. > > key_verify failed for server_host_key So I've narrowed this down a bit, but not yet sufficiently to actually diagnose. The issue only occurs when the userspace is Debian Jessie. Debian Wheezy does not, for some reason, expose this. It seems unlikely (although not impossible) to be a real issue in Jessie vs Wheezy, more likely some different behaviour in Jessie's sshd just exposes some issue somewhere else. When running Jessie userspace the issue only appeared somewhere between Linux v3.18 and v3.19, I'm currently looking at bisecting that range in case the commit which exposed the issue gives a hint (I fear it wont though). The attached ts-fetch-check-file exposes this pretty readily against either dom0 or domU (slightly differing symptoms) on effected versions. I can reproduce on the cubietruck on my desk as well as in the COLO. I cannot reproduce on the same cubietruck on my desk when running native 4.1, it only happens when running under Xen. I cannot reproduce on the arndale on my desk. I've not tried in the COLO since the test results suggest there would be no point -- there is no sign of this class of failure in the colo on arndale nor on any x86 box. Ian. [-- Attachment #2: 0001-ts-fetch-check-file-new-ts-to-fetch-a-file-and-check.patch --] [-- Type: text/x-patch, Size: 4400 bytes --] From 337f663c13e46f815ce1f13b070b492f8d248b0c Mon Sep 17 00:00:00 2001 From: Ian Campbell <ian.campbell@citrix.com> Date: Tue, 2 Feb 2016 10:54:42 +0000 Subject: [PATCH] ts-fetch-check-file: new ts to fetch a file and check for corruption Compares a checksum computed on the target with one computed after cat'ting the file over ssh. Picks up on network corruption errors etc which might be missed with smaller interactions. Works for guests or hosts. To support this add a variant of target_cmd_output which returns the file descriptor instead of the actual data (which could be large), allowing us to pipe it to the local sum. Signed-off-by: Ian Campbell <ian.campbell@citrix.com> --- Osstest/TestSupport.pm | 10 ++++++-- ts-fetch-check-file | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+), 2 deletions(-) create mode 100755 ts-fetch-check-file diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm index 2141905..3c287b2 100644 --- a/Osstest/TestSupport.pm +++ b/Osstest/TestSupport.pm @@ -51,6 +51,7 @@ BEGIN { target_cmd_root target_cmd target_cmd_build target_cmd_output_root target_cmd_output + target_cmd_stdoutfd target_cmd_stdoutfd_root target_cmd_inputfh_root sshuho target_getfile target_getfile_root target_putfile target_putfile_root @@ -646,9 +647,11 @@ sub target_cmd ($$;$$) { tcmd(undef,undef,'osstest',@_); } sub target_cmd_root ($$;$$) { tcmd(undef,undef,'root',@_); } sub tcmdout { + my $wantfd = shift; my $stdout= IO::File::new_tmpfile(); tcmd(undef,$stdout,@_); $stdout->seek(0,0) or die "$stdout $!"; + return $stdout if $wantfd; my $r; { local ($/) = undef; $r= <$stdout>; } @@ -657,8 +660,11 @@ sub tcmdout { return $r; } -sub target_cmd_output ($$;$) { tcmdout('osstest',@_); } -sub target_cmd_output_root ($$;$) { tcmdout('root',@_); } +sub target_cmd_output ($$;$) { tcmdout(0,'osstest',@_); } +sub target_cmd_output_root ($$;$) { tcmdout(0,'root',@_); } + +sub target_cmd_stdoutfd ($$;$$) { tcmdout(1,'osstest',@_); } +sub target_cmd_stdoutfd_root ($$;$$) { tcmdout(1,'root',@_); } sub target_cmd_inputfh_root ($$$;$$) { my ($tho,$stdinfh,$tcmd,@rest) = @_; diff --git a/ts-fetch-check-file b/ts-fetch-check-file new file mode 100755 index 0000000..bfceb6b --- /dev/null +++ b/ts-fetch-check-file @@ -0,0 +1,68 @@ +#!/usr/bin/perl -w +# This is part of "osstest", an automated testing framework for Xen. +# Copyright (C) 2016 Citrix Inc. +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +use strict qw(vars); +use DBI; +use Osstest; +use Osstest::TestSupport; + +use IO::Pipe; + +tsreadconfig(); + +our ($whhost,$guest) = @ARGV; +$whhost ||= 'host'; + +our ($ho,$gho); +our $fn = "/bin/bash"; # reasonable size, present in most guests and hosts + +$ho= selecthost($whhost); +$gho= selectguest($guest,$ho) if $guest; + +sub fetch_and_check_file ($$) { + my ($t,$fn) = @_; + + target_check_ip($t); + + target_cmd_root($t, "ls -lH $fn"); + + my $expect = target_cmd_output_root($t,"sum $fn"); + logm($expect); + + my $stdout= target_cmd_stdoutfd_root($t,"cat $fn",5,[qw(-v)]); + + my $pipe= IO::Pipe->new(); + my $child= fork; die $! unless defined $child; + if (!$child) { + $pipe->writer(); + open STDIN, "<&", $stdout or die "STDIN $!"; + open STDOUT, ">&", $pipe or die "STDOUT $!"; + exec("sum") or die "pipe writer $!"; + } + + $pipe->reader(); + + my $got = <$pipe>; + chomp($got); + + logm("expected $expect"); + logm("got $got"); + + die unless $expect eq $got; +} + +fetch_and_check_file($gho ? $gho : $ho, $fn); -- 2.6.1 [-- Attachment #3: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [linux-4.1 test] 79008: regressions - FAIL 2016-02-02 14:10 ` Ian Campbell @ 2016-02-05 14:51 ` Ian Campbell 0 siblings, 0 replies; 5+ messages in thread From: Ian Campbell @ 2016-02-05 14:51 UTC (permalink / raw) To: osstest service owner, xen-devel, Ian Jackson; +Cc: Stefano Stabellini On Tue, 2016-02-02 at 14:10 +0000, Ian Campbell wrote: > > When running Jessie userspace the issue only appeared somewhere between > Linux v3.18 and v3.19, I'm currently looking at bisecting that range in > case the commit which exposed the issue gives a hint (I fear it wont > though). Bisecting the dom0 failure lead me to: 3567258d281b5b515d5165ed23851d9f84087e7d is the first bad commit commit 3567258d281b5b515d5165ed23851d9f84087e7d Author: Stefano Stabellini <stefano.stabellini@eu.citrix.com> Date: Fri Nov 21 11:05:39 2014 +0000 xen/arm: use hypercall to flush caches in map_page In xen_dma_map_page, if the page is a local page, call the native map_page dma_ops. If the page is foreign, call __xen_dma_map_page that issues any required cache maintenane operations via hypercall. The reason for doing this is that the native dma_ops map_page could allocate buffers than need to be freed. If the page is foreign we don't call the native unmap_page dma_ops function, resulting in a memory leak. Suggested-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> However AIUI this commit is supposed to be a NOP for all dom0 initiated I/O, which is all which should be occurring in a test which only involves ssh to dom0. Something to do with cache flushes, dma and/or barriers does seem like a plausible candidate for there error though. Ian. NB last few were combined with git cherry-pick --no-commit 28603d13997e2ef47f18589cc9a44553aad49c86 else the NIC driver just crashes on boot. git bisect start # bad: [bfa76d49576599a4b9f9b7a71f23d73d6dcff735] Linux 3.19 git bisect bad bfa76d49576599a4b9f9b7a71f23d73d6dcff735 # good: [b2776bf7149bddd1f4161f14f79520f17fc1d71d] Linux 3.18 git bisect good b2776bf7149bddd1f4161f14f79520f17fc1d71d # bad: [54850e73e86e3bc092680d1bdb84eb322f982ab1] zram: change parameter from vaild_io_request() git bisect bad 54850e73e86e3bc092680d1bdb84eb322f982ab1 # good: [6b9e2cea428cf7af93a84bcb865e478d8bf1c165] Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost git bisect good 6b9e2cea428cf7af93a84bcb865e478d8bf1c165 # good: [b5f185f33d0432cef6ff78765e033dfa8f4de068] Merge tag 'master-2014-12-08' of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next git bisect good b5f185f33d0432cef6ff78765e033dfa8f4de068 # good: [bae41e45b7400496b9bf0c70c6004419d9987819] Merge tag 'sound-3.19-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound git bisect good bae41e45b7400496b9bf0c70c6004419d9987819 # good: [c0222ac086669a631814bbf857f8c8023452a4d7] Merge branch 'upstream' of git://git.linux-mips.org/pub/scm/ralf/upstream-linus git bisect good c0222ac086669a631814bbf857f8c8023452a4d7 # bad: [a7cb7bb664543e4562ab0e9a072470d2d18c761f] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial git bisect bad a7cb7bb664543e4562ab0e9a072470d2d18c761f # bad: [9bfccec24e31f4f83445cfe0c1b0a5ef97900628] Merge tag 'ext4_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4 git bisect bad 9bfccec24e31f4f83445cfe0c1b0a5ef97900628 # bad: [4e8790f77f051d4cc745a57b48a73052521e8dfc] Merge branch 'for-3.19' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/libata git bisect bad 4e8790f77f051d4cc745a57b48a73052521e8dfc # skip: [b1df4a56bf4a61113e8928f932d346bed6eef553] xen/pciback: Restore configuration space when detaching from a guest. git bisect skip b1df4a56bf4a61113e8928f932d346bed6eef553 # bad: [9d050966e2eb37a643ac15904b6a8fda7fcfabe9] Merge tag 'stable/for-linus-3.19-rc0-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip git bisect bad 9d050966e2eb37a643ac15904b6a8fda7fcfabe9 # bad: [9490c6c67e2f41760de8ece4e4f56f75f84ceb9e] swiotlb-xen: call xen_dma_sync_single_for_device when appropriate git bisect bad 9490c6c67e2f41760de8ece4e4f56f75f84ceb9e # good: [a0f2dee0cd651efb5fac6a1d35b0a14460ebcdd4] xen: add a dma_addr_t dev_addr argument to xen_dma_map_page git bisect good a0f2dee0cd651efb5fac6a1d35b0a14460ebcdd4 # bad: [a4dba130891271084344c12537731542ec77cb85] xen/arm/arm64: introduce xen_arch_need_swiotlb git bisect bad a4dba130891271084344c12537731542ec77cb85 # bad: [3567258d281b5b515d5165ed23851d9f84087e7d] xen/arm: use hypercall to flush caches in map_page git bisect bad 3567258d281b5b515d5165ed23851d9f84087e7d # first bad commit: [3567258d281b5b515d5165ed23851d9f84087e7d] xen/arm: use hypercall to flush caches in map_page _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-02-05 14:51 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-01-26 13:11 [linux-4.1 test] 79008: regressions - FAIL osstest service owner 2016-01-27 11:18 ` Ian Campbell 2016-01-27 12:05 ` Ian Campbell 2016-02-02 14:10 ` Ian Campbell 2016-02-05 14:51 ` Ian Campbell
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).