Re: [xen-unstable test] 59040: regressions - FAIL

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-07-03 at 21:59 +0000, osstest service owner wrote:
> flight 59040 xen-unstable real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/59040/
> 
> Regressions :-(



> 
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm 14 guest-localmigrate.2 fail REGR. vs. 58958

http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm/xen-unstable.html

This one seems to have fallen into the merlot trap.

>  test-armhf-armhf-xl-credit2  15 guest-start/debian.repeat fail REGR. vs. 58965

From
http://logs.test-lab.xenproject.org/osstest/results/history/test-armhf-armhf-xl-credit2/xen-unstable.html it appears this one is sporadically unreliable on cubietruck, but works just fine on arndale.

Comparing to
http://logs.test-lab.xenproject.org/osstest/results/history/test-armhf-armhf-xl/xen-unstable.html 
it looks to be credit2 specific.

d8v0 has a PC at 0xffff000c, which will be the entry point for the
prefetch abort handler. ABT_LR shows it came from 0xffff0010 which is
the data abt handler, which uses the same banked LR as prefetch abt so
we don't know where it came from.

SVR_LR shows that the last function call done in kernel mode was a call
to fdt_check_header from early_init_dt_verify, but that might have been
ages ago.

Guest logs ends with it shutting down, but I think that is the previous
test step completing.

>  test-amd64-amd64-xl-qemuu-debianhvm-amd64 17 guest-start.2 fail REGR. vs. 58965

http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-qemuu-debianhvm-amd64/xen-unstable.html

2015-07-03 15:40:16 Z guest debianhvm.guest.osstest 5a:36:0e:a0:00:1e 22 link/ip/tcp: ping gave (256): PING 172.16.146.71 (172.16.146.71) 56(84) bytes of data. | From 172.16.144.3 icmp_seq=1 Destination Host Unreachable | From 172.16.144.3 icmp_seq=2 Destination Host Unreachable | From 172.16.144.3 icmp_seq=3 Destination Host Unreachable | From 172.16.144.3 icmp_seq=4 Destination Host Unreachable | From 172.16.144.3 icmp_seq=5 Destination Host Unreachable |  | --- 172.16.146.71 ping statistics --- | 5 packets transmitted, 0 received, +5 errors, 100% packet loss, time XXXms | pipe 3 |  (waiting) ...
2015-07-03 15:40:27 Z guest debianhvm.guest.osstest 5a:36:0e:a0:00:1e 22 link/ip/tcp: ok. (31s)
2015-07-03 15:40:27 Z executing ssh ... root@172.16.146.71 echo guest debianhvm.guest.osstest: ok
ssh: connect to host 172.16.146.71 port 22: Connection refused

Ian, I believe you observed some other instances of this in triaging
another failure last week.

http://logs.test-lab.xenproject.org/osstest/logs/59040/test-amd64-amd64-xl-qemuu-debianhvm-amd64/fiano0---var-log-xen-qemu-dm-debianhvm.guest.osstest.log

is pretty uninteresting, we should perhaps arrange to remove some
"quiet"'s from guest commands lines somewhere, or to frob the guest
console to include ttyS0, which it isn't clear it does.

vnc snapshot just shows the login prompt.

The bisector looks to be working on this one
http://logs.test-lab.xenproject.org/osstest/results/bisect/xen-unstable/test-amd64-amd64-xl-qemuu-debianhvm-amd64.guest-start.2.html
but I don't think it is going to conclude anything, since it looks intermittent.

>  test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm 9 debian-hvm-install fail REGR. vs. 58965

http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm/xen-unstable.html

2015-07-03 15:39:29 Z executing ssh ... root@172.16.144.41 xl list
2015-07-03 15:39:30 Z guest debianhvm.guest.osstest not present on this host
2015-07-03 15:39:30 Z FAILURE: guest unexpectedly shutdown; state is ''
failure: guest unexpectedly shutdown; state is ''

The xl log just says:

Waiting for domain debianhvm.guest.osstest (domid 1) to die [pid 4412]
Domain 1 has shut down, reason code 0 0x0
Action for shutdown reason code 0 is destroy
Domain 1 needs to be cleaned up: destroying the domain
Done. Exiting now

Ian.