Re: [linux-4.1 test] 79008: regressions - FAIL

[Ian Campbell <ian.campbell@citrix.com>]

On Tue, 2016-01-26 at 13:11 +0000, osstest service owner wrote:
> flight 79008 linux-4.1 real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/79008/
> 
> Regressions :-(
> 
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  test-armhf-armhf-xl-credit2  15 guest-start/debian.repeat fail REGR. vs. 66399
>  test-armhf-armhf-xl-xsm      15 guest-start/debian.repeat fail REGR. vs. 66399

These were both:

2016-01-26 01:20:33 Z executing ssh ... root@172.16.147.101 echo guest debian.guest.osstest: ok 
Warning: Permanently added '172.16.147.101' (ECDSA) to the list of known hosts.
key_verify failed for server_host_key

Searching for that message online shows a variety of results for the last
15 years or so, although it's not clear that any of them are actually the
same, many of them also involve a "hash mismatch" message, which is not
present here.

As far as our own tests go:

$ echo ~osstest/pub/logs/*/*/[0-9]*.log | xargs grep -l "key_verify failed for server_host_key"
/home/osstest/pub/logs/78033/test-armhf-armhf-xl-cubietruck/15.ts-repeat-test.log

/home/osstest/pub/logs/78925/test-armhf-armhf-xl-credit2/11.ts-guest-start.log
/home/osstest/pub/logs/78925/test-armhf-armhf-xl-cubietruck/15.ts-repeat-test.log
/home/osstest/pub/logs/78925/test-armhf-armhf-xl-xsm/15.ts-repeat-test.log

/home/osstest/pub/logs/79008/test-armhf-armhf-xl-credit2/15.ts-repeat-test.log
/home/osstest/pub/logs/79008/test-armhf-armhf-xl-cubietruck/11.ts-guest-start.log
/home/osstest/pub/logs/79008/test-armhf-armhf-xl-xsm/15.ts-repeat-test.log

/home/osstest/pub/logs/79088/test-armhf-armhf-xl-credit2/15.ts-repeat-test.log

/home/osstest/pub/logs/79090/test-armhf-armhf-xl-credit2/11.ts-guest-start.log
/home/osstest/pub/logs/79090/test-armhf-armhf-xl-cubietruck/11.ts-guest-start.log
/home/osstest/pub/logs/79090/test-armhf-armhf-xl-xsm/11.ts-guest-start.log

/home/osstest/pub/logs/79120/test-armhf-armhf-xl-xsm/15.ts-repeat-test.log

From flights (machines in same order as tests above):

[linux-linus bisection] 78033: tested test-armhf-armhf-xl-cubietruck
    cubietruck-braque
[linux-4.1 test] 78925: regressions - trouble: broken/fail/pass
    cubietruck-metzinger
    cubietruck-gleizes
    cubietruck-gleizes
[linux-4.1 test] 79008: regressions - FAIL
    cubietruck-metzinger
[linux-4.1 bisection] 79088: testing test-armhf-armhf-xl-credit2
    cubietruck-metzinger
79090 is still in progress (nearly complete) but is on linux-4.1
    cubietruck-metzinger
    cubietruck-gleizes
    cubietruck-gleizes
[linux-4.1 bisection] 79120: testing test-armhf-armhf-xl-xsm
    cubietruck-gleizes

Neither of the bisects look likely to make any progress:
http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-4.1/test-armhf-armhf-xl-xsm.guest-start--debian.repeat.html
http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-4.1/test-armhf-armhf-xl-credit2.guest-start--debian.repeat.html

We only have logs back as far as 20 Jan, so it's possible there isn't
enough data here to decide anything, but the span does include some logs
from linux-3.18 which seems to not be showing this. OTOH linux-linus has
only hit it once, maybe this was fixed already upstream.

I suspect linux-4.1 has a cubietruck specific hardware issue (perhaps in
the network driver). I'm not sure how best to approach it though,
especially given that it doesn't seem to be reproducible by the bisector.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel