Re: [xen-unstable test] 57852: regressions - FAIL

[Ian Campbell <ian.campbell@citrix.com>]

On Mon, 2015-06-08 at 09:07 +0100, Jan Beulich wrote:
> >>> On 05.06.15 at 12:48, <ian.campbell@citrix.com> wrote:
> > On Fri, 2015-06-05 at 10:18 +0100, Jan Beulich wrote:
> >> >>> On 05.06.15 at 11:07, <ian.campbell@citrix.com> wrote:
> >> > WRT the move to the colo, flights in 5xxxx are in the new one, while
> >> > 3xxxx are in the old one,
> >> > http://logs.test-lab.xenproject.org/osstest/results/history.test-amd64-amd64 
> >> > -xl-qemuu-win7-amd64.xen-unstable.html
> >> > shows that things seemed ok for 8 consecutive runs after the move
> >> > (ignoring blockages).
> >> 
> >> And when it went live, all systems being in use now got immediately
> >> deployed?
> > 
> > All the flights in the new colo seem to have been on fiano[01].
> 
> So are there just two hosts to run all x86 tests on? I thought one
> of the purposes of the switch was to have a wider pool of test
> systems...

There are about a dozen, but when a test is failing osstest will have a
preference for the host on which it failed last time (i.e. failures
become sticky to the host), in order to catch host specific failures I
think.

I think it was just coincidence that the first group of runs which
passed were on fiano0, although perhaps the pool was smaller then since
the colo was in the process of being commissioned.

The stickiness does make it a bit harder to know if a failure is host
specific though, since you often don't get results for other systems.

> > But having looked at the page again the early success was all on fiano0
> > while the later failures were all on fiano1.
> 
> But that's for the unstable install failures only as it looks. At the
> example of flight 57955 (testing 4.2) a local migration failure was
> observed on fiano0. Which would seem to support your earlier
> assumption that the install and migration issues are likely unrelated
> (yet their coincidence still strikes me as odd).

http://logs.test-lab.xenproject.org/osstest/results/history.test-amd64-amd64-xl-qemuu-win7-amd64.html has the cross branch history for this test case. With one exception (on chardonay0, in a linux-next test) all the fails were on fiano[01] and they were all on branches which would use xen-unstable as the Xen version (xen-unstable itself and linux-* + qemu-mainline which both use the current xen.git#master as their Xen).

I've got some adhoc results over the weekend, all can be found at
http://logs.test-lab.xenproject.org/osstest/logs/<NNNNN>/test-amd64-amd64-xl-qemuu-win7-amd64/info.html for flight <NNNNN>. All of them are using the binaries from 57852.

I messed up my first command line and ran them all on fiano0 by mistake,
so there are more results than I was planning for.

Flight	Host	Failed at	Install step duration
57940	fiano0	ts-guest-stop	1483
57945	fiano0	ts-guest-stop	1640
57953	fiano0	ts-guest-stop	1473
57958	fiano0	ts-guest-stop	1472
57962	fiano0	windows-install	7512
57973	fiano0	windows-install	7693
57080	fiano0	ts-guest-stop	1534
57986	fiano0	windows-install	7203
57933	fiano0	ts-guest-stop	1529
57997	fiano0	ts-guest-stop	1494
58004	fiano0	ts-guest-stop	1492

58011	fiano1	ts-guest-stop	1408
58012	fiano1	ts-guest-stop	1529
58017	fiano1	ts-guest-stop	1466
58023	fiano1	ts-guest-stop	1624
58028	fiano1	windows-install	7208
58038	fiano1	ts-guest-stop	1479
58043	fiano1	ts-guest-stop	1493

58053	fiano0	windows-install	7439
58062	fiano0	windows-install	1916
58063	fiano0	windows-install	1477

58067	fiano1	ts-guest-stop	1453
58071	fiano1	ts-guest-stop	1550
58077	fiano1	windows-install	7156

That's 6/14 (43%) failure rate on fiano0 and 2/10 (20%) on fiano1. Which
differs form the apparent xen-unstable failure rate. But I wouldn't take
this as evidence that the two systems differ significantly, despite how
the unstable results looked at first glance.

On successful install the test step takes 1450-1650s, with one outlier
at 1916. The failures take 7000-7500s (test case timeout is 7000, so
with slop that fits). So on success it takes <30mins and on fail it has
been given nearly 2hours.

Ian.