Re: [xen-unstable test] 57852: regressions - FAIL

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-06-05 at 10:00 +0100, Jan Beulich wrote:
> >>> On 05.06.15 at 10:45, <ian.campbell@citrix.com> wrote:
> > On Thu, 2015-06-04 at 12:01 +0000, osstest service user wrote:
> >> flight 57852 xen-unstable real [real]
> >> http://logs.test-lab.xenproject.org/osstest/logs/57852/ 
> >> 
> >> Regressions :-(
> >> 
> >> Tests which did not succeed and are blocking,
> >> including tests which could not be run:
> >>  test-amd64-amd64-xl-qemuu-win7-amd64  9 windows-install   fail REGR. vs. 57419
> > 
> > Is anyone looking into this?
> 
> Not actively, to be honest.
> 
> > It seems to have been intermittent for a long time but the probability
> > of failure seems to have increased significantly some time around flight
> > 52633 (see [0]). Before that it failed <5% of the time and since then it
> > looks to be closer to 45-50%. 5% could be put down to infrastructure or
> > guest flakiness, 50% seems more like something on the Xen (or qemu etc)
> > side.
> > 
> > The bisector is taking a look[1] but TBH given a 50% pass rate I think
> > it is unlikely to get anywhere (I suspect this isn't its first attempt
> > at this either, pretty sure I saw a failed attempt on an earlier range).
> > 
> > Taking 50370 as a rough baseline (4 consecutive passes before the first
> > of the more frequent failures) gives a range of
> > b6e7fbadbda4..5c44b5cf352e which is quite a few. It's noteworthy though
> > that qemuu didn't change during the interval 50370..52633 (again, from
> > [0]).
> > 
> > None of the vnc snapshots look interesting, just the windows login
> > screen. Neither do any of the logs look interesting.
> 
> Which is the main reason for it being difficult to look into without
> seeing it oneself. Two things are possibly noteworthy: This again
> is an issue only ever seen with qemuu (just like the migration issue
> on the stable branches), and the other day there was a report of
> posted interrupts causing spurious hangs, which raises the question
> whether the increased failure rate was perhaps due to the new
> osstest host system pool having got extended at around that time.
> (As noted in a reply to that report, this possible issue can't be an
> explanation for the issue on the stable trees, as 4.3 doesn't support
> posted interrupts yet.)

From
http://logs.test-lab.xenproject.org/osstest/results/history.test-amd64-amd64-xl-qemuu-win7-amd64.xen-4.3-testing.html
it doesn't seem like 4.3-testing is suffering from the higher incidence
of windows-install failures, just the background noise which unstable
had prior to 52633.

From:
http://logs.test-lab.xenproject.org/osstest/results/history.test-amd64-amd64-xl-qemuu-win7-amd64.xen-4.4-testing.html
http://logs.test-lab.xenproject.org/osstest/results/history.test-amd64-amd64-xl-qemuu-win7-amd64.xen-4.5-testing.html
it looks like none of the stable branches suffer from the install issue.
I'd be inclined to discount any possible link with the migration issue
based on that.

WRT the move to the colo, flights in 5xxxx are in the new one, while
3xxxx are in the old one,
http://logs.test-lab.xenproject.org/osstest/results/history.test-amd64-amd64-xl-qemuu-win7-amd64.xen-unstable.html
shows that things seemed ok for 8 consecutive runs after the move
(ignoring blockages).

Ian.