Re: [xen-4.5-testing test] 56898: regressions - FAIL

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-05-22 at 08:11 +0100, Jan Beulich wrote:
> >>> On 21.05.15 at 21:30, <osstest@xenbits.xen.org> wrote:
> > flight 56898 xen-4.5-testing real [real]
> > http://logs.test-lab.xenproject.org/osstest/logs/56898/ 
> > 
> > Regressions :-(
> > 
> > Tests which did not succeed and are blocking,
> > including tests which could not be run:
> >  test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop         fail REGR. vs. 56728
> 
> This is recurring (i.e. presumably real), but none of the few changes
> under test appear to be related in any way. And going through the
> logs I can't spot anything suspicious either. Does anyone else have
> a clue?

For a long time this test was marked "never passed". However I recently
added osstest support for using the ACPI shutdown method against certain
guests when configured to do so and configured win7.

However it is starting to look like the ACPI shutdown method is
unreliable, since this now seems to be failing intermittently. I haven't
had a chance to analyse it yet, but it seems like it might also be
specific to Windows 7.

In this case, as with the other one I looked at earlier in the week, the
guest vnc screenshot shows no sign that it is considering shutting down.

> >  test-amd64-amd64-rumpuserxen-amd64 15 
> > rumpuserxen-demo-xenstorels/xenstorels.repeat fail REGR. vs. 56728
> 
> I suppose this (also recurring, and also seemingly unrelated to
> any of the commits under test)

Correct, this is a long standing heisenbug. I recently increased the
number of iterations used in this test to (hopefully) reduce the
incidences of false passes.

I would force push any flight which failed only this case.

> (As a side note - can anyone explain why
> elbling0---var-log-xen-console-guest-xenstorels.log.gz is gzip-ed
> twice?)

osstest zips logs over some size threshold. I'm not sure what would be
responsible for the other one -- perhaps either logrotate on the
original host or something in the webserver giving out the logs?

> > Tests which are failing intermittently (not blocking):
> >  test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stop          fail pass in 56821
> > 
> > Regressions which are regarded as allowable (not blocking):
> >  test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop              fail like 56728
> 
> It is interesting to see that these seem to intermittently fail, so
> perhaps the problem above is an older one...

I think so (and more evidence that it is win7 only).