Re: [xen-4.4-testing test] 30367: regressions - FAIL

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 24/09/14 10:25, Ian Campbell wrote:
> On Wed, 2014-09-24 at 10:21 +0100, Andrew Cooper wrote:
>> On 24/09/14 08:20, xen.org wrote:
>>> flight 30367 xen-4.4-testing real [real]
>>> http://www.chiark.greenend.org.uk/~xensrcts/logs/30367/
>>>
>>> Regressions :-(
>>>
>>> Tests which did not succeed and are blocking,
>>> including tests which could not be run:
>>>  test-amd64-amd64-xl-win7-amd64  7 windows-install         fail REGR. vs. 30239
>> Can we just nuke this test?  It has never worked reliably, and now only
>> serves to delay movement from staging-4.4 to stable-4.4
> Surely then we should fix the issue

Yes absolutely, but this is *never* done OSSTest.  This is largely as a
result of the absurd notion of "tests which are expected to fail, so
lets exit 1 in the hope someone will implement a test in the future"
which a) waste time, and b) actively bad, as serves to train people to
ignore failures.

These tests should be a single pass/fail with no intermediate delineation.

Any failure is then either
  a) a bad test (delete/fix test)
  b) an infrastructure issues (new/fix infrastructure)
  c) a genuine regression between staging and master

At this point, the expected result from OSSTest is PASS, and any
failures should be investigated ASAP.  This is the whole point of a
smoketest.

At the moment, I see "[Xen-devel] [xen-4.4-testing test] $FLIGHT:
regressions - FAIL" and think to myself "Ah - that will be the
intermittent test-amd64-amd64-xl-win7-amd64 again, wont it?".  And
clearly, this is a similar thought as everyone else on xen-devel by
virtue of the the issue having never even been investigated.


We had a similar issue in XenRT.  Over time, there were an increasing
number of tests which were intermittently failing, which caused
uncertainty at whether a new change had caused a regression or not. 
About a year ago, the intermittent tests were disabled and put to one
side.  This immediately upped the confidence in the reliable tests
making it much clearer whether regressions had occurred.  Over time, the
tests were reviewed by QA and either declared good (i.e. showed real
intermittent product problems), fixed up (so as to be reliable again),
or discarded due to serving no useful purpose.

The overall result is that we still have almost all the same tests as
previously, but a far higher confidence in the result, and well as it
being much more obvious when issues have occurred.

>  (be it a xen.git or an osstest.git
> thing), not sweep the issue under the carpet.
>
> Ian.
>

I can state with absolute certainty that Windows 7 is not broken under
XenServer, which is currently Xen 4.4.1, and with almost complete
certainly that we don't have a patch in our patch queue which would be
relevant.

That would make this issue either an xl, libxl (or possibly
qemu-upstream if I could work out which qemu is actually in use). 
However, the best I can work from the OSSTest logs is that it timed out
attempting to boot.

~Andrew