Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree

[Ian Jackson <ian.jackson@eu.citrix.com>]

Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
> On 04/08/16 16:21, Ian Jackson wrote:
> > Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
> >> We should also clarify the force push criteria.  It is moderately likely
> >> that we get a fix or extension to an existing test which starts showing
> >> up a new bug in the code under test.
> > Can this not be made into a new test ?
> 
> That very much depends, and probably needs deciding on a case by case basis.
> 
> What absolutely shouldn't happen is ending up with test-$FOO,
> test-$FOO-2, test-more-$FOO, test-$FOO-again because that will result in
> the same logical test being split up in ad-hoc ways.  While XTF is
> useful for automation, it is first and foremost a tool for humans.

You mean that this would contort the code for the test ?

Here is another approach that could be used to satisfy osstest's
desire for stability in the meaning of test names: you could
explicitly copy the entire source of the test to a new test name, and
deprecate the old test.  Later, if the old versions of Xen are fixed,
or after they are retired you would delete the old test.

The result would be that the old, less-good, test would still exist
and still detect regressions.  The new, better test, would be seen to
have never passed on old branches.

Force pushing XTF is not an answer to this problem, for two reasons:

Firstly, if the improved test correctly fails on old versions of Xen,
this will appear as a regression in each old version of Xen.  Each old
Xen tree would need to be force pushed, every time this happened.

Secondly, in any case, force pushing XTF might not be necessary.  In
the usual case, one would expect xen-unstable (which is what would
drive the XTF push gate) to be fixed so that the improved test
passes.  So osstest wouldn't spot the issue until it ran the new XTF
with old Xen.

Another possibility would be to have the capability for an single XTF
test execution to return multiple statuses.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel