Re: [OSSTEST Nested PATCH v11 6/7] Compose the main recipe of nested test job

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-06-12 at 17:00 +0800, Robert Hu wrote:
> On Fri, 2015-06-12 at 09:44 +0100, Ian Campbell wrote:
> > On Fri, 2015-06-12 at 11:42 +0800, Robert Hu wrote:
> > > Hi Ian J., because nested Xen isn't that matured at present; it is
> > > currently 'tech preview' phase. We now just add some sanity test case to
> > > defend current work fruits. We can add more complicated test cases later
> > > as nested Xen development moves forward.
> > 
> > I don't think it needs doing as part of this series, but I do think it
> > would be worth adding the standard suite of tests steps to L2 guests
> > soon after.
> Agree. We can separately add those part in a later patch series.  
> > 
> > The way osstest deals with test failures i.e. classifying them into "has
> > always failed" vs. "regression" means that it is fine to add tests for
> > features which aren't mature yet, since they will fall into the "has
> > always failed" set and not block pushes. In fact it is in some sense
> > good to do this since it gives a concrete set of (necessary but not
> > necessarily sufficient) things which need to be fixed for the feature to
> > reach maturity.
> We would like this test job's failure to be marked 'regression', which
> shall block related breaking code's commitment.

This is all done automatically, by comparison with previous flights.

In essence if it passed last time then it will be considered a
regression if it fails and for as long as it keeps failing (unless it is
manually overridden via a forced push).

So there is nothing to be "marked".

> We can consider to contribute more code later, including proved/success
> test cases (which will fall into regression as this case), and fail test
> cases (which shall soon be fixed/developed).

You should just add the cases, await the first results and investigate
if it is not a pass as expected.

Ian.