Re: [OSSTEST PATCH 3/3] smoke tests: Consider osstest revision when reusing builds

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-10-09 at 12:48 +0100, Ian Jackson wrote:
> Build results done with one version of osstest are not necessarily
> reuseable with a different version of osstest.  For example, the suite
> may have changed.  The smoke tests try to reuse builds from
> xen-unstable but if osstest changes incompatibly, the smoke tests
> might repeatedly fail until a xen-unstable flight using the new
> osstest completes.
> 
> (This issue is a problem for bisection too but that's less critical
> and there is less of an easy answer.)

Probably a bad idea, but I wonder: would comparing the hostflags required
of the two build hosts take care of some of this?

e.g. some random job I just pulled up:

    share-build-wheezy-amd64,arch-amd64,suite-wheezy,purpose-build

This is a bit tenuous though, since really it is $r{$ident_suite} //
$c{DebianSuite} which matters.

> 
[...]
> 3. After a manual force push of an untested osstest, there no suitable

                                                            ^ are

> builds on either xen-unstable or osstest.  The first
> xen-unstable-smoke run will have to do all the builds.  However,
> subsequent xen-unstable-smoke runs can just pick up those builds.
> These same builds will be reused until a xen-unstable flight using the
> new osstest produces a passing build.

4. After a push from another tree whose gated output is used by xen
-unstable-smoke (e.g. the linux-X.Y for the default kernel revision) then
there will be no suitable builds in either the latest xen-unstable or
osstest (neither of which are likely to have seen the linux push and built
it before a smoke run occurs) in which case xen-unstable-smoke will do that
build and then subsequently reuse it until a xen-unstable or osstest flight
occurs which uses that Linux tree.

(is that worth mentioning? is it correct?)

> We honour an environemnt variable SMOKE_HARNESS_REV to override the

"environment"

> automatic determination of the desired test harness revision.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Despite the above:

Acked-by: Ian Campbell <ian.campbell@citrix.com>