[OSSTEST PATCH 1/3] Move get_harness_rev to Osstest from Osstest::Executive.

[OSSTEST PATCH 1/3] Move get_harness_rev to Osstest from Osstest::Executive.

[Ian Jackson]

No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest.pm           |   17 +++++++++++++++++
 Osstest/Executive.pm |   18 +-----------------
 2 files changed, 18 insertions(+), 17 deletions(-)

diff --git a/Osstest.pm b/Osstest.pm
index 4a763c6..969a2d0 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -38,6 +38,7 @@ BEGIN {
                       main_revision_job_cond other_revision_job_suffix
                       $dbh_tests db_retry db_retry_retry db_retry_abort
                       db_begin_work db_prepare
+                      get_harness_rev
                       ensuredir get_filecontents_core_quiet system_checked
                       nonempty visible_undef show_abs_time
                       %arch_debian2xen %arch_xen2debian $cfgvar_re
@@ -333,6 +334,22 @@ sub main_revision_job_cond ($) {
     return "(${\ other_revision_job_suffix($jobfield,'x') } = '')";
 }
 
+sub get_harness_rev () {
+    $!=0; $?=0;  my $rev= `git rev-parse HEAD^0`;
+    die "$? $!" unless defined $rev;
+
+    $rev =~ s/\n$//;
+    die "$rev ?" unless $rev =~ m/^[0-9a-f]+$/;
+
+    my $diffr= system 'git diff --exit-code HEAD >/dev/null';
+    if ($diffr) {
+        die "$diffr $! ?" if $diffr != 256;
+        $rev .= '+';
+    }
+
+    return $rev;
+}
+
 sub get_filecontents_core_quiet ($) { # ENOENT => undef
     my ($path) = @_;
     if (!open GFC, '<', $path) {
diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index bb2663a..01915ac 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -44,7 +44,7 @@ BEGIN {
     our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
     $VERSION     = 1.00;
     @ISA         = qw(Exporter);
-    @EXPORT      = qw(get_harness_rev grabrepolock_reexec
+    @EXPORT      = qw(grabrepolock_reexec
                       findtask @all_lock_tables
                       restrictflight_arg restrictflight_cond
                       report_run_getinfo report_altcolour
@@ -128,22 +128,6 @@ sub grabrepolock_reexec {
     }
 }
 
-sub get_harness_rev () {
-    $!=0; $?=0;  my $rev= `git rev-parse HEAD^0`;
-    die "$? $!" unless defined $rev;
-
-    $rev =~ s/\n$//;
-    die "$rev ?" unless $rev =~ m/^[0-9a-f]+$/;
-
-    my $diffr= system 'git diff --exit-code HEAD >/dev/null';
-    if ($diffr) {
-        die "$diffr $! ?" if $diffr != 256;
-        $rev .= '+';
-    }
-
-    return $rev;
-}
-
 #---------- database access ----------#
 
 sub opendb_state () {
-- 
1.7.10.4

[OSSTEST PATCH 2/3] sg-check-tested: Honour multiple branches (comma-separated)

[Ian Jackson]

No functional change with existing invocations.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 sg-check-tested |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/sg-check-tested b/sg-check-tested
index 1a3afa3..43f2854 100755
--- a/sg-check-tested
+++ b/sg-check-tested
@@ -80,8 +80,9 @@ END
                AND   r.flight = flights.flight)
 END
     } elsif (m/^--branch=(.*)$/) {
-        push @conds_vars, $1;
-        push @conds, "branch = ?";
+        my @branches = split /\,/, $1;
+        push @conds_vars, @branches;
+        push @conds, "(".(join " OR ", map { "branch = ?" } @branches).")";
     } elsif (m/^--blessings=(.*)$/) {
         my @blessings= split /\,/, $1;
         push @conds_vars, @blessings;
-- 
1.7.10.4

[OSSTEST PATCH 3/3] smoke tests: Consider osstest revision when reusing builds

[Ian Jackson]

Build results done with one version of osstest are not necessarily
reuseable with a different version of osstest.  For example, the suite
may have changed.  The smoke tests try to reuse builds from
xen-unstable but if osstest changes incompatibly, the smoke tests
might repeatedly fail until a xen-unstable flight using the new
osstest completes.

(This issue is a problem for bisection too but that's less critical
and there is less of an easy answer.)

Address this by also considering the osstest revision when searching
for builds to reuse, so the smoke tests only reuse builds made with
the same version of osstest.  (If we are running with uncommitted
changes, we ignore that aspect and instead insist on only builds which
used the committed revision of osstest: this makes testing the
xen-unstable-smoke machinery marginally easier, and will make no
difference to production runs.)

This introduces a new problem, though: after an osstest push, there
will be no suitable builds until the next xen-unstable flight passes.
So each smoke test would run its own build.  This would delay the
smoke tests, and waste capacity.

To address this we permit the smoke tests to reuse (i) builds from a
suitable osstest flight (hopefully there will be an osstest flight
which justified the push of the osstest version we are running) or
(ii) previous builds done by a xen-unstable-smoke test (this is a
useful backstop).

sg-check-tested always reports the highest-numbered flight which
matches all the specified conditions, so overall this means that:

1. Normally, the most recent relevant build for each job will be a
xen-unstable build; xen-unstable-smoke will reuse those.  Recent
flights on the osstest branch will be unsuitable because they use
different osstest; and there will be no recent relevant builds on
xen-unstable-smoke because xen-unstable-smoke will prefer to reuse its
own old builds rather than make new ones.

2. After a normal osstest push (whether force-pushed manually on the
basis of a test flight, or automatically pushed), the xen-unstable
builds are unsuitable.  However, the osstest push _is_ suitable, and
its builds will be used.

3. After a manual force push of an untested osstest, there no suitable
builds on either xen-unstable or osstest.  The first
xen-unstable-smoke run will have to do all the builds.  However,
subsequent xen-unstable-smoke runs can just pick up those builds.
These same builds will be reused until a xen-unstable flight using the
new osstest produces a passing build.

(If the force pushed osstest causes a build to break, then
xen-unstable-smoke will keep retrying and failing that individual
build until a fix is pushed to osstest#production or xen#staging.)

We honour an environemnt variable SMOKE_HARNESS_REV to override the
automatic determination of the desired test harness revision.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cr-daily-branch |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/cr-daily-branch b/cr-daily-branch
index c7cc33b..364238c 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -281,9 +281,14 @@ flight=`$makeflight $branch $xenbranch $OSSTEST_BLESSING "$@"`
 
 case $branch in
 xen-unstable-smoke)
+	harness_rev=$(perl -e 'use Osstest; print get_harness_rev();')
+	: ${SMOKE_HARNESS_REV:=${harness_rev%+}}
+
 	./mg-adjust-flight-makexrefs -v $flight \
 		'!build-amd64 !build-amd64-libvirt !build-armhf build-*' \
-		--debug --branch=xen-unstable --blessings=real
+		--debug --blessings=real				\
+		--branch=xen-unstable,xen-unstable-smoke,osstest	\
+		--revision-osstest=$SMOKE_HARNESS_REV
 	# Even adhoc or play flights ought to reuse only real
 	# previous builds.
 	;;
-- 
1.7.10.4

Re: [OSSTEST PATCH 3/3] smoke tests: Consider osstest revision when reusing builds

[Ian Campbell]

On Fri, 2015-10-09 at 12:48 +0100, Ian Jackson wrote:
> Build results done with one version of osstest are not necessarily
> reuseable with a different version of osstest.  For example, the suite
> may have changed.  The smoke tests try to reuse builds from
> xen-unstable but if osstest changes incompatibly, the smoke tests
> might repeatedly fail until a xen-unstable flight using the new
> osstest completes.
> 
> (This issue is a problem for bisection too but that's less critical
> and there is less of an easy answer.)

Probably a bad idea, but I wonder: would comparing the hostflags required
of the two build hosts take care of some of this?

e.g. some random job I just pulled up:

    share-build-wheezy-amd64,arch-amd64,suite-wheezy,purpose-build

This is a bit tenuous though, since really it is $r{$ident_suite} //
$c{DebianSuite} which matters.

> 
[...]
> 3. After a manual force push of an untested osstest, there no suitable

                                                            ^ are

> builds on either xen-unstable or osstest.  The first
> xen-unstable-smoke run will have to do all the builds.  However,
> subsequent xen-unstable-smoke runs can just pick up those builds.
> These same builds will be reused until a xen-unstable flight using the
> new osstest produces a passing build.

4. After a push from another tree whose gated output is used by xen
-unstable-smoke (e.g. the linux-X.Y for the default kernel revision) then
there will be no suitable builds in either the latest xen-unstable or
osstest (neither of which are likely to have seen the linux push and built
it before a smoke run occurs) in which case xen-unstable-smoke will do that
build and then subsequently reuse it until a xen-unstable or osstest flight
occurs which uses that Linux tree.

(is that worth mentioning? is it correct?)

> We honour an environemnt variable SMOKE_HARNESS_REV to override the

"environment"

> automatic determination of the desired test harness revision.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Despite the above:

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 1/3] Move get_harness_rev to Osstest from Osstest::Executive.

[Ian Campbell]

On Fri, 2015-10-09 at 12:48 +0100, Ian Jackson wrote:
> No functional change.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 2/3] sg-check-tested: Honour multiple branches (comma-separated)

[Ian Campbell]

On Fri, 2015-10-09 at 12:48 +0100, Ian Jackson wrote:
> No functional change with existing invocations.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 3/3] smoke tests: Consider osstest revision when reusing builds

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 3/3] smoke tests: Consider osstest revision when reusing builds"):
> Probably a bad idea, but I wonder: would comparing the hostflags required
> of the two build hosts take care of some of this?
> 
> e.g. some random job I just pulled up:
> 
>     share-build-wheezy-amd64,arch-amd64,suite-wheezy,purpose-build
> 
> This is a bit tenuous though, since really it is $r{$ident_suite} //
> $c{DebianSuite} which matters.

Also I don't think grobbling around in the runvars looking at
all_hostflags etc. is right.

> [...]
> > 3. After a manual force push of an untested osstest, there no suitable

Fixed (and the other typo).

> > builds on either xen-unstable or osstest.  The first
> > xen-unstable-smoke run will have to do all the builds.  However,
> > subsequent xen-unstable-smoke runs can just pick up those builds.
> > These same builds will be reused until a xen-unstable flight using the
> > new osstest produces a passing build.
> 
> 4. After a push from another tree whose gated output is used by xen
> -unstable-smoke (e.g. the linux-X.Y for the default kernel revision) then
> there will be no suitable builds in either the latest xen-unstable or
> osstest (neither of which are likely to have seen the linux push and built
> it before a smoke run occurs) in which case xen-unstable-smoke will do that
> build and then subsequently reuse it until a xen-unstable or osstest flight
> occurs which uses that Linux tree.
> 
> (is that worth mentioning? is it correct?)

No.  This reuse machinery does not consider the versions of anything -
except, now, osstest itself.  This is because the other trees'
versions are supposed to be intercompatible.

> Acked-by: Ian Campbell <ian.campbell@citrix.com>

Thanks,
Ian.

Re: [OSSTEST PATCH 3/3] smoke tests: Consider osstest revision when reusing builds

[Ian Campbell]

On Fri, 2015-10-09 at 14:22 +0100, Ian Jackson wrote:
> > (is that worth mentioning? is it correct?)
> 
> No.  This reuse machinery does not consider the versions of anything -
> except, now, osstest itself.  This is because the other trees'
> versions are supposed to be intercompatible.

Ah, I'm always forgetting that. Sorry for the noise.

Ian.