[OSSTEST PATCH 00/27] xen.git#staging smoke tests

[OSSTEST PATCH 00/27] xen.git#staging smoke tests

[Ian Jackson]

This is (finally!) my series to provide fast smoke tests of xen-unstable.

The first two thirds of the series shaves a number of yaks I
encountered on the way.  Many of these (particularly the ones up to
patch 11 or so) are generally useful and could perhaps go in before
the rest.

One outstanding bikeshed question about naming: I have called the job
`xen-unstable-smoke' and the output ref xen.git#smoked.  I tripped
over the presence or absence of the `d' a few times.  Should one of
these two be renamed, and if so which ?

I have verified that this series from
 "standalone-generate-dump-flight-runvars: memoise ap-fetch"
onwards generates no unexpected changes to any runvars.  The
diff -bu (from there to tip) in the output from
  OSSTEST_CONFIG=standalone-config-example \
  ./standalone-generate-dump-flight-runvars
is included below.  At the time of running this, xen.git had
  340ade2f0ef692efce01a708ad98629af985f978 refs/heads/smoked
  26646f34c31f663fa23076ed0433195edec50d9f refs/heads/staging
and the only change to the existing flights' runvars is that
xen-unstable's non-prev revision_xen values referring to staging now
refer to smoked instead.

--- ../play-testing.git/before	2015-09-16 14:07:53.493236378 +0100
+++ after	2015-09-16 14:08:05.577518049 +0100
@@ -23100,16 +23100,16 @@
 xen-unstable               build-armhf-xsm                                       revision_seabios                                                                          
 xen-unstable               build-i386                                            revision_seabios                                                                          
 xen-unstable               build-i386-xsm                                        revision_seabios                                                                          
-xen-unstable               build-amd64                                           revision_xen                26646f34c31f663fa23076ed0433195edec50d9f                      
+xen-unstable               build-amd64                                           revision_xen                 340ade2f0ef692efce01a708ad98629af985f978                      
 xen-unstable               build-amd64-prev                                      revision_xen                a7b39c8bd6cba3fe1c8012987b9e28bdbac7e92d                      
-xen-unstable               build-amd64-rumpuserxen                               revision_xen                26646f34c31f663fa23076ed0433195edec50d9f                      
-xen-unstable               build-amd64-xsm                                       revision_xen                26646f34c31f663fa23076ed0433195edec50d9f                      
-xen-unstable               build-armhf                                           revision_xen                26646f34c31f663fa23076ed0433195edec50d9f                      
-xen-unstable               build-armhf-xsm                                       revision_xen                26646f34c31f663fa23076ed0433195edec50d9f                      
-xen-unstable               build-i386                                            revision_xen                26646f34c31f663fa23076ed0433195edec50d9f                      
+xen-unstable               build-amd64-rumpuserxen                               revision_xen                 340ade2f0ef692efce01a708ad98629af985f978                      
+xen-unstable               build-amd64-xsm                                       revision_xen                 340ade2f0ef692efce01a708ad98629af985f978                      
+xen-unstable               build-armhf                                           revision_xen                 340ade2f0ef692efce01a708ad98629af985f978                      
+xen-unstable               build-armhf-xsm                                       revision_xen                 340ade2f0ef692efce01a708ad98629af985f978                      
+xen-unstable               build-i386                                            revision_xen                 340ade2f0ef692efce01a708ad98629af985f978                      
 xen-unstable               build-i386-prev                                       revision_xen                a7b39c8bd6cba3fe1c8012987b9e28bdbac7e92d                      
-xen-unstable               build-i386-rumpuserxen                                revision_xen                26646f34c31f663fa23076ed0433195edec50d9f                      
-xen-unstable               build-i386-xsm                                        revision_xen                26646f34c31f663fa23076ed0433195edec50d9f                      
+xen-unstable               build-i386-rumpuserxen                                revision_xen                 340ade2f0ef692efce01a708ad98629af985f978                      
+xen-unstable               build-i386-xsm                                        revision_xen                 340ade2f0ef692efce01a708ad98629af985f978                      
 xen-unstable               test-amd64-amd64-rumpuserxen-amd64                    rump_builtimage             rumpuserxen:/usr/local/lib/xen/rump-kernel/rump-kernel        
 xen-unstable               test-amd64-i386-rumpuserxen-i386                      rump_builtimage             rumpuserxen:/usr/local/lib/xen/rump-kernel/rump-kernel        
 xen-unstable               test-amd64-amd64-rumpuserxen-amd64                    rump_cmdline                3                                                             
@@ -23365,3 +23365,77 @@
 xen-unstable               test-amd64-i386-rumpuserxen-i386                      xenstorels_builtimage       :/usr/local/bin/xenstore                                      
 xen-unstable               test-amd64-amd64-rumpuserxen-amd64                    xenstorels_cmdline          ls -fp device                                                 
 xen-unstable               test-amd64-i386-rumpuserxen-i386                      xenstorels_cmdline          ls -fp device                                                 
+xen-unstable-smoke         test-amd64-amd64-libvirt                 all_hostflags                 arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test           
+xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 all_hostflags                 arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test,hvm       
+xen-unstable-smoke         test-armhf-armhf-xl                      all_hostflags                 arch-armhf,arch-xen-armhf,suite-wheezy,purpose-test           
+xen-unstable-smoke         build-amd64                              arch                          amd64                                                         
+xen-unstable-smoke         build-armhf                              arch                          armhf                                                         
+xen-unstable-smoke         test-amd64-amd64-libvirt                 arch                          amd64                                                         
+xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 arch                          amd64                                                         
+xen-unstable-smoke         test-armhf-armhf-xl                      arch                          armhf                                                         
+xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 bios                          seabios                                                       
+xen-unstable-smoke         build-amd64                              build_lvextend_max            50                                                            
+xen-unstable-smoke         build-armhf                              build_lvextend_max            50                                                            
+xen-unstable-smoke         test-amd64-amd64-libvirt                 buildjob                      build-amd64                                                   
+xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 buildjob                      build-amd64                                                   
+xen-unstable-smoke         test-armhf-armhf-xl                      buildjob                      build-armhf                                                   
+xen-unstable-smoke         test-amd64-amd64-libvirt                 debian_arch                   amd64                                                         
+xen-unstable-smoke         test-armhf-armhf-xl                      debian_arch                   armhf                                                         
+xen-unstable-smoke         test-amd64-amd64-libvirt                 debian_kernkind               pvops                                                         
+xen-unstable-smoke         test-armhf-armhf-xl                      debian_kernkind               pvops                                                         
+xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 debianhvm_image               debian-7.2.0-i386-CD-1.iso                                    
+xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 device_model_version          qemu-xen                                                      
+xen-unstable-smoke         build-amd64                              enable_ovmf                   true                                                          
+xen-unstable-smoke         build-armhf                              enable_ovmf                   true                                                          
+xen-unstable-smoke         build-amd64                              enable_xend                   false                                                         
+xen-unstable-smoke         build-armhf                              enable_xend                   false                                                         
+xen-unstable-smoke         build-amd64                              enable_xsm                    false                                                         
+xen-unstable-smoke         build-armhf                              enable_xsm                    false                                                         
+xen-unstable-smoke         test-amd64-amd64-libvirt                 enable_xsm                    false                                                         
+xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 enable_xsm                    false                                                         
+xen-unstable-smoke         test-armhf-armhf-xl                      enable_xsm                    false                                                         
+xen-unstable-smoke         build-amd64                              host_hostflags                share-build-wheezy-amd64,arch-amd64,suite-wheezy,purpose-build
+xen-unstable-smoke         build-armhf                              host_hostflags                share-build-wheezy-armhf,arch-armhf,suite-wheezy,purpose-build
+xen-unstable-smoke         test-amd64-amd64-libvirt                 kernbuildjob                  build-amd64-pvops                                             
+xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 kernbuildjob                  build-amd64-pvops                                             
+xen-unstable-smoke         test-armhf-armhf-xl                      kernbuildjob                  build-armhf-pvops                                             
+xen-unstable-smoke         test-amd64-amd64-libvirt                 kernkind                      pvops                                                         
+xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 kernkind                      pvops                                                         
+xen-unstable-smoke         test-armhf-armhf-xl                      kernkind                      pvops                                                         
+xen-unstable-smoke         test-amd64-amd64-libvirt                 libvirtbuildjob               build-amd64-libvirt                                           
+xen-unstable-smoke         build-amd64                              revision_ovmf                                                                               
+xen-unstable-smoke         build-armhf                              revision_ovmf                                                                               
+xen-unstable-smoke         build-amd64                              revision_qemu                                                                               
+xen-unstable-smoke         build-armhf                              revision_qemu                                                                               
+xen-unstable-smoke         build-amd64                              revision_qemuu                b05befcbea71a979509ce04f02929969a790c923                      
+xen-unstable-smoke         build-armhf                              revision_qemuu                b05befcbea71a979509ce04f02929969a790c923                      
+xen-unstable-smoke         build-amd64                              revision_seabios                                                                            
+xen-unstable-smoke         build-armhf                              revision_seabios                                                                            
+xen-unstable-smoke         build-amd64                              revision_xen                  a7b39c8bd6cba3fe1c8012987b9e28bdbac7e92d                      
+xen-unstable-smoke         build-armhf                              revision_xen                  a7b39c8bd6cba3fe1c8012987b9e28bdbac7e92d                      
+xen-unstable-smoke         test-amd64-amd64-libvirt                 toolstack                     libvirt                                                       
+xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 toolstack                     xl                                                            
+xen-unstable-smoke         test-armhf-armhf-xl                      toolstack                     xl                                                            
+xen-unstable-smoke         build-amd64                              tree_ovmf                                                                                   
+xen-unstable-smoke         build-armhf                              tree_ovmf                                                                                   
+xen-unstable-smoke         build-amd64                              tree_qemu                     git://xenbits.xen.org/staging/qemu-xen-unstable.git           
+xen-unstable-smoke         build-armhf                              tree_qemu                     git://xenbits.xen.org/staging/qemu-xen-unstable.git           
+xen-unstable-smoke         build-amd64                              tree_qemuu                    git://xenbits.xen.org/staging/qemu-upstream-unstable.git      
+xen-unstable-smoke         build-armhf                              tree_qemuu                    git://xenbits.xen.org/staging/qemu-upstream-unstable.git      
+xen-unstable-smoke         build-amd64                              tree_seabios                                                                                
+xen-unstable-smoke         build-armhf                              tree_seabios                                                                                
+xen-unstable-smoke         build-amd64                              tree_xen                      git://xenbits.xen.org/xen.git                                 
+xen-unstable-smoke         build-armhf                              tree_xen                      git://xenbits.xen.org/xen.git                                 
+xen-unstable-smoke         test-amd64-amd64-libvirt                 xenbuildjob                   build-amd64                                                   
+xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 xenbuildjob                   build-amd64                                                   
+xen-unstable-smoke         test-armhf-armhf-xl                      xenbuildjob                   build-armhf                                                   
+xen-unstable-smoke         build-amd64                              hostalloc_bonus_previousfail~ 0                                                             
+xen-unstable-smoke         build-armhf                              hostalloc_bonus_previousfail~ 0                                                             
+xen-unstable-smoke         test-amd64-amd64-libvirt                 hostalloc_bonus_previousfail~ 0                                                             
+xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 hostalloc_bonus_previousfail~ 0                                                             
+xen-unstable-smoke         test-armhf-armhf-xl                      hostalloc_bonus_previousfail~ 0                                                             
+xen-unstable-smoke         build-amd64                              hostalloc_maxbonus_variation~ 0                                                             
+xen-unstable-smoke         build-armhf                              hostalloc_maxbonus_variation~ 0                                                             
+xen-unstable-smoke         test-amd64-amd64-libvirt                 hostalloc_maxbonus_variation~ 0                                                             
+xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 hostalloc_maxbonus_variation~ 0                                                             
+xen-unstable-smoke         test-armhf-armhf-xl                      hostalloc_maxbonus_variation~ 0                                                             

[OSSTEST PATCH 01/27] docs: Document OSSTEST_RESOURCE_PRIORITY values in README.planner

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 README.planner |    7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/README.planner b/README.planner
index 21b3415..dfa623e 100644
--- a/README.planner
+++ b/README.planner
@@ -84,6 +84,13 @@ no resources actually get allocated), so that there is a report for
 the administrator showing an idea of what the system thinks may happen
 in the more distant future.
 
+Values for OSSTEST_RESOURCE_PRIORITY:
+  mg-allocate                                    -1000000
+  mg-blockage                                    -2000000
+  default Executive.pm with controlling terminal      -10
+  mg-execute-flight                                    -8
+  default                                               0
+
 
 ms-ownerdaemon and `ownd' tasks
 -------------------------------
-- 
1.7.10.4

[OSSTEST PATCH 02/27] Executive: cs-job-create: Check that OSSTEST_FLIGHT is not set

[Ian Jackson]

If OSSTEST_FLIGHT is set on entry to JobDB::Executive::flight_create,
fail.  This obviously represents some kind of mistake.

One effect is that standalone-generate-dump-flight-runvars won't fill
an Executive database with test flights.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/JobDB/Executive.pm |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index cc52f57..6ba3d37 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -89,6 +89,8 @@ END
 
 sub flight_create ($$$) { #method
     my ($jd,$intended,$branch) = @_;
+    die "OSSTEST_FLIGHT set on entry to JobDB::Executive::flight_create\n"
+	if $ENV{'OSSTEST_FLIGHT'};
     $dbh_tests->do(<<END, {}, $branch, $intended);
              INSERT INTO flights
                          (flight,  started, blessing,       branch, intended)
-- 
1.7.10.4

[OSSTEST PATCH 03/27] mg-list-all-branches: Suppress `maintjobs'

[Ian Jackson]

This is not a real branch; it's a cron working directory, only.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 mg-list-all-branches |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/mg-list-all-branches b/mg-list-all-branches
index 1549f81..87703c7 100755
--- a/mg-list-all-branches
+++ b/mg-list-all-branches
@@ -16,4 +16,6 @@ foreach my $f (qw(cr-for-branches crontab)) {
     close C or die $!;
 }
 
+delete $branches{maintjobs};
+
 print $_,"\n" or die $! foreach sort { versioncmp($a, $b) } keys %branches;
-- 
1.7.10.4

[OSSTEST PATCH 04/27] standalone: Do not blunder on after errors

[Ian Jackson]

./standalone's with_logging function would _log_ errors, but it
wouldn't exit immediately.  As a result, the script would blunder on.

Normally it wouldn't do very much more since most of the with_logging
calls are the last thing it does - but the exit status would be wrong
(0, from echo).

As a result, for example, standalone-generate-dump-flight-runvars
would never properly report make-flight failures.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 standalone |    1 +
 1 file changed, 1 insertion(+)

diff --git a/standalone b/standalone
index 873616a..60b6666 100755
--- a/standalone
+++ b/standalone
@@ -194,6 +194,7 @@ with_logging() {
     rc=${PIPESTATUS[0]}
     if [ $rc -ne 0 ] ; then
 	echo "FAILED rc=${rc}" >&2
+	exit $rc
     fi
 }
 
-- 
1.7.10.4

[OSSTEST PATCH 05/27] standalone: Set very long SQLite3 busy timeout in Perl

[Ian Jackson]

Without this, big standalone-generate-dump-flight-runvars jobs may
trying to serialise so much work that SQLite3 times out.  And we are
about to introduce an optimisation which makes this much more likely.

In standalone mode we probably don't care much about this timeout at
all.  (It might even be that the user is using sqlite(3) and has
effectively locked the database interactively for an extended period.)

We would prefer to rely on the user to stop anything that seems to
have become stuck.  So set the timeout to 10ks.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/JobDB/Standalone.pm |    3 +++
 1 file changed, 3 insertions(+)

diff --git a/Osstest/JobDB/Standalone.pm b/Osstest/JobDB/Standalone.pm
index 45ec9ca..431ba5a 100644
--- a/Osstest/JobDB/Standalone.pm
+++ b/Osstest/JobDB/Standalone.pm
@@ -52,6 +52,9 @@ sub open ($) {
         ShowErrorStatement => 1,
         })
         or die "could not open standalone db $dbi";
+
+    $dbh->sqlite_busy_timeout(10E6); # 10ks
+
     return $dbh;
 }
 
-- 
1.7.10.4

[OSSTEST PATCH 06/27] cr-daily-branch: Honour AP_FETCH_PFX

[Ian Jackson]

This environment variable is prefixed to all cr-daily-branch's command
lines involving ap-fetch.  We are going to use this for memoisation.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cr-daily-branch |   12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/cr-daily-branch b/cr-daily-branch
index ade515d..9500bdc 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -51,12 +51,12 @@ determine_version () {
                 if [ "x$FORCE_REVISION" != x ]; then
                         tversion="$FORCE_REVISION"
                 else
-        		tversion=`./ap-fetch-version "$tbranch"`
+			tversion=`$AP_FETCH_PFX ./ap-fetch-version "$tbranch"`
                 fi
 		determine_tree "$treevarwhich" "" _${treevarwhich}
 		determine_tree "$treevarwhich" "" _${treevarwhich}_THIS
 	else
-		tversion=`./ap-fetch-version-baseline "$tbranch"`
+		tversion=`$AP_FETCH_PFX ./ap-fetch-version-baseline "$tbranch"`
 		determine_tree "$treevarwhich" BASE_ _${treevarwhich}
 		determine_tree "$treevarwhich" BASE_ _${treevarwhich}_THIS
 	fi
@@ -98,7 +98,7 @@ if [ "x$OLD_REVISION" = x ]; then
 		# last test will have tested the previous upstream baseline
 		OLD_REVISION=`check_tested --print-revision=$tree`
 	    else
-		OLD_REVISION="`./ap-fetch-version-old $branch`"
+		OLD_REVISION="`$AP_FETCH_PFX ./ap-fetch-version-old $branch`"
 	    fi
 	    ;;
     esac
@@ -131,7 +131,8 @@ if [ "x$REVISION_XEN" = x ]; then
         export REVISION_XEN
 fi
 if [ "x$REVISION_PREVXEN" = x -a "x$prevxenbranch" != "x" ]; then
-	REVISION_PREVXEN="`./ap-fetch-version-baseline $prevxenbranch`"
+	REVISION_PREVXEN="`\
+		$AP_FETCH_PFX ./ap-fetch-version-baseline $prevxenbranch`"
 	export REVISION_PREVXEN
 fi
 if [ "x$REVISION_QEMU_UPSTREAM" = x ]; then
@@ -252,7 +253,8 @@ ovmf)
 esac
 
 if [ "x$OLD_REVISION" = xdetermine-late ]; then
-	OLD_REVISION="`./ap-fetch-version-baseline-late $branch $NEW_REVISION`"
+	OLD_REVISION="`\
+ $AP_FETCH_PFX ./ap-fetch-version-baseline-late $branch $NEW_REVISION`"
 fi
 
 case $branch in
-- 
1.7.10.4

[OSSTEST PATCH 07/27] memoise: New utility

[Ian Jackson]

Give this a GPLv2+ licence so that we can move it into some other
FLOSS package later.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 memoise |   56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)
 create mode 100755 memoise

diff --git a/memoise b/memoise
new file mode 100755
index 0000000..ff999d5
--- /dev/null
+++ b/memoise
@@ -0,0 +1,56 @@
+#!/bin/sh
+#
+# memoise - a command line output memoisation tool
+#
+# usage:
+#    memoise datadir cmd arg arg ...
+#
+# will create datadir if necessary; delete datadir when you're done
+#
+#    Copyright 2015 Citrix Inc.
+#
+#    This file is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License along
+#    with this program; if not, write to the Free Software Foundation, Inc.,
+#    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+#    NB that the rest of osstest is AGPLv3+, so when this file is
+#    distributed together with osstest, the AGPLv3 applies.
+
+set -e
+
+case "$#.$1" in
+[01].)	echo >&2 "memoise: too few args"; exit 127 ;;
+*.-)	echo >&2 "memoise: unknown option $1"; exit 127 ;;
+esac
+
+datadir=$1; shift
+
+mkdir -p -- "$datadir"
+
+id=`for arg in "$@"; do printf '%s\0' "$arg"; done | sha256sum`
+id=${id%  -}
+
+f="$datadir/$id"
+
+if ! [ -f "$f.o" ]; then
+	with-lock-ex -w "$f.l" sh -ec '
+		f=$1; shift
+		if [ -f "$f.o" ]; then exit 0; fi
+		exec </dev/null >"$f.t"
+		"$@"
+		mv -- "$f.t" "$f.o"
+		rm "$f.l"
+	' x "$f" "$@"
+fi
+
+cat -- "$f.o"
-- 
1.7.10.4

[OSSTEST PATCH 08/27] standalone-generate-dump-flight-runvars: memoise ap-fetch

[Ian Jackson]

This makes it _much_ faster.

Also you can say AP_FETCH_MEMO_KEEP=1 to make it reuse old
information, which is useful for making comparisons.

For a further speed improvement, use `eatmydata'.  This is not the
default because it risks corruption of `standalone.db' which is used
for other purposes too.  Add a comment about possibly improving this.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 standalone-generate-dump-flight-runvars |   11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/standalone-generate-dump-flight-runvars b/standalone-generate-dump-flight-runvars
index 4405720..ed94792 100755
--- a/standalone-generate-dump-flight-runvars
+++ b/standalone-generate-dump-flight-runvars
@@ -36,6 +36,17 @@ if [ $# = 0 ]; then
    set `./mg-list-all-branches`
 fi
 
+if [ "x$AP_FETCH_MEMO_KEEP" = x ]; then
+	rm -rf tmp/apmemo
+	mkdir tmp/apmemo
+fi
+export AP_FETCH_PFX='./memoise tmp/apmemo'
+
+# In the future it might be nice for this script to arrange to use a
+# separate standalone.db, in tmp/ probably, for each different branch.
+# That would avoid a lot of needless write contention over the single
+# db, and make it possible to use `eatmydata' if it is installed.
+
 perbranch () {
     log=tmp/make-flight.$branch.log
     flight=check_${branch//[-._]/_}
-- 
1.7.10.4

[OSSTEST PATCH 09/27] mfi-common: Use job_create_build for build-*-prev

[Ian Jackson]

This causes it to call job_create_build_filter_callback but has no
ultimate functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 mfi-common |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mfi-common b/mfi-common
index 991baf5..aaaeac9 100644
--- a/mfi-common
+++ b/mfi-common
@@ -216,7 +216,7 @@ create_build_jobs () {
         # only pass tree_xen and revision_xen here and therefore
         # pickup the versions implicitly selected by the
         # $REVISION_PREVXEN.
-        ./cs-job-create $flight build-$arch-prev build                       \
+        job_create_build build-$arch-prev build                       \
                     arch=$arch enable_xend=false enable_ovmf=$enable_ovmf    \
                     enable_xsm=false                                         \
             tree_xen=$TREE_XEN                                               \
-- 
1.7.10.4

[OSSTEST PATCH 10/27] ap-fetch-version*: Unwrap xen-unstable lines

[Ian Jackson]

Whitespace and \ change only.  Makes the next patch a bit clearer.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ap-fetch-version     |    3 +--
 ap-fetch-version-old |    3 +--
 2 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/ap-fetch-version b/ap-fetch-version
index 8b41acf..488ea55 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -40,8 +40,7 @@ xen-3.*)
 	' x
 	;;
 xen-unstable)
-	repo_tree_rev_fetch_git xen \
-		$TREE_XEN staging $LOCALREV_XEN
+	repo_tree_rev_fetch_git xen $TREE_XEN staging $LOCALREV_XEN
 	;;
 xen-4.*-testing)
 	branchcore=${branch#xen-}
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index 0b4739b..7b918e3 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -48,8 +48,7 @@ xen-3.*)
 	'
 	;;
 xen-unstable)
-	repo_tree_rev_fetch_git xen \
-		$TREE_XEN master $LOCALREV_XEN
+	repo_tree_rev_fetch_git xen $TREE_XEN master $LOCALREV_XEN
 	;;
 xen-4.*-testing)
 	branchcore=${branch#xen-}
-- 
1.7.10.4

[OSSTEST PATCH 11/27] ap-push: Use refs/heads/ for destinations

[Ian Jackson]

When the destination is a branch, specify refs/heads/ explicitly.
This makes ap-push work even if the ref does not yet exist on the
destination.

There is no functional change for an existing installation pushing to
an existing branch.  But for a hypothetical new installation, this
would be necessary.

And, more relevantly, when new "branches" are invented, the use of an
existing ap-push case as a template will generate a new case which
creates the branch as is necessary.

I leave the more complex osstest case alone.  It's not clear to me
whether the destination ref not existing is an installation problem of
such severity that indeed ap-push should fail.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 ap-push |   12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/ap-push b/ap-push
index 5967b42..1dd5b05 100755
--- a/ap-push
+++ b/ap-push
@@ -52,14 +52,14 @@ xen-3.*)
 	;;
 xen-unstable)
 	cd $repos/xen
-	git push $TREE_XEN $revision:master
+	git push $TREE_XEN $revision:refs/heads/master
 	;;
 xen-*-testing)
 	cd $repos/xen
 	xenversion=$branch
 	xenversion=${branch%-testing}
 	xenversion=${xenversion#xen-}
-	git push $TREE_XEN $revision:stable-$xenversion
+	git push $TREE_XEN $revision:refs/heads/stable-$xenversion
 	;;
 qemu-mainline)
 	cd $repos/qemu-mainline
@@ -67,11 +67,11 @@ qemu-mainline)
 	;;
 qemu-upstream-*)
 	cd $repos/$branch
-        git push $TREE_QEMU_UPSTREAM $revision:master
+        git push $TREE_QEMU_UPSTREAM $revision:refs/heads/master
         ;;
 linux-3.*)
 	cd $repos/linux
-	git push $TREE_LINUX $revision:tested/$branch
+	git push $TREE_LINUX $revision:refs/heads/tested/$branch
 	;;
 linux-2.6.39)
 	cd $repos/linux
@@ -83,11 +83,11 @@ linux-xen-*)
         ;;
 libvirt)
 	cd $repos/libvirt
-	git push $TREE_LIBVIRT $revision:xen-tested-master
+	git push $TREE_LIBVIRT $revision:refs/heads/xen-tested-master
 	;;
 rumpuserxen)
 	cd $repos/rumpuserxen
-	git push $TREE_RUMPUSERXEN $revision:xen-tested-master
+	git push $TREE_RUMPUSERXEN $revision:refs/heads/xen-tested-master
 	;;
 seabios)
 	cd $repos/seabios
-- 
1.7.10.4

[OSSTEST PATCH 12/27] cr-ensure-disk-space: Honour -D

[Ian Jackson]

Provide DEBUG and use db_prepare.  For now that is the only debugging
output.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cr-ensure-disk-space |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index 304f262..ced9777 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -27,11 +27,15 @@ use Fcntl qw(:flock);
 
 our $dryrun= 0;
 
+open DEBUG, ">/dev/null" or die $!;
+
 while (@ARGV && $ARGV[0] =~ m/^\-/) {
     $_= shift @ARGV;
     last if $_ eq '--';
     if (m/^-n$/) {
         $dryrun= 1;
+    } elsif (m/^-D$/) {
+        open DEBUG, ">&2" or die $!;
     } else {
 	die "$_ ?";
     }
@@ -50,7 +54,7 @@ flock LOCK, LOCK_EX or die $!;
 
 $|=1;
 
-my $chkq= $dbh_tests->prepare("SELECT * FROM flights WHERE flight=?");
+my $chkq= db_prepare("SELECT * FROM flights WHERE flight=?");
 our @flights;
 
 for (;;) {
-- 
1.7.10.4

[OSSTEST PATCH 13/27] cr-ensure-disk-space: Look at referring flights

[Ian Jackson]

Previously the flight to delete was simply the one with the lowest
flight number.  Now we sort flights not by their own flight number,
but by the highest flight number of any referencing flight.

This means that flights whose builds are being reused are kept as long
as the reusing flights.

This almost-entirely fixes a largely-theoretical race in the way
cs-bisection-step works (where the flight's logs and build outputs
might be deleted between the setup and execution of the referring
flight).

A smaller race still exists because the stash check in
cs-bisection-step occurs before the being-created flight is visible to
other db clients.  We will have to fix this by taking the flights
lock.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cr-ensure-disk-space |   43 ++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 38 insertions(+), 5 deletions(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index ced9777..0314e7a 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -55,7 +55,17 @@ flock LOCK, LOCK_EX or die $!;
 $|=1;
 
 my $chkq= db_prepare("SELECT * FROM flights WHERE flight=?");
+
+my $refq= db_prepare(<<END);
+    SELECT flight, val
+      FROM runvars
+     WHERE name like '%job'
+       AND val like '%.%'
+       AND flight >= ?
+END
+
 our @flights;
+our %latestref;
 
 for (;;) {
     open P, "-|", onloghost "df --block-size=1M -P $logdir" or die $!;
@@ -69,23 +79,46 @@ for (;;) {
     last if $space >= logcfg('MinSpaceMby');
 
     if (!@flights) {
+	%latestref = ();
 	open P, "-|", onloghost "ls -1 $logdir" or die $!;
+
+	my $minflight = undef;
 	while (<P>) {
 	    next unless m/^(\d+)\n$/;
-	    push @flights, $1;
+	    $latestref{$1} = $1;
+	    $minflight //= $1;
+	    $minflight = $1 if $1 < $minflight;
 	}
 	$!=$?=0; close P or die "ls: $? $!";
-	@flights = sort { $b <=> $a } @flights;
+
+	print DEBUG "MINFLIGHT $minflight\n";
+
+	$refq->execute($minflight);
+	while (my $rr= $refq->fetchrow_hashref) {
+	    my $testflight = $rr->{flight};
+	    next unless $rr->{val} =~ m/^(\d+)\./;
+	    my $buildflight = $1;
+	    next unless exists $latestref{$buildflight};
+	    if ($testflight > $latestref{$buildflight}) {
+		print DEBUG "REF $buildflight <- $testflight\n";
+		$latestref{$buildflight} = $testflight;
+	    }
+	}
+
+	@flights =
+	    sort { $latestref{$b} <=> $latestref{$a} }
+	    keys %latestref;
         printf "(%d flights) ", scalar @flights;
         die unless @flights;
     }
     my $flight = pop @flights;
-    printf "selected %s ", $flight;
+    my $latestref = $latestref{$flight};
+    printf "selected %s (latest ref %s) ", $flight, $latestref;
 
-    $chkq->execute($flight);
+    $chkq->execute($latestref);
     my $row= $chkq->fetchrow_hashref();
     $chkq->finish();
-    die $flight unless defined $row;
+    die "$flight $latestref" unless defined $row;
     my $age= time - $row->{started};
 
     printf "(age %dd) ", $age / 86400;
-- 
1.7.10.4

[OSSTEST PATCH 14/27] cr-ensure-disk-space: Make main loop body into a subroutine

[Ian Jackson]

No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cr-ensure-disk-space |   11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index 0314e7a..8c5a708 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -67,7 +67,7 @@ END
 our @flights;
 our %latestref;
 
-for (;;) {
+sub iteration_continue () {
     open P, "-|", onloghost "df --block-size=1M -P $logdir" or die $!;
     $_= <P>;
     m/^filesystem/i or die "$_ ?";
@@ -76,7 +76,7 @@ for (;;) {
     $!=0; $?=0; close P or die "$! $?";
     my $space= $1;
     printf "space: %8d, wanted: %8d ", $space, logcfg('MinSpaceMby');
-    last if $space >= logcfg('MinSpaceMby');
+    return 0 if $space >= logcfg('MinSpaceMby');
 
     if (!@flights) {
 	%latestref = ();
@@ -141,6 +141,13 @@ for (;;) {
 END
 
     printf "done.\n";
+
+    return 1;
+}
+
+for (;;) {
+    iteration_continue()
+	or last;
 }
 
 printf "ok.\n";
-- 
1.7.10.4

[OSSTEST PATCH 15/27] cr-ensure-disk-space: Take the flights db lock

[Ian Jackson]

This eliminates the race with cs-bisection-step (and other
flight-construction tools which might reuse previous flights, provided
that they also do not pass previous flight numbers from hand to hand
with the db unlocked).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cr-ensure-disk-space |   11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index 8c5a708..138f3f4 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -145,10 +145,13 @@ END
     return 1;
 }
 
-for (;;) {
-    iteration_continue()
-	or last;
-}
+db_retry($dbh_tests,[qw(flights)], sub {
+    @flights = ();
+    for (;;) {
+	iteration_continue()
+	    or last;
+    }
+});
 
 printf "ok.\n";
 exit 0;
-- 
1.7.10.4

[OSSTEST PATCH 16/27] cs-job-create: Permit creation of `synth' runvars

[Ian Jackson]

This will be useful if we want to set hostalloc_* runvars.  Normally
we would want to set those only on flights generated by make-flight.

Flights generated by cs-bisection-step or cs-adjust-flight ought not
to copy them, because cs-bisection-step makes its own arrangements for
host specification, as should the caller of cs-adjust-flight (perhaps
via the blessing system).

Using `synth' for this is arguably slightly wrong but it does the
right thing in all existing cases.  The alternative would be a schema
change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cs-job-create |   16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/cs-job-create b/cs-job-create
index df2738c..9812bf2 100755
--- a/cs-job-create
+++ b/cs-job-create
@@ -2,6 +2,11 @@
 #
 # args:
 #   <flight> <job> <recipe> <runvar>=<value> ...
+#
+# If <runvar> ends in ~, the ~ is stripped and the runvar is marked
+# `synth'.  (Normally runvars marked `synth' are defined during test
+# execution.  They are not taken as part of the runvar set copied
+# when cloning flights, eg for bisection.)
 
 # This is part of "osstest", an automated testing framework for Xen.
 # Copyright (C) 2009-2013 Citrix Inc.
@@ -47,9 +52,9 @@ foreach my $rv (@runvars) {
         $suppress{$1}= 1;
         next;
     }
-    $rv =~ m/^([a-z][0-9a-z_]*)\=(.*)$/ or die "$rv ?";
-    my ($name,$val) = ($1,$2);
-    $runvars{$name}= $val;
+    $rv =~ m/^([a-z][0-9a-z_]*)(\~?)\=(.*)$/ or die "$rv ?";
+    my ($name,$synth,$val) = ($1,$2,$3);
+    $runvars{$name}= [$val,$synth];
 }
 
 foreach my $name (keys %suppress) {
@@ -61,9 +66,10 @@ db_retry($flight,'constructing', $dbh_tests,[qw(flights)], sub {
         INSERT INTO jobs VALUES (?,'$job','$recipe','queued')
 END
     my $q= $dbh_tests->
-        prepare("INSERT INTO runvars VALUES (?,'$job',?,?,'f')");
+        prepare("INSERT INTO runvars VALUES (?,'$job',?,?,?)");
     foreach my $name (keys %runvars) {
-        $q->execute($flight, $name, $runvars{$name});
+        $q->execute($flight, $name, $runvars{$name}[0],
+	            $runvars{$name}[1] ? 't' : 'f');
         $q->finish();
     }
 });
-- 
1.7.10.4

[OSSTEST PATCH 17/27] mg-show-flight-runvars: Decorate synth runvar names with ~

[Ian Jackson]

Make mg-show-flight-runvars -a append ~ to the names of synth runvars.
(This is consistent with the new syntax in cs-job-create.)

We do this by editing $row->[1] (and $colws[1]) so we can avoid
disturbing the general column format calcuation and printing; this
means we have to copy $row (because the value in the DB $row is
readonly).

We have to check for $synth eq 'f' as well as $synth being boolean
false, because SQLite's typeless nature (or, to put it another way,
DBD::SQLite's failure to look at the schema) means that a boolean
field's value of 'f' or 't' is simply returned as a string to Perl.
But of course "f" is trueish in Perl.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 mg-show-flight-runvars |   10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/mg-show-flight-runvars b/mg-show-flight-runvars
index 62d5cba..aa4cc25 100755
--- a/mg-show-flight-runvars
+++ b/mg-show-flight-runvars
@@ -26,6 +26,7 @@ use Osstest;
 csreadconfig();
 
 my $synthcond = '(NOT synth)';
+my $synthsufx = '';
 
 for (;;) {
     last unless @ARGV;
@@ -33,7 +34,7 @@ for (;;) {
     $_ = shift @ARGV;
     last if m/^\-\-?$/;
     if (m/^-a$/) {
-	$synthcond = 'TRUE';
+	$synthsufx = '~';
     } else {
 	die "$_ ?";
     }
@@ -52,10 +53,15 @@ my @colws = $dbh_tests->selectrow_array
     ("SELECT ".(join ',', map { "max(length($_))" } @cols)." $qfrom");
 
 my $q = $dbh_tests->prepare
-    ("SELECT ".(join ',', @cols)." $qfrom ORDER BY name, job");
+    ("SELECT synth, ".(join ',', @cols)." $qfrom ORDER BY synth, name, job");
 $q->execute();
 
+$colws[1] += length $synthsufx;
+
 while (my $row = $q->fetchrow_arrayref()) {
+    $row = [ @$row ];
+    my $synth = shift @$row;
+    $row->[1] .= $synthsufx if $synth && $synth ne 'f'; # sqlite3 is typeless
     printf "%-*s %-*s %-*s\n", map { $colws[$_], $row->[$_] } qw(0 1 2)
         or die $!;
 }
-- 
1.7.10.4

[OSSTEST PATCH 18/27] standalone-generate-dump-flight-runvars: Show synth runvars

[Ian Jackson]

Pass -a to mg-show-flight-runvars.  That way when we use the new
cs-job-create feature to set synth runvars during creation, we will
see them in the dump.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 standalone-generate-dump-flight-runvars |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/standalone-generate-dump-flight-runvars b/standalone-generate-dump-flight-runvars
index ed94792..da3c0f8 100755
--- a/standalone-generate-dump-flight-runvars
+++ b/standalone-generate-dump-flight-runvars
@@ -76,5 +76,5 @@ done
 
 for proc in $procs; do
     perproc
-    ./mg-show-flight-runvars $flight | sed "s/^/`printf '%-26s ' $branch`/"
+    ./mg-show-flight-runvars -a $flight | sed "s/^/`printf '%-26s ' $branch`/"
 done
-- 
1.7.10.4

[OSSTEST PATCH 19/27] ts-host-alloc-Executive: Honour various hostalloc_* runvars

[Ian Jackson]

We honour
  hostalloc_maxbonus_variation
  hostalloc_bonus_previousfail
  hostalloc_bonus_sharereuse
and make them default to their previous values.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 ts-hosts-allocate-Executive |   13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index 781ddaf..eafcce2 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -541,14 +541,19 @@ sub hid_recurse ($$) {
 
     my $log_variation_age = log(1+$variation_age/86400);
     my $variation_bonus = $log_variation_age * 3600*2;
-    my $max_variation_bonus = 12*3600;
+    my $max_variation_bonus = $r{hostalloc_maxbonus_variation} // 12*3600;
     $variation_bonus=$max_variation_bonus
 	if $variation_bonus>$max_variation_bonus;
 
+    my $prevfail_bonus = $r{hostalloc_bonus_previousfail} // 7.0*86400;
+    my $prevfail_equiv_bonus = $prevfail_bonus * (6.5 / 7.0);
+
+    my $share_reuse_bonus = $r{hostalloc_bonus_sharereuse} // 10000;
+
     my $cost= $start_time
 	+ $duration_for_cost
-        - ($previously_failed      ==@hids ?   7*86400 :
-	   $previously_failed_equiv==@hids ? 6.5*86400 :
+        - ($previously_failed      ==@hids ? $prevfail_bonus       :
+	   $previously_failed_equiv==@hids ? $prevfail_equiv_bonus :
 	   # We wait 7d extra to try a failing test on the same
 	   # hardware, or 6.5d on `equivalent' hardware (as defined by
 	   # equiv-* flags).  Compared to `equivalent' hardware, we
@@ -556,7 +561,7 @@ sub hid_recurse ($$) {
 	   0)
         + ($previously_failed || $previously_failed_equiv
 	   ? (-$max_variation_bonus+$variation_bonus) : -$variation_bonus)
-	- $share_reuse * 10000;
+	- $share_reuse * $share_reuse_bonus;
     
     print DEBUG "$dbg FINAL start=$start_time va=$variation_age".
 	" variation_bonus=$variation_bonus".
-- 
1.7.10.4

[OSSTEST PATCH 20/27] make-flight: Allow separate specification of pre-built Xen vs others

[Ian Jackson]

Make it possible to separately specify
 - reuse previous flight's builds of xen.git
 - reuse previous flight's builds of all other stuff

Specifically:

 * Match the 4th argument to make-flight with case rather than if.  If
   it contains a slash, the first part is for Xen and the second part
   for everything else.

 * Split the variable `bfi' into `bfi' and `xenbfi'

 * Introduce job_create_build_filter_callback, whose initial contents
   check the recipe (to see what we're building) and then the relevant
   bfi variable.  (NB that this is wrong for historical kernel builds
   using the xen.git build system, which have recipe=build, but we do
   not care about that.)

No functional change for existing call patterns.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 make-flight |   30 +++++++++++++++++++++++-------
 mfi-common  |    4 ++--
 2 files changed, 25 insertions(+), 9 deletions(-)

diff --git a/make-flight b/make-flight
index 00f797e..6db3550 100755
--- a/make-flight
+++ b/make-flight
@@ -22,7 +22,7 @@ set -e
 branch=$1
 xenbranch=$2
 blessing=$3
-buildflight=$4
+buildflight=$4 # can be [<xen.git-build-flight>/<general-build-flight>]
 
 flight=`./cs-flight-create $blessing $branch`
 
@@ -35,18 +35,34 @@ defsuite=`getconfig DebianSuite`
 defguestsuite=`getconfig GuestDebianSuite`
 
 job_create_build_filter_callback () {
-    :
-}
+  local job=$1; shift
+  local recipe=$1; shift
 
-if [ x$buildflight = x ]; then
+  local relevant_bfi
 
-  create_build_jobs
+  case "$recipe" in
+    build) relevant_bfi=$xenbfi ;;
+    *)     relevant_bfi=$bfi    ;;
+  esac
+  if [ "x$relevant_bfi" != x ]; then return 1; fi # reusing
 
-else
+  return 0
+}
 
+case "$buildflight" in
+'')
+  ;;
+*/*)
+  bfi=${buildflight#*/}; bfi=$bfi${bfi:+.}
+  xenbfi=${buildflight%%/*}; xenbfi=$xenbfi${xenbfi:+.}
+  ;;
+*)
   bfi=$buildflight.
+  xenbfi=$buildflight. # applies to `buildjob' and `xenbuildjob'
+  ;;
+esac
 
-fi
+create_build_jobs
 
 job_create_test_filter_callback () {
   local job=$1; shift
diff --git a/mfi-common b/mfi-common
index aaaeac9..af98f2d 100644
--- a/mfi-common
+++ b/mfi-common
@@ -325,8 +325,8 @@ job_create_test () {
   esac
 
   job="$job$xsm_suffix"
-  xenbuildjob="${bfi}build-$xenarch$xsm_suffix"
-  buildjob="${bfi}build-$dom0arch$xsm_suffix"
+  xenbuildjob="${xenbfi}build-$xenarch$xsm_suffix"
+  buildjob="${xenbfi}build-$dom0arch$xsm_suffix"
   tsbuildjob=
 
   case "$xenbranch:$toolstack" in
-- 
1.7.10.4

[OSSTEST PATCH 21/27] make-flight: Contemplate varying architecture for Debian HVM

[Ian Jackson]

In the Debian hvm tests, ask branch_debianhvm_arch for the
architecture to use.  This currently prints only amd64.

While we are here, provide some comments about the (somewhat
confusing) argument conventions of do_hvm_debian_test_one.

No functional change yet.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 make-flight |   16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/make-flight b/make-flight
index 6db3550..2246411 100755
--- a/make-flight
+++ b/make-flight
@@ -236,12 +236,20 @@ do_hvm_win7_x64_tests () {
             all_hostflags=$most_hostflags,hvm
 }
 
+branch_debianhvm_arch () {
+  case $branch in
+    *) echo amd64;;
+  esac
+}
+
 do_hvm_debian_test_one () {
   testname=$1
   toolstack=$2
   bios=$3
-  xsm=$4
-  stubdom=$5
+  xsm=$4 # 'false' or 'true'
+  stubdom=$5 # '' (or unset) or 'true'
+
+  local arch=$(branch_debianhvm_arch)
 
   stubdom_suffix=""
   stubdom_runvar=""
@@ -250,11 +258,11 @@ do_hvm_debian_test_one () {
       stubdom_runvar="debianhvm_stubdom=$stubdom"
   fi
 
-  job_create_test test-$xenarch$kern-$dom0arch-$toolstack$qemuu_suffix$stubdom_suffix-$testname-amd64\
+  job_create_test test-$xenarch$kern-$dom0arch-$toolstack$qemuu_suffix$stubdom_suffix-$testname-$arch\
     test-debianhvm $toolstack $xenarch $dom0arch $qemuu_runvar \
     enable_xsm=$xsm                             \
     $stubdom_runvar                             \
-    debianhvm_image=debian-7.2.0-amd64-CD-1.iso \
+    debianhvm_image=debian-7.2.0-$arch-CD-1.iso \
     bios=$bios \
     all_hostflags=$most_hostflags,hvm
 }
-- 
1.7.10.4

[OSSTEST PATCH 22/27] make-flight: Run job_create_test_filter_callback on true job name

[Ian Jackson]

job_create_test would pass $job to job_create_test_filter_callback but
then later maybe append -xsm to it.  Fix this.

No functional change for existing in-tree code because all existing
tests of the $job end in *.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 mfi-common |    7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/mfi-common b/mfi-common
index af98f2d..6ffa9e6 100644
--- a/mfi-common
+++ b/mfi-common
@@ -311,8 +311,6 @@ create_build_jobs () {
 }
 
 job_create_test () {
-  job_create_test_filter_callback "$@" || return 0
-
   local job=$1; shift
   local recipe=$1; shift
   local toolstack=$1; shift
@@ -323,8 +321,11 @@ job_create_test () {
   case " $* " in
     *" enable_xsm=true "*) xsm_suffix="-xsm";
   esac
-
   job="$job$xsm_suffix"
+
+  job_create_test_filter_callback \
+    "$job" "$recipe" "$toolstack" "$xenarch" "$dom0arch" "$@" || return 0
+
   xenbuildjob="${xenbfi}build-$xenarch$xsm_suffix"
   buildjob="${xenbfi}build-$dom0arch$xsm_suffix"
   tsbuildjob=
-- 
1.7.10.4

[OSSTEST PATCH 23/27] make-flight: Provide xen-unstable-smoke branch

[Ian Jackson]

This contains a very limited set of jobs:
    build-amd64
    build-armhf
    test-amd64-amd64-libvirt
    test-amd64-amd64-xl-qemuu-debianhvm-i386
    test-armhf-armhf-xl

The debianhvm job exists only in this flight, and is generated by
having branch_debianhvm_arch return i386 instead of amd64.

We override host allocator parameters to make this flight not care
about host stickiness: it just takes whatever comes to hand.

Deployment note: This requires images/debian-7.2.0-i386-CD-1.iso
which I have already downloaded to the Cambridge instance using jigdo.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 make-flight |   26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/make-flight b/make-flight
index 2246411..09f3304 100755
--- a/make-flight
+++ b/make-flight
@@ -46,6 +46,16 @@ job_create_build_filter_callback () {
   esac
   if [ "x$relevant_bfi" != x ]; then return 1; fi # reusing
 
+  case "$branch" in
+    xen-unstable-smoke)
+      case "$job" in
+        build-amd64) ;;
+        build-armhf) ;;
+        *)           return 1;;
+      esac
+    ;;
+  esac
+
   return 0
 }
 
@@ -62,6 +72,13 @@ case "$buildflight" in
   ;;
 esac
 
+case "$branch" in
+xen-unstable-smoke)
+	global_runvars+=" hostalloc_maxbonus_variation~=0 "
+	global_runvars+=" hostalloc_bonus_previousfail~=0 "
+        ;;
+esac
+
 create_build_jobs
 
 job_create_test_filter_callback () {
@@ -72,6 +89,14 @@ job_create_test_filter_callback () {
   local dom0arch=$1; shift
 
   case "$branch" in
+    xen-unstable-smoke)
+      case "$job" in
+        test-amd64-amd64-libvirt)                  return 0 ;;
+        test-armhf-armhf-xl)                       return 0 ;;
+        test-amd64-amd64-xl-qemuu-debianhvm-i386)  return 0 ;;
+        *)                                         return 1 ;;
+      esac
+      ;;
     qemu-upstream-4.2-testing)
       case " $* " in
         *" device_model_version=qemu-xen "*)
@@ -238,6 +263,7 @@ do_hvm_win7_x64_tests () {
 
 branch_debianhvm_arch () {
   case $branch in
+    xen-unstable-smoke) echo i386;;
     *) echo amd64;;
   esac
 }
-- 
1.7.10.4

[OSSTEST PATCH 24/27] make-flight: mfi-common: Honour $global_runvars

[Ian Jackson]

Provide a way for the main script to set some runvars on all jobs.  We expect this to be mostly set with +=.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 mfi-common |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mfi-common b/mfi-common
index 6ffa9e6..1ef4f26 100644
--- a/mfi-common
+++ b/mfi-common
@@ -69,7 +69,7 @@ job_create_build () {
   local job=$1; shift
   local recipe=$1; shift
 
-  ./cs-job-create $flight $job $recipe "$@"
+  ./cs-job-create $flight $job $recipe $global_runvars "$@"
 }
 
 create_build_jobs () {
@@ -344,7 +344,7 @@ job_create_test () {
   esac
 
   ./cs-job-create $flight $job $recipe toolstack=$toolstack       \
-    $RUNVARS $TEST_RUNVARS $most_runvars                          \
+    $RUNVARS $TEST_RUNVARS $global_runvars $most_runvars          \
     xenbuildjob=$xenbuildjob buildjob=$buildjob $tsbuildjob "$@"
 }
 
-- 
1.7.10.4

[OSSTEST PATCH 25/27] ap-*: Be able to fetch and push xen.git#smoked

[Ian Jackson]

The branches and push gates are now:
 xen.git#staging -[xen-unstable-smoke]-> #smoked -[xen-unstable]-> #master

Deployment note: When this passes the osstest self-push-gate, the main
xen-unstable flight will start using smoked as an input.  Therefore,
until the new cronjob is installed to run the xen-unstable-smoke
tests, an automatic process should keep xen.git#smoked up to date with
xen.git#staging.  Eg, running in screen in xen@xenbits:~/git/xen.git:
    while sleep 1800; do git fetch . staging:smoked; done

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 ap-fetch-version     |    3 +++
 ap-fetch-version-old |    3 +++
 ap-push              |    4 ++++
 3 files changed, 10 insertions(+)

diff --git a/ap-fetch-version b/ap-fetch-version
index 488ea55..a239968 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -40,6 +40,9 @@ xen-3.*)
 	' x
 	;;
 xen-unstable)
+	repo_tree_rev_fetch_git xen $TREE_XEN smoked $LOCALREV_XEN
+	;;
+xen-unstable-smoke)
 	repo_tree_rev_fetch_git xen $TREE_XEN staging $LOCALREV_XEN
 	;;
 xen-4.*-testing)
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index 7b918e3..b1e2cd9 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -50,6 +50,9 @@ xen-3.*)
 xen-unstable)
 	repo_tree_rev_fetch_git xen $TREE_XEN master $LOCALREV_XEN
 	;;
+xen-unstable-smoke)
+	repo_tree_rev_fetch_git xen $TREE_XEN smoked $LOCALREV_XEN
+	;;
 xen-4.*-testing)
 	branchcore=${branch#xen-}
 	branchcore=${branchcore%-testing}
diff --git a/ap-push b/ap-push
index 1dd5b05..bcb2f51 100755
--- a/ap-push
+++ b/ap-push
@@ -54,6 +54,10 @@ xen-unstable)
 	cd $repos/xen
 	git push $TREE_XEN $revision:refs/heads/master
 	;;
+xen-unstable-smoke)
+	cd $repos/xen
+	git push $TREE_XEN $revision:refs/heads/smoked
+	;;
 xen-*-testing)
 	cd $repos/xen
 	xenversion=$branch
-- 
1.7.10.4

[OSSTEST PATCH 26/27] cri-common: Support xen-unstable-smoke

[Ian Jackson]

Add this branch to select_xenbranch.  This works like xen-unstable in
most respects.  We are going to sort out the push gate ref plumbing in
xen.git in the next osstest patch.

Also, use a branch-settings file to set its resource priority to -20
to make it run ahead of anything else automatic.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 README.planner                     |    1 +
 branch-settings.xen-unstable-smoke |    1 +
 cri-common                         |    1 +
 3 files changed, 3 insertions(+)
 create mode 100644 branch-settings.xen-unstable-smoke

diff --git a/README.planner b/README.planner
index dfa623e..3a491a4 100644
--- a/README.planner
+++ b/README.planner
@@ -87,6 +87,7 @@ in the more distant future.
 Values for OSSTEST_RESOURCE_PRIORITY:
   mg-allocate                                    -1000000
   mg-blockage                                    -2000000
+  xen-unstable-smoke cr-daily-branch                  -20
   default Executive.pm with controlling terminal      -10
   mg-execute-flight                                    -8
   default                                               0
diff --git a/branch-settings.xen-unstable-smoke b/branch-settings.xen-unstable-smoke
new file mode 100644
index 0000000..b73bc6a
--- /dev/null
+++ b/branch-settings.xen-unstable-smoke
@@ -0,0 +1 @@
+export OSSTEST_RESOURCE_PRIORITY=-20
diff --git a/cri-common b/cri-common
index 94696ab..a31004a 100644
--- a/cri-common
+++ b/cri-common
@@ -80,6 +80,7 @@ select_prevxenbranch () {
 
 select_xenbranch () {
 	case "$branch" in
+	xen-unstable-smoke)	tree=xen;	xenbranch=xen-unstable ;;
 	xen-*)			tree=xen;	xenbranch=$branch ;;
 	qemu-mainline)		tree=qemuu;	xenbranch=xen-unstable	qemuubranch=qemu-mainline;;
         qemu-upstream-*)    tree=qemuu; xenbranch=xen-${branch#qemu-upstream-};;
-- 
1.7.10.4

[OSSTEST PATCH 27/27] crontab: Enable xen-unstable-smoke branch every 3h at <1mod3>:51 UTC

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 crontab |    1 +
 1 file changed, 1 insertion(+)

diff --git a/crontab b/crontab
index 436645b..b9d4ad1 100644
--- a/crontab
+++ b/crontab
@@ -2,6 +2,7 @@ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
 MAILTO=ian.jackson@citrix.com,ian.campbell@eu.citrix.com
 # m		h	dom mon dow	command
 49		1	* * *		cd testing.git && BRANCHES_ALWAYS=xen-unstable	./cr-for-branches branches -w "./cr-daily-branch --real"
+51		1-23/3	* * *		cd testing.git && BRANCHES=xen-unstable-smoke	./cr-for-branches branches -w "./cr-daily-branch --real"
 4-59/30		*	* * *		cd testing.git &&				./cr-for-branches branches -q "./cr-daily-branch --real"
 18		9	* * 1,3,5	cd testing.git && BRANCHES=linux-next		./cr-for-branches branches -w "./cr-daily-branch --real"
 18		4	* * *		cd testing.git && BRANCHES='linux-linus linux-mingo-tip-master linux-3.0 libvirt rumpuserxen' ./cr-for-branches branches -w "./cr-daily-branch --real"
-- 
1.7.10.4

Re: [OSSTEST PATCH 01/27] docs: Document OSSTEST_RESOURCE_PRIORITY values in README.planner

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 02/27] Executive: cs-job-create: Check that OSSTEST_FLIGHT is not set

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> If OSSTEST_FLIGHT is set on entry to JobDB::Executive::flight_create,
> fail.  This obviously represents some kind of mistake.
> 
> One effect is that standalone-generate-dump-flight-runvars won't fill
> an Executive database with test flights.

Speaking from experience there?


> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  Osstest/JobDB/Executive.pm |    2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
> index cc52f57..6ba3d37 100644
> --- a/Osstest/JobDB/Executive.pm
> +++ b/Osstest/JobDB/Executive.pm
> @@ -89,6 +89,8 @@ END
>  
>  sub flight_create ($$$) { #method
>      my ($jd,$intended,$branch) = @_;
> +    die "OSSTEST_FLIGHT set on entry to
> JobDB::Executive::flight_create\n"
> +	if $ENV{'OSSTEST_FLIGHT'};
>      $dbh_tests->do(<<END, {}, $branch, $intended);
>               INSERT INTO flights
>                           (flight,  started, blessing,       branch,
> intended)

Re: [OSSTEST PATCH 03/27] mg-list-all-branches: Suppress `maintjobs'

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> This is not a real branch; it's a cron working directory, only.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 04/27] standalone: Do not blunder on after errors

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> ./standalone's with_logging function would _log_ errors, but it
> wouldn't exit immediately.  As a result, the script would blunder on.
> 
> Normally it wouldn't do very much more since most of the with_logging
> calls are the last thing it does - but the exit status would be wrong
> (0, from echo).
> 
> As a result, for example, standalone-generate-dump-flight-runvars
> would never properly report make-flight failures.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  standalone |    1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/standalone b/standalone
> index 873616a..60b6666 100755
> --- a/standalone
> +++ b/standalone
> @@ -194,6 +194,7 @@ with_logging() {
>      rc=${PIPESTATUS[0]}
>      if [ $rc -ne 0 ] ; then
>  	echo "FAILED rc=${rc}" >&2
> +	exit $rc
>      fi
>  }
>  

Re: [OSSTEST PATCH 05/27] standalone: Set very long SQLite3 busy timeout in Perl

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> Without this, big standalone-generate-dump-flight-runvars jobs may
> trying to serialise so much work that SQLite3 times out.  And we are
> about to introduce an optimisation which makes this much more likely.
> 
> In standalone mode we probably don't care much about this timeout at
> all.  (It might even be that the user is using sqlite(3) and has
> effectively locked the database interactively for an extended period.)
> 
> We would prefer to rely on the user to stop anything that seems to
> have become stuck.  So set the timeout to 10ks.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 06/27] cr-daily-branch: Honour AP_FETCH_PFX

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> This environment variable is prefixed to all cr-daily-branch's command
> lines involving ap-fetch.  We are going to use this for memoisation.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 02/27] Executive: cs-job-create: Check that OSSTEST_FLIGHT is not set

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 02/27] Executive: cs-job-create: Check that OSSTEST_FLIGHT is not set"):
> On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> > If OSSTEST_FLIGHT is set on entry to JobDB::Executive::flight_create,
> > fail.  This obviously represents some kind of mistake.
> > 
> > One effect is that standalone-generate-dump-flight-runvars won't fill
> > an Executive database with test flights.
> 
> Speaking from experience there?

I managed to /mostly/ catch it because it seemed to be taking rather
long ...

Ian.

Re: [OSSTEST PATCH 07/27] memoise: New utility

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> Give this a GPLv2+ licence so that we can move it into some other
> FLOSS package later.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>
[...]

> +#    You should have received a copy of the GNU General Public License along
> +#    with this program; if not, write to the Free Software Foundation, Inc.,
> +#    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

These days http://www.gnu.org/licenses/gpl-howto.en.html recommends:

    You should have received a copy of the GNU General Public License
    along with Foobar.  If not, see <http://www.gnu.org/licenses/>.

FWIW.

Ian.

Re: [OSSTEST PATCH 08/27] standalone-generate-dump-flight-runvars: memoise ap-fetch

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> This makes it _much_ faster.
> 
> Also you can say AP_FETCH_MEMO_KEEP=1 to make it reuse old
> information, which is useful for making comparisons.
> 
> For a further speed improvement, use `eatmydata'.  This is not the

                                  ^one could/should ?

Because this patch doesn't do so (even as a non-default option) afaict.

> default because it risks corruption of `standalone.db' which is used
> for other purposes too.  Add a comment about possibly improving this.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Other than that:
Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 09/27] mfi-common: Use job_create_build for build-*-prev

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> This causes it to call job_create_build_filter_callback but has no
> ultimate functional change.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 10/27] ap-fetch-version*: Unwrap xen-unstable lines

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> Whitespace and \ change only.  Makes the next patch a bit clearer.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 11/27] ap-push: Use refs/heads/ for destinations

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> When the destination is a branch, specify refs/heads/ explicitly.
> This makes ap-push work even if the ref does not yet exist on the
> destination.
> 
> There is no functional change for an existing installation pushing to
> an existing branch.  But for a hypothetical new installation, this
> would be necessary.
> 
> And, more relevantly, when new "branches" are invented, the use of an
> existing ap-push case as a template will generate a new case which
> creates the branch as is necessary.
> 
> I leave the more complex osstest case alone.  It's not clear to me
> whether the destination ref not existing is an installation problem of
> such severity that indeed ap-push should fail.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 07/27] memoise: New utility

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 07/27] memoise: New utility"):
> These days http://www.gnu.org/licenses/gpl-howto.en.html recommends:
> 
>     You should have received a copy of the GNU General Public License
>     along with Foobar.  If not, see <http://www.gnu.org/licenses/>.

I will fix this.

Also of course this statement was, with my original patch, wrong,
because osstest.git didn't contain a copy of any version of the
vanilla GPL.  I have added a copy of GPL-3 to this patch.

Ian.

Re: [OSSTEST PATCH 12/27] cr-ensure-disk-space: Honour -D

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> Provide DEBUG and use db_prepare.  For now that is the only debugging
> output.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 08/27] standalone-generate-dump-flight-runvars: memoise ap-fetch

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 08/27] standalone-generate-dump-flight-runvars: memoise ap-fetch"):
> On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> > This makes it _much_ faster.
> > 
> > Also you can say AP_FETCH_MEMO_KEEP=1 to make it reuse old
> > information, which is useful for making comparisons.
> > 
> > For a further speed improvement, use `eatmydata'.  This is not the
> 
>                                   ^one could/should ?
> 
> Because this patch doesn't do so (even as a non-default option) afaict.

Indeed.  Wrong grammar.  Fixed.

Thanks,
Ian.

Re: [OSSTEST PATCH 07/27] memoise: New utility

[Ian Campbell]

On Wed, 2015-09-16 at 14:58 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [OSSTEST PATCH 07/27] memoise: New utility"):
> > These days http://www.gnu.org/licenses/gpl-howto.en.html recommends:
> > 
> >     You should have received a copy of the GNU General Public License
> >     along with Foobar.  If not, see <http://www.gnu.org/licenses/>.
> 
> I will fix this.
> 
> Also of course this statement was, with my original patch, wrong,
> because osstest.git didn't contain a copy of any version of the
> vanilla GPL.  I have added a copy of GPL-3 to this patch.

NB all the text in the patch talks about v2, FWIW...

> 
> Ian.

Re: [OSSTEST PATCH 13/27] cr-ensure-disk-space: Look at referring flights

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> Previously the flight to delete was simply the one with the lowest
> flight number.  Now we sort flights not by their own flight number,
> but by the highest flight number of any referencing flight.
> 
> This means that flights whose builds are being reused are kept as long
> as the reusing flights.
> 
> This almost-entirely fixes a largely-theoretical race in the way
> cs-bisection-step works (where the flight's logs and build outputs
> might be deleted between the setup and execution of the referring
> flight).
> 
> A smaller race still exists because the stash check in
> cs-bisection-step occurs before the being-created flight is visible to
> other db clients.  We will have to fix this by taking the flights
> lock.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 14/27] cr-ensure-disk-space: Make main loop body into a subroutine

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> No functional change.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 15/27] cr-ensure-disk-space: Take the flights db lock

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> This eliminates the race with cs-bisection-step (and other
> flight-construction tools which might reuse previous flights, provided
> that they also do not pass previous flight numbers from hand to hand
> with the db unlocked).
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>


> ---
>  cr-ensure-disk-space |   11 +++++++----
>  1 file changed, 7 insertions(+), 4 deletions(-)
> 
> diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
> index 8c5a708..138f3f4 100755
> --- a/cr-ensure-disk-space
> +++ b/cr-ensure-disk-space
> @@ -145,10 +145,13 @@ END
>      return 1;
>  }
>  
> -for (;;) {
> -    iteration_continue()
> -	or last;
> -}
> +db_retry($dbh_tests,[qw(flights)], sub {
> +    @flights = ();
> +    for (;;) {
> +	iteration_continue()
> +	    or last;
> +    }
> +});
>  
>  printf "ok.\n";
>  exit 0;

Re: [OSSTEST PATCH 16/27] cs-job-create: Permit creation of `synth' runvars

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> This will be useful if we want to set hostalloc_* runvars.  Normally
> we would want to set those only on flights generated by make-flight.
> 
> Flights generated by cs-bisection-step or cs-adjust-flight ought not
> to copy them, because cs-bisection-step makes its own arrangements for
> host specification, as should the caller of cs-adjust-flight (perhaps
> via the blessing system).
> 
> Using `synth' for this is arguably slightly wrong but it does the
> right thing in all existing cases.  The alternative would be a schema
> change.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

./standalone set-paths could usefully use this too I think.

Re: [OSSTEST PATCH 17/27] mg-show-flight-runvars: Decorate synth runvar names with ~

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> Make mg-show-flight-runvars -a append ~ to the names of synth runvars.
> (This is consistent with the new syntax in cs-job-create.)
> 
> We do this by editing $row->[1] (and $colws[1]) so we can avoid
> disturbing the general column format calcuation and printing; this

"calculation"

> means we have to copy $row (because the value in the DB $row is
> readonly).

fetchrow_array instead of fetchrow_arrayref would avoid the copy, at the
expense of removing -> from the usages (or taking the ref yourself).


> We have to check for $synth eq 'f' as well as $synth being boolean
> false, because SQLite's typeless nature (or, to put it another way,
> DBD::SQLite's failure to look at the schema) means that a boolean
> field's value of 'f' or 't' is simply returned as a string to Perl.
> But of course "f" is trueish in Perl.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Nonetheless:
    Acked-by: Ian Campbell <    ian.campbell@citrix.com    >

Re: [OSSTEST PATCH 18/27] standalone-generate-dump-flight-runvars: Show synth runvars

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> Pass -a to mg-show-flight-runvars.  That way when we use the new
> cs-job-create feature to set synth runvars during creation, we will
> see them in the dump.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 19/27] ts-host-alloc-Executive: Honour various hostalloc_* runvars

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> We honour
>   hostalloc_maxbonus_variation
>   hostalloc_bonus_previousfail
>   hostalloc_bonus_sharereuse
> and make them default to their previous values.

Took me a while to grok what was happening with $prevfail_equiv_bonus.
still:

> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 20/27] make-flight: Allow separate specification of pre-built Xen vs others

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> Make it possible to separately specify
>  - reuse previous flight's builds of xen.git
>  - reuse previous flight's builds of all other stuff
> 
> Specifically:
> 
>  * Match the 4th argument to make-flight with case rather than if.  If
>    it contains a slash, the first part is for Xen and the second part
>    for everything else.
> 
>  * Split the variable `bfi' into `bfi' and `xenbfi'
> 
>  * Introduce job_create_build_filter_callback, whose initial contents
>    check the recipe (to see what we're building) and then the relevant
>    bfi variable.  (NB that this is wrong for historical kernel builds
>    using the xen.git build system, which have recipe=build, but we do
>    not care about that.)
> 
> No functional change for existing call patterns.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 21/27] make-flight: Contemplate varying architecture for Debian HVM

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> In the Debian hvm tests, ask branch_debianhvm_arch for the
> architecture to use.  This currently prints only amd64.
> 
> While we are here, provide some comments about the (somewhat
> confusing) argument conventions of do_hvm_debian_test_one.
> 
> No functional change yet.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 22/27] make-flight: Run job_create_test_filter_callback on true job name

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> job_create_test would pass $job to job_create_test_filter_callback but
> then later maybe append -xsm to it.  Fix this.
> 
> No functional change for existing in-tree code because all existing
> tests of the $job end in *.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 23/27] make-flight: Provide xen-unstable-smoke branch

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> This contains a very limited set of jobs:
>     build-amd64
>     build-armhf
>     test-amd64-amd64-libvirt
>     test-amd64-amd64-xl-qemuu-debianhvm-i386
>     test-armhf-armhf-xl
> 
> The debianhvm job exists only in this flight, and is generated by
> having branch_debianhvm_arch return i386 instead of amd64.

any particular reason for this?

> 
> We override host allocator parameters to make this flight not care
> about host stickiness: it just takes whatever comes to hand.
> 
> Deployment note: This requires images/debian-7.2.0-i386-CD-1.iso
> which I have already downloaded to the Cambridge instance using jigdo.

But not the main colo?

> +case "$branch" in
> +xen-unstable-smoke)
> +	global_runvars+=" hostalloc_maxbonus_variation~=0 "
> +	global_runvars+=" hostalloc_bonus_previousfail~=0 "

Why are these synthetic? 

Re: [OSSTEST PATCH 24/27] make-flight: mfi-common: Honour $global_runvars

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> Provide a way for the main script to set some runvars on all jobs.  We
> expect this to be mostly set with +=.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

You've used this facility in #23, so maybe you want to reorder?

In any case:

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 23/27] make-flight: Provide xen-unstable-smoke branch

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> This contains a very limited set of jobs:
>     build-amd64
>     build-armhf
>     test-amd64-amd64-libvirt

Contrary to the previous ack -- this will need build-amd64-libvirt, won't
it?

Re: [OSSTEST PATCH 23/27] make-flight: Provide xen-unstable-smoke branch

[Ian Campbell]

On Wed, 2015-09-16 at 15:31 +0100, Ian Campbell wrote:
> On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> > This contains a very limited set of jobs:
> >     build-amd64
> >     build-armhf
> >     test-amd64-amd64-libvirt
> 
> Contrary to the previous ack -- this will need build-amd64-libvirt, won't
> it?

And what about kernels for both cases?

Re: [OSSTEST PATCH 25/27] ap-*: Be able to fetch and push xen.git#smoked

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> The branches and push gates are now:
>  xen.git#staging -[xen-unstable-smoke]-> #smoked -[xen-unstable]->
> #master
> 
> Deployment note: When this passes the osstest self-push-gate, the main
> xen-unstable flight will start using smoked as an input.  Therefore,
> until the new cronjob is installed to run the xen-unstable-smoke
> tests, an automatic process should keep xen.git#smoked up to date with
> xen.git#staging.  Eg, running in screen in xen@xenbits:~/git/xen.git:
>     while sleep 1800; do git fetch . staging:smoked; done
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 26/27] cri-common: Support xen-unstable-smoke

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> Add this branch to select_xenbranch.  This works like xen-unstable in
> most respects.  We are going to sort out the push gate ref plumbing in
> xen.git in the next osstest patch.
> 
> Also, use a branch-settings file to set its resource priority to -20
> to make it run ahead of anything else automatic.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 27/27] crontab: Enable xen-unstable-smoke branch every 3h at <1mod3>:51 UTC

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 27/27] crontab: Enable xen-unstable-smoke branch every 3h at <1mod3>:51 UTC

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:

> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH 07/27] memoise: New utility

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 07/27] memoise: New utility"):
> On Wed, 2015-09-16 at 14:58 +0100, Ian Jackson wrote:
> > Also of course this statement was, with my original patch, wrong,
> > because osstest.git didn't contain a copy of any version of the
> > vanilla GPL.  I have added a copy of GPL-3 to this patch.
> 
> NB all the text in the patch talks about v2, FWIW...

You mean it talks about `version 2 or, at your option, any later
version'.  Clearly we don't want to be pushing GPL-2 at this stage, so
GPL-3 is the right licence version text to include.  But likewise we
don't want to rule out distribution under GPL-2.

Ian.

Re: [OSSTEST PATCH 16/27] cs-job-create: Permit creation of `synth' runvars

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 16/27] cs-job-create: Permit creation of `synth' runvars"):
> On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> > This will be useful if we want to set hostalloc_* runvars.  Normally
> > we would want to set those only on flights generated by make-flight.
> 
> Acked-by: Ian Campbell <ian.campbell@citrix.com>

Thanks.

> ./standalone set-paths could usefully use this too I think.

Could it ?  I don't think I would object.  I never use set-paths...

Ian.

Re: [OSSTEST PATCH 07/27] memoise: New utility

[Ian Campbell]

On Wed, 2015-09-16 at 15:38 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [OSSTEST PATCH 07/27] memoise: New utility"):
> > On Wed, 2015-09-16 at 14:58 +0100, Ian Jackson wrote:
> > > Also of course this statement was, with my original patch, wrong,
> > > because osstest.git didn't contain a copy of any version of the
> > > vanilla GPL.  I have added a copy of GPL-3 to this patch.
> > 
> > NB all the text in the patch talks about v2, FWIW...
> 
> You mean it talks about `version 2 or, at your option, any later
> version'.  Clearly we don't want to be pushing GPL-2 at this stage, so
> GPL-3 is the right licence version text to include.  But likewise we
> don't want to rule out distribution under GPL-2.

Yep, just wanted to check it was intentional.

Re: [OSSTEST PATCH 17/27] mg-show-flight-runvars: Decorate synth runvar names with ~

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 17/27] mg-show-flight-runvars: Decorate synth runvar names with ~"):
> On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> > Make mg-show-flight-runvars -a append ~ to the names of synth runvars.
> > (This is consistent with the new syntax in cs-job-create.)
> > 
> > We do this by editing $row->[1] (and $colws[1]) so we can avoid
> > disturbing the general column format calcuation and printing; this
> 
> "calculation"

Fixed.

> > means we have to copy $row (because the value in the DB $row is
> > readonly).
> 
> fetchrow_array instead of fetchrow_arrayref would avoid the copy, at the
> expense of removing -> from the usages (or taking the ref yourself).

Yes.  I have done that and it's nicer.

Ian.

Re: [OSSTEST PATCH 16/27] cs-job-create: Permit creation of `synth' runvars

[Ian Campbell]

On Wed, 2015-09-16 at 15:41 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [OSSTEST PATCH 16/27] cs-job-create: Permit
> creation of `synth' runvars"):
> > On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> > > This will be useful if we want to set hostalloc_* runvars.  Normally
> > > we would want to set those only on flights generated by make-flight.
> > 
> > Acked-by: Ian Campbell <ian.campbell@citrix.com>
> 
> Thanks.
> 
> > ./standalone set-paths could usefully use this too I think.
> 
> Could it ?  I don't think I would object.  I never use set-paths...

IIRC if you use set-paths and then subsequently have a need to actually re
-run the job then it barfs trying to write the new runvars for path etc.

Ian.

Re: [OSSTEST PATCH 23/27] make-flight: Provide xen-unstable-smoke branch

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 23/27] make-flight: Provide xen-unstable-smoke branch"):
> On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> > This contains a very limited set of jobs:
> >     build-amd64
> >     build-armhf
> >     test-amd64-amd64-libvirt
> >     test-amd64-amd64-xl-qemuu-debianhvm-i386
> >     test-armhf-armhf-xl
> > 
> > The debianhvm job exists only in this flight, and is generated by
> > having branch_debianhvm_arch return i386 instead of amd64.
> 
> any particular reason for this?

I wanted to test at least one 32-bit guest.  I have added a note about
this to the commit message.
> 
> > Deployment note: This requires images/debian-7.2.0-i386-CD-1.iso
> > which I have already downloaded to the Cambridge instance using jigdo.
> 
> But not the main colo?

Not yet.  I will do that before I start trying to run this there.  I
think I ought to do such a --real run, by hand, when the series seems
about ready to go into pretest.

> > +case "$branch" in
> > +xen-unstable-smoke)
> > +	global_runvars+=" hostalloc_maxbonus_variation~=0 "
> > +	global_runvars+=" hostalloc_bonus_previousfail~=0 "
> 
> Why are these synthetic? 

I wrote in the commit message that introduced the ~ notation in
cs-job-create:

    This will be useful if we want to set hostalloc_* runvars.  Normally
    we would want to set those only on flights generated by make-flight.
    
    Flights generated by cs-bisection-step or cs-adjust-flight ought not
    to copy them, because cs-bisection-step makes its own arrangements for
    host specification, as should the caller of cs-adjust-flight (perhaps
    via the blessing system).
    
    Using `synth' for this is arguably slightly wrong but it does the
    right thing in all existing cases.  The alternative would be a schema
    change.

Perhaps I should move some of this text to this commit message.

Ian.

Re: [OSSTEST PATCH 24/27] make-flight: mfi-common: Honour $global_runvars

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 24/27] make-flight: mfi-common: Honour $global_runvars"):
> On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> > Provide a way for the main script to set some runvars on all jobs.  We
> > expect this to be mostly set with +=.
> 
> You've used this facility in #23, so maybe you want to reorder?

Oops, yes, I will swap this with #23.

> In any case:
> 
> Acked-by: Ian Campbell <ian.campbell@citrix.com>

Thanks,
Ian.

Re: [OSSTEST PATCH 23/27] make-flight: Provide xen-unstable-smoke branch

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 23/27] make-flight: Provide xen-unstable-smoke branch"):
> On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> > This contains a very limited set of jobs:
> >     build-amd64
> >     build-armhf
> >     test-amd64-amd64-libvirt
> 
> Contrary to the previous ack -- this will need build-amd64-libvirt, won't
> it?

The intent was to reuse previous build jobs for everything.  That's
what
  make-flight: Allow separate specification of pre-built Xen vs others
was for.  But I don't in fact seem to have produced any patch which
actually does that!

Ian.

Re: [OSSTEST PATCH 23/27] make-flight: Provide xen-unstable-smoke branch

[Ian Campbell]

On Wed, 2015-09-16 at 15:51 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [OSSTEST PATCH 23/27] make-flight: Provide xen
> -unstable-smoke branch"):
> > On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> > > This contains a very limited set of jobs:
> > >     build-amd64
> > >     build-armhf
> > >     test-amd64-amd64-libvirt
> > >     test-amd64-amd64-xl-qemuu-debianhvm-i386
> > >     test-armhf-armhf-xl
> > > 
> > > The debianhvm job exists only in this flight, and is generated by
> > > having branch_debianhvm_arch return i386 instead of amd64.
> > 
> > any particular reason for this?
> 
> I wanted to test at least one 32-bit guest.  I have added a note about
> this to the commit message.

OK.

I wonder if we should also add this to the regular flights? Seems odd to
smoke test it for unstable but not for e.g. real 4.6 flights.

> > > Deployment note: This requires images/debian-7.2.0-i386-CD-1.iso
> > > which I have already downloaded to the Cambridge instance using
> > > jigdo.
> > 
> > But not the main colo?
> 
> Not yet.  I will do that before I start trying to run this there.  I
> think I ought to do such a --real run, by hand, when the series seems
> about ready to go into pretest.
> 
> > > +case "$branch" in
> > > +xen-unstable-smoke)
> > > +	global_runvars+=" hostalloc_maxbonus_variation~=0 "
> > > +	global_runvars+=" hostalloc_bonus_previousfail~=0 "
> > 
> > Why are these synthetic? 
> 
> I wrote in the commit message that introduced the ~ notation in
> cs-job-create:
> 
>     This will be useful if we want to set hostalloc_* runvars.  Normally
>     we would want to set those only on flights generated by make-flight.
>     
>     Flights generated by cs-bisection-step or cs-adjust-flight ought not
>     to copy them, because cs-bisection-step makes its own arrangements
> for
>     host specification, as should the caller of cs-adjust-flight (perhaps
>     via the blessing system).
>     
>     Using `synth' for this is arguably slightly wrong but it does the
>     right thing in all existing cases.  The alternative would be a schema
>     change.
> 
> Perhaps I should move some of this text to this commit message.

Yes, I'd forgotten all about it by now, which I think it is when it is
relevant.

> 
> Ian.

Re: [OSSTEST PATCH 23/27] make-flight: Provide xen-unstable-smoke branch

[Ian Campbell]

On Wed, 2015-09-16 at 15:57 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [OSSTEST PATCH 23/27] make-flight: Provide xen
> -unstable-smoke branch"):
> > On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> > > This contains a very limited set of jobs:
> > >     build-amd64
> > >     build-armhf
> > >     test-amd64-amd64-libvirt
> > 
> > Contrary to the previous ack -- this will need build-amd64-libvirt,
> > won't
> > it?
> 
> The intent was to reuse previous build jobs for everything.  That's
> what
>   make-flight: Allow separate specification of pre-built Xen vs others
> was for.

Right, makes sense.

>   But I don't in fact seem to have produced any patch which
> actually does that!

Indeed!

You would also need to handle the case where the was no build remaining
which you could use (it have might been gc'd between two smoke runs)

> 
> Ian.

Re: [OSSTEST PATCH 00/27] xen.git#staging smoke tests

[Ian Campbell]

On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> One outstanding bikeshed question about naming: I have called the job
> `xen-unstable-smoke' and the output ref xen.git#smoked.  I tripped
> over the presence or absence of the `d' a few times.  Should one of
> these two be renamed, and if so which ?

Yes, having the "doing" name vs the "done" name does lead to this.

I'm not sure, the disconnect seems unfortunate, but either of the solutions
are clumsy when reading one context or the other.

Leave it?


> 
> I have verified that this series from
>  "standalone-generate-dump-flight-runvars: memoise ap-fetch"
> onwards generates no unexpected changes to any runvars.  The
> diff -bu (from there to tip) in the output from
>   OSSTEST_CONFIG=standalone-config-example \
>   ./standalone-generate-dump-flight-runvars
> is included below.  At the time of running this, xen.git had
>   340ade2f0ef692efce01a708ad98629af985f978 refs/heads/smoked
>   26646f34c31f663fa23076ed0433195edec50d9f refs/heads/staging
> and the only change to the existing flights' runvars is that
> xen-unstable's non-prev revision_xen values referring to staging now
> refer to smoked instead.
> 
> --- ../play-testing.git/before	2015-09-16 14:07:53.493236378 +0100
> +++ after	2015-09-16 14:08:05.577518049 +0100
> @@ -23100,16 +23100,16 @@
>  xen-unstable               build-armhf-xsm                              
>          revision_seabios                                                
>                           
>  xen-unstable               build-i386                                   
>          revision_seabios                                                
>                           
>  xen-unstable               build-i386-xsm                               
>          revision_seabios                                                
>                           
> -xen-unstable               build-amd64                                  
>          revision_xen               
>  26646f34c31f663fa23076ed0433195edec50d9f                      
> +xen-unstable               build-amd64                                  
>          revision_xen                
>  340ade2f0ef692efce01a708ad98629af985f978                      
>  xen-unstable               build-amd64-prev                             
>          revision_xen               
>  a7b39c8bd6cba3fe1c8012987b9e28bdbac7e92d                      
> -xen-unstable               build-amd64-rumpuserxen                      
>          revision_xen               
>  26646f34c31f663fa23076ed0433195edec50d9f                      
> -xen-unstable               build-amd64-xsm                              
>          revision_xen               
>  26646f34c31f663fa23076ed0433195edec50d9f                      
> -xen-unstable               build-armhf                                  
>          revision_xen               
>  26646f34c31f663fa23076ed0433195edec50d9f                      
> -xen-unstable               build-armhf-xsm                              
>          revision_xen               
>  26646f34c31f663fa23076ed0433195edec50d9f                      
> -xen-unstable               build-i386                                   
>          revision_xen               
>  26646f34c31f663fa23076ed0433195edec50d9f                      
> +xen-unstable               build-amd64-rumpuserxen                      
>          revision_xen                
>  340ade2f0ef692efce01a708ad98629af985f978                      
> +xen-unstable               build-amd64-xsm                              
>          revision_xen                
>  340ade2f0ef692efce01a708ad98629af985f978                      
> +xen-unstable               build-armhf                                  
>          revision_xen                
>  340ade2f0ef692efce01a708ad98629af985f978                      
> +xen-unstable               build-armhf-xsm                              
>          revision_xen                
>  340ade2f0ef692efce01a708ad98629af985f978                      
> +xen-unstable               build-i386                                   
>          revision_xen                
>  340ade2f0ef692efce01a708ad98629af985f978                      
>  xen-unstable               build-i386-prev                              
>          revision_xen               
>  a7b39c8bd6cba3fe1c8012987b9e28bdbac7e92d                      
> -xen-unstable               build-i386-rumpuserxen                       
>          revision_xen               
>  26646f34c31f663fa23076ed0433195edec50d9f                      
> -xen-unstable               build-i386-xsm                               
>          revision_xen               
>  26646f34c31f663fa23076ed0433195edec50d9f                      
> +xen-unstable               build-i386-rumpuserxen                       
>          revision_xen                
>  340ade2f0ef692efce01a708ad98629af985f978                      
> +xen-unstable               build-i386-xsm                               
>          revision_xen                
>  340ade2f0ef692efce01a708ad98629af985f978                      
>  xen-unstable               test-amd64-amd64-rumpuserxen-amd64           
>          rump_builtimage             rumpuserxen:/usr/local/lib/xen/rump
> -kernel/rump-kernel        
>  xen-unstable               test-amd64-i386-rumpuserxen-i386             
>          rump_builtimage             rumpuserxen:/usr/local/lib/xen/rump
> -kernel/rump-kernel        
>  xen-unstable               test-amd64-amd64-rumpuserxen-amd64           
>          rump_cmdline                3                                   
>                           
> @@ -23365,3 +23365,77 @@
>  xen-unstable               test-amd64-i386-rumpuserxen-i386             
>          xenstorels_builtimage       :/usr/local/bin/xenstore            
>                           
>  xen-unstable               test-amd64-amd64-rumpuserxen-amd64           
>          xenstorels_cmdline          ls -fp device                       
>                           
>  xen-unstable               test-amd64-i386-rumpuserxen-i386             
>          xenstorels_cmdline          ls -fp device                       
>                           
> +xen-unstable-smoke         test-amd64-amd64-libvirt                
>  all_hostflags                 arch-amd64,arch-xen-amd64,suite
> -wheezy,purpose-test           
> +xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386
> all_hostflags                 arch-amd64,arch-xen-amd64,suite
> -wheezy,purpose-test,hvm       
> +xen-unstable-smoke         test-armhf-armhf-xl                     
>  all_hostflags                 arch-armhf,arch-xen-armhf,suite
> -wheezy,purpose-test           
> +xen-unstable-smoke         build-amd64                              arch
>                           amd64                                          
>                
> +xen-unstable-smoke         build-armhf                              arch
>                           armhf                                          
>                
> +xen-unstable-smoke         test-amd64-amd64-libvirt                 arch
>                           amd64                                          
>                
> +xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 arch
>                           amd64                                          
>                
> +xen-unstable-smoke         test-armhf-armhf-xl                      arch
>                           armhf                                          
>                
> +xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386 bios
>                           seabios                                        
>                
> +xen-unstable-smoke         build-amd64                             
>  build_lvextend_max            50                                        
>                     
> +xen-unstable-smoke         build-armhf                             
>  build_lvextend_max            50                                        
>                     
> +xen-unstable-smoke         test-amd64-amd64-libvirt                
>  buildjob                      build-amd64                               
>                     
> +xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386
> buildjob                      build-amd64                                
>                    
> +xen-unstable-smoke         test-armhf-armhf-xl                     
>  buildjob                      build-armhf                               
>                     
> +xen-unstable-smoke         test-amd64-amd64-libvirt                
>  debian_arch                   amd64                                     
>                     
> +xen-unstable-smoke         test-armhf-armhf-xl                     
>  debian_arch                   armhf                                     
>                     
> +xen-unstable-smoke         test-amd64-amd64-libvirt                
>  debian_kernkind               pvops                                     
>                     
> +xen-unstable-smoke         test-armhf-armhf-xl                     
>  debian_kernkind               pvops                                     
>                     
> +xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386
> debianhvm_image               debian-7.2.0-i386-CD-1.iso                 
>                    
> +xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386
> device_model_version          qemu-xen                                   
>                    
> +xen-unstable-smoke         build-amd64                             
>  enable_ovmf                   true                                      
>                     
> +xen-unstable-smoke         build-armhf                             
>  enable_ovmf                   true                                      
>                     
> +xen-unstable-smoke         build-amd64                             
>  enable_xend                   false                                     
>                     
> +xen-unstable-smoke         build-armhf                             
>  enable_xend                   false                                     
>                     
> +xen-unstable-smoke         build-amd64                             
>  enable_xsm                    false                                     
>                     
> +xen-unstable-smoke         build-armhf                             
>  enable_xsm                    false                                     
>                     
> +xen-unstable-smoke         test-amd64-amd64-libvirt                
>  enable_xsm                    false                                     
>                     
> +xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386
> enable_xsm                    false                                      
>                    
> +xen-unstable-smoke         test-armhf-armhf-xl                     
>  enable_xsm                    false                                     
>                     
> +xen-unstable-smoke         build-amd64                             
>  host_hostflags                share-build-wheezy-amd64,arch-amd64,suite
> -wheezy,purpose-build
> +xen-unstable-smoke         build-armhf                             
>  host_hostflags                share-build-wheezy-armhf,arch-armhf,suite
> -wheezy,purpose-build
> +xen-unstable-smoke         test-amd64-amd64-libvirt                
>  kernbuildjob                  build-amd64-pvops                         
>                     
> +xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386
> kernbuildjob                  build-amd64-pvops                          
>                    
> +xen-unstable-smoke         test-armhf-armhf-xl                     
>  kernbuildjob                  build-armhf-pvops                         
>                     
> +xen-unstable-smoke         test-amd64-amd64-libvirt                
>  kernkind                      pvops                                     
>                     
> +xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386
> kernkind                      pvops                                      
>                    
> +xen-unstable-smoke         test-armhf-armhf-xl                     
>  kernkind                      pvops                                     
>                     
> +xen-unstable-smoke         test-amd64-amd64-libvirt                
>  libvirtbuildjob               build-amd64-libvirt                       
>                     
> +xen-unstable-smoke         build-amd64                             
>  revision_ovmf                                                           
>                     
> +xen-unstable-smoke         build-armhf                             
>  revision_ovmf                                                           
>                     
> +xen-unstable-smoke         build-amd64                             
>  revision_qemu                                                           
>                     
> +xen-unstable-smoke         build-armhf                             
>  revision_qemu                                                           
>                     
> +xen-unstable-smoke         build-amd64                             
>  revision_qemuu                b05befcbea71a979509ce04f02929969a790c923  
>                     
> +xen-unstable-smoke         build-armhf                             
>  revision_qemuu                b05befcbea71a979509ce04f02929969a790c923  
>                     
> +xen-unstable-smoke         build-amd64                             
>  revision_seabios                                                        
>                     
> +xen-unstable-smoke         build-armhf                             
>  revision_seabios                                                        
>                     
> +xen-unstable-smoke         build-amd64                             
>  revision_xen                  a7b39c8bd6cba3fe1c8012987b9e28bdbac7e92d  
>                     
> +xen-unstable-smoke         build-armhf                             
>  revision_xen                  a7b39c8bd6cba3fe1c8012987b9e28bdbac7e92d  
>                     
> +xen-unstable-smoke         test-amd64-amd64-libvirt                
>  toolstack                     libvirt                                   
>                     
> +xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386
> toolstack                     xl                                         
>                    
> +xen-unstable-smoke         test-armhf-armhf-xl                     
>  toolstack                     xl                                        
>                     
> +xen-unstable-smoke         build-amd64                             
>  tree_ovmf                                                               
>                     
> +xen-unstable-smoke         build-armhf                             
>  tree_ovmf                                                               
>                     
> +xen-unstable-smoke         build-amd64                             
>  tree_qemu                     git://xenbits.xen.org/staging/qemu-xen
> -unstable.git           
> +xen-unstable-smoke         build-armhf                             
>  tree_qemu                     git://xenbits.xen.org/staging/qemu-xen
> -unstable.git           
> +xen-unstable-smoke         build-amd64                             
>  tree_qemuu                    git://xenbits.xen.org/staging/qemu
> -upstream-unstable.git      
> +xen-unstable-smoke         build-armhf                             
>  tree_qemuu                    git://xenbits.xen.org/staging/qemu
> -upstream-unstable.git      
> +xen-unstable-smoke         build-amd64                             
>  tree_seabios                                                            
>                     
> +xen-unstable-smoke         build-armhf                             
>  tree_seabios                                                            
>                     
> +xen-unstable-smoke         build-amd64                             
>  tree_xen                      git://xenbits.xen.org/xen.git             
>                     
> +xen-unstable-smoke         build-armhf                             
>  tree_xen                      git://xenbits.xen.org/xen.git             
>                     
> +xen-unstable-smoke         test-amd64-amd64-libvirt                
>  xenbuildjob                   build-amd64                               
>                     
> +xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386
> xenbuildjob                   build-amd64                                
>                    
> +xen-unstable-smoke         test-armhf-armhf-xl                     
>  xenbuildjob                   build-armhf                               
>                     
> +xen-unstable-smoke         build-amd64                             
>  hostalloc_bonus_previousfail~ 0                                         
>                     
> +xen-unstable-smoke         build-armhf                             
>  hostalloc_bonus_previousfail~ 0                                         
>                     
> +xen-unstable-smoke         test-amd64-amd64-libvirt                
>  hostalloc_bonus_previousfail~ 0                                         
>                     
> +xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386
> hostalloc_bonus_previousfail~ 0                                          
>                    
> +xen-unstable-smoke         test-armhf-armhf-xl                     
>  hostalloc_bonus_previousfail~ 0                                         
>                     
> +xen-unstable-smoke         build-amd64                             
>  hostalloc_maxbonus_variation~ 0                                         
>                     
> +xen-unstable-smoke         build-armhf                             
>  hostalloc_maxbonus_variation~ 0                                         
>                     
> +xen-unstable-smoke         test-amd64-amd64-libvirt                
>  hostalloc_maxbonus_variation~ 0                                         
>                     
> +xen-unstable-smoke         test-amd64-amd64-xl-qemuu-debianhvm-i386
> hostalloc_maxbonus_variation~ 0                                          
>                    
> +xen-unstable-smoke         test-armhf-armhf-xl                     
>  hostalloc_maxbonus_variation~ 0                                         
>                     
> 

Re: [OSSTEST PATCH 00/27] xen.git#staging smoke tests

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 00/27] xen.git#staging smoke tests"):
> On Wed, 2015-09-16 at 14:35 +0100, Ian Jackson wrote:
> > One outstanding bikeshed question about naming: I have called the job
> > `xen-unstable-smoke' and the output ref xen.git#smoked.  I tripped
> > over the presence or absence of the `d' a few times.  Should one of
> > these two be renamed, and if so which ?
> 
> Yes, having the "doing" name vs the "done" name does lead to this.
> 
> I'm not sure, the disconnect seems unfortunate, but either of the solutions
> are clumsy when reading one context or the other.

Changing the osstest branch name (flight report title) to
`xen-unstable-smoked' would clearly be wrong.

I'm tempted to change the xen.git ref name to `smoke' to match.  It
also means you can say entertaining things like `are you using the
latest smoke' ?

Ian.