[OSSTEST PATCH 1/6] rumpuserxen: Increase build timeout for grievous bodge

[OSSTEST PATCH 1/6] rumpuserxen: Increase build timeout for grievous bodge

[Ian Jackson]

This involves copying 90Mby of data.  If the test machine is busy (in
the worst case, running the kernel side of resize2fs which due to
Linux's poor IO scheduling hogs the entire IO system) it can take a
long time.  In flight 26359 it was still running after 280s, when the
machine was released.

Increase the timeout to 2000s.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 ts-rumpuserxen-build |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ts-rumpuserxen-build b/ts-rumpuserxen-build
index e6adc75..3f72719 100755
--- a/ts-rumpuserxen-build
+++ b/ts-rumpuserxen-build
@@ -67,7 +67,7 @@ sub massage() {
 	    print EO or die $!;
 	}
     });
-    target_cmd_build($ho, 200, $rux, "bash -x grievous-bodge-nblibs");
+    target_cmd_build($ho, 2000, $rux, "bash -x grievous-bodge-nblibs");
 
     # Rather poor
     target_editfile($ho, "$rux/Config.mk", sub {
-- 
1.7.10.4

[OSSTEST PATCH 2/6] crontab: Rearrange actually-daily branches

[Ian Jackson]

Make them start at roughly the same time, by using cr-for-branches for
them all in one go.  That increases the extent to which build hosts
will be reused (saving on reinstallations), and hence also the extent
to which the build hosts' ccache will be effective (saving on cpu).

Set them off at 04:18 UK time (currently equivalent to 21:18 Pacific
time), in the hope that at least some of them might have produced
answers during following the UK working day.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 crontab |    8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/crontab b/crontab
index 5a621e2..99e45bc 100644
--- a/crontab
+++ b/crontab
@@ -4,11 +4,7 @@ MAILTO=ian.jackson@citrix.com,ian.campbell@eu.citrix.com
 49		1	* * *		cd testing.git && BRANCHES_ALWAYS=xen-unstable	./cr-for-branches branches -w "./cr-daily-branch --real"
 4-59/30		*	* * *		cd testing.git &&				./cr-for-branches branches -q "./cr-daily-branch --real"
 18		9	* * 1-5		cd testing.git && BRANCHES=linux-next		./cr-for-branches branches -w "./cr-daily-branch --real"
-18		10	* * *		cd testing.git && BRANCHES=linux-linus		./cr-for-branches branches -w "./cr-daily-branch --real"
-18		11	* * *		cd testing.git && BRANCHES=linux-mingo-tip-master ./cr-for-branches branches -w "./cr-daily-branch --real"
-18		12	* * *		cd testing.git && BRANCHES=linux-3.0		./cr-for-branches branches -w "./cr-daily-branch --real"
-18		13	* * *		cd testing.git && BRANCHES=libvirt		./cr-for-branches branches -w "./cr-daily-branch --real"
-18		14	* * *		cd testing.git && BRANCHES=rumpuserxen		./cr-for-branches branches -w "./cr-daily-branch --real"
+18		4	* * *		cd testing.git && BRANCHES='linux-linus linux-mingo-tip-master linux-3.0 libvirt rumpuserxen' ./cr-for-branches branches -w "./cr-daily-branch --real"
 6-59/15   	*	* * *		cd testing.git && EXTRA_BRANCHES='linux-linus linux-3.0 rumpuserxen libvirt' ./cr-for-branches bisects -w "./cr-try-bisect --real"
 #8-59/5		*	* * *		cd bisects/adhoc.git &&	with-lock-ex -q data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
-3		5	* * *		savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
+3		4	* * *		savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
-- 
1.7.10.4

[OSSTEST PATCH 3/6] make-flight: rumpuserxen amd64 too

[Ian Jackson]

Build and test amd64 rump kernel.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 make-flight |    5 +++--
 mfi-common  |    2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/make-flight b/make-flight
index f812a2f..ba1bc5b 100755
--- a/make-flight
+++ b/make-flight
@@ -292,13 +292,14 @@ test_matrix_do_one () {
   *)               test_xend=n ;;
   esac
 
-  for rumparch in i386; do
+  rumparch=$dom0arch
+  if true; then
       job_create_test test-$xenarch$kern-$dom0arch-rumpuserxen-$rumparch \
                       test-rumpuserxen xl \
             $xenarch $dom0arch                                       \
             rump_rumpuserxenbuildjob=build-$rumparch-rumpuserxen     \
             all_hostflags=$most_hostflags
-  done
+  fi
 
   # xend PV guest test on x86 only
   if [ x$test_xend = xy -a \( $dom0arch = "i386" -o $dom0arch = "amd64" \) ]; then
diff --git a/mfi-common b/mfi-common
index b08a675..e73da67 100644
--- a/mfi-common
+++ b/mfi-common
@@ -181,7 +181,7 @@ create_build_jobs () {
     fi
 
     case $arch in
-    i386)
+    i386 amd64)
     ./cs-job-create $flight build-$arch-rumpuserxen build-rumpuserxen        \
                 arch=$arch                                                   \
         tree_xen=$TREE_XEN                                                   \
-- 
1.7.10.4

Re: [OSSTEST PATCH 3/6] make-flight: rumpuserxen amd64 too

[Ian Campbell]

On Wed, 2014-05-21 at 12:42 +0100, Ian Jackson wrote:
> Build and test amd64 rump kernel.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> ---
>  make-flight |    5 +++--
>  mfi-common  |    2 +-
>  2 files changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/make-flight b/make-flight
> index f812a2f..ba1bc5b 100755
> --- a/make-flight
> +++ b/make-flight
> @@ -292,13 +292,14 @@ test_matrix_do_one () {
>    *)               test_xend=n ;;
>    esac
>  
> -  for rumparch in i386; do
> +  rumparch=$dom0arch
> +  if true; then

Intentional or left over debug?

>        job_create_test test-$xenarch$kern-$dom0arch-rumpuserxen-$rumparch \
>                        test-rumpuserxen xl \
>              $xenarch $dom0arch                                       \
>              rump_rumpuserxenbuildjob=build-$rumparch-rumpuserxen     \
>              all_hostflags=$most_hostflags
> -  done
> +  fi
>  
>    # xend PV guest test on x86 only
>    if [ x$test_xend = xy -a \( $dom0arch = "i386" -o $dom0arch = "amd64" \) ]; then
> diff --git a/mfi-common b/mfi-common
> index b08a675..e73da67 100644
> --- a/mfi-common
> +++ b/mfi-common
> @@ -181,7 +181,7 @@ create_build_jobs () {
>      fi
>  
>      case $arch in
> -    i386)
> +    i386 amd64)
>      ./cs-job-create $flight build-$arch-rumpuserxen build-rumpuserxen        \
>                  arch=$arch                                                   \
>          tree_xen=$TREE_XEN                                                   \

Re: [OSSTEST PATCH 3/6] make-flight: rumpuserxen amd64 too

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 3/6] make-flight: rumpuserxen amd64 too"):
> On Wed, 2014-05-21 at 12:42 +0100, Ian Jackson wrote:
> > -  for rumparch in i386; do
> > +  rumparch=$dom0arch
> > +  if true; then
> 
> Intentional or left over debug?

Intentional.  I anticipate that we might want to filter these four
jobs in the future, or iterate over different dom0s.  But if you
object I'll take it out (and reindent).

Ian.

Re: [OSSTEST PATCH 3/6] make-flight: rumpuserxen amd64 too

[Ian Campbell]

On Wed, 2014-05-21 at 14:52 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [OSSTEST PATCH 3/6] make-flight: rumpuserxen amd64 too"):
> > On Wed, 2014-05-21 at 12:42 +0100, Ian Jackson wrote:
> > > -  for rumparch in i386; do
> > > +  rumparch=$dom0arch
> > > +  if true; then
> > 
> > Intentional or left over debug?
> 
> Intentional.  I anticipate that we might want to filter these four
> jobs in the future, or iterate over different dom0s.  But if you
> object I'll take it out (and reindent).

Leaving it is fine by me.

Ian.

[OSSTEST PATCH 4/6] FIX MAKE-FLIGHT

[Ian Jackson]

---
 mfi-common |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mfi-common b/mfi-common
index e73da67..d1b2737 100644
--- a/mfi-common
+++ b/mfi-common
@@ -181,7 +181,7 @@ create_build_jobs () {
     fi
 
     case $arch in
-    i386 amd64)
+    i386|amd64)
     ./cs-job-create $flight build-$arch-rumpuserxen build-rumpuserxen        \
                 arch=$arch                                                   \
         tree_xen=$TREE_XEN                                                   \
-- 
1.7.10.4

Re: [OSSTEST PATCH 4/6] FIX MAKE-FLIGHT

[Ian Campbell]

Did you mean to fold this into the predecessor?

On Wed, 2014-05-21 at 12:42 +0100, Ian Jackson wrote:
> ---
>  mfi-common |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/mfi-common b/mfi-common
> index e73da67..d1b2737 100644
> --- a/mfi-common
> +++ b/mfi-common
> @@ -181,7 +181,7 @@ create_build_jobs () {
>      fi
>  
>      case $arch in
> -    i386 amd64)
> +    i386|amd64)
>      ./cs-job-create $flight build-$arch-rumpuserxen build-rumpuserxen        \
>                  arch=$arch                                                   \
>          tree_xen=$TREE_XEN                                                   \

Re: [OSSTEST PATCH 4/6] FIX MAKE-FLIGHT

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 4/6] FIX MAKE-FLIGHT"):
> Did you mean to fold this into the predecessor?

Oops, yes.  When sending it, I forgot that I hadn't squashed the
series.

Ian.

[OSSTEST PATCH 5/6] TestSupport::broken: Update status of `preparing' jobs

[Ian Jackson]

If the current job is preparing, and we call broken, then we should
update the status to that supplied by broken's caller.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index e906dff..4187b6c 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -206,7 +206,7 @@ sub broken ($;$) {
         $affected= $dbh_tests->do(<<END, {}, $newst, $flight, $job);
             UPDATE jobs SET status=?
              WHERE flight=? AND job=?
-               AND (status='queued' OR status='running')
+               AND (status='queued' OR status='running' OR status='preparing')
 END
     });
     die "BROKEN: $m; ". ($affected>0 ? "marked $flight.$job $newst"
-- 
1.7.10.4

[OSSTEST PATCH 6/6] sg-run-job: Do blocked check globally, in run-job

[Ian Jackson]

Only run-job can fully abandon the whole job.  With the previous
arrangements (in bc63a901), a blocked build job would go on to try to
run ts-logs-capture and become broken (rather than blocked).

The downside of this is that it is not possible in the future to
introduce a kind of job which can selectively carry on if some of its
dependencies fail.  This notion is IMO an acceptable casualty.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 sg-run-job |    9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/sg-run-job b/sg-run-job
index c1dfa86..e903869 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -39,9 +39,10 @@ proc run-job {job} {
         set need_build_host 0
     }
 
+    catching-otherwise blocked            check-not-blocked
+    if {!$ok} return
+
     if {[llength $need_xen_hosts]} {
-               catching-otherwise blocked   check-not-blocked
-	if {!$ok} return
         eval run-ts broken  =             ts-hosts-allocate + $need_xen_hosts
     }
 
@@ -340,9 +341,7 @@ proc run-job/build-rumpuserxen {} {
 }
 
 proc prepare-build-host {} {
-    global jobinfo ok
-    catching-otherwise blocked check-not-blocked
-    if {!$ok} return
+    global jobinfo
     run-ts broken = ts-hosts-allocate + host
     run-ts broken host-install(*) ts-host-install-twice
     run-ts . host-build-prep ts-xen-build-prep
-- 
1.7.10.4

Re: [OSSTEST PATCH 6/6] sg-run-job: Do blocked check globally, in run-job

[Ian Campbell]

On Wed, 2014-05-21 at 12:42 +0100, Ian Jackson wrote:
> Only run-job can fully abandon the whole job.  With the previous
> arrangements (in bc63a901), a blocked build job would go on to try to
> run ts-logs-capture and become broken (rather than blocked).
> 
> The downside of this is that it is not possible in the future to
> introduce a kind of job which can selectively carry on if some of its
> dependencies fail.  This notion is IMO an acceptable casualty.

Just to be clear: This doesn't exclude the existing behaviour of
ts-migrate-support-check, does it? (I think not because you talk about
jobs and this is a step)

Ian.

> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> ---
>  sg-run-job |    9 ++++-----
>  1 file changed, 4 insertions(+), 5 deletions(-)
> 
> diff --git a/sg-run-job b/sg-run-job
> index c1dfa86..e903869 100755
> --- a/sg-run-job
> +++ b/sg-run-job
> @@ -39,9 +39,10 @@ proc run-job {job} {
>          set need_build_host 0
>      }
>  
> +    catching-otherwise blocked            check-not-blocked
> +    if {!$ok} return
> +
>      if {[llength $need_xen_hosts]} {
> -               catching-otherwise blocked   check-not-blocked
> -	if {!$ok} return
>          eval run-ts broken  =             ts-hosts-allocate + $need_xen_hosts
>      }
>  
> @@ -340,9 +341,7 @@ proc run-job/build-rumpuserxen {} {
>  }
>  
>  proc prepare-build-host {} {
> -    global jobinfo ok
> -    catching-otherwise blocked check-not-blocked
> -    if {!$ok} return
> +    global jobinfo
>      run-ts broken = ts-hosts-allocate + host
>      run-ts broken host-install(*) ts-host-install-twice
>      run-ts . host-build-prep ts-xen-build-prep

Re: [OSSTEST PATCH 6/6] sg-run-job: Do blocked check globally, in run-job

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 6/6] sg-run-job: Do blocked check globally, in run-job"):
> On Wed, 2014-05-21 at 12:42 +0100, Ian Jackson wrote:
> > Only run-job can fully abandon the whole job.  With the previous
> > arrangements (in bc63a901), a blocked build job would go on to try to
> > run ts-logs-capture and become broken (rather than blocked).
> > 
> > The downside of this is that it is not possible in the future to
> > introduce a kind of job which can selectively carry on if some of its
> > dependencies fail.  This notion is IMO an acceptable casualty.
> 
> Just to be clear: This doesn't exclude the existing behaviour of
> ts-migrate-support-check, does it? (I think not because you talk about
> jobs and this is a step)

Indeed, that remains unaffected.  We're talking here only about
inter-job dependencies which show up in "*buildjob" runvars.

Ian.