[PATCH OSSTEST 0/5] make-flight job filter cleanups

[PATCH OSSTEST 0/5] make-flight job filter cleanups

[Ian Campbell]

The mechanisms for omitting certain archs from jobs are inconsistently
applied to build vs test jobs. 

This series tries to tidy that up somewhat.

I've check the output of standalone-generate-dump-flight-runvars at each
step (i.e. covering the main make-flight).

I also did a before and after the entire flight check for the branches
handled by make-distro-flight and there was no difference (as expected).

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[PATCH OSSTEST 1/5] mg-adjust-flight-makexrefs: Make a nop in Standalone mode

[Ian Campbell]

It is not useful to try and drop jobs based on scrobbling around in
the standalone history.

This makes it possible to "./standalone make-flight
xen-unstable-smoke" and fixes standalone-generate-dump-flight-runvars

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 mg-adjust-flight-makexrefs | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/mg-adjust-flight-makexrefs b/mg-adjust-flight-makexrefs
index ec555bc..3e02ef3 100755
--- a/mg-adjust-flight-makexrefs
+++ b/mg-adjust-flight-makexrefs
@@ -42,6 +42,13 @@ done
 
 if [ $# -le 2 ]; then badusage; fi
 
+. ./cri-getconfig
+
+jobdb=`getconfig JobDB`
+if [ "$jobdb" = Standalone ]; then
+    exit 0
+fi
+
 flight=$1; shift
 keepjobs=$1; shift
 
-- 
2.6.1

[PATCH OSSTEST 2/5] mfi-common: Drop armhf build jobs on older linux-* branches

[Ian Campbell]

test_matrix_branch_filter_callback was recently updated to exclude
testing of linux-3.10 (54f237784d4b) and 3.14 (b0c5663a03e7), however
this only excludes test and not build jobs so we were still trying to
build the arm kernel there.

This changes the build job filtering to be in sync with the test job
filtering, the net result is to remove build-armhf* from linux-3.10
and linux-3.14.

A subsequent patch will try and combine those two filters into one to
prevent this skew happening again.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 mfi-common | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/mfi-common b/mfi-common
index 5fbe195..3463430 100644
--- a/mfi-common
+++ b/mfi-common
@@ -94,6 +94,8 @@ create_build_jobs () {
       case "$branch" in
       linux-3.0) continue;;
       linux-3.4) continue;;
+      linux-3.10) continue;;
+      linux-3.14) continue;;
       linux-mingo-tip-master) continue;;
       linux-*) ;;
       qemu-upstream-4.2-testing) continue;;
-- 
2.6.1

[PATCH OSSTEST 3/5] make-*flight: Reorder.

[Ian Campbell]

I just got tripped up again by putting a build job filter definition
after the call to create_build_jobs. Reorder the make-*flight scripts
to reduce the probability of me doing so any more times.

The general order of these scripts is now:
  - job filter callbacks
  - test job creation
  - top-level code which drives the process.

No change to the output of standalone-generate-dump-flight-runvars.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 make-distros-flight | 24 ++++++++++++------------
 make-flight         | 20 ++++++++++----------
 2 files changed, 22 insertions(+), 22 deletions(-)

diff --git a/make-distros-flight b/make-distros-flight
index 90e28c3..9e1d77e 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -50,18 +50,6 @@ job_create_build_filter_callback () {
   return 0
 }
 
-if [ x$buildflight = x ]; then
-
-  WANT_XEND=false REVISION_LINUX_OLD=disable
-
-  create_build_jobs
-
-else
-
-  bfi=$buildflight.
-
-fi
-
 job_create_test_filter_callback () {
   if [ "$xenarch" = "i386" ]; then return 1; fi
   return 0
@@ -201,6 +189,18 @@ test_matrix_do_one () {
   done
 }
 
+if [ x$buildflight = x ]; then
+
+  WANT_XEND=false REVISION_LINUX_OLD=disable
+
+  create_build_jobs
+
+else
+
+  bfi=$buildflight.
+
+fi
+
 test_matrix_iterate
 
 echo $flight
diff --git a/make-flight b/make-flight
index 8523995..5d8284a 100755
--- a/make-flight
+++ b/make-flight
@@ -58,16 +58,6 @@ job_create_build_filter_callback () {
   return 0
 }
 
-if [ x$buildflight = x ]; then
-
-  create_build_jobs
-
-else
-
-  bfi=$buildflight.
-
-fi
-
 job_create_test_filter_callback () {
   local job=$1; shift
   local recipe=$1; shift
@@ -674,6 +664,16 @@ test_matrix_do_one () {
   do_pvgrub_tests
 }
 
+if [ x$buildflight = x ]; then
+
+  create_build_jobs
+
+else
+
+  bfi=$buildflight.
+
+fi
+
 test_matrix_iterate
 
 echo $flight
-- 
2.6.1

[PATCH OSSTEST 4/5] make-flight: consolidate branch_filter_callback for builds and tests

[Ian Campbell]

Currently we have a test_matrix_branch_filter_callback which filters
jobs based on the $xenarch and $branch and a separate more adhoc
filter inline for the build jobs.

This has lead to things getting out of sync in the past (e.g. recently
we dropped armhf tests from the linux-3.10 and -3.14 branches but not
the build jobs).

Add a new build_matrix_branch_filter_callback and for make-flight
cause this and test_matrix_branch_filter_callback to use a common
helper.

The adhoc filtering in the build loop remains and will be tidied up
next.

For make-distros-flight just add a nop build filter alongside the test
filter.

No change to the output of standalone-generate-dump-flight-runvars.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 make-distros-flight |  4 ++++
 make-flight         | 14 ++++++++++++--
 mfi-common          |  2 ++
 3 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/make-distros-flight b/make-distros-flight
index 9e1d77e..9d04d3b 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -55,6 +55,10 @@ job_create_test_filter_callback () {
   return 0
 }
 
+build_matrix_branch_filter_callback () {
+    :
+}
+
 test_matrix_branch_filter_callback () {
     :
 }
diff --git a/make-flight b/make-flight
index 5d8284a..8febb9f 100755
--- a/make-flight
+++ b/make-flight
@@ -133,8 +133,9 @@ job_create_test_filter_callback () {
   return 0;
 }
 
-test_matrix_branch_filter_callback () {
-  case "$xenarch" in
+arch_branch_filter_callback () {
+  local arch=$1
+  case "$arch" in
   armhf)
         case "$branch" in
         linux-3.0) return 1;;
@@ -154,7 +155,16 @@ test_matrix_branch_filter_callback () {
         esac
         ;;
   esac
+  return 0
+}
 
+build_matrix_branch_filter_callback () {
+  if ! arch_branch_filter_callback $arch ; then return 1 ; fi
+  return 0
+}
+
+test_matrix_branch_filter_callback () {
+  if ! arch_branch_filter_callback $xenarch ; then return 1 ; fi
   return 0
 }
 
diff --git a/mfi-common b/mfi-common
index 3463430..44cd13b 100644
--- a/mfi-common
+++ b/mfi-common
@@ -89,6 +89,8 @@ create_build_jobs () {
 
     if [ "x$arch" = xdisable ]; then continue; fi
 
+    build_matrix_branch_filter_callback || continue
+
     case "$arch" in
     armhf)
       case "$branch" in
-- 
2.6.1

[PATCH OSSTEST 5/5] make-flight: move in-lined branch vs arch filtering into callbacks

[Ian Campbell]

No change to the output of standalone-generate-dump-flight-runvars

The inlined xenbranch vs arch filters remain where they are since they
are common (in that they reflect the addition and removal of arches)
and apply equally to all make-*-flight.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 make-flight |  3 +++
 mfi-common  | 17 -----------------
 2 files changed, 3 insertions(+), 17 deletions(-)

diff --git a/make-flight b/make-flight
index 8febb9f..6f462ad 100755
--- a/make-flight
+++ b/make-flight
@@ -147,6 +147,9 @@ arch_branch_filter_callback () {
         qemu-upstream-4.2-testing) return 1;;
         qemu-upstream-4.3-testing) return 1;;
         qemu-upstream-4.4-testing) return 1;;
+        rumpuserxen) return 1;;
+        seabios) return 1;;
+        ovmf) return 1;;
         esac
         ;;
   i386|amd64)
diff --git a/mfi-common b/mfi-common
index 44cd13b..4ed040e 100644
--- a/mfi-common
+++ b/mfi-common
@@ -93,20 +93,6 @@ create_build_jobs () {
 
     case "$arch" in
     armhf)
-      case "$branch" in
-      linux-3.0) continue;;
-      linux-3.4) continue;;
-      linux-3.10) continue;;
-      linux-3.14) continue;;
-      linux-mingo-tip-master) continue;;
-      linux-*) ;;
-      qemu-upstream-4.2-testing) continue;;
-      qemu-upstream-4.3-testing) continue;;
-      qemu-upstream-4.4-testing) continue;;
-      rumpuserxen) continue;;
-      seabios) continue;;
-      ovmf) continue;;
-      esac
       case "$xenbranch" in
       xen-3.*-testing) continue;;
       xen-4.0-testing) continue;;
@@ -122,9 +108,6 @@ create_build_jobs () {
       "
       ;;
     *)
-      case "$branch" in
-      linux-arm-xen) continue;;
-      esac
       pvops_kernel="
         tree_linux=$TREE_LINUX
         revision_linux=${REVISION_LINUX:-${DEFAULT_REVISION_LINUX}}
-- 
2.6.1

Re: [PATCH OSSTEST 1/5] mg-adjust-flight-makexrefs: Make a nop in Standalone mode

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST 1/5] mg-adjust-flight-makexrefs: Make a nop in Standalone mode"):
> It is not useful to try and drop jobs based on scrobbling around in
> the standalone history.
> 
> This makes it possible to "./standalone make-flight
> xen-unstable-smoke" and fixes standalone-generate-dump-flight-runvars

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH OSSTEST 2/5] mfi-common: Drop armhf build jobs on older linux-* branches

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST 2/5] mfi-common: Drop armhf build jobs on older linux-* branches"):
> test_matrix_branch_filter_callback was recently updated to exclude
> testing of linux-3.10 (54f237784d4b) and 3.14 (b0c5663a03e7), however
> this only excludes test and not build jobs so we were still trying to
> build the arm kernel there.
> 
> This changes the build job filtering to be in sync with the test job
> filtering, the net result is to remove build-armhf* from linux-3.10
> and linux-3.14.
> 
> A subsequent patch will try and combine those two filters into one to
> prevent this skew happening again.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH OSSTEST 3/5] make-*flight: Reorder.

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST 3/5] make-*flight: Reorder."):
> I just got tripped up again by putting a build job filter definition
> after the call to create_build_jobs. Reorder the make-*flight scripts
> to reduce the probability of me doing so any more times.
> 
> The general order of these scripts is now:
>   - job filter callbacks
>   - test job creation
>   - top-level code which drives the process.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH OSSTEST 5/5] make-flight: move in-lined branch vs arch filtering into callbacks

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST 5/5] make-flight: move in-lined branch vs arch filtering into callbacks"):
> No change to the output of standalone-generate-dump-flight-runvars
> 
> The inlined xenbranch vs arch filters remain where they are since they
> are common (in that they reflect the addition and removal of arches)
> and apply equally to all make-*-flight.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>