[OSSTEST PATCH] make-flight: Trim the matrix of disk format flights

[OSSTEST PATCH] make-flight: Trim the matrix of disk format flights

[Ian Jackson]

We don't need to test every combination of toolstack, architecture,
and disk format.  We don't expect many architecture-specific bugs in
the per-disk-format code in the toolstack layers.

We _do_ want to test every combination of toolstack and disk format
(since the format configuration machinery is toolstack specific) and a
reasonable selection of architectures for each disk format (since
arch-specific bugs in actual underlying disk drivers are a
possibility).

The implementation strategy is for do_pv_debian_tests to select a
particular architecture for each combination of toolstack and format.
(Because the architecture is actually in an outer loop, we recalculate
that selection multiple times, and skip inner iterations for the other
architectures.  This is all in bash code so the wasted computation is
not particularly important.)

We have a safety catch which spots if any architecture is entirely
untested in any of these combinations; this would happen if a new
architecture is introduced elsewhere and not added to the list.  We do
not have a safety catch which spots when a (toolstack,format)
combination becomes untested due to deletion of an architecture.
(That would be more fiddly to implement without restructuring.)

We list armhf twice because we would like to do at least as many ARM
as x86 tests (particularly given our current workload and capacity).

The result is that the set of generated jobs is adjusted as follows:

   keep  test-amd64-i386-xl-raw
     -   test-amd64-i386-xl-vhd
     -   test-amd64-i386-xl-qcow2
     -   test-amd64-i386-libvirt-raw
     -   test-amd64-i386-libvirt-vhd
     -   test-amd64-i386-libvirt-qcow2
     -   test-amd64-amd64-xl-raw
     -   test-amd64-amd64-xl-vhd
   keep  test-amd64-amd64-xl-qcow2
     -   test-amd64-amd64-libvirt-raw
   keep  test-amd64-amd64-libvirt-vhd
     -   test-amd64-amd64-libvirt-qcow2
     -   test-armhf-armhf-xl-raw
   keep  test-armhf-armhf-xl-vhd
     -   test-armhf-armhf-xl-qcow2
   keep  test-armhf-armhf-libvirt-raw
     -   test-armhf-armhf-libvirt-vhd
   keep  test-armhf-armhf-libvirt-qcow2

(Where `-' means `drop this job'.)

Or to look at it another way:

                         i386   amd64   armhf

                 raw     keep     -       -
                 vhd       -      -     keep
                 qcow2     -    keep      -

         libvirt raw       -      -     keep
         libvirt vhd       -    keep      -
         libvirt qcow2     -      -     keep

I have diffed standalone-generate-dump-flight-runvars output before
and after and it looks plausible.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 make-flight |   21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/make-flight b/make-flight
index 70c909d..8aab28d 100755
--- a/make-flight
+++ b/make-flight
@@ -460,10 +460,29 @@ do_pv_debian_tests () {
     do_pv_debian_test_one libvirt '' libvirt '' enable_xsm=$xsm
   done
 
+  # We compute a desired architecture for each combination of ts and fmt
+  # Within each fmt we rotate through the list of arches
+  # The starting list rotates once per ts, so that we try to
+  # exercise each fmt on each arch family.
+  local fmtarches_outer="i386 armhf amd64 armhf"
+  local endfmt="do_pv_debian_tests-missing-ts-fmt-for-dom0arch="
+
   for ts in xl libvirt ; do
 
+    local fmtarches=$fmtarches_outer
+    fmtarches_outer="${fmtarches_outer#* } ${fmtarches_outer%% *}"
+
     for fmt in raw vhd qcow2 ; do
 
+      local fmtarch="${fmtarches%% *}"
+      fmtarches="${fmtarches#* } $fmtarch"
+
+      if [ "x$fmtarch" != "x$dom0arch" ]; then
+        continue
+      else
+        endfmt=": "
+      fi
+
       fmt_runvar="debian_diskfmt=$fmt"
 
       do_pv_debian_test_one $ts-$fmt '-di' $ts '' \
@@ -475,6 +494,8 @@ do_pv_debian_tests () {
 
     done
   done
+
+  $endfmt$dom0arch
 }
 
 test_matrix_do_one () {
-- 
1.7.10.4

Re: [OSSTEST PATCH] make-flight: Trim the matrix of disk format flights

[Ian Campbell]

On Wed, 2015-09-30 at 15:04 +0100, Ian Jackson wrote:
> We don't need to test every combination of toolstack, architecture,
> and disk format.  We don't expect many architecture-specific bugs in
> the per-disk-format code in the toolstack layers.
> 
> We _do_ want to test every combination of toolstack and disk format
> (since the format configuration machinery is toolstack specific) and a
> reasonable selection of architectures for each disk format (since
> arch-specific bugs in actual underlying disk drivers are a
> possibility).
> 
> The implementation strategy is for do_pv_debian_tests to select a
> particular architecture for each combination of toolstack and format.
> (Because the architecture is actually in an outer loop, we recalculate
> that selection multiple times, and skip inner iterations for the other
> architectures.  This is all in bash code so the wasted computation is
> not particularly important.)
> 
> We have a safety catch which spots if any architecture is entirely
> untested in any of these combinations; this would happen if a new
> architecture is introduced elsewhere and not added to the list.  We do
> not have a safety catch which spots when a (toolstack,format)
> combination becomes untested due to deletion of an architecture.
> (That would be more fiddly to implement without restructuring.)
> 
> We list armhf twice because we would like to do at least as many ARM
> as x86 tests (particularly given our current workload and capacity).
> 
> The result is that the set of generated jobs is adjusted as follows:
> 
>    keep  test-amd64-i386-xl-raw
>      -   test-amd64-i386-xl-vhd
>      -   test-amd64-i386-xl-qcow2
>      -   test-amd64-i386-libvirt-raw
>      -   test-amd64-i386-libvirt-vhd
>      -   test-amd64-i386-libvirt-qcow2
>      -   test-amd64-amd64-xl-raw
>      -   test-amd64-amd64-xl-vhd
>    keep  test-amd64-amd64-xl-qcow2
>      -   test-amd64-amd64-libvirt-raw
>    keep  test-amd64-amd64-libvirt-vhd
>      -   test-amd64-amd64-libvirt-qcow2
>      -   test-armhf-armhf-xl-raw
>    keep  test-armhf-armhf-xl-vhd
>      -   test-armhf-armhf-xl-qcow2
>    keep  test-armhf-armhf-libvirt-raw
>      -   test-armhf-armhf-libvirt-vhd
>    keep  test-armhf-armhf-libvirt-qcow2
> 
> (Where `-' means `drop this job'.)
> 
> Or to look at it another way:
> 
>                          i386   amd64   armhf
> 
>                  raw     keep     -       -
>                  vhd       -      -     keep
>                  qcow2     -    keep      -
> 
>          libvirt raw       -      -     keep
>          libvirt vhd       -    keep      -
>          libvirt qcow2     -      -     keep
> 
> I have diffed standalone-generate-dump-flight-runvars output before
> and after and it looks plausible.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [OSSTEST PATCH] make-flight: Trim the matrix of disk format flights

[Ian Campbell]

On Wed, 2015-09-30 at 15:42 +0100, Ian Campbell wrote:
> >                          i386   amd64   armhf
> > 
> >                  raw     keep     -       -
> >                  vhd       -      -     keep
> >                  qcow2     -    keep      -
> > 
> >          libvirt raw       -      -     keep
> >          libvirt vhd       -    keep      -
> >          libvirt qcow2     -      -     keep
> > 
[...]
> Acked-by: Ian Campbell <ian.campbell@citrix.com>

It occurred to me just now (i.e. rather too late) that as things stand
debian-di-install does not work for ARM in production, since it is using
Wheezy but Jessie is required for ARM guest support.

Hence the above change effectively dropped testing of xl.vhd and
libvirt.{raw,qcow2}.

Now, we are at least testing something with each format, just not quite
what we expected.

Since I have patches pending for switching Jessie I don't suppose those
matters too much.


Ian.