Re: [OSSTEST PATCH] make-flight: Trim the matrix of disk format flights

[Ian Campbell <ian.campbell@citrix.com>]

On Wed, 2015-09-30 at 15:04 +0100, Ian Jackson wrote:
> We don't need to test every combination of toolstack, architecture,
> and disk format.  We don't expect many architecture-specific bugs in
> the per-disk-format code in the toolstack layers.
> 
> We _do_ want to test every combination of toolstack and disk format
> (since the format configuration machinery is toolstack specific) and a
> reasonable selection of architectures for each disk format (since
> arch-specific bugs in actual underlying disk drivers are a
> possibility).
> 
> The implementation strategy is for do_pv_debian_tests to select a
> particular architecture for each combination of toolstack and format.
> (Because the architecture is actually in an outer loop, we recalculate
> that selection multiple times, and skip inner iterations for the other
> architectures.  This is all in bash code so the wasted computation is
> not particularly important.)
> 
> We have a safety catch which spots if any architecture is entirely
> untested in any of these combinations; this would happen if a new
> architecture is introduced elsewhere and not added to the list.  We do
> not have a safety catch which spots when a (toolstack,format)
> combination becomes untested due to deletion of an architecture.
> (That would be more fiddly to implement without restructuring.)
> 
> We list armhf twice because we would like to do at least as many ARM
> as x86 tests (particularly given our current workload and capacity).
> 
> The result is that the set of generated jobs is adjusted as follows:
> 
>    keep  test-amd64-i386-xl-raw
>      -   test-amd64-i386-xl-vhd
>      -   test-amd64-i386-xl-qcow2
>      -   test-amd64-i386-libvirt-raw
>      -   test-amd64-i386-libvirt-vhd
>      -   test-amd64-i386-libvirt-qcow2
>      -   test-amd64-amd64-xl-raw
>      -   test-amd64-amd64-xl-vhd
>    keep  test-amd64-amd64-xl-qcow2
>      -   test-amd64-amd64-libvirt-raw
>    keep  test-amd64-amd64-libvirt-vhd
>      -   test-amd64-amd64-libvirt-qcow2
>      -   test-armhf-armhf-xl-raw
>    keep  test-armhf-armhf-xl-vhd
>      -   test-armhf-armhf-xl-qcow2
>    keep  test-armhf-armhf-libvirt-raw
>      -   test-armhf-armhf-libvirt-vhd
>    keep  test-armhf-armhf-libvirt-qcow2
> 
> (Where `-' means `drop this job'.)
> 
> Or to look at it another way:
> 
>                          i386   amd64   armhf
> 
>                  raw     keep     -       -
>                  vhd       -      -     keep
>                  qcow2     -    keep      -
> 
>          libvirt raw       -      -     keep
>          libvirt vhd       -    keep      -
>          libvirt qcow2     -      -     keep
> 
> I have diffed standalone-generate-dump-flight-runvars output before
> and after and it looks plausible.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>