[OSSTEST PATCH RFC] do not install xend for xl tests

[OSSTEST PATCH RFC] do not install xend for xl tests

[Ian Campbell]

We need to check that xl works correctly when xend is not even installed (in
case we are subtly relying on some file which xend installs).

Therefore for xen 4.4 onwards never build xend in the default build job and
instead create two new build jobs (for i386 and amd64) with xend enabled.
Update the tests to use the correct xenbuildjob.

Tested only to the extent of running make-flight for xen-4.{2,3,4}-testing and
xen-unstable and observing that the jobs do not differ for 4.2 and 4.3 and the
4.4 and unstable have gained the new build-{i386,and64}-xend jobs and that the
relevant tests have switched their xenbuildjob runvar to have the suffix.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
Relies on the xen-create-image workaround which I send earier today
---
 make-flight |   72 +++++++++++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 55 insertions(+), 17 deletions(-)

diff --git a/make-flight b/make-flight
index 5b96153..703006e 100755
--- a/make-flight
+++ b/make-flight
@@ -83,9 +83,22 @@ if [ x$buildflight = x ]; then
         suite_runvars=
     fi
 
+    # In 4.4 onwards xend is off by default. If necessary we build a
+    # separate set of binaries with xend enabled in order to run those
+    # tests which use xend.
     case "$arch" in
-    i386|amd64) enable_xend=true;;
-    *) enable_xend=false;;
+    i386|amd64) want_xend=true;;
+    *) want_xend=false;;
+    esac
+
+    case "$xenbranch" in
+    xen-3.*-testing) build_defxend=$want_xend; build_extraxend=false;;
+    xen-4.0-testing) build_defxend=$want_xend; build_extraxend=false;;
+    xen-4.1-testing) build_defxend=$want_xend; build_extraxend=false;;
+    xen-4.2-testing) build_defxend=$want_xend; build_extraxend=false;;
+    xen-4.3-testing) build_defxend=$want_xend; build_extraxend=false;;
+    *) build_defxend=false;
+       build_extraxend=$want_xend
     esac
 
     case "$xenbranch" in
@@ -104,7 +117,7 @@ if [ x$buildflight = x ]; then
     build_hostflags=share-build-$suite-$arch,arch-$arch,suite-$suite,purpose-build
 
     ./cs-job-create $flight build-$arch build				     \
-		arch=$arch enable_xend=$enable_xend enable_ovmf=$enable_ovmf	     \
+		arch=$arch enable_xend=$build_defxend enable_ovmf=$enable_ovmf	     \
 	tree_qemu=$TREE_QEMU	     \
 	tree_qemuu=$TREE_QEMU_UPSTREAM	     \
 	tree_xen=$TREE_XEN		     \
@@ -115,6 +128,20 @@ if [ x$buildflight = x ]; then
 		revision_qemu=$REVISION_QEMU				     \
 		revision_qemuu=$REVISION_QEMU_UPSTREAM
 
+    if [ $build_extraxend = "true" ] ; then
+    ./cs-job-create $flight build-$arch-xend build			     \
+		arch=$arch enable_xend=true enable_ovmf=$enable_ovmf	     \
+	tree_qemu=$TREE_QEMU	     \
+	tree_qemuu=$TREE_QEMU_UPSTREAM	     \
+	tree_xen=$TREE_XEN		     \
+		$RUNVARS $BUILD_RUNVARS $BUILD_XEN_RUNVARS $arch_runvars     \
+		$suite_runvars                                               \
+		host_hostflags=$build_hostflags    \
+		revision_xen=$REVISION_XEN				     \
+		revision_qemu=$REVISION_QEMU				     \
+		revision_qemuu=$REVISION_QEMU_UPSTREAM
+    fi
+
     ./cs-job-create $flight build-$arch-pvops build-kern		     \
 		arch=$arch kconfighow=xen-enable-xen-config		     \
 	tree_xen=$TREE_XEN		     \
@@ -198,10 +225,20 @@ job_create_test () {
 	local job=$1; shift
 	local recipe=$1; shift
 	local toolstack=$1; shift
+	local xenbuildjob=$1; shift
 
         local job_md5=`echo "$job" | md5sum`
         job_md5="${job_md5%  -}"
 
+        case "$xenbranch:$toolstack" in
+        xen-3.*-testing:*) ;;
+        xen-4.0-testing:*) ;;
+        xen-4.1-testing:*) ;;
+        xen-4.2-testing:*) ;;
+        xen-4.3-testing:*) ;;
+        *:xend) xenbuildjob="$xenbuildjob-xend";;
+        esac
+
         if [ "x$JOB_MD5_PATTERN" != x ]; then
                 case "$job_md5" in
                 $JOB_MD5_PATTERN)       ;;
@@ -237,7 +274,7 @@ job_create_test () {
         esac
 
 	./cs-job-create $flight $job $recipe toolstack=$toolstack \
-		$RUNVARS $TEST_RUNVARS $most_runvars "$@"
+		$RUNVARS $TEST_RUNVARS $most_runvars xenbuildjob=$xenbuildjob "$@"
 }
 
 for xenarch in ${TEST_ARCHES- i386 amd64 armhf } ; do
@@ -329,28 +366,29 @@ for xenarch in ${TEST_ARCHES- i386 amd64 armhf } ; do
 
       most_hostflags="arch-$dom0arch,arch-xen-$xenarch,suite-$suite,purpose-test"
 
+      xenbuildjob="${bfi}build-$xenarch"
+
       most_runvars="
 		arch=$dom0arch			        	\
-		xenbuildjob=${bfi}build-$xenarch        	\
 		kernbuildjob=${bfi}build-$dom0arch-$kernbuild 	\
 		buildjob=${bfi}build-$dom0arch	        	\
 		kernkind=$kernkind		        	\
 		$arch_runvars $suite_runvars
 		"
       if [ $dom0arch = armhf ]; then
-	  job_create_test test-$xenarch$kern-$dom0arch-xl test-debian xl \
+	  job_create_test test-$xenarch$kern-$dom0arch-xl test-debian xl "$xenbuildjob" \
 		debian_kernkind=$kernkind				  \
 		debian_arch=$dom0arch   				  \
 		all_hostflags=$most_hostflags
 	  continue
       fi
 
-      job_create_test test-$xenarch$kern-$dom0arch-pv test-debian xend \
+      job_create_test test-$xenarch$kern-$dom0arch-pv test-debian xend "$xenbuildjob" \
 		debian_kernkind=$kernkind				  \
 		debian_arch=$dom0arch   				  \
 		all_hostflags=$most_hostflags
 
-      job_create_test test-$xenarch$kern-$dom0arch-xl test-debian xl \
+      job_create_test test-$xenarch$kern-$dom0arch-xl test-debian xl "$xenbuildjob" \
 		debian_kernkind=$kernkind				  \
 		debian_arch=$dom0arch   				  \
 		all_hostflags=$most_hostflags
@@ -360,7 +398,7 @@ for xenarch in ${TEST_ARCHES- i386 amd64 armhf } ; do
         for freebsdarch in amd64 i386; do
 
  job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-freebsd10-$freebsdarch \
-			test-freebsd xl \
+			test-freebsd xl "$xenbuildjob" \
 			freebsd_arch=$freebsdarch \
  freebsd_image=${FREEBSD_IMAGE_PREFIX-FreeBSD-10.0-BETA3-}$freebsdarch${FREEBSD_IMAGE_SUFFIX--20131103-r257580.qcow2.xz} \
 			all_hostflags=$most_hostflags
@@ -406,7 +444,7 @@ for xenarch in ${TEST_ARCHES- i386 amd64 armhf } ; do
 
       job_create_test \
                 test-$xenarch$kern-$dom0arch-$toolstack$qemuu_suffix-winxpsp3$vcpus_suffix \
-                test-win $toolstack $qemuu_runvar \
+                test-win $toolstack "$xenbuildjob" $qemuu_runvar \
 		win_image=winxpsp3.iso $vcpus_runvars	\
 		all_hostflags=$most_hostflags,hvm
 
@@ -416,7 +454,7 @@ for xenarch in ${TEST_ARCHES- i386 amd64 armhf } ; do
       if [ $xenarch = amd64 ]; then
 
       job_create_test test-$xenarch$kern-$dom0arch-xl$qemuu_suffix-win7-amd64 \
-                test-win xl $qemuu_runvar \
+                test-win xl "$xenbuildjob" $qemuu_runvar \
 		win_image=win7-x64.iso \
 		all_hostflags=$most_hostflags,hvm
 
@@ -427,7 +465,7 @@ for xenarch in ${TEST_ARCHES- i386 amd64 armhf } ; do
 	for cpuvendor in amd intel; do
 
     job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-rhel6hvm-$cpuvendor \
-						test-rhelhvm xl \
+						test-rhelhvm xl "$xenbuildjob" \
 		redhat_image=rhel-server-6.1-i386-dvd.iso		\
 		all_hostflags=$most_hostflags,hvm-$cpuvendor \
                 $qemuu_runvar
@@ -439,7 +477,7 @@ for xenarch in ${TEST_ARCHES- i386 amd64 armhf } ; do
       done # qemuu_suffix
 
       job_create_test test-$xenarch$kern-$dom0arch-pair test-pair \
-		$onetoolstack \
+		$onetoolstack "$xenbuildjob" \
                 !host !host_hostflags \
 		debian_kernkind=$kernkind				  \
 		debian_arch=$dom0arch   				  \
@@ -450,7 +488,7 @@ for xenarch in ${TEST_ARCHES- i386 amd64 armhf } ; do
        for pin in '' -pin; do
 
       job_create_test test-$xenarch$kern-$dom0arch-xl-sedf$pin  \
-           test-debian xl guests_vcpus=4 xen_boot_append="sched=sedf loglvl=all ${pin:+dom0_vcpus_pin}" linux_boot_append='loglevel=9 debug' \
+           test-debian xl "$xenbuildjob" guests_vcpus=4 xen_boot_append="sched=sedf loglvl=all ${pin:+dom0_vcpus_pin}" linux_boot_append='loglevel=9 debug' \
 		debian_kernkind=$kernkind				  \
 		debian_arch=$dom0arch   				  \
 		all_hostflags=$most_hostflags
@@ -462,13 +500,13 @@ for xenarch in ${TEST_ARCHES- i386 amd64 armhf } ; do
       if [ $xenarch = amd64 -a $dom0arch = i386 ]; then
 
       job_create_test test-$xenarch$kern-$dom0arch-xl-multivcpu \
-                        test-debian xl guests_vcpus=4 \
+                        test-debian xl "$xenbuildjob" guests_vcpus=4 \
 		debian_kernkind=$kernkind				  \
 		debian_arch=$dom0arch   				  \
 		all_hostflags=$most_hostflags
 
       job_create_test test-$xenarch$kern-$dom0arch-xl-credit2  \
-           test-debian xl guests_vcpus=4 xen_boot_append='sched=credit2'      \
+           test-debian xl "$xenbuildjob" guests_vcpus=4 xen_boot_append='sched=credit2'      \
 		debian_kernkind=$kernkind				  \
 		debian_arch=$dom0arch   				  \
 		all_hostflags=$most_hostflags
@@ -480,7 +518,7 @@ for xenarch in ${TEST_ARCHES- i386 amd64 armhf } ; do
         for cpuvendor in intel; do
 
       job_create_test test-$xenarch$kern-$dom0arch-xl-pcipt-intel \
-                        test-debian-nomigr xl guests_vcpus=4 \
+                        test-debian-nomigr xl "$xenbuildjob" guests_vcpus=4 \
 		debian_kernkind=$kernkind				  \
 		debian_arch=$dom0arch   				  \
 		debian_pcipassthrough_nic=host				  \
-- 
1.7.10.4

Re: [OSSTEST PATCH RFC] do not install xend for xl tests

[Ian Jackson]

Ian Campbell writes ("[OSSTEST PATCH RFC] do not install xend for xl tests"):
> We need to check that xl works correctly when xend is not even installed (in
> case we are subtly relying on some file which xend installs).
> 
> Therefore for xen 4.4 onwards never build xend in the default build job and
> instead create two new build jobs (for i386 and amd64) with xend enabled.
> Update the tests to use the correct xenbuildjob.
> 
> Tested only to the extent of running make-flight for
> xen-4.{2,3,4}-testing and xen-unstable and observing that the jobs
> do not differ for 4.2 and 4.3 and the 4.4 and unstable have gained
> the new build-{i386,and64}-xend jobs and that the relevant tests
> have switched their xenbuildjob runvar to have the suffix.

Can we solve this problem by having a variant of "make install" which
provides a differentiated output ?  That way we could use the same
build, split into several outputs.

Ian.

Re: [OSSTEST PATCH RFC] do not install xend for xl tests

[Ian Campbell]

On Wed, 2013-12-18 at 14:56 +0000, Ian Jackson wrote:
> Ian Campbell writes ("[OSSTEST PATCH RFC] do not install xend for xl tests"):
> > We need to check that xl works correctly when xend is not even installed (in
> > case we are subtly relying on some file which xend installs).
> > 
> > Therefore for xen 4.4 onwards never build xend in the default build job and
> > instead create two new build jobs (for i386 and amd64) with xend enabled.
> > Update the tests to use the correct xenbuildjob.
> > 
> > Tested only to the extent of running make-flight for
> > xen-4.{2,3,4}-testing and xen-unstable and observing that the jobs
> > do not differ for 4.2 and 4.3 and the 4.4 and unstable have gained
> > the new build-{i386,and64}-xend jobs and that the relevant tests
> > have switched their xenbuildjob runvar to have the suffix.
> 
> Can we solve this problem by having a variant of "make install" which
> provides a differentiated output ?  That way we could use the same
> build, split into several outputs.

I suppose anything is possible/ASMOP ;-) But it seems like it would
involve a lot of fiddly build system stuff which would only ever be
exercised by osstest.

What I really want is for the non-xend test cases to be using the same
build environment as what end users will be using by default, which
doesn't involve a split make install (e.g. what if we get a critical
file in the wrong list?).

Ian.

Re: [OSSTEST PATCH RFC] do not install xend for xl tests

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH RFC] do not install xend for xl tests"):
> What I really want is for the non-xend test cases to be using the same
> build environment as what end users will be using by default, which
> doesn't involve a split make install (e.g. what if we get a critical
> file in the wrong list?).

Good point.  OK, you've convinced me of the soundness of the
principle.

I'm slightly concerned by the repetitive nature of your changes to
make-flight, but something along these lines is a good idea.

Ian.

Re: [OSSTEST PATCH RFC] do not install xend for xl tests

[Ian Campbell]

On Thu, 2013-12-19 at 15:17 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [OSSTEST PATCH RFC] do not install xend for xl tests"):
> > What I really want is for the non-xend test cases to be using the same
> > build environment as what end users will be using by default, which
> > doesn't involve a split make install (e.g. what if we get a critical
> > file in the wrong list?).
> 
> Good point.  OK, you've convinced me of the soundness of the
> principle.

Thanks.

> I'm slightly concerned by the repetitive nature of your changes to
> make-flight, but something along these lines is a good idea.

I wasn't sure how to avoid it -- ultimately I needed to make the build
depend on the toolstack, but $most_runvars is used with multiple
toolstacks, so ultimately everywhere which used it needed to be changed.

Ian.

Re: [OSSTEST PATCH RFC] do not install xend for xl tests

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH RFC] do not install xend for xl tests"):
> I wasn't sure how to avoid it -- ultimately I needed to make the build
> depend on the toolstack, but $most_runvars is used with multiple
> toolstacks, so ultimately everywhere which used it needed to be changed.

Do it in job_create_test ?

Ian.

Re: [OSSTEST PATCH RFC] do not install xend for xl tests

[Ian Campbell]

On Thu, 2013-12-19 at 16:14 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [OSSTEST PATCH RFC] do not install xend for xl tests"):
> > I wasn't sure how to avoid it -- ultimately I needed to make the build
> > depend on the toolstack, but $most_runvars is used with multiple
> > toolstacks, so ultimately everywhere which used it needed to be changed.
> 
> Do it in job_create_test ?

The necessary info to create the base build job name aren't available
there, or so I thought. In particular $xenarch.

Ian.

Re: [OSSTEST PATCH RFC] do not install xend for xl tests

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH RFC] do not install xend for xl tests"):
> On Thu, 2013-12-19 at 16:14 +0000, Ian Jackson wrote:
> > Do it in job_create_test ?
> 
> The necessary info to create the base build job name aren't available
> there, or so I thought. In particular $xenarch.

It could be made a positional parameter to job_create_test.

Ian.

Re: [OSSTEST PATCH RFC] do not install xend for xl tests

[Ian Campbell]

On Fri, 2013-12-20 at 11:48 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [OSSTEST PATCH RFC] do not install xend for xl tests"):
> > On Thu, 2013-12-19 at 16:14 +0000, Ian Jackson wrote:
> > > Do it in job_create_test ?
> > 
> > The necessary info to create the base build job name aren't available
> > there, or so I thought. In particular $xenarch.
> 
> It could be made a positional parameter to job_create_test.

That would need the same amount of repetitive changes as the current
patch (since it also adds a positional parameter). If you would prefer
the new param to be the arch rather than the build job then I'm happy to
make that change.

Re: [OSSTEST PATCH RFC] do not install xend for xl tests

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH RFC] do not install xend for xl tests"):
> On Fri, 2013-12-20 at 11:48 +0000, Ian Jackson wrote:
> > It could be made a positional parameter to job_create_test.
> 
> That would need the same amount of repetitive changes as the current
> patch (since it also adds a positional parameter). If you would prefer
> the new param to be the arch rather than the build job then I'm happy to
> make that change.

Yes, the change is just as repetitive but the resulting code less so:
the resulting code doesn't need to specify the toolstack twice
(effectively) in its calls to job_create_test.

Ian.