[PATCH v8 0/2] OSSTEST: introduce a raisin build test

[PATCH v8 0/2] OSSTEST: introduce a raisin build test

[Stefano Stabellini]

Hi all,

the first patch introduces a Raisin build job, meant to test Raisin
itself. The second patch push-gates Raisin on successful builds.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>


Changes in v8:
- update commit message
- do not install-builddep in ts-raisin-build, rely on ts-xen-build-dep
- add raisin to BRANCHES at the top of cr-for-branches
- do not run jobs on the raisin flight
- switch to the common staging/master branch names

Changes in v7:
- update raisin git url after http://marc.info/?l=xen-devel&m=143472251602803
- add a patch to push-gate raisin

Changes in v6:
- move divide to Osstest/BuildSupport.pm and rename it to
divide_xen_build; use divide_xen_build in ts-raisin-build and
ts-xen-build.

Changes in v5:
- remove setting DEFAULT_REVISION_RAISIN to master
- disable raisin when branch is not xen*
- do not overwrite raisin default variables unless both revision_* and
tree_* are set of a given component

Changes in v4:
- introduce enable_raisin in mfi-common: only build raisin when building
xen-unstable
- start off from the default raisin config, then append osstest config
options to it
- do not write variable to the raisin config if the conrresponding
runvar is not set
- remove TREE_OVMF and TREE_SEABIOS

Changes in v3:
- use $raisindir throughout ts-raisin-build
- do not specify ENABLED_COMPONENTS so that empty REVISION variables can
be used to disable building a raisin component

Changes in v2:
- set revision_* variables in mfi-common;
- in ts-raisin-build set the *_REVISION config options based on the
  revision_* variables;
- in ts-raisin-build, call store_revision appropriately;
- divide the output in an hypervisor and a tools tarball.


Stefano Stabellini (2):
      OSSTEST: introduce a raisin build test
      OSSTest: push successful raisin builds

 Osstest/BuildSupport.pm |   48 ++++++++++++++++++
 ap-common               |    5 ++
 ap-fetch-version        |    3 ++
 ap-fetch-version-old    |    5 ++
 ap-print-url            |    3 ++
 ap-push                 |    5 ++
 cr-daily-branch         |    8 +++
 cr-for-branches         |    2 +-
 cri-common              |    1 +
 make-flight             |    3 ++
 mfi-common              |   41 +++++++++++++++
 sg-run-job              |    5 ++
 ts-raisin-build         |  128 +++++++++++++++++++++++++++++++++++++++++++++++
 ts-xen-build            |   44 +---------------
 14 files changed, 258 insertions(+), 43 deletions(-)
 create mode 100755 ts-raisin-build

[PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Stefano Stabellini]

Introduce a raisin build test to OSSTest. The build is meant to test
Raisin itself.

Enable the raisin build just for xen-unstable. Disable it for all the
older Xen branches and for all the non-Xen branches.

The components built by the raisin job are static; the list is written
by ts-raisin-build. Don't automatically generate the list as the
component names are not regular. The versions of the components and
their git tree urls are passed by mfi-common to ts-raisin-build.
mfi-common passes the same TREE_ and REVISION_ variables used with the
other jobs.  ts-raisin-build writes them to the raisin config file,
overwriting the defaults, only if both the tree and revision variables
for a component are set, to avoid inconsistencies.  For example
mfi-common can pass a revision only present in a specific git tree, but
not the corresponding git tree.

Make divide_xen_build common between ts-raisin-build and ts-xen-build.
It is used to split the build output in two archives: one with the
hypervisor binary and another with the userspace tools.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

---

Changes in v8:
- update commit message
- do not install-builddep in ts-raisin-build, rely on ts-xen-build-dep

Changes in v7:
- update raisin git url after http://marc.info/?l=xen-devel&m=143472251602803

Changes in v6:
- move divide to Osstest/BuildSupport.pm and rename it to
divide_xen_build; use divide_xen_build in ts-raisin-build and
ts-xen-build.

Changes in v5:
- remove setting DEFAULT_REVISION_RAISIN to master
- disable raisin when branch is not xen*
- do not overwrite raisin default variables unless both revision_* and
tree_* are set of a given component

Changes in v4:
- introduce enable_raisin in mfi-common: only build raisin when building
xen-unstable
- start off from the default raisin config, then append osstest config
options to it
- do not write variable to the raisin config if the conrresponding
runvar is not set
- remove TREE_OVMF and TREE_SEABIOS

Changes in v3:
- use $raisindir throughout ts-raisin-build
- do not specify ENABLED_COMPONENTS so that empty REVISION variables can
be used to disable building a raisin component

Changes in v2:
- set revision_* variables in mfi-common;
- in ts-raisin-build set the *_REVISION config options based on the
  revision_* variables;
- in ts-raisin-build, call store_revision appropriately;
- divide the output in an hypervisor and a tools tarball.
---
 Osstest/BuildSupport.pm |   48 ++++++++++++++++++
 ap-common               |    2 +
 mfi-common              |   41 +++++++++++++++
 sg-run-job              |    5 ++
 ts-raisin-build         |  128 +++++++++++++++++++++++++++++++++++++++++++++++
 ts-xen-build            |   44 +---------------
 6 files changed, 226 insertions(+), 42 deletions(-)
 create mode 100755 ts-raisin-build

diff --git a/Osstest/BuildSupport.pm b/Osstest/BuildSupport.pm
index 933f6e1..76a3feb 100644
--- a/Osstest/BuildSupport.pm
+++ b/Osstest/BuildSupport.pm
@@ -43,6 +43,10 @@ BEGIN {
                       xendist
                       $xendist
 
+                      divide_xen_build
+                      trapping
+                      die_if_fail
+
                       submodulefixup submodule_have submodule_find
 
                       );
@@ -84,6 +88,50 @@ sub xendist () {
 	($ho, 'xendist', '', $r{"buildjob"});
 }
 
+sub divide_xen_build ($$) {
+    my ($subdir, $postfix) = @_;
+    # Only move hv to xeninstall, so that we can have
+    # xenpolicy in tools tarball.
+    #
+    # The files inside boot/ after `make dist' are
+    # xen-$XEN_VERSION: Xen binary
+    # xen.gz/xen: symlink to xen-$XEN_VERSION
+    # xen-$MAJOR: symlink to xen-$XEN_VERSION
+    # xen-$MAJOR.$MINOR: symlink to xen-$XEN_VERSION
+    # xen-sym-$XEN_VERSION: Xen symbol
+    # xenpolicy-$XEN_VERSION: flask policy binary if xsm is enabled
+    #
+    # So the following snippet will leave xenpolicy* in
+    # install/boot and get packaged to tools tarball.
+    target_cmd_build($ho, 100, $builddir, <<END);
+        cd $subdir
+        mkdir xen$postfix
+        for f in *$postfix; do
+            mkdir -p \$f/lib
+        done
+        if test -d $postfix/boot; then
+            if test -f $postfix/boot/xen.gz || test -f $postfix/boot/xen; then
+                mkdir xen$postfix/boot
+                mvfiles=`find $postfix/boot -name 'xen[a-z]*' -prune -o -name 'xen*' -print`
+                mv \$mvfiles xen$postfix/boot/.
+            fi
+        fi
+END
+}
+
+our @probs;
+
+sub trapping ($) {
+    my ($sub) = @_;
+    my $tok= eval { $sub->(); 1; };
+    if (!$tok) { push @probs, $@; print STDERR "failure (trapped): $@\n"; }
+}
+
+sub die_if_fail () {
+    die "*** something failed:\n\n".(join "\n\n",@probs)."\n** something failed"
+        if @probs;
+}
+
 #----- submodules -----
 
 sub submodulefixup ($$$$) {
diff --git a/ap-common b/ap-common
index 64749e3..6e1c472 100644
--- a/ap-common
+++ b/ap-common
@@ -47,6 +47,8 @@
 # rumpsrc-related runvars needed only for old rumpuser-xen
 # (ie ones which need $bodges=1 in ts-rumpuserxen-build)
 
+: ${TREE_RAISIN:=git://xenbits.xen.org/raisin.git}
+
 : ${TREE_SEABIOS_UPSTREAM:=git://git.seabios.org/seabios.git}
 : ${PUSH_TREE_SEABIOS:=$XENBITS:/home/xen/git/osstest/seabios.git}
 : ${BASE_TREE_SEABIOS:=git://xenbits.xen.org/osstest/seabios.git}
diff --git a/mfi-common b/mfi-common
index 16fc8c5..d75823d 100644
--- a/mfi-common
+++ b/mfi-common
@@ -148,6 +148,22 @@ create_build_jobs () {
     *) enable_ovmf=true;
     esac
 
+    case "$xenbranch" in
+    xen-3.*-testing) enable_raisin=false;;
+    xen-4.0-testing) enable_raisin=false;;
+    xen-4.1-testing) enable_raisin=false;;
+    xen-4.2-testing) enable_raisin=false;;
+    xen-4.3-testing) enable_raisin=false;;
+    xen-4.4-testing) enable_raisin=false;;
+    xen-4.5-testing) enable_raisin=false;;
+    *)               enable_raisin=true;
+    esac
+    case "$branch" in
+    xen*) ;;
+    *) enable_raisin=false;;
+    esac
+
+
     eval "
         arch_runvars=\"\$ARCH_RUNVARS_$arch\"
     "
@@ -215,6 +231,31 @@ create_build_jobs () {
 
     fi
 
+    if [ "x$REVISION_RAISIN" != xdisable ] && [ "$enable_raisin" = "true" ]; then
+
+    ./cs-job-create $flight build-$arch-raisin build-raisin                  \
+                arch=$arch                                                   \
+        tree_xen=$TREE_XEN                                                   \
+                $RUNVARS $BUILD_RUNVARS $BUILD_RAISIN_RUNVARS $arch_runvars \
+                $suite_runvars                                               \
+                host_hostflags=$build_hostflags                              \
+                buildjob=${bfi}build-$arch                                   \
+                tree_raisin=$TREE_RAISIN                                     \
+                tree_qemuu=$TREE_QEMU_UPSTREAM                               \
+                tree_qemu=$TREE_QEMU                                         \
+                tree_seabios=$TREE_SEABIOS                                   \
+                tree_libvirt=$TREE_LIBVIRT                                   \
+                tree_ovmf=$TREE_OVMF                                         \
+                revision_xen=$REVISION_XEN                                   \
+                revision_qemu=$REVISION_QEMU                                 \
+                revision_qemuu=$REVISION_QEMU_UPSTREAM                       \
+                revision_seabios=$REVISION_SEABIOS                           \
+                revision_ovmf=$REVISION_OVMF                                 \
+                revision_libvirt=$REVISION_LIBVIRT                           \
+                revision_raisin=${REVISION_RAISIN:-${DEFAULT_REVISION_RAISIN}}\
+
+    fi
+
     if branch_wants_rumpkernel_tests; then
 
     case $arch in
diff --git a/sg-run-job b/sg-run-job
index eae159d..449118d 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -346,6 +346,7 @@ proc need-hosts/build {} { return BUILD }
 proc need-hosts/build-kern {} { return BUILD }
 proc need-hosts/build-libvirt {} { return BUILD }
 proc need-hosts/build-rumpuserxen {} { return BUILD }
+proc need-hosts/build-raisin {} { return BUILD }
 
 proc run-job/build {} {
     run-ts . = ts-xen-build
@@ -364,6 +365,10 @@ proc run-job/build-rumpuserxen {} {
     run-ts . = ts-xen-build + host tools
 }
 
+proc run-job/build-raisin {} {
+    run-ts . = ts-raisin-build
+}
+
 proc prepare-build-host {} {
     global jobinfo
     run-ts broken = ts-hosts-allocate + host
diff --git a/ts-raisin-build b/ts-raisin-build
new file mode 100755
index 0000000..85f12d4
--- /dev/null
+++ b/ts-raisin-build
@@ -0,0 +1,128 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use File::Path;
+use POSIX;
+use Osstest::TestSupport;
+use Osstest::BuildSupport;
+
+tsreadconfig();
+selectbuildhost(\@ARGV);
+# remaining arguments are passed as targets to "make"
+builddirsprops();
+
+my $raisindir="$builddir/raisin";
+
+sub checkout () {
+    prepbuilddirs();
+    build_clone($ho, 'raisin', $builddir, 'raisin');
+
+    target_cmd_build($ho, 100, $builddir, <<END.
+        cd $raisindir
+        cp defconfig config
+    >>config
+
+    echo >>config MAKE=\\"make $makeflags\\"
+    echo >>config PREFIX=\\"/usr\\"
+    echo >>config DESTDIR=dist
+    echo >>config ENABLED_COMPONENTS=\\"seabios ovmf xen qemu qemu_traditional libvirt\\"
+END
+               (nonempty($r{tree_xen}) && nonempty($r{revision_xen}) ? <<END : '').
+    echo >>config XEN_URL=\\"$r{tree_xen}\\"
+    echo >>config XEN_REVISION=\\"$r{revision_xen}\\"
+END
+               (nonempty($r{tree_qemuu}) && nonempty($r{revision_qemuu}) ? <<END : '').
+    echo >>config QEMU_URL=\\"$r{tree_qemuu}\\"
+    echo >>config QEMU_REVISION=\\"$r{revision_qemuu}\\"
+END
+               (nonempty($r{tree_qemu}) && nonempty($r{revision_qemu}) ? <<END : '').
+    echo >>config QEMU_TRADITIONAL_URL=\\"$r{tree_qemu}\\"
+    echo >>config QEMU_TRADITIONAL_REVISION=\\"$r{revision_qemu}\\"
+END
+               (nonempty($r{tree_seabios}) && nonempty($r{revision_seabios}) ? <<END : '').
+    echo >>config SEABIOS_URL=\\"$r{tree_seabios}\\"
+    echo >>config SEABIOS_REVISION=\\"$r{revision_seabios}\\"
+END
+               (nonempty($r{tree_libvirt}) && nonempty($r{revision_libvirt}) ? <<END : '').
+    echo >>config LIBVIRT_URL=\\"$r{tree_libvirt}\\"
+    echo >>config LIBVIRT_REVISION=\\"$r{revision_libvirt}\\"
+END
+               (nonempty($r{tree_ovmf}) && nonempty($r{revision_ovmf}) ? <<END : '')
+    echo >>config OVMF_URL=\\"$r{tree_ovmf}\\"
+    echo >>config OVMF_REVISION=\\"$r{revision_ovmf}\\"
+END
+               );
+}
+
+sub build () {
+    target_cmd_build($ho, 9000, $builddir, <<END);
+        cd $raisindir
+        ./raise -n build
+END
+#/;
+}
+
+sub checkoutput () {
+    target_cmd_build($ho, 100, $builddir, <<END);
+            cd $raisindir/dist
+            ls boot/xen.gz
+            ls usr/sbin/xenstored
+            ls usr/sbin/xenconsoled
+            ls usr/lib/libxenlight.so
+            ls usr/sbin/xl
+            ls usr/lib/xen/boot/hvmloader
+            ls usr/lib/xen/bin/qemu-dm
+            ls usr/lib/xen/bin/qemu-system-i386
+            ls usr/sbin/libvirtd
+END
+#/;
+}
+
+sub collectversions () {
+    store_revision($ho, 'xen', "$raisindir/xen-dir", 1);
+    store_revision($ho, 'qemu', "$raisindir/qemu-traditional-dir", 1);
+    store_revision($ho, 'qemuu', "$raisindir/qemu-dir", 1);
+    store_revision($ho, 'seabios', "$raisindir/seabios-dir", 1);
+    store_revision($ho, 'ovmf', "$raisindir/ovmf-dir", 1);
+    store_revision($ho, 'libvirt', "$raisindir/libvirt-dir", 1);
+}
+
+sub stash () {
+    foreach my $part ('', 'xen') {
+        built_stash($ho, $builddir,
+                    "raisin/${part}dist",
+                    "${part}dist");
+    }
+    built_stash_file($ho, $builddir, "xen-syms", "raisin/xen-dir/xen/xen-syms", 1);
+    built_stash_file($ho, $builddir, "raisin-config", "raisin/config", 1);
+    built_stash_file($ho, $builddir, "seabios-config",
+             "raisin/seabios-dir/.config", 1);
+    built_compress_stashed("xen-syms");
+}
+
+
+checkout();
+trapping(\&build);
+trapping(\&checkoutput);
+trapping(\&collectversions);
+divide_xen_build('raisin', 'dist');
+stash();
+die_if_fail();
+
diff --git a/ts-xen-build b/ts-xen-build
index e757f68..10fc95b 100755
--- a/ts-xen-build
+++ b/ts-xen-build
@@ -145,36 +145,6 @@ sub collectversions () {
     store_revision($ho, 'ovmf', "$tools/firmware/ovmf-dir", 1);
 }
 
-sub divide () {
-    # Only move hv to xeninstall, so that we can have
-    # xenpolicy in tools tarball.
-    #
-    # The files inside boot/ after `make dist' are
-    # xen-$XEN_VERSION: Xen binary
-    # xen.gz/xen: symlink to xen-$XEN_VERSION
-    # xen-$MAJOR: symlink to xen-$XEN_VERSION
-    # xen-$MAJOR.$MINOR: symlink to xen-$XEN_VERSION
-    # xen-sym-$XEN_VERSION: Xen symbol
-    # xenpolicy-$XEN_VERSION: flask policy binary if xsm is enabled
-    #
-    # So the following snippet will leave xenpolicy* in
-    # install/boot and get packaged to tools tarball.
-    target_cmd_build($ho, 100, $builddir, <<END);
-        cd xen/dist
-        mkdir xeninstall
-	for f in *install; do
-	    mkdir -p \$f/lib
-	done
-        if test -d install/boot; then
-            if test -f install/boot/xen.gz || test -f install/boot/xen; then
-                mkdir xeninstall/boot
-                mvfiles=`find install/boot -name 'xen[a-z]*' -prune -o -name 'xen*' -print`
-                mv \$mvfiles xeninstall/boot/.
-            fi
-        fi
-END
-}
-
 sub stash () {
     foreach my $part ('', 'xen') {
         built_stash($ho, $builddir,
@@ -212,22 +182,12 @@ END
     $chk->finish();
 }
 
-our @probs;
-
-sub trapping ($) {
-    my ($sub) = @_;
-    my $tok= eval { $sub->(); 1; };
-    if (!$tok) { push @probs, $@; print STDERR "failure (trapped): $@\n"; }
-}
-
 checkout();
 
 trapping(\&build);
 trapping(\&collectversions);
+die_if_fail();
 
-die "*** something failed:\n\n".(join "\n\n",@probs)."\n** something failed"
-    if @probs;
-
-divide();
+divide_xen_build('xen/dist', 'install');
 stash();
 checkversions();
-- 
1.7.10.4

[PATCH v8 2/2] OSSTest: push successful raisin builds

[Stefano Stabellini]

Determine the most recent raisin revision that needs to be tested, by comparing
the staging with the master branches.  Push to raisin.git:master when the build
is successful.

Do not run any jobs on the raisin flight.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

---

Changes in v8:
- add raisin to BRANCHES at the top of cr-for-branches
- do not run jobs on the raisin flight
- switch to the common staging/master branch names
---
 ap-common            |    3 +++
 ap-fetch-version     |    3 +++
 ap-fetch-version-old |    5 +++++
 ap-print-url         |    3 +++
 ap-push              |    5 +++++
 cr-daily-branch      |    8 ++++++++
 cr-for-branches      |    2 +-
 cri-common           |    1 +
 make-flight          |    3 +++
 9 files changed, 32 insertions(+), 1 deletion(-)

diff --git a/ap-common b/ap-common
index 6e1c472..34613bf 100644
--- a/ap-common
+++ b/ap-common
@@ -48,6 +48,8 @@
 # (ie ones which need $bodges=1 in ts-rumpuserxen-build)
 
 : ${TREE_RAISIN:=git://xenbits.xen.org/raisin.git}
+: ${PUSH_TREE_RAISIN:=$XENBITS:/home/xen/git/raisin.git}
+: ${BASE_TREE_RAISIN:=$TREE_RAISIN}
 
 : ${TREE_SEABIOS_UPSTREAM:=git://git.seabios.org/seabios.git}
 : ${PUSH_TREE_SEABIOS:=$XENBITS:/home/xen/git/osstest/seabios.git}
@@ -84,6 +86,7 @@ fi
 : ${LOCALREV_RUMPUSERXEN:=daily-cron.$branch}
 : ${LOCALREV_SEABIOS:=daily-cron.$branch}
 : ${LOCALREV_OVMF:=daily-cron.$branch}
+: ${LOCALREV_RAISIN:=daily-cron.$branch}
 
 : ${TREEBASE_LINUX_XCP:=http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27}
 
diff --git a/ap-fetch-version b/ap-fetch-version
index 33aaf00..2788d9f 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -80,6 +80,9 @@ libvirt)
 rumpuserxen)
 	repo_tree_rev_fetch_git rumpuserxen \
 		$TREE_RUMPUSERXEN master $LOCALREV_RUMPUSERXEN
+raisin)
+	repo_tree_rev_fetch_git raisin \
+		$TREE_RAISIN staging $LOCALREV_RAISIN
 	;;
 seabios)
 	repo_tree_rev_fetch_git seabios \
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index 0b4739b..f0c9b7e 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -29,6 +29,7 @@ select_xenbranch
 : ${BASE_LOCALREV_LINUX:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_LIBVIRT:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_RUMPUSERXEN:=daily-cron.$branch.old}
+: ${BASE_LOCALREV_RAISIN:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_SEABIOS:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_OVMF:=daily-cron.$branch.old}
 
@@ -89,6 +90,10 @@ rumpuserxen)
 	repo_tree_rev_fetch_git rumpuserxen \
 		$BASE_TREE_RUMPUSERXEN xen-tested-master $BASE_LOCALREV_RUMPUSERXEN
 	;;
+raisin)
+	repo_tree_rev_fetch_git raisin \
+		$BASE_TREE_RAISIN master $BASE_LOCALREV_RAISIN
+	;;
 seabios)
 	repo_tree_rev_fetch_git seabios \
 		$BASE_TREE_SEABIOS xen-tested-master $BASE_LOCALREV_SEABIOS
diff --git a/ap-print-url b/ap-print-url
index 3db2375..e999b65 100755
--- a/ap-print-url
+++ b/ap-print-url
@@ -55,6 +55,9 @@ libvirt)
 rumpuserxen)
 	echo $TREE_RUMPUSERXEN
 	;;
+raisin)
+	echo $TREE_RAISIN
+	;;
 seabios)
 	echo $TREE_SEABIOS_UPSTREAM
 	;;
diff --git a/ap-push b/ap-push
index c141464..2cc3287 100755
--- a/ap-push
+++ b/ap-push
@@ -37,6 +37,7 @@ TREE_LIBVIRT=$PUSH_TREE_LIBVIRT
 TREE_RUMPUSERXEN=$PUSH_TREE_RUMPUSERXEN
 TREE_SEABIOS=$PUSH_TREE_SEABIOS
 TREE_OVMF=$PUSH_TREE_OVMF
+TREE_RAISIN=$PUSH_TREE_RAISIN
 
 if info_linux_tree "$branch"; then
 	cd $repos/linux
@@ -89,6 +90,10 @@ rumpuserxen)
 	cd $repos/rumpuserxen
 	git push $TREE_RUMPUSERXEN $revision:xen-tested-master
 	;;
+raisin)
+	cd $repos/raisin
+	git push $TREE_RAISIN $revision:master
+	;;
 seabios)
 	cd $repos/seabios
 	git push $TREE_SEABIOS $revision:refs/heads/xen-tested-master
diff --git a/cr-daily-branch b/cr-daily-branch
index c7d1071..0b7bb78 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -164,6 +164,10 @@ if [ "x$REVISION_RUMPUSERXEN" = x ]; then
 	determine_version REVISION_RUMPUSERXEN rumpuserxen RUMPUSERXEN
 	export REVISION_RUMPUSERXEN
 fi
+if [ "x$REVISION_RAISIN" = x ]; then
+	determine_version REVISION_RAISIN raisin RAISIN
+	export REVISION_RAISIN
+fi
 if [ "x$REVISION_LINUXFIRMWARE" = x ]; then
 	determine_version REVISION_LINUXFIRMWARE linuxfirmware LINUXFIRMWARE
         export REVISION_LINUXFIRMWARE
@@ -209,6 +213,10 @@ rumpuserxen)
 	export REVISION_LINUX_OLD=disable
 	export REVISION_LIBVIRT=disable
 	;;
+raisin)
+	realtree=raisin
+	NEW_REVISION=$REVISION_RAISIN
+	;;
 seabios)
 	realtree=seabios
 	NEW_REVISION=$REVISION_SEABIOS
diff --git a/cr-for-branches b/cr-for-branches
index 7a93a62..37b37b9 100755
--- a/cr-for-branches
+++ b/cr-for-branches
@@ -31,7 +31,7 @@ scriptoptions="$1"; shift
 LOGFILE=tmp/cr-for-branches.log
 export LOGFILE
 
-: ${BRANCHES:=osstest xen-4.0-testing xen-4.1-testing xen-4.2-testing xen-4.3-testing xen-4.4-testing xen-4.5-testing xen-unstable qemu-mainline qemu-upstream-unstable qemu-upstream-4.2-testing qemu-upstream-4.3-testing qemu-upstream-4.4-testing qemu-upstream-4.5-testing linux-3.16 linux-3.14 linux-3.10 linux-3.4 linux-arm-xen seabios ovmf ${EXTRA_BRANCHES}}
+: ${BRANCHES:=osstest xen-4.0-testing xen-4.1-testing xen-4.2-testing xen-4.3-testing xen-4.4-testing xen-4.5-testing xen-unstable qemu-mainline qemu-upstream-unstable qemu-upstream-4.2-testing qemu-upstream-4.3-testing qemu-upstream-4.4-testing qemu-upstream-4.5-testing linux-3.16 linux-3.14 linux-3.10 linux-3.4 linux-arm-xen seabios ovmf raisin ${EXTRA_BRANCHES}}
 export BRANCHES
 
 fetchwlem=$wlem
diff --git a/cri-common b/cri-common
index ad44546..c091622 100644
--- a/cri-common
+++ b/cri-common
@@ -70,6 +70,7 @@ select_xenbranch () {
 	linuxfirmware)	    tree=linuxfirmware;	xenbranch=xen-unstable ;;
 	libvirt)		tree=libvirt;	xenbranch=xen-unstable ;;
 	rumpuserxen)	      tree=rumpuserxen; xenbranch=xen-unstable ;;
+	raisin)			tree=raisin;	xenbranch=xen-unstable ;;
 	seabios)		tree=seabios;	xenbranch=xen-unstable ;;
 	ovmf)			tree=ovmf;	xenbranch=xen-unstable ;;
 	osstest)		tree=osstest;	xenbranch=xen-unstable ;;
diff --git a/make-flight b/make-flight
index cc8ecdb..bd4052d 100755
--- a/make-flight
+++ b/make-flight
@@ -89,6 +89,9 @@ job_create_test_filter_callback () {
           *) return 1;;
       esac
       ;;
+    raisin)
+      # no tests jobs with raisin yet
+      return 1;;
     *)
       case "$job" in
         *-qemuu-*)
-- 
1.7.10.4

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Ian Jackson]

Stefano Stabellini writes ("[PATCH v8 1/2] OSSTEST: introduce a raisin build test"):
> Make divide_xen_build common between ts-raisin-build and ts-xen-build.
> It is used to split the build output in two archives: one with the
> hypervisor binary and another with the userspace tools.

This refactoring should be in its own patch.

> Introduce a raisin build test to OSSTest. The build is meant to test
> Raisin itself.
> 
> Enable the raisin build just for xen-unstable. Disable it for all the
> older Xen branches and for all the non-Xen branches.
> 
> The components built by the raisin job are static; the list is written
> by ts-raisin-build. Don't automatically generate the list as the
> component names are not regular. The versions of the components and
> their git tree urls are passed by mfi-common to ts-raisin-build.
> mfi-common passes the same TREE_ and REVISION_ variables used with the
> other jobs.  ts-raisin-build writes them to the raisin config file,
> overwriting the defaults, only if both the tree and revision variables
> for a component are set, to avoid inconsistencies.  For example
> mfi-common can pass a revision only present in a specific git tree, but
> not the corresponding git tree.
> 
> 
> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> 
> ---
> 
> Changes in v8:
> - update commit message
> - do not install-builddep in ts-raisin-build, rely on ts-xen-build-dep
> 
> Changes in v7:
> - update raisin git url after http://marc.info/?l=xen-devel&m=143472251602803
> 
> Changes in v6:
> - move divide to Osstest/BuildSupport.pm and rename it to
> divide_xen_build; use divide_xen_build in ts-raisin-build and
> ts-xen-build.
> 
> Changes in v5:
> - remove setting DEFAULT_REVISION_RAISIN to master
> - disable raisin when branch is not xen*
> - do not overwrite raisin default variables unless both revision_* and
> tree_* are set of a given component
> 
> Changes in v4:
> - introduce enable_raisin in mfi-common: only build raisin when building
> xen-unstable
> - start off from the default raisin config, then append osstest config
> options to it
> - do not write variable to the raisin config if the conrresponding
> runvar is not set
> - remove TREE_OVMF and TREE_SEABIOS
> 
> Changes in v3:
> - use $raisindir throughout ts-raisin-build
> - do not specify ENABLED_COMPONENTS so that empty REVISION variables can
> be used to disable building a raisin component
> 
> Changes in v2:
> - set revision_* variables in mfi-common;
> - in ts-raisin-build set the *_REVISION config options based on the
>   revision_* variables;
> - in ts-raisin-build, call store_revision appropriately;
> - divide the output in an hypervisor and a tools tarball.
> ---
>  Osstest/BuildSupport.pm |   48 ++++++++++++++++++
>  ap-common               |    2 +
>  mfi-common              |   41 +++++++++++++++
>  sg-run-job              |    5 ++
>  ts-raisin-build         |  128 +++++++++++++++++++++++++++++++++++++++++++++++
>  ts-xen-build            |   44 +---------------
>  6 files changed, 226 insertions(+), 42 deletions(-)
>  create mode 100755 ts-raisin-build
> 
> diff --git a/Osstest/BuildSupport.pm b/Osstest/BuildSupport.pm
> index 933f6e1..76a3feb 100644
> --- a/Osstest/BuildSupport.pm
> +++ b/Osstest/BuildSupport.pm
> @@ -43,6 +43,10 @@ BEGIN {
>                        xendist
>                        $xendist
>  
> +                      divide_xen_build
> +                      trapping
> +                      die_if_fail
> +
>                        submodulefixup submodule_have submodule_find
>  
>                        );
> @@ -84,6 +88,50 @@ sub xendist () {
>  	($ho, 'xendist', '', $r{"buildjob"});
>  }
>  
> +sub divide_xen_build ($$) {
> +    my ($subdir, $postfix) = @_;
> +    # Only move hv to xeninstall, so that we can have
> +    # xenpolicy in tools tarball.
> +    #
> +    # The files inside boot/ after `make dist' are
> +    # xen-$XEN_VERSION: Xen binary
> +    # xen.gz/xen: symlink to xen-$XEN_VERSION
> +    # xen-$MAJOR: symlink to xen-$XEN_VERSION
> +    # xen-$MAJOR.$MINOR: symlink to xen-$XEN_VERSION
> +    # xen-sym-$XEN_VERSION: Xen symbol
> +    # xenpolicy-$XEN_VERSION: flask policy binary if xsm is enabled
> +    #
> +    # So the following snippet will leave xenpolicy* in
> +    # install/boot and get packaged to tools tarball.
> +    target_cmd_build($ho, 100, $builddir, <<END);
> +        cd $subdir
> +        mkdir xen$postfix
> +        for f in *$postfix; do
> +            mkdir -p \$f/lib
> +        done
> +        if test -d $postfix/boot; then
> +            if test -f $postfix/boot/xen.gz || test -f $postfix/boot/xen; then
> +                mkdir xen$postfix/boot
> +                mvfiles=`find $postfix/boot -name 'xen[a-z]*' -prune -o -name 'xen*' -print`
> +                mv \$mvfiles xen$postfix/boot/.
> +            fi
> +        fi
> +END
> +}
> +
> +our @probs;
> +
> +sub trapping ($) {
> +    my ($sub) = @_;
> +    my $tok= eval { $sub->(); 1; };
> +    if (!$tok) { push @probs, $@; print STDERR "failure (trapped): $@\n"; }
> +}
> +
> +sub die_if_fail () {
> +    die "*** something failed:\n\n".(join "\n\n",@probs)."\n** something failed"
> +        if @probs;
> +}
> +
>  #----- submodules -----
>  
>  sub submodulefixup ($$$$) {
> diff --git a/ap-common b/ap-common
> index 64749e3..6e1c472 100644
> --- a/ap-common
> +++ b/ap-common
> @@ -47,6 +47,8 @@
>  # rumpsrc-related runvars needed only for old rumpuser-xen
>  # (ie ones which need $bodges=1 in ts-rumpuserxen-build)
>  
> +: ${TREE_RAISIN:=git://xenbits.xen.org/raisin.git}
> +
>  : ${TREE_SEABIOS_UPSTREAM:=git://git.seabios.org/seabios.git}
>  : ${PUSH_TREE_SEABIOS:=$XENBITS:/home/xen/git/osstest/seabios.git}
>  : ${BASE_TREE_SEABIOS:=git://xenbits.xen.org/osstest/seabios.git}
> diff --git a/mfi-common b/mfi-common
> index 16fc8c5..d75823d 100644
> --- a/mfi-common
> +++ b/mfi-common
> @@ -148,6 +148,22 @@ create_build_jobs () {
>      *) enable_ovmf=true;
>      esac
>  
> +    case "$xenbranch" in
> +    xen-3.*-testing) enable_raisin=false;;
> +    xen-4.0-testing) enable_raisin=false;;
> +    xen-4.1-testing) enable_raisin=false;;
> +    xen-4.2-testing) enable_raisin=false;;
> +    xen-4.3-testing) enable_raisin=false;;
> +    xen-4.4-testing) enable_raisin=false;;
> +    xen-4.5-testing) enable_raisin=false;;

How about

> +    xen-4.[0-5]-testing) enable_raisin=false;;

?

> @@ -215,6 +231,31 @@ create_build_jobs () {
>  
>      fi
>  
> +    if [ "x$REVISION_RAISIN" != xdisable ] && [ "$enable_raisin" = "true" ]; then
> +
> +    ./cs-job-create $flight build-$arch-raisin build-raisin                  \
> +                arch=$arch                                                   \
> +        tree_xen=$TREE_XEN                                                   \
> +                $RUNVARS $BUILD_RUNVARS $BUILD_RAISIN_RUNVARS $arch_runvars \
> +                $suite_runvars                                               \
> +                host_hostflags=$build_hostflags                              \
> +                buildjob=${bfi}build-$arch                                   \
> +                tree_raisin=$TREE_RAISIN                                     \
> +                tree_qemuu=$TREE_QEMU_UPSTREAM                               \
> +                tree_qemu=$TREE_QEMU                                         \
> +                tree_seabios=$TREE_SEABIOS                                   \
> +                tree_libvirt=$TREE_LIBVIRT                                   \
> +                tree_ovmf=$TREE_OVMF                                         \
> +                revision_xen=$REVISION_XEN                                   \
> +                revision_qemu=$REVISION_QEMU                                 \
> +                revision_qemuu=$REVISION_QEMU_UPSTREAM                       \
> +                revision_seabios=$REVISION_SEABIOS                           \
> +                revision_ovmf=$REVISION_OVMF                                 \
> +                revision_libvirt=$REVISION_LIBVIRT                           \
> +                revision_raisin=${REVISION_RAISIN:-${DEFAULT_REVISION_RAISIN}}\
> +
> +    fi
> +
>      if branch_wants_rumpkernel_tests; then
>  
>      case $arch in
> diff --git a/sg-run-job b/sg-run-job
> index eae159d..449118d 100755
> --- a/sg-run-job
> +++ b/sg-run-job
> @@ -346,6 +346,7 @@ proc need-hosts/build {} { return BUILD }
>  proc need-hosts/build-kern {} { return BUILD }
>  proc need-hosts/build-libvirt {} { return BUILD }
>  proc need-hosts/build-rumpuserxen {} { return BUILD }
> +proc need-hosts/build-raisin {} { return BUILD }
>  
>  proc run-job/build {} {
>      run-ts . = ts-xen-build
> @@ -364,6 +365,10 @@ proc run-job/build-rumpuserxen {} {
>      run-ts . = ts-xen-build + host tools
>  }
>  
> +proc run-job/build-raisin {} {
> +    run-ts . = ts-raisin-build
> +}
> +
>  proc prepare-build-host {} {
>      global jobinfo
>      run-ts broken = ts-hosts-allocate + host
> diff --git a/ts-raisin-build b/ts-raisin-build
> new file mode 100755
> index 0000000..85f12d4
> --- /dev/null
> +++ b/ts-raisin-build
> @@ -0,0 +1,128 @@
> +#!/usr/bin/perl -w
> +# This is part of "osstest", an automated testing framework for Xen.
> +# Copyright (C) 2009-2013 Citrix Inc.
> +# 
> +# This program is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU Affero General Public License as published by
> +# the Free Software Foundation, either version 3 of the License, or
> +# (at your option) any later version.
> +# 
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU Affero General Public License for more details.
> +# 
> +# You should have received a copy of the GNU Affero General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +
> +use strict qw(vars);
> +use DBI;
> +use Osstest;
> +use File::Path;
> +use POSIX;
> +use Osstest::TestSupport;
> +use Osstest::BuildSupport;
> +
> +tsreadconfig();
> +selectbuildhost(\@ARGV);
> +# remaining arguments are passed as targets to "make"
> +builddirsprops();
> +
> +my $raisindir="$builddir/raisin";
> +
> +sub checkout () {
> +    prepbuilddirs();
> +    build_clone($ho, 'raisin', $builddir, 'raisin');
> +
> +    target_cmd_build($ho, 100, $builddir, <<END.
> +        cd $raisindir
> +        cp defconfig config
> +    >>config
> +
> +    echo >>config MAKE=\\"make $makeflags\\"
> +    echo >>config PREFIX=\\"/usr\\"
> +    echo >>config DESTDIR=dist
> +    echo >>config ENABLED_COMPONENTS=\\"seabios ovmf xen qemu qemu_traditional libvirt\\"
> +END
> +               (nonempty($r{tree_xen}) && nonempty($r{revision_xen}) ? <<END : '').
> +    echo >>config XEN_URL=\\"$r{tree_xen}\\"
> +    echo >>config XEN_REVISION=\\"$r{revision_xen}\\"
> +END
> +               (nonempty($r{tree_qemuu}) && nonempty($r{revision_qemuu}) ? <<END : '').
> +    echo >>config QEMU_URL=\\"$r{tree_qemuu}\\"
> +    echo >>config QEMU_REVISION=\\"$r{revision_qemuu}\\"
> +END
> +               (nonempty($r{tree_qemu}) && nonempty($r{revision_qemu}) ? <<END : '').
> +    echo >>config QEMU_TRADITIONAL_URL=\\"$r{tree_qemu}\\"
> +    echo >>config QEMU_TRADITIONAL_REVISION=\\"$r{revision_qemu}\\"
> +END
> +               (nonempty($r{tree_seabios}) && nonempty($r{revision_seabios}) ? <<END : '').
> +    echo >>config SEABIOS_URL=\\"$r{tree_seabios}\\"
> +    echo >>config SEABIOS_REVISION=\\"$r{revision_seabios}\\"
> +END
> +               (nonempty($r{tree_libvirt}) && nonempty($r{revision_libvirt}) ? <<END : '').
> +    echo >>config LIBVIRT_URL=\\"$r{tree_libvirt}\\"
> +    echo >>config LIBVIRT_REVISION=\\"$r{revision_libvirt}\\"
> +END
> +               (nonempty($r{tree_ovmf}) && nonempty($r{revision_ovmf}) ? <<END : '')
> +    echo >>config OVMF_URL=\\"$r{tree_ovmf}\\"
> +    echo >>config OVMF_REVISION=\\"$r{revision_ovmf}\\"
> +END
> +               );
> +}
> +
> +sub build () {
> +    target_cmd_build($ho, 9000, $builddir, <<END);
> +        cd $raisindir
> +        ./raise -n build
> +END
> +#/;
> +}
> +
> +sub checkoutput () {
> +    target_cmd_build($ho, 100, $builddir, <<END);
> +            cd $raisindir/dist
> +            ls boot/xen.gz
> +            ls usr/sbin/xenstored
> +            ls usr/sbin/xenconsoled
> +            ls usr/lib/libxenlight.so
> +            ls usr/sbin/xl
> +            ls usr/lib/xen/boot/hvmloader
> +            ls usr/lib/xen/bin/qemu-dm
> +            ls usr/lib/xen/bin/qemu-system-i386
> +            ls usr/sbin/libvirtd
> +END
> +#/;
> +}
> +
> +sub collectversions () {
> +    store_revision($ho, 'xen', "$raisindir/xen-dir", 1);
> +    store_revision($ho, 'qemu', "$raisindir/qemu-traditional-dir", 1);
> +    store_revision($ho, 'qemuu', "$raisindir/qemu-dir", 1);
> +    store_revision($ho, 'seabios', "$raisindir/seabios-dir", 1);
> +    store_revision($ho, 'ovmf', "$raisindir/ovmf-dir", 1);
> +    store_revision($ho, 'libvirt', "$raisindir/libvirt-dir", 1);
> +}
> +
> +sub stash () {
> +    foreach my $part ('', 'xen') {
> +        built_stash($ho, $builddir,
> +                    "raisin/${part}dist",
> +                    "${part}dist");
> +    }
> +    built_stash_file($ho, $builddir, "xen-syms", "raisin/xen-dir/xen/xen-syms", 1);
> +    built_stash_file($ho, $builddir, "raisin-config", "raisin/config", 1);
> +    built_stash_file($ho, $builddir, "seabios-config",
> +             "raisin/seabios-dir/.config", 1);
> +    built_compress_stashed("xen-syms");
> +}
> +
> +
> +checkout();
> +trapping(\&build);
> +trapping(\&checkoutput);
> +trapping(\&collectversions);
> +divide_xen_build('raisin', 'dist');
> +stash();
> +die_if_fail();
> +
> diff --git a/ts-xen-build b/ts-xen-build
> index e757f68..10fc95b 100755
> --- a/ts-xen-build
> +++ b/ts-xen-build
> @@ -145,36 +145,6 @@ sub collectversions () {
>      store_revision($ho, 'ovmf', "$tools/firmware/ovmf-dir", 1);
>  }
>  
> -sub divide () {
> -    # Only move hv to xeninstall, so that we can have
> -    # xenpolicy in tools tarball.
> -    #
> -    # The files inside boot/ after `make dist' are
> -    # xen-$XEN_VERSION: Xen binary
> -    # xen.gz/xen: symlink to xen-$XEN_VERSION
> -    # xen-$MAJOR: symlink to xen-$XEN_VERSION
> -    # xen-$MAJOR.$MINOR: symlink to xen-$XEN_VERSION
> -    # xen-sym-$XEN_VERSION: Xen symbol
> -    # xenpolicy-$XEN_VERSION: flask policy binary if xsm is enabled
> -    #
> -    # So the following snippet will leave xenpolicy* in
> -    # install/boot and get packaged to tools tarball.
> -    target_cmd_build($ho, 100, $builddir, <<END);
> -        cd xen/dist
> -        mkdir xeninstall
> -	for f in *install; do
> -	    mkdir -p \$f/lib
> -	done
> -        if test -d install/boot; then
> -            if test -f install/boot/xen.gz || test -f install/boot/xen; then
> -                mkdir xeninstall/boot
> -                mvfiles=`find install/boot -name 'xen[a-z]*' -prune -o -name 'xen*' -print`
> -                mv \$mvfiles xeninstall/boot/.
> -            fi
> -        fi
> -END
> -}
> -
>  sub stash () {
>      foreach my $part ('', 'xen') {
>          built_stash($ho, $builddir,
> @@ -212,22 +182,12 @@ END
>      $chk->finish();
>  }
>  
> -our @probs;
> -
> -sub trapping ($) {
> -    my ($sub) = @_;
> -    my $tok= eval { $sub->(); 1; };
> -    if (!$tok) { push @probs, $@; print STDERR "failure (trapped): $@\n"; }
> -}
> -
>  checkout();
>  
>  trapping(\&build);
>  trapping(\&collectversions);
> +die_if_fail();
>  
> -die "*** something failed:\n\n".(join "\n\n",@probs)."\n** something failed"
> -    if @probs;
> -
> -divide();
> +divide_xen_build('xen/dist', 'install');
>  stash();
>  checkversions();
> -- 
> 1.7.10.4
> 

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Ian Jackson]

Ian Jackson writes ("Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test"):
> Stefano Stabellini writes ("[PATCH v8 1/2] OSSTEST: introduce a raisin build test"):
> > Make divide_xen_build common between ts-raisin-build and ts-xen-build.
> > It is used to split the build output in two archives: one with the
> > hypervisor binary and another with the userspace tools.
> 
> This refactoring should be in its own patch.

Sorry, I sent that mail before it was finished.  I'll send a followup.

Ian.

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Ian Jackson]

(Please disregard my previous incomplete mail about this patch.  This
reply has all of my comments.  Thanks.)

Stefano Stabellini writes ("[PATCH v8 1/2] OSSTEST: introduce a raisin build test"):
> Make divide_xen_build common between ts-raisin-build and ts-xen-build.
> It is used to split the build output in two archives: one with the
> hypervisor binary and another with the userspace tools.

This refactoring should be in its own patch.

> +    case "$xenbranch" in
> +    xen-3.*-testing) enable_raisin=false;;
> +    xen-4.0-testing) enable_raisin=false;;
> +    xen-4.1-testing) enable_raisin=false;;
> +    xen-4.2-testing) enable_raisin=false;;
> +    xen-4.3-testing) enable_raisin=false;;
> +    xen-4.4-testing) enable_raisin=false;;
> +    xen-4.5-testing) enable_raisin=false;;

How about

> +    xen-4.[0-5]-testing) enable_raisin=false;;

?

> +    echo >>config ENABLED_COMPONENTS=\\"seabios ovmf xen qemu qemu_traditional libvirt\\"
> +END
> +               (nonempty($r{tree_xen}) && nonempty($r{revision_xen}) ? <<END : '').
> +    echo >>config XEN_URL=\\"$r{tree_xen}\\"
> +    echo >>config XEN_REVISION=\\"$r{revision_xen}\\"
> +END

This URL/REVISION setting is too repetetive.  If you improved it you
could combine it with your setting of ENABLED_COMPONENTS.

It's true that qemuu/QEMU_URL and qemu/QEMU_TRADITIONAL_URL etc. need
special handling but their specialness should be minimised.  Maybe an
anonymous subref for generating each stanza, which you call with
matching arguments in a loop for most components, and explicitly with
nonmatching arguments for the two qemus.

> +sub checkoutput () {
> +    target_cmd_build($ho, 100, $builddir, <<END);
> +            cd $raisindir/dist
> +            ls boot/xen.gz
> +            ls usr/sbin/xenstored
> +            ls usr/sbin/xenconsoled
> +            ls usr/lib/libxenlight.so
> +            ls usr/sbin/xl
> +            ls usr/lib/xen/boot/hvmloader
> +            ls usr/lib/xen/bin/qemu-dm
> +            ls usr/lib/xen/bin/qemu-system-i386
> +            ls usr/sbin/libvirtd
> +END

I'm not sure why not `ls -l'.

> +sub collectversions () {
> +    store_revision($ho, 'xen', "$raisindir/xen-dir", 1);
> +    store_revision($ho, 'qemu', "$raisindir/qemu-traditional-dir", 1);
> +    store_revision($ho, 'qemuu', "$raisindir/qemu-dir", 1);
> +    store_revision($ho, 'seabios', "$raisindir/seabios-dir", 1);
> +    store_revision($ho, 'ovmf', "$raisindir/ovmf-dir", 1);
> +    store_revision($ho, 'libvirt', "$raisindir/libvirt-dir", 1);

This list of revisions should come from the same place as the list of
config settings, above.

Thanks,
Ian.

Re: [PATCH v8 2/2] OSSTest: push successful raisin builds

[Ian Jackson]

Stefano Stabellini writes ("[PATCH v8 2/2] OSSTest: push successful raisin builds"):
> Determine the most recent raisin revision that needs to be tested,
> by comparing the staging with the master branches.  Push to
> raisin.git:master when the build is successful.
> 
> Do not run any jobs on the raisin flight.

The raisin flight should probably lack most of the other build jobs,
too ?

Ian.

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Stefano Stabellini]

On Fri, 31 Jul 2015, Ian Jackson wrote:
> (Please disregard my previous incomplete mail about this patch.  This
> reply has all of my comments.  Thanks.)
> 
> Stefano Stabellini writes ("[PATCH v8 1/2] OSSTEST: introduce a raisin build test"):
> > Make divide_xen_build common between ts-raisin-build and ts-xen-build.
> > It is used to split the build output in two archives: one with the
> > hypervisor binary and another with the userspace tools.
> 
> This refactoring should be in its own patch.

OK


> > +    case "$xenbranch" in
> > +    xen-3.*-testing) enable_raisin=false;;
> > +    xen-4.0-testing) enable_raisin=false;;
> > +    xen-4.1-testing) enable_raisin=false;;
> > +    xen-4.2-testing) enable_raisin=false;;
> > +    xen-4.3-testing) enable_raisin=false;;
> > +    xen-4.4-testing) enable_raisin=false;;
> > +    xen-4.5-testing) enable_raisin=false;;
> 
> How about
> 
> > +    xen-4.[0-5]-testing) enable_raisin=false;;
> 
> ?

OK


> > +    echo >>config ENABLED_COMPONENTS=\\"seabios ovmf xen qemu qemu_traditional libvirt\\"
> > +END
> > +               (nonempty($r{tree_xen}) && nonempty($r{revision_xen}) ? <<END : '').
> > +    echo >>config XEN_URL=\\"$r{tree_xen}\\"
> > +    echo >>config XEN_REVISION=\\"$r{revision_xen}\\"
> > +END
> 
> This URL/REVISION setting is too repetetive.  If you improved it you
> could combine it with your setting of ENABLED_COMPONENTS.
> 
> It's true that qemuu/QEMU_URL and qemu/QEMU_TRADITIONAL_URL etc. need
> special handling but their specialness should be minimised.  Maybe an
> anonymous subref for generating each stanza, which you call with
> matching arguments in a loop for most components, and explicitly with
> nonmatching arguments for the two qemus.

I disagree on this: some things are not worth optimizing.

Nonetheless I would make the changes you are requesting, but as
explained in the past, what you are asking is beyond my knowledge of the
language. Furthermore the snippet of code that was provided to me on the
13th of May did not actually work.

In conclusion, given that this is v8, I'll leave this code as is. Feel
free to change it the way you think is best.



> > +sub checkoutput () {
> > +    target_cmd_build($ho, 100, $builddir, <<END);
> > +            cd $raisindir/dist
> > +            ls boot/xen.gz
> > +            ls usr/sbin/xenstored
> > +            ls usr/sbin/xenconsoled
> > +            ls usr/lib/libxenlight.so
> > +            ls usr/sbin/xl
> > +            ls usr/lib/xen/boot/hvmloader
> > +            ls usr/lib/xen/bin/qemu-dm
> > +            ls usr/lib/xen/bin/qemu-system-i386
> > +            ls usr/sbin/libvirtd
> > +END
> 
> I'm not sure why not `ls -l'.

OK


> > +sub collectversions () {
> > +    store_revision($ho, 'xen', "$raisindir/xen-dir", 1);
> > +    store_revision($ho, 'qemu', "$raisindir/qemu-traditional-dir", 1);
> > +    store_revision($ho, 'qemuu', "$raisindir/qemu-dir", 1);
> > +    store_revision($ho, 'seabios', "$raisindir/seabios-dir", 1);
> > +    store_revision($ho, 'ovmf', "$raisindir/ovmf-dir", 1);
> > +    store_revision($ho, 'libvirt', "$raisindir/libvirt-dir", 1);
> 
> This list of revisions should come from the same place as the list of
> config settings, above.

I guess you mean the list of revisions to store? As in xen, qemu, qemuu,
seabios, ovmf, libvirt? The list is currently hardcoded and names of the
corresponding repositories are not regular, so for similar reasons as
above I would prefer to keep it as is.

Re: [PATCH v8 2/2] OSSTest: push successful raisin builds

[Stefano Stabellini]

On Fri, 31 Jul 2015, Ian Jackson wrote:
> Stefano Stabellini writes ("[PATCH v8 2/2] OSSTest: push successful raisin builds"):
> > Determine the most recent raisin revision that needs to be tested,
> > by comparing the staging with the master branches.  Push to
> > raisin.git:master when the build is successful.
> > 
> > Do not run any jobs on the raisin flight.
> 
> The raisin flight should probably lack most of the other build jobs,
> too ?

Yes, that's right. I think that the following, part of the first patch,
should already take care of that?


diff --git a/mfi-common b/mfi-common
index 16fc8c5..8d6ab3e 100644
--- a/mfi-common
+++ b/mfi-common
@@ -148,6 +148,17 @@ create_build_jobs () {
     *) enable_ovmf=true;
     esac
 
+    case "$xenbranch" in
+    xen-3.*-testing) enable_raisin=false;;
+    xen-4.[0-5]-testing) enable_raisin=false;;
+    *)               enable_raisin=true;
+    esac
+    case "$branch" in
+    xen*) ;;
+    *) enable_raisin=false;;
+    esac
+
+
     eval "
         arch_runvars=\"\$ARCH_RUNVARS_$arch\"
     "

Re: [PATCH v8 2/2] OSSTest: push successful raisin builds

[Ian Campbell]

On Mon, 2015-08-03 at 18:09 +0100, Stefano Stabellini wrote:
> On Fri, 31 Jul 2015, Ian Jackson wrote:
> > Stefano Stabellini writes ("[PATCH v8 2/2] OSSTest: push successful 
> > raisin builds"):
> > > Determine the most recent raisin revision that needs to be tested,
> > > by comparing the staging with the master branches.  Push to
> > > raisin.git:master when the build is successful.
> > > 
> > > Do not run any jobs on the raisin flight.
> > 
> > The raisin flight should probably lack most of the other build jobs,
> > too ?
> 
> Yes, that's right. I think that the following, part of the first patch,
> should already take care of that?

That enables raisin builds, but doesn't (AFAICT) disable the other builds.

In the latest osstest you can use make
-flight:job_create_build_filter_callback to remove non raisin builds,
something like:

if [ "$branch" = "raisin" ] then
    case " $* " in
    *" enable_raisin=false "*) return 1;;
    esac
fi

(look for the usage in make-distros-flight for an example, also
job_create_test_filter_callback has similar constructs)

Ian.

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Ian Campbell]

On Thu, 2015-07-23 at 17:52 +0100, Stefano Stabellini wrote:
> diff --git a/ts-xen-build b/ts-xen-build
> index e757f68..10fc95b 100755
> --- a/ts-xen-build
> +++ b/ts-xen-build
> @@ -145,36 +145,6 @@ sub collectversions () {
>      store_revision($ho, 'ovmf', "$tools/firmware/ovmf-dir", 1);
>  }
>  
> -sub divide () {

FYI this needs rebasing over "Collect xen.efi into xendist and install in
appropriate place".

> -    # Only move hv to xeninstall, so that we can have
> -    # xenpolicy in tools tarball.
> -    #
> -    # The files inside boot/ after `make dist' are
> -    # xen-$XEN_VERSION: Xen binary
> -    # xen.gz/xen: symlink to xen-$XEN_VERSION
> -    # xen-$MAJOR: symlink to xen-$XEN_VERSION
> -    # xen-$MAJOR.$MINOR: symlink to xen-$XEN_VERSION
> -    # xen-sym-$XEN_VERSION: Xen symbol
> -    # xenpolicy-$XEN_VERSION: flask policy binary if xsm is enabled
> -    #
> -    # So the following snippet will leave xenpolicy* in
> -    # install/boot and get packaged to tools tarball.
> -    target_cmd_build($ho, 100, $builddir, <<END);
> -        cd xen/dist
> -        mkdir xeninstall
> -       for f in *install; do
> -           mkdir -p \$f/lib
> -       done
> -        if test -d install/boot; then
> -            if test -f install/boot/xen.gz || test -f install/boot/xen; 
> then
> -                mkdir xeninstall/boot
> -                mvfiles=`find install/boot -name 'xen[a-z]*' -prune -o 
> -name 'xen*' -print`
> -                mv \$mvfiles xeninstall/boot/.
> -            fi
> -        fi
> -END
> -}
> -

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Ian Campbell]

On Tue, 2015-08-04 at 09:35 +0100, Ian Campbell wrote:
> On Thu, 2015-07-23 at 17:52 +0100, Stefano Stabellini wrote:
> > diff --git a/ts-xen-build b/ts-xen-build
> > index e757f68..10fc95b 100755
> > --- a/ts-xen-build
> > +++ b/ts-xen-build
> > @@ -145,36 +145,6 @@ sub collectversions () {
> >      store_revision($ho, 'ovmf', "$tools/firmware/ovmf-dir", 1);
> >  }
> >  
> > -sub divide () {
> 
> FYI this needs rebasing over "Collect xen.efi into xendist and install in
> appropriate place".

The right hunk to the moved function is, I think:

@@ -116,6 +116,13 @@ sub divide_xen_build ($$) {
                 mv \$mvfiles xen$postfix/boot/.
             fi
         fi
+       if test -d $postfix/usr/lib64/efi/; then
+            if test -f $postfix/usr/lib64/efi/xen.efi; then
+                mkdir -p xen$postfix/usr/lib64/efi
+                mvfiles=`find $postfix/usr/lib64/efi -name 'xen[a-z]*' -prune -o -name 'xen*' -print`
+                mv \$mvfiles xen$postfix/usr/lib64/efi/.
+            fi
+       fi
 END
 }

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Ian Campbell]

On Thu, 2015-07-23 at 17:52 +0100, Stefano Stabellini wrote:
> 
> +    ./cs-job-create $flight build-$arch-raisin build-raisin                  \
> +                arch=$arch                                                   \
> +        tree_xen=$TREE_XEN                                                   \
> +                $RUNVARS $BUILD_RUNVARS $BUILD_RAISIN_RUNVARS $arch_runvars \
> +                $suite_runvars                                               \
> +                host_hostflags=$build_hostflags                              \
> +                buildjob=${bfi}build-$arch                                   \

This line means you can't run the build-$arch-raisin job unless the build
-$arch job is complete. Is that intentional?

I can see an argument for not testing raisin if the underlying Xen tree
doesn't build, it's a bit annoying in standalone mode is all...

Re: [PATCH v8 2/2] OSSTest: push successful raisin builds

[Ian Campbell]

On Mon, 2015-08-03 at 18:09 +0100, Stefano Stabellini wrote:
> +    case "$branch" in
> +    xen*) ;;
> +    *) enable_raisin=false;;
> +    esac

BTW, this doesn't actually enable raisin for builds on the raisin branch...

Ian.

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Ian Campbell]

On Mon, 2015-08-03 at 17:59 +0100, Stefano Stabellini wrote:
> 
> > > +    echo >>config ENABLED_COMPONENTS=\\"seabios ovmf xen qemu 
> > > qemu_traditional libvirt\\"
> > > +END
> > > +               (nonempty($r{tree_xen}) && nonempty($r{revision_xen}) 
> > > ? <<END : '').
> > > +    echo >>config XEN_URL=\\"$r{tree_xen}\\"
> > > +    echo >>config XEN_REVISION=\\"$r{revision_xen}\\"
> > > +END
> > 
> > This URL/REVISION setting is too repetetive.  If you improved it you
> > could combine it with your setting of ENABLED_COMPONENTS.
> > 
> > It's true that qemuu/QEMU_URL and qemu/QEMU_TRADITIONAL_URL etc. need
> > special handling but their specialness should be minimised.  Maybe an
> > anonymous subref for generating each stanza, which you call with
> > matching arguments in a loop for most components, and explicitly with
> > nonmatching arguments for the two qemus.
> 
> I disagree on this: some things are not worth optimizing.

This is not an optimisation, it is a maintainability improvement, having
multiple places which operate in an open coded way on a list of things
which are all supposed to remain in sync just means they will get out of
sync.

> Nonetheless I would make the changes you are requesting, but as
> explained in the past, what you are asking is beyond my knowledge of the
> language. Furthermore the snippet of code that was provided to me on the
> 13th of May did not actually work.

Perl is really not _so_ obscure or hard that this is much of an excuse,
even if you don't want to write it in a Perl-ish way you could get this
done just using constructs common to most languages, i.e. hashes, arrays
and loops.

Nevertheless, since this seems to be going nowhere I'll take your series
and fix this up for you.

> In conclusion, given that this is v8, I'll leave this code as is. Feel
> free to change it the way you think is best.

The fact we have made it to v8 without a piece of review commentary being
implemented is not a reason to continue to ignore it, nor to just accept
the patch as is IMHO.

> > > +sub collectversions () {
> > > +    store_revision($ho, 'xen', "$raisindir/xen-dir", 1);
> > > +    store_revision($ho, 'qemu', "$raisindir/qemu-traditional-dir", 
> > > 1);
> > > +    store_revision($ho, 'qemuu', "$raisindir/qemu-dir", 1);
> > > +    store_revision($ho, 'seabios', "$raisindir/seabios-dir", 1);
> > > +    store_revision($ho, 'ovmf', "$raisindir/ovmf-dir", 1);
> > > +    store_revision($ho, 'libvirt', "$raisindir/libvirt-dir", 1);
> > 
> > This list of revisions should come from the same place as the list of
> > config settings, above.
> 
> I guess you mean the list of revisions to store? As in xen, qemu, qemuu,
> seabios, ovmf, libvirt? The list is currently hardcoded and names of the
> corresponding repositories are not regular, so for similar reasons as
> above I would prefer to keep it as is.

As above, I'll fix it.

Ian.

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Ian Campbell]

On Thu, 2015-07-23 at 17:52 +0100, Stefano Stabellini wrote:
> +sub build () {
> +    target_cmd_build($ho, 9000, $builddir, <<END);
> +        cd $raisindir
> +        ./raise -n build
> +END
> +#/;

Running this is pretty quiet it seems, which is normally good but I think
for automated test we likely want to crank up the verbosity somewhat. Is
that possible?

Or else if this is being captured into some logfile somewhere we should
capture that.

It also said:
    [raisin] Missing packages: pciutils-dev
which I guess is something which needs to be added to ts-xen-build-prep?

It hasn't actually failed though, is that normal?

It also said
    [raisin] Raisin requires sudo to install build dependencies for you.
    [raisin] You can only build without it.
but I think that is desirable/expected. Not installing sudo on osstest
build hosts (as we do) seems like a good safety catch to maintain to me.

Ian.

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Stefano Stabellini]

On Tue, 4 Aug 2015, Ian Campbell wrote:
> On Thu, 2015-07-23 at 17:52 +0100, Stefano Stabellini wrote:
> > +sub build () {
> > +    target_cmd_build($ho, 9000, $builddir, <<END);
> > +        cd $raisindir
> > +        ./raise -n build
> > +END
> > +#/;
> 
> Running this is pretty quiet it seems, which is normally good but I think
> for automated test we likely want to crank up the verbosity somewhat. Is
> that possible?

-v for verbose

./raise -v -n build


> Or else if this is being captured into some logfile somewhere we should
> capture that.
> 
> It also said:
>     [raisin] Missing packages: pciutils-dev
> which I guess is something which needs to be added to ts-xen-build-prep?

This is a mismatch between raisin's build dependency list and osstest's.
I thought that OSSTest might not want to add pciutils-dev as a
dependency, this is why I didn't add it. On the other hand I would like
to keep it in Raisin as it is required for passthrough.


> It hasn't actually failed though, is that normal?
 
It is an "optional" dependency, without it passthrough is disabled in
qemu-traditional. Passing -n tell raisin to "shut up, do nothing and
continue", so at the end the build still succeed..


> It also said
>     [raisin] Raisin requires sudo to install build dependencies for you.
>     [raisin] You can only build without it.
> but I think that is desirable/expected. Not installing sudo on osstest
> build hosts (as we do) seems like a good safety catch to maintain to me.

Yes, I agree.

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Ian Campbell]

On Tue, 2015-08-04 at 12:18 +0100, Stefano Stabellini wrote:
> On Tue, 4 Aug 2015, Ian Campbell wrote:
> > On Thu, 2015-07-23 at 17:52 +0100, Stefano Stabellini wrote:
> > > +sub build () {
> > > +    target_cmd_build($ho, 9000, $builddir, <<END);
> > > +        cd $raisindir
> > > +        ./raise -n build
> > > +END
> > > +#/;
> > 
> > Running this is pretty quiet it seems, which is normally good but I 
> > think
> > for automated test we likely want to crank up the verbosity somewhat. 
> > Is
> > that possible?
> 
> -v for verbose
> 
> ./raise -v -n build
> 
> 
> > Or else if this is being captured into some logfile somewhere we should
> > capture that.
> > 
> > It also said:
> >     [raisin] Missing packages: pciutils-dev
> > which I guess is something which needs to be added to ts-xen-build
> > -prep?
> 
> This is a mismatch between raisin's build dependency list and osstest's.
> I thought that OSSTest might not want to add pciutils-dev as a
> dependency, this is why I didn't add it.

On the contrary I think we probably do want it, especially if without it
passthrough doesn't work, we should be at least build testing that!

> > It hasn't actually failed though, is that normal?
>  
> It is an "optional" dependency, without it passthrough is disabled in
> qemu-traditional. Passing -n tell raisin to "shut up, do nothing and
> continue", so at the end the build still succeed.

What other sorts of things are optional? I think there could be an argument
for osstest insisting on as much stuff as possible being enabled and
therefore for asking raisin to be pedantic about optional deps.

Ian

Re: [PATCH v8 2/2] OSSTest: push successful raisin builds

[Ian Campbell]

On Thu, 2015-07-23 at 17:52 +0100, Stefano Stabellini wrote:
> diff --git a/ap-fetch-version b/ap-fetch-version
> index 33aaf00..2788d9f 100755
> --- a/ap-fetch-version
> +++ b/ap-fetch-version
> @@ -80,6 +80,9 @@ libvirt)
>  rumpuserxen)
>  	repo_tree_rev_fetch_git rumpuserxen \
>  		$TREE_RUMPUSERXEN master $LOCALREV_RUMPUSERXEN

Missing ";;" here.

[PATCH 1/2] ts-raisin-build: consolidate open coded iterations over enabled components

[Ian Campbell]

This removes the possibility of the various bits (ENABLED_COMPONENTS,
TREE/REVISION_* and store_revision) getting out of sync. Use a hash
table to store overrides where the Raising and runvar names for a
component differ.

Also switch to using a shell heredoc to populate config, this was
necessary as part of how the TREE/REVISION setup was rewritten and we
may as well pull the other settings into that too (reduces the amount
of quoting too).

Also add a comment to config to indicate where the osstest overrides
start.

The config file produced here is identical to before except for the
added command and an extra trailing newline.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
It would probably be best to fold this into the base patch rather than
doing it seperately, maybe after Ian has had a chance to comment on
the incremental patch?

The use of $treerev_override{$_}//$_ could be simplified by doing
something like:
    $treerev_override{$_} //= $_ foreach @components;
at some point after the hash is declared but before it is used. I'm
not sure if that would be preferable or not.
---
 ts-raisin-build | 75 +++++++++++++++++++++++++++++----------------------------
 1 file changed, 38 insertions(+), 37 deletions(-)

diff --git a/ts-raisin-build b/ts-raisin-build
index 137ee37..2f3d4eb 100755
--- a/ts-raisin-build
+++ b/ts-raisin-build
@@ -30,45 +30,46 @@ builddirsprops();
 
 my $raisindir="$builddir/raisin";
 
+# Components to enable, using the Raisin naming scheme
+my @components = qw(seabios ovmf xen qemu qemu_traditional libvirt);
+# Where they differ, map raisin component names to the osstest runvar names
+my %treerev_override = (
+    qemu => 'qemuu',
+    qemu_traditional => 'qemu'
+    );
+
 sub checkout () {
     prepbuilddirs();
     build_clone($ho, 'raisin', $builddir, 'raisin');
 
-    target_cmd_build($ho, 100, $builddir, <<END.
+    my $enabled = join(" ", @components);
+    my $treerevs;
+    foreach (@components) {
+	my $treerev = $treerev_override{$_}//$_;
+	logm("Mapping component $_ to runvar $treerev");
+	# We require both a tree and revision
+	next unless nonempty($r{"tree_$treerev"});
+	next unless nonempty($r{"revision_$treerev"});
+
+	my $url = $r{"tree_$treerev"};
+	my $rev = $r{"revision_$treerev"};
+	$treerevs .= uc($_)."_URL=\"$url\"\n";
+	$treerevs .= uc($_)."_REVISION=\"$rev\"\n";
+    }
+
+    target_cmd_build($ho, 100, $builddir, <<END);
         cd $raisindir
         cp defconfig config
-    >>config
+    cat >>config <<EOF
 
-    echo >>config MAKE=\\"make $makeflags\\"
-    echo >>config PREFIX=\\"/usr\\"
-    echo >>config DESTDIR=dist
-    echo >>config ENABLED_COMPONENTS=\\"seabios ovmf xen qemu qemu_traditional libvirt\\"
-END
-               (nonempty($r{tree_xen}) && nonempty($r{revision_xen}) ? <<END : '').
-    echo >>config XEN_URL=\\"$r{tree_xen}\\"
-    echo >>config XEN_REVISION=\\"$r{revision_xen}\\"
+# osstest overrides
+MAKE=\"make $makeflags\"
+PREFIX=\"/usr\"
+DESTDIR=dist
+ENABLED_COMPONENTS=\"$enabled\"
+$treerevs
+EOF
 END
-               (nonempty($r{tree_qemuu}) && nonempty($r{revision_qemuu}) ? <<END : '').
-    echo >>config QEMU_URL=\\"$r{tree_qemuu}\\"
-    echo >>config QEMU_REVISION=\\"$r{revision_qemuu}\\"
-END
-               (nonempty($r{tree_qemu}) && nonempty($r{revision_qemu}) ? <<END : '').
-    echo >>config QEMU_TRADITIONAL_URL=\\"$r{tree_qemu}\\"
-    echo >>config QEMU_TRADITIONAL_REVISION=\\"$r{revision_qemu}\\"
-END
-               (nonempty($r{tree_seabios}) && nonempty($r{revision_seabios}) ? <<END : '').
-    echo >>config SEABIOS_URL=\\"$r{tree_seabios}\\"
-    echo >>config SEABIOS_REVISION=\\"$r{revision_seabios}\\"
-END
-               (nonempty($r{tree_libvirt}) && nonempty($r{revision_libvirt}) ? <<END : '').
-    echo >>config LIBVIRT_URL=\\"$r{tree_libvirt}\\"
-    echo >>config LIBVIRT_REVISION=\\"$r{revision_libvirt}\\"
-END
-               (nonempty($r{tree_ovmf}) && nonempty($r{revision_ovmf}) ? <<END : '')
-    echo >>config OVMF_URL=\\"$r{tree_ovmf}\\"
-    echo >>config OVMF_REVISION=\\"$r{revision_ovmf}\\"
-END
-               );
 }
 
 sub build () {
@@ -96,12 +97,12 @@ END
 }
 
 sub collectversions () {
-    store_revision($ho, 'xen', "$raisindir/xen-dir", 1);
-    store_revision($ho, 'qemu', "$raisindir/qemu-traditional-dir", 1);
-    store_revision($ho, 'qemuu', "$raisindir/qemu-dir", 1);
-    store_revision($ho, 'seabios', "$raisindir/seabios-dir", 1);
-    store_revision($ho, 'ovmf', "$raisindir/ovmf-dir", 1);
-    store_revision($ho, 'libvirt', "$raisindir/libvirt-dir", 1);
+    foreach ( @components ) {
+	my $tree = $treerev_override{$_}//$_;
+	# raisin uses - in dir names, not _ as in component names
+	tr/_/-/;
+	store_revision($ho, $tree, "$raisindir/$_-dir", 1);
+    }
 }
 
 sub stash () {
-- 
2.1.4

[PATCH 2/2] ts-raisin-build: Use git_massage_url

[Ian Campbell]

Sends clones via any configured proxy.

XXX this only affects URLs explicitly configured by osstest, so not
any implicit ones. Perhaps there is a better Raisin way to achieve
this?

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ts-raisin-build | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ts-raisin-build b/ts-raisin-build
index 2f3d4eb..63519c6 100755
--- a/ts-raisin-build
+++ b/ts-raisin-build
@@ -51,7 +51,7 @@ sub checkout () {
 	next unless nonempty($r{"tree_$treerev"});
 	next unless nonempty($r{"revision_$treerev"});
 
-	my $url = $r{"tree_$treerev"};
+	my $url = git_massage_url($r{"tree_$treerev"});
 	my $rev = $r{"revision_$treerev"};
 	$treerevs .= uc($_)."_URL=\"$url\"\n";
 	$treerevs .= uc($_)."_REVISION=\"$rev\"\n";
-- 
2.1.4

Re: [PATCH v8 2/2] OSSTest: push successful raisin builds

[Ian Campbell]

On Tue, 2015-08-04 at 13:56 +0100, Ian Campbell wrote:
> On Thu, 2015-07-23 at 17:52 +0100, Stefano Stabellini wrote:
> > diff --git a/ap-fetch-version b/ap-fetch-version
> > index 33aaf00..2788d9f 100755
> > --- a/ap-fetch-version
> > +++ b/ap-fetch-version
> > @@ -80,6 +80,9 @@ libvirt)
> >  rumpuserxen)
> >  	repo_tree_rev_fetch_git rumpuserxen \
> >  		$TREE_RUMPUSERXEN master $LOCALREV_RUMPUSERXEN
> 
> Missing ";;" here.

If you are looking to test the ap-* stuff in a more realistic (i.e. closer
to how production would call it) environment then you can do:

For a flight named "raisin" testing the raisin branch:
$ ./standalone make-flight -f raisin raisin

For a flight named "standalone" (the default) testing xen-unstable:
$ ./standalone make-flight xen-unstable

You can then examine the contents of those two with mg-show-flight-runvars.

You may need some options in your ~/.xen-osstest/config which you didn't
before. The main one is Repos which should point to a scratch directory to
use for cloning trees while determining revisions to use.

Ian.

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Stefano Stabellini]

On Tue, 4 Aug 2015, Ian Campbell wrote:
> On Tue, 2015-08-04 at 12:18 +0100, Stefano Stabellini wrote:
> > On Tue, 4 Aug 2015, Ian Campbell wrote:
> > > On Thu, 2015-07-23 at 17:52 +0100, Stefano Stabellini wrote:
> > > > +sub build () {
> > > > +    target_cmd_build($ho, 9000, $builddir, <<END);
> > > > +        cd $raisindir
> > > > +        ./raise -n build
> > > > +END
> > > > +#/;
> > > 
> > > Running this is pretty quiet it seems, which is normally good but I 
> > > think
> > > for automated test we likely want to crank up the verbosity somewhat. 
> > > Is
> > > that possible?
> > 
> > -v for verbose
> > 
> > ./raise -v -n build
> > 
> > 
> > > Or else if this is being captured into some logfile somewhere we should
> > > capture that.
> > > 
> > > It also said:
> > >     [raisin] Missing packages: pciutils-dev
> > > which I guess is something which needs to be added to ts-xen-build
> > > -prep?
> > 
> > This is a mismatch between raisin's build dependency list and osstest's.
> > I thought that OSSTest might not want to add pciutils-dev as a
> > dependency, this is why I didn't add it.
> 
> On the contrary I think we probably do want it, especially if without it
> passthrough doesn't work, we should be at least build testing that!

fair enough, I can add them


> > > It hasn't actually failed though, is that normal?
> >  
> > It is an "optional" dependency, without it passthrough is disabled in
> > qemu-traditional. Passing -n tell raisin to "shut up, do nothing and
> > continue", so at the end the build still succeed.
> 
> What other sorts of things are optional?

There isn't really a distinction between mandatory dependencies and
optional dependencies in raisin. This is what raisin is currently
complaining about:

bridge-utils pciutils-dev libnl-3-dev libnl-route-3-dev

bridge-utils is actually a run time dependency for Xen, while
libnl-3-dev and libnl-route-3-dev are build time dependencies for
libvirt. I think they are needed for vlan support. Should I add them
too?


> I think there could be an argument
> for osstest insisting on as much stuff as possible being enabled and
> therefore for asking raisin to be pedantic about optional deps.

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Stefano Stabellini]

On Tue, 4 Aug 2015, Ian Campbell wrote:
> On Thu, 2015-07-23 at 17:52 +0100, Stefano Stabellini wrote:
> > 
> > +    ./cs-job-create $flight build-$arch-raisin build-raisin                  \
> > +                arch=$arch                                                   \
> > +        tree_xen=$TREE_XEN                                                   \
> > +                $RUNVARS $BUILD_RUNVARS $BUILD_RAISIN_RUNVARS $arch_runvars \
> > +                $suite_runvars                                               \
> > +                host_hostflags=$build_hostflags                              \
> > +                buildjob=${bfi}build-$arch                                   \
> 
> This line means you can't run the build-$arch-raisin job unless the build
> -$arch job is complete. Is that intentional?
> 
> I can see an argument for not testing raisin if the underlying Xen tree
> doesn't build, it's a bit annoying in standalone mode is all...

Fair enough. I can remove the buildjob=${bfi}build-$arch line.

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Ian Campbell]

On Tue, 2015-08-04 at 14:18 +0100, Stefano Stabellini wrote:
> On Tue, 4 Aug 2015, Ian Campbell wrote:
> > On Tue, 2015-08-04 at 12:18 +0100, Stefano Stabellini wrote:
> > > > It hasn't actually failed though, is that normal?
> > >  
> > > It is an "optional" dependency, without it passthrough is disabled in
> > > qemu-traditional. Passing -n tell raisin to "shut up, do nothing and
> > > continue", so at the end the build still succeed.
> > 
> > What other sorts of things are optional?
> 
> There isn't really a distinction between mandatory dependencies and
> optional dependencies in raisin. This is what raisin is currently
> complaining about:
> 
> bridge-utils pciutils-dev libnl-3-dev libnl-route-3-dev
> 
> bridge-utils is actually a run time dependency for Xen, while
> libnl-3-dev and libnl-route-3-dev are build time dependencies for
> libvirt. I think they are needed for vlan support. Should I add them
> too?

I think so, I have a feeling Remus might use them too? cf "[PATCH] osstest:
install libnl3 packages" from Roger.

Ian.

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Stefano Stabellini]

On Tue, 4 Aug 2015, Ian Campbell wrote:
> On Tue, 2015-08-04 at 14:18 +0100, Stefano Stabellini wrote:
> > On Tue, 4 Aug 2015, Ian Campbell wrote:
> > > On Tue, 2015-08-04 at 12:18 +0100, Stefano Stabellini wrote:
> > > > > It hasn't actually failed though, is that normal?
> > > >  
> > > > It is an "optional" dependency, without it passthrough is disabled in
> > > > qemu-traditional. Passing -n tell raisin to "shut up, do nothing and
> > > > continue", so at the end the build still succeed.
> > > 
> > > What other sorts of things are optional?
> > 
> > There isn't really a distinction between mandatory dependencies and
> > optional dependencies in raisin. This is what raisin is currently
> > complaining about:
> > 
> > bridge-utils pciutils-dev libnl-3-dev libnl-route-3-dev
> > 
> > bridge-utils is actually a run time dependency for Xen, while
> > libnl-3-dev and libnl-route-3-dev are build time dependencies for
> > libvirt. I think they are needed for vlan support. Should I add them
> > too?
> 
> I think so, I have a feeling Remus might use them too? cf "[PATCH] osstest:
> install libnl3 packages" from Roger.

Actually this is why I cannot remove the buildjob=${bfi}build-$arch
line: otherwise the build deps don't get installed, right?

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Roger Pau Monné]

El 04/08/15 a les 15.41, Ian Campbell ha escrit:
> On Tue, 2015-08-04 at 14:18 +0100, Stefano Stabellini wrote:
>> On Tue, 4 Aug 2015, Ian Campbell wrote:
>>> On Tue, 2015-08-04 at 12:18 +0100, Stefano Stabellini wrote:
>>>>> It hasn't actually failed though, is that normal?
>>>>  
>>>> It is an "optional" dependency, without it passthrough is disabled in
>>>> qemu-traditional. Passing -n tell raisin to "shut up, do nothing and
>>>> continue", so at the end the build still succeed.
>>>
>>> What other sorts of things are optional?
>>
>> There isn't really a distinction between mandatory dependencies and
>> optional dependencies in raisin. This is what raisin is currently
>> complaining about:
>>
>> bridge-utils pciutils-dev libnl-3-dev libnl-route-3-dev
>>
>> bridge-utils is actually a run time dependency for Xen, while
>> libnl-3-dev and libnl-route-3-dev are build time dependencies for
>> libvirt. I think they are needed for vlan support. Should I add them
>> too?
> 
> I think so, I have a feeling Remus might use them too? cf "[PATCH] osstest:
> install libnl3 packages" from Roger.

Yes, Remus needs libnl-3-dev and libnl-route-3-dev. They are available
since Wheezy, but only the ones on Squeeze are new enough for the Remus
code. Feel free to pick that patch, it only needs a comment in the
description to mention that support for anything older than Wheezy is
dropped.

Roger.

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Ian Campbell]

On Tue, 2015-08-04 at 15:47 +0200, Roger Pau Monné wrote:
> El 04/08/15 a les 15.41, Ian Campbell ha escrit:
> > On Tue, 2015-08-04 at 14:18 +0100, Stefano Stabellini wrote:
> > > On Tue, 4 Aug 2015, Ian Campbell wrote:
> > > > On Tue, 2015-08-04 at 12:18 +0100, Stefano Stabellini wrote:
> > > > > > It hasn't actually failed though, is that normal?
> > > > >  
> > > > > It is an "optional" dependency, without it passthrough is 
> > > > > disabled in
> > > > > qemu-traditional. Passing -n tell raisin to "shut up, do nothing 
> > > > > and
> > > > > continue", so at the end the build still succeed.
> > > > 
> > > > What other sorts of things are optional?
> > > 
> > > There isn't really a distinction between mandatory dependencies and
> > > optional dependencies in raisin. This is what raisin is currently
> > > complaining about:
> > > 
> > > bridge-utils pciutils-dev libnl-3-dev libnl-route-3-dev
> > > 
> > > bridge-utils is actually a run time dependency for Xen, while
> > > libnl-3-dev and libnl-route-3-dev are build time dependencies for
> > > libvirt. I think they are needed for vlan support. Should I add them
> > > too?
> > 
> > I think so, I have a feeling Remus might use them too? cf "[PATCH] 
> > osstest:
> > install libnl3 packages" from Roger.
> 
> Yes, Remus needs libnl-3-dev and libnl-route-3-dev. They are available
> since Wheezy, but only the ones on Squeeze are new enough for the Remus
> code.

Squeeze was before Wheezy. I think you meant s/Squeeze/Jessie/?

>  Feel free to pick that patch, it only needs a comment in the
> description to mention that support for anything older than Wheezy is
> dropped.

Right, I think Ian was expecting a resend with that adding.

I was considering picking it up and doing that myself as I applied, but I'm
currently waiting for osstest to settle down a bit and for Wei's libvirt
changes to go through the push gate.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v8 1/2] OSSTEST: introduce a raisin build test

[Ian Campbell]

On Tue, 2015-08-04 at 14:47 +0100, Stefano Stabellini wrote:
> On Tue, 4 Aug 2015, Ian Campbell wrote:
> > On Tue, 2015-08-04 at 14:18 +0100, Stefano Stabellini wrote:
> > > On Tue, 4 Aug 2015, Ian Campbell wrote:
> > > > On Tue, 2015-08-04 at 12:18 +0100, Stefano Stabellini wrote:
> > > > > > It hasn't actually failed though, is that normal?
> > > > >  
> > > > > It is an "optional" dependency, without it passthrough is 
> > > > > disabled in
> > > > > qemu-traditional. Passing -n tell raisin to "shut up, do nothing 
> > > > > and
> > > > > continue", so at the end the build still succeed.
> > > > 
> > > > What other sorts of things are optional?
> > > 
> > > There isn't really a distinction between mandatory dependencies and
> > > optional dependencies in raisin. This is what raisin is currently
> > > complaining about:
> > > 
> > > bridge-utils pciutils-dev libnl-3-dev libnl-route-3-dev
> > > 
> > > bridge-utils is actually a run time dependency for Xen, while
> > > libnl-3-dev and libnl-route-3-dev are build time dependencies for
> > > libvirt. I think they are needed for vlan support. Should I add them
> > > too?
> > 
> > I think so, I have a feeling Remus might use them too? cf "[PATCH] 
> > osstest:
> > install libnl3 packages" from Roger.
> 
> Actually this is why I cannot remove the buildjob=${bfi}build-$arch
> line: otherwise the build deps don't get installed, right?

I don't think so, ts-xen-build-prep is badly named and is run for any build
job (formally I think anything in sg-run-job where need-hosts/FOO returns
BUILD).

Ian.

Re: [PATCH 1/2] ts-raisin-build: consolidate open coded iterations over enabled components

[Ian Campbell]

On Tue, 2015-08-04 at 14:02 +0100, Ian Campbell wrote:
> +my %treerev_override = (

This is a kindofcrappy name actually. Maybe %comp2runvar or
%raisin2osstest or (perhaps going too far) just %r2o?

Ian.