[PATCH OSSTEST v2 00/15] add distro domU testing flight

[PATCH OSSTEST v2 00/15] add distro domU testing flight

[Ian Campbell]

The following series adds a new flight type intended to test distro domU
support, i.e. running distro installers and kernels on top of current
Xen to ensure things work. I intend for the flight to be run on a weekly
basis, and I've scheduled it here to run at the weekend when in theory
things might be quieter.

I've initially just done Debian PV domU support since Debian is what I
know, but I hope others will add other distros as time goes on.

The series introduces support for running the Debian Installer in a PV
domU booting from netboot (PXE-ish) and netinst (ISO media) and adding
jobs to test all of those for Squeeze onwards up to Sid and the
daily/weekly builds.

The flight is a bit unusual since distro installer builds are not really
versioned in a way which is useful to osstest, so instead each flight
will be compared against the previous run only (this stuff needs testing
in production).

The new Debian install functionality here also gives us the opportunity
to test pygrub and pvgrub, so a couple of jobs are added to the regular
tests using installer images versioned via TftpDiVersion in the usual
way. This requires an early patch "mg-debian-installer-update: grab Xen
PV domU capable images too" to go in and a patch to update TftpDiVersion
to a refreshed version to be inserted before "Test pygrub and pvgrub on
the regular flights" is committed.

I've also pushed a branch, pull details at the end.

v1 of this[0] wasn't really a proper posting, but calling this v2 to
avoid any potential confusion.

Ian.

[0] http://article.gmane.org/gmane.comp.emulators.xen.devel/180019


The following changes since commit 0b5094e69de4c1a6cfb866d0cfe4d10ba655285c:

  crontab: Add a daily run of the libvirt branch. (2014-04-25 10:07:56 +0100)

are available in the git repository at:

  git://xenbits.xen.org/people/ianc/osstest.git distro-flight-v2

for you to fetch changes up to 95006fcc3c921467d85794d7f27f67232ad6597a:

  distros: Run a flight over the weekend. (2014-04-26 09:47:36 +0100)

----------------------------------------------------------------
Ian Campbell (15):
      cr-external-linux: remove
      Build against liblzma to support .xz compressed kernels
      mg-debian-installer-update: grab Xen PV domU capable images too
      TestSupport: Add helper to fetch a URL on a host
      TestSupport: Add helper to wait for a guest to shutdown
      TestSupport: allow overring of on_* in prepareguest_part_xencfg
      TestSupport: allow caller of prepareguest_part_xencfg to specify viftype
      Debian: add preseed_create_guest helper
      distros: add support for installing Debian PV guests via d-i, flight and jobs
      distros: support booting Debian PV (d-i installed) guests with pvgrub.
      distros: attempt pvgrub support for PV Wheezy too.
      distros: support PV guest install from Debian netinst media.
      Test pygrub and pvgrub on the regular flights
      distros: add branch infrastructure
      distros: Run a flight over the weekend.

 Osstest/Debian.pm          |  83 +++++++++++++++-
 Osstest/TestSupport.pm     |  32 ++++++-
 ap-fetch-version           |   4 +
 ap-fetch-version-old       |   3 +
 ap-print-url               |   3 +
 cr-daily-branch            |  11 ++-
 cr-external-linux          |  50 ----------
 cri-common                 |   1 +
 crontab                    |   1 +
 make-distros-flight        | 140 +++++++++++++++++++++++++++
 make-flight                |  39 ++++++++
 mg-debian-installer-update |  11 ++-
 sg-run-job                 |  11 +++
 ts-debian-di-install       | 233 +++++++++++++++++++++++++++++++++++++++++++++
 ts-xen-build-prep          |   2 +-
 ts-xen-install             |   2 +-
 16 files changed, 562 insertions(+), 64 deletions(-)
 delete mode 100755 cr-external-linux
 create mode 100755 make-distros-flight
 create mode 100755 ts-debian-di-install

[PATCH OSSTEST v2 01/15] cr-external-linux: remove

[Ian Campbell]

Seems to be unused.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 cr-external-linux | 50 --------------------------------------------------
 1 file changed, 50 deletions(-)
 delete mode 100755 cr-external-linux

diff --git a/cr-external-linux b/cr-external-linux
deleted file mode 100755
index 073fc86..0000000
--- a/cr-external-linux
+++ /dev/null
@@ -1,50 +0,0 @@
-#!/bin/bash
-
-# This is part of "osstest", an automated testing framework for Xen.
-# Copyright (C) 2009-2013 Citrix Inc.
-# 
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-# 
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-# 
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-
-set -ex
-
-. cri-args-hostlists
-
-branch=$1; shift
-tree=linux
-xenbranch=xen-unstable
-
-case "$branch" in
-linux-next)
-	url=git://git.kernel.org/pub/scm/linux/kernel/git/sfr/linux-next.git
-	;;
-*)
-	echo >&2 "unknown cr-external-linux branch"
-	exit 1
-	;;
-esac
-
-select_branch
-
-check_stop external-linux.
-
-export REVISION_LINUX_OLD=disable
-export REVISION_XEN="`./ap-fetch-version-baseline $xenbranch`"
-export TREE_LINUX="$url"
-
-flight=`./make-flight $branch $xenbranch $OSSTEST_BLESSING "$@"`
-
-execute_flight $flight $OSSTEST_BLESSING
-start_email $flight "$branch"
-publish_send_email $flight
-- 
1.9.0

Re: [PATCH OSSTEST v2 01/15] cr-external-linux: remove

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 01/15] cr-external-linux: remove"):
> Seems to be unused.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

[PATCH OSSTEST v2 02/15] Build against liblzma to support .xz compressed kernels

[Ian Campbell]

Install the library at runtime.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ts-xen-build-prep | 2 +-
 ts-xen-install    | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index a4e166e..0e68cd3 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -171,7 +171,7 @@ sub prep () {
                                libpci-dev libncurses5-dev libssl-dev python-dev
                                libx11-dev git-core uuid-dev gettext gawk
                                libsdl-dev libyajl-dev libaio-dev libpixman-1-dev
-                               libglib2.0-dev pkg-config
+                               libglib2.0-dev liblzma-dev pkg-config
                                autoconf automake libtool xsltproc
                                libxml2-utils libxml2-dev libnl-dev
                                libdevmapper-dev w3c-dtd-xhtml
diff --git a/ts-xen-install b/ts-xen-install
index 3f7ade8..0b84b7b 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -51,7 +51,7 @@ my %distpath;
 sub packages () {
     target_install_packages($ho,
                             qw(bridge-utils vncsnapshot libaio1 libpixman-1-0
-                               libsdl1.2debian libglib2.0-0));
+                               libsdl1.2debian libglib2.0-0 liblzma5));
     target_install_packages($ho,
 			    $ho->{Suite} =~ /squeeze/ ? "libyajl1" : "libyajl2");
     if ($ho->{Suite} !~ m/lenny|squeeze/) {
-- 
1.9.0

Re: [PATCH OSSTEST v2 02/15] Build against liblzma to support .xz compressed kernels

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 02/15] Build against liblzma to support .xz compressed kernels"):
> Install the library at runtime.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

[PATCH OSSTEST v2 03/15] mg-debian-installer-update: grab Xen PV domU capable images too

[Ian Campbell]

Some test cases are going to want this, so they should be grabbed and
versioned here.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 mg-debian-installer-update | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/mg-debian-installer-update b/mg-debian-installer-update
index 452a2ad..7d24ed2 100755
--- a/mg-debian-installer-update
+++ b/mg-debian-installer-update
@@ -34,8 +34,8 @@ sbase=$site/dists/$suite
 
 case ${suite}_${arch} in
     *_i386|*_amd64)
-        src=$sbase/main/installer-$arch/current/images/netboot/debian-installer/$arch
-        files='initrd.gz linux'
+        src=$sbase/main/installer-$arch/current/images/netboot/
+        files="debian-installer/$arch/initrd.gz:initrd.gz debian-installer/$arch/linux:linux xen/vmlinuz:vmlinuz-xen xen/initrd.gz:initrd.gz-xen"
         ;;
     lenny_armhf|squeeze_armhf)
         # No such thing.
@@ -61,7 +61,9 @@ cd $dst
 
 for f in $files; do
         echo >&2 "fetching $f"
-        curl -s $src/$f >$f.new
+        s=${f/:*} ; d=${f/*:}
+        echo "$src/$s => $d.new"
+        curl -s $src/$s >$d.new
 done
 
 curl -s $pfile >Packages.bz2
@@ -131,7 +133,8 @@ if [ $arch = armhf ]; then
 fi
 
 for f in $files; do
-        mv -f $f.new $f
+        s=${f/:*} ; d=${f/*:}
+        mv -f $d.new $d
 done
 
 rm Packages.bz2
-- 
1.9.0

Re: [PATCH OSSTEST v2 03/15] mg-debian-installer-update: grab Xen PV domU capable images too

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 03/15] mg-debian-installer-update: grab Xen PV domU capable images too"):
> Some test cases are going to want this, so they should be grabbed and
> versioned here.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

[PATCH OSSTEST v2 04/15] TestSupport: Add helper to fetch a URL on a host

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/TestSupport.pm | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 6558fc8..35838a7 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -54,6 +54,7 @@ BEGIN {
                       target_putfile target_putfile_root
                       target_putfilecontents_stash
 		      target_putfilecontents_root_stash
+                      target_fetchurl
                       target_put_guest_image
                       target_editfile_root target_file_exists
                       target_run_apt
@@ -1431,6 +1432,13 @@ END
     return $cfgpath;
 }
 
+sub target_fetchurl($$$;$) {
+    my ($ho, $url, $path, $timeo) = @_;
+    $timeo ||= 2000;
+    target_cmd_root($ho, "wget --progress=dot:mega -O $path $url", $timeo);
+}
+
+
 sub target_put_guest_image ($$$) {
     my ($ho, $gho, $default) = @_;
     my $specimage = $r{"$gho->{Guest}_image"};
-- 
1.9.0

Re: [PATCH OSSTEST v2 04/15] TestSupport: Add helper to fetch a URL on a host

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 04/15] TestSupport: Add helper to fetch a URL on a host"):
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Do we make sure that wget is installed, somehow ?  I think it needs to
be added to some of the lists of packages.

Ian.

Re: [PATCH OSSTEST v2 04/15] TestSupport: Add helper to fetch a URL on a host

[Ian Campbell]

On Fri, 2014-05-02 at 12:25 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 04/15] TestSupport: Add helper to fetch a URL on a host"):
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> 
> Do we make sure that wget is installed, somehow ?  I think it needs to
> be added to some of the lists of packages.

It is there in practice (perhaps due to being priority important?). But
it would certainly be fine to list it in the preseed, I'll do that.

Ian.

[PATCH OSSTEST v2 05/15] TestSupport: Add helper to wait for a guest to shutdown

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/TestSupport.pm | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 35838a7..b283e07 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -91,7 +91,8 @@ BEGIN {
                       guest_umount_lv guest_await guest_await_dhcp_tcp
                       guest_checkrunning guest_check_ip guest_find_ether
                       guest_find_domid guest_check_up guest_check_up_quick
-                      guest_get_state guest_await_reboot guest_destroy
+                      guest_get_state guest_await_reboot
+                      guest_await_shutdown guest_destroy
                       guest_vncsnapshot_begin guest_vncsnapshot_stash
 		      guest_check_remus_ok guest_editconfig
                       host_involves_pcipassthrough host_get_pcipassthrough_devs
@@ -1249,6 +1250,17 @@ sub guest_await_reboot ($$$) {
         return undef if $st eq 'sr';
         fail("guest unexpectedly shutdown; state is '$st'")
             if $st =~ m/^s/ || $st eq '';
+        return "guest state is \"$st\"";
+    });
+}
+
+sub guest_await_shutdown ($$$) {
+    my ($ho,$gho, $timeout) = @_;
+    poll_loop($timeout, 30, "await shutdown request from $gho->{Guest}", sub {
+        my $st= guest_get_state($ho,$gho);
+        return undef if $st eq 's';
+        fail("guest unexpectedly shutdown; state is '$st'")
+            if $st =~ m/^s/ || $st eq '';
         return "guest state is $st";
     });
 }
-- 
1.9.0

Re: [PATCH OSSTEST v2 05/15] TestSupport: Add helper to wait for a guest to shutdown

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 05/15] TestSupport: Add helper to wait for a guest to shutdown"):
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
...
> @@ -1249,6 +1250,17 @@ sub guest_await_reboot ($$$) {
>          return undef if $st eq 'sr';
>          fail("guest unexpectedly shutdown; state is '$st'")
>              if $st =~ m/^s/ || $st eq '';
> +        return "guest state is \"$st\"";
> +    });
> +}
> +
> +sub guest_await_shutdown ($$$) {
> +    my ($ho,$gho, $timeout) = @_;
> +    poll_loop($timeout, 30, "await shutdown request from $gho->{Guest}", sub {
> +        my $st= guest_get_state($ho,$gho);
> +        return undef if $st eq 's';
> +        fail("guest unexpectedly shutdown; state is '$st'")
> +            if $st =~ m/^s/ || $st eq '';

This function is practically identical to guest_await_reboot.

Ian.

[PATCH OSSTEST v2 06/15] TestSupport: allow overring of on_* in prepareguest_part_xencfg

[Ian Campbell]

Currently only on_reboot can be overridden

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/TestSupport.pm | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index b283e07..132e8e9 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1416,6 +1416,8 @@ sub prepareguest_part_lvmdisk ($$$) {
 sub prepareguest_part_xencfg ($$$$$) {
     my ($ho, $gho, $ram_mb, $xopts, $cfgrest) = @_;
     my $onreboot= $xopts->{OnReboot} || 'restart';
+    my $onpoweroff= $xopts->{OnPowerOff} || 'destroy';
+    my $oncrash= $xopts->{OnCrash} || 'preserve';
     my $vcpus= guest_var($gho, 'vcpus', $xopts->{DefVcpus} || 2);
     my $xoptcfg= $xopts->{ExtraConfig};
     $xoptcfg='' unless defined $xoptcfg;
@@ -1424,9 +1426,9 @@ name        = '$gho->{Name}'
 memory = ${ram_mb}
 vif         = [ 'type=ioemu,mac=$gho->{Ether}' ]
 #
-on_poweroff = 'destroy'
+on_poweroff = '$onpoweroff'
 on_reboot   = '$onreboot'
-on_crash    = 'preserve'
+on_crash    = '$oncrash'
 #
 vcpus = $vcpus
 #
-- 
1.9.0

Re: [PATCH OSSTEST v2 06/15] TestSupport: allow overring of on_* in prepareguest_part_xencfg

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 06/15] TestSupport: allow overring of on_* in prepareguest_part_xencfg"):
> Currently only on_reboot can be overridden

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

[PATCH OSSTEST v2 07/15] TestSupport: allow caller of prepareguest_part_xencfg to specify viftype

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/TestSupport.pm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 132e8e9..9d5762f 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1419,12 +1419,13 @@ sub prepareguest_part_xencfg ($$$$$) {
     my $onpoweroff= $xopts->{OnPowerOff} || 'destroy';
     my $oncrash= $xopts->{OnCrash} || 'preserve';
     my $vcpus= guest_var($gho, 'vcpus', $xopts->{DefVcpus} || 2);
+    my $viftype= "type=$xopts->{viftype}," if $xopts->{DefVcpus};
     my $xoptcfg= $xopts->{ExtraConfig};
     $xoptcfg='' unless defined $xoptcfg;
     my $xencfg= <<END;
 name        = '$gho->{Name}'
 memory = ${ram_mb}
-vif         = [ 'type=ioemu,mac=$gho->{Ether}' ]
+vif         = [ '${viftype}mac=$gho->{Ether}' ]
 #
 on_poweroff = '$onpoweroff'
 on_reboot   = '$onreboot'
@@ -1519,6 +1520,7 @@ END
         $cfg .= "bios='$bios'\n";
     }
 
+    $xopts{'viftype'} ||= "ioemu";
     my $cfgpath= prepareguest_part_xencfg($ho, $gho, $ram_mb, \%xopts, $cfg);
     target_cmd_root($ho, <<END);
         (echo $passwd; echo $passwd) | vncpasswd $gho->{Guest}.vncpw
-- 
1.9.0

Re: [PATCH OSSTEST v2 07/15] TestSupport: allow caller of prepareguest_part_xencfg to specify viftype

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 07/15] TestSupport: allow caller of prepareguest_part_xencfg to specify viftype"):
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  Osstest/TestSupport.pm | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
> index 132e8e9..9d5762f 100644
> --- a/Osstest/TestSupport.pm
> +++ b/Osstest/TestSupport.pm
> @@ -1419,12 +1419,13 @@ sub prepareguest_part_xencfg ($$$$$) {
>      my $onpoweroff= $xopts->{OnPowerOff} || 'destroy';
>      my $oncrash= $xopts->{OnCrash} || 'preserve';
>      my $vcpus= guest_var($gho, 'vcpus', $xopts->{DefVcpus} || 2);
> +    my $viftype= "type=$xopts->{viftype}," if $xopts->{DefVcpus};

Why is this conditional on DefVcpus ?

> +    $xopts{'viftype'} ||= "ioemu";

The other xopts have CamelCase names.  And you don't need to quote it
there.

Ian.

Re: [PATCH OSSTEST v2 07/15] TestSupport: allow caller of prepareguest_part_xencfg to specify viftype

[Ian Campbell]

On Fri, 2014-05-02 at 12:28 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 07/15] TestSupport: allow caller of prepareguest_part_xencfg to specify viftype"):
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> >  Osstest/TestSupport.pm | 4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
> > index 132e8e9..9d5762f 100644
> > --- a/Osstest/TestSupport.pm
> > +++ b/Osstest/TestSupport.pm
> > @@ -1419,12 +1419,13 @@ sub prepareguest_part_xencfg ($$$$$) {
> >      my $onpoweroff= $xopts->{OnPowerOff} || 'destroy';
> >      my $oncrash= $xopts->{OnCrash} || 'preserve';
> >      my $vcpus= guest_var($gho, 'vcpus', $xopts->{DefVcpus} || 2);
> > +    my $viftype= "type=$xopts->{viftype}," if $xopts->{DefVcpus};
> 
> Why is this conditional on DefVcpus ?

I think I wasn't concentrating and read the line above... It should have
been viftype to match the line bellow.

> > +    $xopts{'viftype'} ||= "ioemu";
> 
> The other xopts have CamelCase names.  And you don't need to quote it
> there.

Ack.

[PATCH OSSTEST v2 08/15] Debian: add preseed_create_guest helper

[Ian Campbell]

Creates a preseed file suitable for use in a PV guest

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 63 insertions(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index ab09abb..043ab0d 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -35,6 +35,7 @@ BEGIN {
                       %preseed_cmds
                       preseed_base
                       preseed_create
+                      preseed_create_guest
                       preseed_hook_command preseed_hook_installscript
                       di_installcmdline_core
                       );
@@ -429,6 +430,8 @@ sub di_installcmdline_core ($$;@) {
 sub preseed_base ($$;@) {
     my ($suite,$extra_packages,%xopts) = @_;
 
+    $extra_packages ||= '';
+
     return <<"END";
 d-i mirror/suite string $suite
 
@@ -498,7 +501,66 @@ $xopts{ExtraPreseed}
 ### END OF DEBIAN PRESEED BASE
 
 END
-}          
+}
+
+sub preseed_create_guest ($$;@) {
+    my ($ho, $sfx, %xopts) = @_;
+
+    my $suite= $xopts{Suite} || $c{DebianSuite};
+
+    my $extra_packages;
+
+    my $preseed_file= preseed_base($suite, $extra_packages, %xopts);
+    $preseed_file.= (<<END);
+d-i     partman-auto/method             string regular
+d-i     partman-auto/choose_recipe \\
+                select All files in one partition (recommended for new users)
+
+d-i     grub-installer/bootdev          string /dev/xvda
+
+END
+
+    my $authkeys= authorized_keys();
+
+    my $hostkeyfile= "$c{OverlayLocal}/etc/ssh/ssh_host_rsa_key";
+    my $host_rsa_key= get_filecontents($hostkeyfile);
+    chomp($host_rsa_key); $host_rsa_key.="\n";
+
+    preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+r=/target/root
+cd \$r
+
+umask 022
+mkdir .ssh
+cat <<'ENDKEYS' >.ssh/authorized_keys
+$authkeys
+ENDKEYS
+
+u=osstest
+h=/home/\$u
+mkdir /target\$h/.ssh
+cp .ssh/authorized_keys /target\$h/.ssh
+chroot /target chown -R \$u.\$u \$h/.ssh
+
+rm -f /target/etc/ssh/ssh_host_*_key
+rm -f /target/etc/ssh/ssh_host_*_key.pub
+
+cat <<'ENDKEYS' > /target/etc/ssh/ssh_host_rsa_key
+$host_rsa_key
+ENDKEYS
+chmod 0600 /target/etc/ssh/ssh_host_rsa_key
+END
+
+    foreach my $di_key (keys %preseed_cmds) {
+        $preseed_file .= "d-i preseed/$di_key string ".
+            (join ' && ', @{ $preseed_cmds{$di_key} }). "\n";
+    }
+
+    return create_webfile($ho, "preseed$sfx", $preseed_file);
+}
 
 sub preseed_create ($$;@) {
     my ($ho, $sfx, %xopts) = @_;
-- 
1.9.0

Re: [PATCH OSSTEST v2 08/15] Debian: add preseed_create_guest helper

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 08/15] Debian: add preseed_create_guest helper"):
> Creates a preseed file suitable for use in a PV guest
...
> +    my $authkeys= authorized_keys();
> +
> +    my $hostkeyfile= "$c{OverlayLocal}/etc/ssh/ssh_host_rsa_key";
> +    my $host_rsa_key= get_filecontents($hostkeyfile);
> +    chomp($host_rsa_key); $host_rsa_key.="\n";

Osstest::Debian::preseed_create already has machinery for doing this
(in a slightly different way).  Can you use it ?  (It would
presumably need breaking out.)

I see ts-debian-hvm-install has a yet-different way of doing this,
which I didn't spot.

Ian.

Re: [PATCH OSSTEST v2 08/15] Debian: add preseed_create_guest helper

[Ian Campbell]

On Fri, 2014-05-02 at 12:33 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 08/15] Debian: add preseed_create_guest helper"):
> > Creates a preseed file suitable for use in a PV guest
> ...
> > +    my $authkeys= authorized_keys();
> > +
> > +    my $hostkeyfile= "$c{OverlayLocal}/etc/ssh/ssh_host_rsa_key";
> > +    my $host_rsa_key= get_filecontents($hostkeyfile);
> > +    chomp($host_rsa_key); $host_rsa_key.="\n";
> 
> Osstest::Debian::preseed_create already has machinery for doing this
> (in a slightly different way).  Can you use it ?  (It would
> presumably need breaking out.)

There was something which didn't fit between host and guest usage that I
couldn't figure out, but it was a long time ago and I don't remember.
Let me try again and I'll either succeed or find the problem again.

> I see ts-debian-hvm-install has a yet-different way of doing this,
> which I didn't spot.

I'll see what I can do about this while I'm there.

Ian.

[PATCH OSSTEST v2 09/15] distros: add support for installing Debian PV guests via d-i, flight and jobs

[Ian Campbell]

This patch introduces ts-debian-di-install which can install Debian from a
netboot (PXE) debian installer image. By default it installs from the d-i image
used by osstest (using the special Xen PV guest enabled flavour where
necessary) but it can also install the current release versions from Squeeze
onwards (up to and including Sid) and the d-i daily builds. This is controlled
by runvars {Guest}_diver = osstest|current and {Guest}_dist = squeeze|...|sid.
The resulting guests boot the distro kernel using pygrub (pvgrub will follow).

The distros flights differ substantially from the existing flights. Introduce
make-distros-flight using the functionality previously refactored into
mfi-common. The new flight tests all versions of Debian from Squeeze onward as
an amd64, i386 and armhf guests (armhf from Jessie onwards only) using the
usual smoke tests.

Add the new cases to sg-run-job

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 make-distros-flight  | 101 +++++++++++++++++++++++++++++++
 sg-run-job           |  11 ++++
 ts-debian-di-install | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 279 insertions(+)
 create mode 100755 make-distros-flight
 create mode 100755 ts-debian-di-install

diff --git a/make-distros-flight b/make-distros-flight
new file mode 100755
index 0000000..abc24ff
--- /dev/null
+++ b/make-distros-flight
@@ -0,0 +1,101 @@
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+set -e
+
+branch=$1
+xenbranch=$2
+blessing=$3
+buildflight=$4
+
+flight=`./cs-flight-create $blessing $branch`
+
+. ap-common
+. cri-common
+. mfi-common
+
+defsuite=`getconfig DebianSuite`
+defguestsuite=`getconfig GuestDebianSuite`
+
+if [ x$buildflight = x ]; then
+
+  if [ "x$BUILD_LVEXTEND_MAX" != x ]; then
+     BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
+  fi
+
+  WANT_XEND=false REVISION_LINUX_OLD=disable
+
+  create_build_jobs
+
+else
+
+  bfi=$buildflight.
+
+fi
+
+job_create_test_filter_callback () {
+  if [ "$xenarch" = "i386" ]; then return 1; fi
+  return 0
+}
+
+test_matrix_branch_filter_callback () {
+    :
+}
+
+test_do_one_netboot () {
+  job_create_test test-$xenarch$kern-$dom0arch-$domU-$dist-netboot \
+    test-debian-di xl $xenarch $dom0arch                        \
+      kernbuildjob=${bfi}build-$dom0arch-$kernbuild             \
+      debian_arch=$domU                                         \
+      debian_dist=$dist                                         \
+      debian_method=netboot                                     \
+      debian_diver=current                                      \
+      all_hostflags=$most_hostflags
+}
+
+test_matrix_do_one () {
+  case ${xenarch} in
+  amd64) domUarches="amd64 i386";;
+  armhf) domUarches="armhf";;
+  esac
+
+  for domU in $domUarches ; do
+    for dist in squeeze wheezy jessie sid daily ; do
+      case ${domU}_${dist} in
+      armhf_squeeze) continue;; # No armhf in Squeeze
+      armhf_wheezy) continue;; # No armhf guest support in Wheezy
+      *) ;;
+      esac
+
+      test_do_one_netboot
+
+    done
+
+  done
+}
+
+test_matrix_iterate
+
+echo $flight
+
+# Local variables:
+# mode: sh
+# sh-basic-offset: 2
+# indent-tabs-mode: nil
+# End:
diff --git a/sg-run-job b/sg-run-job
index 3d44331..789315a 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -246,6 +246,17 @@ proc run-job/test-debian {} {
     test-guest debian
 }
 
+proc install-guest-debian-di {} {
+    run-ts . = ts-debian-di-install
+    run-ts . = ts-guest-start + debian
+}
+
+proc need-hosts/test-debian-di {} { return host }
+proc run-job/test-debian-di {} {
+    install-guest-debian-di
+    test-guest debian
+}
+
 proc need-hosts/test-freebsd {} { return host }
 proc run-job/test-freebsd {} {
     run-ts . = ts-freebsd-install
diff --git a/ts-debian-di-install b/ts-debian-di-install
new file mode 100755
index 0000000..a71a336
--- /dev/null
+++ b/ts-debian-di-install
@@ -0,0 +1,167 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost,$gn) = @ARGV;
+$whhost ||= 'host';
+$gn ||= 'debian';
+
+our $ho= selecthost($whhost);
+
+our $ram_mb=    512;
+our $disk_mb= 10000;
+
+our $guesthost= "$gn.guest.osstest";
+our $gho;
+
+sub prep () {
+    target_install_packages_norec($ho, qw(lvm2));
+
+    $gho= prepareguest($ho, $gn, $guesthost, 22,
+                       $disk_mb, 40);
+
+    prepareguest_part_lvmdisk($ho, $gho, $disk_mb);
+
+    target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+}
+
+sub setup_netboot($$$)
+{
+    my ($didir, $arch, $suite) = @_;
+
+    my $di_ver= $r{"$gho->{Guest}_diver"} || "osstest";
+
+    if ( $di_ver eq "osstest" ) {
+	my $di_path = $c{TftpPath}.'/'.$ho->{Tftp}{DiBase}.'/'.$r{arch}.'/'.$c{TftpDiVersion}.'-'.$ho->{Suite};
+
+	my $netboot_kernel = "$di_path/vmlinuz-xen";
+	my $netboot_initrd = "$di_path/initrd.gz-xen";
+
+	target_putfile_root($ho, 60, $netboot_kernel, "$didir/kernel_${suite}_${arch}");
+	target_putfile_root($ho, 60, $netboot_initrd, "$didir/initrd_${suite}_${arch}");
+
+	store_runvar("$gho->{Guest}_netboot_kernel", $netboot_kernel);
+	store_runvar("$gho->{Guest}_netboot_initrd", $netboot_initrd);
+    } else {
+	my $mirror = "http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath}";
+
+	my $di_url = $suite eq "daily" ?
+	    "http://d-i.debian.org/daily-images/$arch/daily/netboot/xen" :
+	    "$mirror/dists/$suite/main/installer-$arch/$di_ver/images/netboot/xen";
+
+	my $netboot_kernel = "$di_url/vmlinuz";
+	my $netboot_initrd = "$di_url/initrd.gz";
+
+	target_fetchurl($ho, $netboot_kernel, "$didir/kernel_${suite}_${arch}");
+	target_fetchurl($ho, $netboot_initrd, "$didir/initrd_${suite}_${arch}");
+
+	store_runvar("$gho->{Guest}_netboot_kernel", $netboot_kernel);
+	store_runvar("$gho->{Guest}_netboot_initrd", $netboot_initrd);
+    }
+
+    return <<END;
+kernel      = "$didir/kernel_${suite}_${arch}"
+ramdisk     = "$didir/initrd_${suite}_${arch}"
+END
+}
+sub ginstall () {
+    my $arch= $r{"$gho->{Guest}_arch"};
+    my $method= $r{"$gho->{Guest}_method"};
+
+    target_cmd_root($ho, <<END);
+rm -rf /root/di
+mkdir /root/di
+END
+
+    my ($method_cfg, $ps_url, $extra_disk);
+
+    if ( $method eq "netboot" )
+    {
+	my $suite= $r{"$gho->{Guest}_dist"};
+	logm("$method $suite/$arch");
+
+	$method_cfg = setup_netboot("/root/di", $arch, $suite);
+
+	$suite = "sid" if $suite eq "daily";
+
+	$ps_url = preseed_create_guest($gho, '', Suite=>$suite);
+
+	$extra_disk = "";
+    }
+    else
+    {
+	die "$method";
+    }
+
+    my $cmdline = join(" ", (
+			   "debian-installer/exit/always_halt=true",
+			   "--",
+			   "console=hvc0",
+			   "auto-install/enable=true",
+			   "hostname=$gho->{Name}",
+			   "domain=$c{TestHostDomain}",
+			   "url=$ps_url",
+			   "DEBIAN_FRONTEND=text",
+			   "netcfg/dhcp_timeout=150",
+			   "netcfg/choose_interface=eth0",
+		       ));
+
+    my %install_xopts = (
+	OnPowerOff => "preserve"
+    );
+
+    prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
+$method_cfg
+extra       = "$cmdline"
+#
+disk        = [
+            $extra_disk 'phy:$gho->{Lvdev},xvda,w'
+            ]
+END
+
+    my $cmd= toolstack()->{Command}." create ".
+        $r{ $gho->{Guest}.'_'. toolstack()->{CfgPathVar} };
+    target_cmd_root($ho, $cmd, 300);
+
+    guest_checkrunning($ho, $gho) or die "$gho->{Name} not running";
+
+    guest_await_shutdown($ho,$gho,2000);
+    guest_destroy($ho,$gho);
+
+    my $blcfg = <<END;
+bootloader = "pygrub"
+END
+
+    prepareguest_part_xencfg($ho, $gho, $ram_mb, {}, <<END);
+$blcfg
+#
+disk        = [
+            'phy:$gho->{Lvdev},xvda,w'
+            ]
+END
+    return;
+}
+
+prep();
+ginstall();
-- 
1.9.0

Re: [PATCH OSSTEST v2 09/15] distros: add support for installing Debian PV guests via d-i, flight and jobs

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 09/15] distros: add support for installing Debian PV guests via d-i, flight and jobs"):
...
> +if [ x$buildflight = x ]; then
> +
> +  if [ "x$BUILD_LVEXTEND_MAX" != x ]; then
> +     BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
> +  fi

This has been clone-and-hacked from make-flight and presumably belongs
in mfi-common (perhaps a subroutine defined there).

...
> +    if ( $di_ver eq "osstest" ) {
...
> +	store_runvar("$gho->{Guest}_netboot_kernel", $netboot_kernel);
> +	store_runvar("$gho->{Guest}_netboot_initrd", $netboot_initrd);
> +    } else {
...
> +	store_runvar("$gho->{Guest}_netboot_kernel", $netboot_kernel);
> +	store_runvar("$gho->{Guest}_netboot_initrd", $netboot_initrd);

Had you not noticed that you'd written out the store_runvars lines
twice ?

> +    target_cmd_root($ho, <<END);
> +rm -rf /root/di
> +mkdir /root/di

You should put the flight and job in this filename in case of future
host-sharing.

> +    my $cmdline = join(" ", (
> +			   "debian-installer/exit/always_halt=true",
> +			   "--",
> +			   "console=hvc0",
> +			   "auto-install/enable=true",
> +			   "hostname=$gho->{Name}",
> +			   "domain=$c{TestHostDomain}",
> +			   "url=$ps_url",
> +			   "DEBIAN_FRONTEND=text",
> +			   "netcfg/dhcp_timeout=150",
> +			   "netcfg/choose_interface=eth0",

I think you want to use Osstest::Debian::di_installcmdline_core.

> +    my $blcfg = <<END;
> +bootloader = "pygrub"
> +END

I'm not sure why you've broken this out.  (I just mention this in case
it's not deliberate.)

Thanks,
Ian.

Re: [PATCH OSSTEST v2 09/15] distros: add support for installing Debian PV guests via d-i, flight and jobs

[Ian Campbell]

On Fri, 2014-05-02 at 12:46 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 09/15] distros: add support for installing Debian PV guests via d-i, flight and jobs"):
> ...
> > +if [ x$buildflight = x ]; then
> > +
> > +  if [ "x$BUILD_LVEXTEND_MAX" != x ]; then
> > +     BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
> > +  fi
> 
> This has been clone-and-hacked from make-flight and presumably belongs
> in mfi-common (perhaps a subroutine defined there).

Ack.

> ...
> > +    if ( $di_ver eq "osstest" ) {
> ...
> > +	store_runvar("$gho->{Guest}_netboot_kernel", $netboot_kernel);
> > +	store_runvar("$gho->{Guest}_netboot_initrd", $netboot_initrd);
> > +    } else {
> ...
> > +	store_runvar("$gho->{Guest}_netboot_kernel", $netboot_kernel);
> > +	store_runvar("$gho->{Guest}_netboot_initrd", $netboot_initrd);
> 
> Had you not noticed that you'd written out the store_runvars lines
> twice ?

The $netboot_foo are local in scope to within the two halves of the
if/else.

> > +    target_cmd_root($ho, <<END);
> > +rm -rf /root/di
> > +mkdir /root/di
> 
> You should put the flight and job in this filename in case of future
> host-sharing.

Ack.

> > +    my $cmdline = join(" ", (
> > +			   "debian-installer/exit/always_halt=true",
> > +			   "--",
> > +			   "console=hvc0",
> > +			   "auto-install/enable=true",
> > +			   "hostname=$gho->{Name}",
> > +			   "domain=$c{TestHostDomain}",
> > +			   "url=$ps_url",
> > +			   "DEBIAN_FRONTEND=text",
> > +			   "netcfg/dhcp_timeout=150",
> > +			   "netcfg/choose_interface=eth0",
> 
> I think you want to use Osstest::Debian::di_installcmdline_core.

I think I do too, thanks for the pointer.

To what extent are $ho and $gho interchangeable? Is it ok to call e.g.
get_host_property on a guest (as that function would do).?

> > +    my $blcfg = <<END;
> > +bootloader = "pygrub"
> > +END
> 
> I'm not sure why you've broken this out.  (I just mention this in case
> it's not deliberate.)

It saved some refactoring noise in a later patch when I add support for
pvgrub.

Ian.

Re: [PATCH OSSTEST v2 09/15] distros: add support for installing Debian PV guests via d-i, flight and jobs

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST v2 09/15] distros: add support for installing Debian PV guests via d-i, flight and jobs"):
> On Fri, 2014-05-02 at 12:46 +0100, Ian Jackson wrote:
> > Had you not noticed that you'd written out the store_runvars lines
> > twice ?
> 
> The $netboot_foo are local in scope to within the two halves of the
> if/else.

You can move declare them (with my) outside the if to fix this
problem.  That's IMO much better than writing out those runvar stores
twice.

> > I think you want to use Osstest::Debian::di_installcmdline_core.
> 
> I think I do too, thanks for the pointer.
> 
> To what extent are $ho and $gho interchangeable? Is it ok to call e.g.
> get_host_property on a guest (as that function would do).?

Hrm.  Arguably according to its name, get_host_property should look
into the $ho->{Host} but it doesn't and that's not what you want here.

Is whether we should do that a function of the property, or a function
of the enquiring call site ?  The code at the moment implies the
former: there is explicit machinery for doing the indirection for
(e.g.) DhcpWatchMethod.

I think until we discover otherwise, we can declare that it's fine to
call get_host_property on a guest object, provided that we understand
that the answer is always going to be "no such property set".  Perhaps
this should be documented.

> > > +    my $blcfg = <<END;
> > > +bootloader = "pygrub"
> > > +END
> > 
> > I'm not sure why you've broken this out.  (I just mention this in case
> > it's not deliberate.)
> 
> It saved some refactoring noise in a later patch when I add support for
> pvgrub.

So I now see.

Thanks,
Ian.

Re: [PATCH OSSTEST v2 09/15] distros: add support for installing Debian PV guests via d-i, flight and jobs

[Ian Campbell]

On Fri, 2014-05-02 at 15:14 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST v2 09/15] distros: add support for installing Debian PV guests via d-i, flight and jobs"):
> > On Fri, 2014-05-02 at 12:46 +0100, Ian Jackson wrote:
> > > Had you not noticed that you'd written out the store_runvars lines
> > > twice ?
> > 
> > The $netboot_foo are local in scope to within the two halves of the
> > if/else.
> 
> You can move declare them (with my) outside the if to fix this
> problem.  That's IMO much better than writing out those runvar stores
> twice.

OK.

> > > I think you want to use Osstest::Debian::di_installcmdline_core.
> > 
> > I think I do too, thanks for the pointer.
> > 
> > To what extent are $ho and $gho interchangeable? Is it ok to call e.g.
> > get_host_property on a guest (as that function would do).?
> 
> Hrm.  Arguably according to its name, get_host_property should look
> into the $ho->{Host} but it doesn't and that's not what you want here.
> 
> Is whether we should do that a function of the property, or a function
> of the enquiring call site ?  The code at the moment implies the
> former: there is explicit machinery for doing the indirection for
> (e.g.) DhcpWatchMethod.
> 
> I think until we discover otherwise, we can declare that it's fine to
> call get_host_property on a guest object, provided that we understand
> that the answer is always going to be "no such property set".

I'm happy to do that.

> Perhaps this should be documented.

I'll try and remember to slip something in.

Ian.

[PATCH OSSTEST v2 10/15] distros: support booting Debian PV (d-i installed) guests with pvgrub.

[Ian Campbell]

This requires the use of the pv-grub-menu package which is in Jessie onwards.
(it is in wheezy-backports which is the subject of a subsequent flight).

The bootloader to use is specified via a runvar {Guest}_bootloader.

Adjust make-distros-flight to use pvgrub for some subset of i386 and amd64
guests to get coverage.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm    |  2 +-
 make-distros-flight  | 16 ++++++++++++++++
 ts-debian-di-install | 13 +++++++++++--
 3 files changed, 28 insertions(+), 3 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 043ab0d..8946ab1 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -508,7 +508,7 @@ sub preseed_create_guest ($$;@) {
 
     my $suite= $xopts{Suite} || $c{DebianSuite};
 
-    my $extra_packages;
+    my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
 
     my $preseed_file= preseed_base($suite, $extra_packages, %xopts);
     $preseed_file.= (<<END);
diff --git a/make-distros-flight b/make-distros-flight
index abc24ff..a6e41e0 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -59,6 +59,21 @@ test_matrix_branch_filter_callback () {
 }
 
 test_do_one_netboot () {
+  stripy bootloader pygrub pvgrub \
+    "$xenarch" "i386" \
+    "$dom0arch" "amd64" \
+    "$domU" "amd64" \
+    "$dist" "sid"
+
+  case ${domU}_${dist} in
+    armhf_*) bootloader="pygrub";; # no pvgrub for arm
+
+    # Needs a menu.lst, not present in Squeeze+ due to switch to grub2,
+    # workedaround in Jessie+ with pv-grub-menu package.
+    *_squeeze) bootloader="pygrub";;
+    *_wheezy) bootloader="pygrub";;
+  esac
+
   job_create_test test-$xenarch$kern-$dom0arch-$domU-$dist-netboot \
     test-debian-di xl $xenarch $dom0arch                        \
       kernbuildjob=${bfi}build-$dom0arch-$kernbuild             \
@@ -66,6 +81,7 @@ test_do_one_netboot () {
       debian_dist=$dist                                         \
       debian_method=netboot                                     \
       debian_diver=current                                      \
+      debian_bootloader=$bootloader                             \
       all_hostflags=$most_hostflags
 }
 
diff --git a/ts-debian-di-install b/ts-debian-di-install
index a71a336..5ed15e4 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -89,6 +89,8 @@ sub ginstall () {
     my $arch= $r{"$gho->{Guest}_arch"};
     my $method= $r{"$gho->{Guest}_method"};
 
+    my $bl= $r{"$gho->{Guest}_bootloader"};
+
     target_cmd_root($ho, <<END);
 rm -rf /root/di
 mkdir /root/di
@@ -105,7 +107,7 @@ END
 
 	$suite = "sid" if $suite eq "daily";
 
-	$ps_url = preseed_create_guest($gho, '', Suite=>$suite);
+	$ps_url = preseed_create_guest($gho, '', Suite=>$suite, PvMenuLst=>($bl eq "pvgrub"));
 
 	$extra_disk = "";
     }
@@ -149,7 +151,14 @@ END
     guest_await_shutdown($ho,$gho,2000);
     guest_destroy($ho,$gho);
 
-    my $blcfg = <<END;
+    my $pvgrub = {
+	i386 => "/usr/local/lib/xen/boot/pv-grub-x86_32.gz",
+	amd64 => "/usr/local/lib/xen/boot/pv-grub-x86_64.gz"
+    };
+    my $blcfg = $bl eq "pvgrub" ? <<END : <<END;
+kernel = "$pvgrub->{$arch}"
+extra = "(hd0,0)/boot/grub/menu.lst"
+END
 bootloader = "pygrub"
 END
 
-- 
1.9.0

Re: [PATCH OSSTEST v2 10/15] distros: support booting Debian PV (d-i installed) guests with pvgrub.

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 10/15] distros: support booting Debian PV (d-i installed) guests with pvgrub."):
> This requires the use of the pv-grub-menu package which is in Jessie onwards.
> (it is in wheezy-backports which is the subject of a subsequent flight).
> 
> The bootloader to use is specified via a runvar {Guest}_bootloader.
> 
> Adjust make-distros-flight to use pvgrub for some subset of i386 and amd64
> guests to get coverage.

This looks reasonable to me.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>


> -    my $blcfg = <<END;
> +    my $pvgrub = {
> +	i386 => "/usr/local/lib/xen/boot/pv-grub-x86_32.gz",
> +	amd64 => "/usr/local/lib/xen/boot/pv-grub-x86_64.gz"
> +    };

I wonder, though, whether this knowledge about arch name mappings
could be combined with that from
ts-kernel-build:config_create_config_sh and put in Osstest.pm or
somewhere.

Ian.

Re: [PATCH OSSTEST v2 10/15] distros: support booting Debian PV (d-i installed) guests with pvgrub.

[Ian Campbell]

On Fri, 2014-05-02 at 14:03 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 10/15] distros: support booting Debian PV (d-i installed) guests with pvgrub."):
> > This requires the use of the pv-grub-menu package which is in Jessie onwards.
> > (it is in wheezy-backports which is the subject of a subsequent flight).
> > 
> > The bootloader to use is specified via a runvar {Guest}_bootloader.
> > 
> > Adjust make-distros-flight to use pvgrub for some subset of i386 and amd64
> > guests to get coverage.
> 
> This looks reasonable to me.
> 
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Ta.

> > -    my $blcfg = <<END;
> > +    my $pvgrub = {
> > +	i386 => "/usr/local/lib/xen/boot/pv-grub-x86_32.gz",
> > +	amd64 => "/usr/local/lib/xen/boot/pv-grub-x86_64.gz"
> > +    };
> 
> I wonder, though, whether this knowledge about arch name mappings
> could be combined with that from
> ts-kernel-build:config_create_config_sh and put in Osstest.pm or
> somewhere.

As in some sort of Osstest::arch_params{PvGrub}, {LinuxDefconfg},
{LinuxKernelImage} ?

I think so.

Re: [PATCH OSSTEST v2 10/15] distros: support booting Debian PV (d-i installed) guests with pvgrub.

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST v2 10/15] distros: support booting Debian PV (d-i installed) guests with pvgrub."):
> On Fri, 2014-05-02 at 14:03 +0100, Ian Jackson wrote:
> > > -    my $blcfg = <<END;
> > > +    my $pvgrub = {
> > > +	i386 => "/usr/local/lib/xen/boot/pv-grub-x86_32.gz",
> > > +	amd64 => "/usr/local/lib/xen/boot/pv-grub-x86_64.gz"
> > > +    };
> > 
> > I wonder, though, whether this knowledge about arch name mappings
> > could be combined with that from
> > ts-kernel-build:config_create_config_sh and put in Osstest.pm or
> > somewhere.
> 
> As in some sort of Osstest::arch_params{PvGrub}, {LinuxDefconfg},
> {LinuxKernelImage} ?
> 
> I think so.

I was thinking

 our %arch_debian2xen = qw(i386 x86_32
                           amd64 x86_64
                           armhf armhf);
 our %arch_xen2debian;
 $arch_xen2debian{$arch_debian2xen{$_}} = $_ foreach keys %arch_debian2xen;

or some such.

Then

   my $xenarch = $arch_debian2xen{$arch};
   my $pvgrub = "/usr/local/lib/xen/boot/pv-grub-$xenarch.gz";

Or is that conceptually wrong ?

Ian.

Re: [PATCH OSSTEST v2 10/15] distros: support booting Debian PV (d-i installed) guests with pvgrub.

[Ian Campbell]

On Fri, 2014-05-02 at 15:18 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST v2 10/15] distros: support booting Debian PV (d-i installed) guests with pvgrub."):
> > On Fri, 2014-05-02 at 14:03 +0100, Ian Jackson wrote:
> > > > -    my $blcfg = <<END;
> > > > +    my $pvgrub = {
> > > > +	i386 => "/usr/local/lib/xen/boot/pv-grub-x86_32.gz",
> > > > +	amd64 => "/usr/local/lib/xen/boot/pv-grub-x86_64.gz"
> > > > +    };
> > > 
> > > I wonder, though, whether this knowledge about arch name mappings
> > > could be combined with that from
> > > ts-kernel-build:config_create_config_sh and put in Osstest.pm or
> > > somewhere.
> > 
> > As in some sort of Osstest::arch_params{PvGrub}, {LinuxDefconfg},
> > {LinuxKernelImage} ?
> > 
> > I think so.
> 
> I was thinking
> 
>  our %arch_debian2xen = qw(i386 x86_32
>                            amd64 x86_64
>                            armhf armhf);
>  our %arch_xen2debian;
>  $arch_xen2debian{$arch_debian2xen{$_}} = $_ foreach keys %arch_debian2xen;

Wowza!

Do we need xen2debian anywhere? Mostly we have a debian thing and need a
Xen (or a Linux) thing.

> or some such.
> 
> Then
> 
>    my $xenarch = $arch_debian2xen{$arch};
>    my $pvgrub = "/usr/local/lib/xen/boot/pv-grub-$xenarch.gz";
> 
> Or is that conceptually wrong ?

Not inherently, I don't think.

You mean to do the same for $arch_debian2linux for the ts-kernel-build
stuff I guess?

Ian.

Re: [PATCH OSSTEST v2 10/15] distros: support booting Debian PV (d-i installed) guests with pvgrub.

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST v2 10/15] distros: support booting Debian PV (d-i installed) guests with pvgrub."):
> On Fri, 2014-05-02 at 15:18 +0100, Ian Jackson wrote:
> >  our %arch_xen2debian;
> >  $arch_xen2debian{$arch_debian2xen{$_}} = $_ foreach keys %arch_debian2xen;
> 
> Wowza!
> 
> Do we need xen2debian anywhere? Mostly we have a debian thing and need a
> Xen (or a Linux) thing.

Maybe not.  I provided it for completeness :-).

> > Then
> >    my $xenarch = $arch_debian2xen{$arch};
> >    my $pvgrub = "/usr/local/lib/xen/boot/pv-grub-$xenarch.gz";
> > Or is that conceptually wrong ?
> 
> Not inherently, I don't think.
> 
> You mean to do the same for $arch_debian2linux for the ts-kernel-build
> stuff I guess?

Right.

Ian.

[PATCH OSSTEST v2 11/15] distros: attempt pvgrub support for PV Wheezy too.

[Ian Campbell]

This requires us to install pv-grub-menu from backports, however the version
there seems to be buggy right now so it doesn't actually work. Introduce the
test anyway, a Debian bug report will follow.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm   | 16 ++++++++++++++--
 make-distros-flight |  5 +++--
 2 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 8946ab1..ad478ab 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -432,7 +432,7 @@ sub preseed_base ($$;@) {
 
     $extra_packages ||= '';
 
-    return <<"END";
+    my $preseed= (<<END);
 d-i mirror/suite string $suite
 
 d-i debian-installer/locale string en_GB
@@ -501,6 +501,14 @@ $xopts{ExtraPreseed}
 ### END OF DEBIAN PRESEED BASE
 
 END
+
+    # deb http://ftp.debian.org/debian/ wheezy-backports main
+    $preseed .= <<END if $xopts{EnableBackports};
+d-i apt-setup/local0/repository string http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
+d-i apt-setup/local0/comment string $suite backports
+END
+
+    return $preseed;
 }
 
 sub preseed_create_guest ($$;@) {
@@ -508,7 +516,11 @@ sub preseed_create_guest ($$;@) {
 
     my $suite= $xopts{Suite} || $c{DebianSuite};
 
-    my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
+    $xopts{EnableBackports} = 1
+	if $xopts{PvMenuLst} and $suite eq "wheezy";
+    my $extra_packages = "pv-grub-menu".
+	($xopts{EnableBackports} ? "/$suite-backports" : "")
+	    if $xopts{PvMenuLst};
 
     my $preseed_file= preseed_base($suite, $extra_packages, %xopts);
     $preseed_file.= (<<END);
diff --git a/make-distros-flight b/make-distros-flight
index a6e41e0..244d582 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -69,9 +69,10 @@ test_do_one_netboot () {
     armhf_*) bootloader="pygrub";; # no pvgrub for arm
 
     # Needs a menu.lst, not present in Squeeze+ due to switch to grub2,
-    # workedaround in Jessie+ with pv-grub-menu package.
+    # workedaround in Wheezy+ with pv-grub-menu package (backports in Wheezy,
+    # in Jessie+ main).
     *_squeeze) bootloader="pygrub";;
-    *_wheezy) bootloader="pygrub";;
+    #*_wheezy) bootloader="pygrub";;
   esac
 
   job_create_test test-$xenarch$kern-$dom0arch-$domU-$dist-netboot \
-- 
1.9.0

Re: [PATCH OSSTEST v2 11/15] distros: attempt pvgrub support for PV Wheezy too.

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 11/15] distros: attempt pvgrub support for PV Wheezy too."):
> diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
> index 8946ab1..ad478ab 100644
> --- a/Osstest/Debian.pm
> +++ b/Osstest/Debian.pm
> @@ -432,7 +432,7 @@ sub preseed_base ($$;@) {
>  
>      $extra_packages ||= '';
>  
> -    return <<"END";
> +    my $preseed= (<<END);

These parens are unnecessary.  (As were the "" before.  Not sure why
they were there.)

> +    $preseed .= <<END if $xopts{EnableBackports};
> +d-i apt-setup/local0/repository string http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main

Perhaps this wants \\ (to wrap it) ?

> -    my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
> +    $xopts{EnableBackports} = 1
> +	if $xopts{PvMenuLst} and $suite eq "wheezy";

I normally write this as   $suite =~ m/wheezy/

> +    my $extra_packages = "pv-grub-menu".
> +	($xopts{EnableBackports} ? "/$suite-backports" : "")
> +	    if $xopts{PvMenuLst};

I think this is a rather strange way of doing things.  It's
backwards.  What you mean is probably something like this:

  +    my $extra_packages = "pv-grub-menu".
  +	($suite =~ m/wheezy/ ? "/$suite-backports" : "")
  +	    if $xopts{PvMenuLst};

  +    $xopts{EnableBackports} ||= $extra_packages =~ m#/\S+-backports\b#;

or even better, do away with the EnableBackports xopt altogether and
have preseed_base do it itself automatically.

Ian.

Re: [PATCH OSSTEST v2 11/15] distros: attempt pvgrub support for PV Wheezy too.

[Ian Campbell]

On Fri, 2014-05-02 at 14:11 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 11/15] distros: attempt pvgrub support for PV Wheezy too."):
> > diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
> > index 8946ab1..ad478ab 100644
> > --- a/Osstest/Debian.pm
> > +++ b/Osstest/Debian.pm
> > @@ -432,7 +432,7 @@ sub preseed_base ($$;@) {
> >  
> >      $extra_packages ||= '';
> >  
> > -    return <<"END";
> > +    my $preseed= (<<END);
> 
> These parens are unnecessary.

OK

>   (As were the "" before.  Not sure why
> they were there.)

Does the quoting of the termination marker have an impact on the way
things within the heredoc are interpreted?

> 
> > +    $preseed .= <<END if $xopts{EnableBackports};
> > +d-i apt-setup/local0/repository string http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
> 
> Perhaps this wants \\ (to wrap it) ?

Sure.

> > -    my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
> > +    $xopts{EnableBackports} = 1
> > +	if $xopts{PvMenuLst} and $suite eq "wheezy";
> 
> I normally write this as   $suite =~ m/wheezy/

OK. (But OOI why?)

> 
> > +    my $extra_packages = "pv-grub-menu".
> > +	($xopts{EnableBackports} ? "/$suite-backports" : "")
> > +	    if $xopts{PvMenuLst};
> 
> I think this is a rather strange way of doing things.  It's
> backwards.  What you mean is probably something like this:
> 
>   +    my $extra_packages = "pv-grub-menu".
>   +	($suite =~ m/wheezy/ ? "/$suite-backports" : "")
>   +	    if $xopts{PvMenuLst};
> 
>   +    $xopts{EnableBackports} ||= $extra_packages =~ m#/\S+-backports\b#;
> 
> or even better, do away with the EnableBackports xopt altogether and
> have preseed_base do it itself automatically.

OK.

Ian.

Re: [PATCH OSSTEST v2 11/15] distros: attempt pvgrub support for PV Wheezy too.

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST v2 11/15] distros: attempt pvgrub support for PV Wheezy too."):
> On Fri, 2014-05-02 at 14:11 +0100, Ian Jackson wrote:
> >   (As were the "" before.  Not sure why they were there.)
> 
> Does the quoting of the termination marker have an impact on the way
> things within the heredoc are interpreted?

Yes, but "" is the same as no quoting.  Saying <<'EOF' prevents
variable interpolation.

> > > -    my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
> > > +    $xopts{EnableBackports} = 1
> > > +	if $xopts{PvMenuLst} and $suite eq "wheezy";
> > 
> > I normally write this as   $suite =~ m/wheezy/
> 
> OK. (But OOI why?)

It makes it easier to add |jessie when the problem remains unfixed.
(Also if someone has some kind of derivative called "wheezy-blarf", it
is more likely to be right than the exact match.)

> > > +    my $extra_packages = "pv-grub-menu".
> > > +	($xopts{EnableBackports} ? "/$suite-backports" : "")
> > > +	    if $xopts{PvMenuLst};
> > 
> > I think this is a rather strange way of doing things.  It's
> > backwards.  What you mean is probably something like this:
> > 
> >   +    my $extra_packages = "pv-grub-menu".
> >   +	($suite =~ m/wheezy/ ? "/$suite-backports" : "")
> >   +	    if $xopts{PvMenuLst};
> > 
> >   +    $xopts{EnableBackports} ||= $extra_packages =~ m#/\S+-backports\b#;
> > 
> > or even better, do away with the EnableBackports xopt altogether and
> > have preseed_base do it itself automatically.
> 
> OK.

Thanks,
Ian.

Re: [PATCH OSSTEST v2 11/15] distros: attempt pvgrub support for PV Wheezy too.

[Ian Campbell]

On Fri, 2014-05-02 at 15:19 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST v2 11/15] distros: attempt pvgrub support for PV Wheezy too."):
> > On Fri, 2014-05-02 at 14:11 +0100, Ian Jackson wrote:
> > >   (As were the "" before.  Not sure why they were there.)
> > 
> > Does the quoting of the termination marker have an impact on the way
> > things within the heredoc are interpreted?
> 
> Yes, but "" is the same as no quoting.  Saying <<'EOF' prevents
> variable interpolation.

Ah, "" is the same as no quoting is what I forget.

> > > > -    my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
> > > > +    $xopts{EnableBackports} = 1
> > > > +	if $xopts{PvMenuLst} and $suite eq "wheezy";
> > > 
> > > I normally write this as   $suite =~ m/wheezy/
> > 
> > OK. (But OOI why?)
> 
> It makes it easier to add |jessie when the problem remains unfixed.
> (Also if someone has some kind of derivative called "wheezy-blarf", it
> is more likely to be right than the exact match.)

Lets hope we never have a legitimate release which is a super/substring
of another ;-) (at least bo and other dead short ones are long ago).

[PATCH OSSTEST v2 12/15] distros: support PV guest install from Debian netinst media.

[Ian Campbell]

The netinst media are iso images containing a base Debian install and some
(image size dependent) additional tasks.

On x86 the "multiarch" iso flavour contains a Xen capable kernel for both i386
and amd64 so use that.

This adds support for two classes of ISO, the CD sized ones which are built
nightly and the DVD sized ones which are built weekly.

The images are downloaded using jigdo which sources the majority of the data
from a local Debian mirror, for this reason I have not worried about the fact
that the i386 and amd64 tests are downloading the same thing (adding a specific
download job would require finding up to 4GB of scratch space for each flight).

The ISOs booted using pygrub which can extract the kernel and initrd from a ISO
image. The resulting guests are also booted with pygrub since the pv-grub-menu
package is not available on the ISO images and we have pvgrub coverage from the
netboot tests.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm    |  7 +++++--
 make-distros-flight  | 22 ++++++++++++++++++++
 ts-debian-di-install | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 84 insertions(+), 2 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index ad478ab..b149e8d 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -433,8 +433,6 @@ sub preseed_base ($$;@) {
     $extra_packages ||= '';
 
     my $preseed= (<<END);
-d-i mirror/suite string $suite
-
 d-i debian-installer/locale string en_GB
 d-i console-keymaps-at/keymap select gb
 d-i keyboard-configuration/xkb-keymap string en_GB
@@ -502,6 +500,11 @@ $xopts{ExtraPreseed}
 
 END
 
+    # For CDROM the suite is part of the image
+    $preseed .= <<END unless $xopts{CDROM};
+d-i mirror/suite string $suite
+END
+
     # deb http://ftp.debian.org/debian/ wheezy-backports main
     $preseed .= <<END if $xopts{EnableBackports};
 d-i apt-setup/local0/repository string http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
diff --git a/make-distros-flight b/make-distros-flight
index 244d582..a679c61 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -86,6 +86,19 @@ test_do_one_netboot () {
       all_hostflags=$most_hostflags
 }
 
+test_do_one_netinst () {
+  # Always pygrub since no pv-grub-menu on CD
+  job_create_test test-$xenarch$kern-$dom0arch-$domU-$cd-netinst \
+    test-debian-di xl $xenarch $dom0arch                        \
+      kernbuildjob=${bfi}build-$dom0arch-$kernbuild             \
+      debian_arch=$domU                                         \
+      debian_cd=$cd                                             \
+      debian_method=netinst                                     \
+      debian_bootloader=pygrub                                  \
+      all_hostflags=$most_hostflags
+
+}
+
 test_matrix_do_one () {
   case ${xenarch} in
   amd64) domUarches="amd64 i386";;
@@ -104,6 +117,15 @@ test_matrix_do_one () {
 
     done
 
+    for cd in current weekly ; do
+      case ${domU}_${dist} in
+      armhf_*) continue;; # No iso targets for armhf
+      *) ;;
+      esac
+
+      test_do_one_netinst
+
+    done
   done
 }
 
diff --git a/ts-debian-di-install b/ts-debian-di-install
index 5ed15e4..56feee6 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -46,6 +46,53 @@ sub prep () {
     target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
 }
 
+sub setup_netinst($$$)
+{
+    my ($didir, $arch, $cd) = @_;
+    my %arch_props = (
+	amd64 => { PathArch => "multi-arch", FileArch => "amd64-i386", IsoPath => "/install.amd/xen" },
+	i386  => { PathArch => "multi-arch", FileArch => "amd64-i386", IsoPath => "/install.386/xen" },
+	armhf => { PathArch => "armhf",      FileArch => "armhf",      IsoPath => "/install.armhf" }
+    );
+    my $props = $arch_props{$arch} or die "Unknown arch $arch";
+
+    target_install_packages($ho, qw(jigdo-file));
+
+    my $baseurl = $cd eq "current" ?
+      "http://cdimage.debian.org/debian-cd/current/$props->{PathArch}/jigdo-cd" :
+      "http://cdimage.debian.org/cdimage/weekly-builds/$props->{PathArch}/jigdo-cd";
+
+    my $filebase;
+
+    # Use the MD5SUMs file as an index
+    logm("Fetch index from $baseurl/MD5SUMS");
+    open C, "curl $baseurl/MD5SUMS|" or die "curl: $!";
+    while(<C>) {
+	m/^[0-9a-f]{32}  (debian-.*-$props->{FileArch}-netinst)\.iso$/ or next;
+	$filebase = $1;
+	last;
+    }
+    close(C);
+
+    die unless $filebase;
+
+    logm("Downloading $baseurl/$filebase.jigdo");
+    # Jigdo seems to use /etc/apt/sources.list or something, so this
+    # just works using the already configured mirror without
+    # additional configuration, which is good because there doesn't
+    # seem to be any support for such things, at least in Squeeze.
+    my $netinst_jigdo = "$baseurl/$filebase.jigdo";
+    target_cmd_root($ho, <<END, 3600);
+    cd $didir && jigdo-lite --noask $netinst_jigdo
+END
+    store_runvar("$gho->{Guest}_netinst_jigdo", $netinst_jigdo);
+
+    return (<<END, "\"file:$didir/$filebase.iso,xvdd:cdrom,r\",");
+bootloader = "pygrub"
+bootloader_args = ["--kernel=$props->{IsoPath}/vmlinuz", "--ramdisk=$props->{IsoPath}/initrd.gz"]
+END
+}
+
 sub setup_netboot($$$)
 {
     my ($didir, $arch, $suite) = @_;
@@ -111,6 +158,16 @@ END
 
 	$extra_disk = "";
     }
+    elsif ($method eq "netinst" )
+    {
+	my $cd = $r{"$gho->{Guest}_cd"};
+
+	logm("$method $cd/$arch");
+
+	($method_cfg,$extra_disk) = setup_netinst("/root/di", $arch, $cd);
+
+	$ps_url = preseed_create_guest($gho, '', CDROM=>1);
+    }
     else
     {
 	die "$method";
-- 
1.9.0

Re: [PATCH OSSTEST v2 12/15] distros: support PV guest install from Debian netinst media.

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 12/15] distros: support PV guest install from Debian netinst media."):
> The netinst media are iso images containing a base Debian install and some
> (image size dependent) additional tasks.
> 
> On x86 the "multiarch" iso flavour contains a Xen capable kernel for both i386
> and amd64 so use that.
> 
> This adds support for two classes of ISO, the CD sized ones which are built
> nightly and the DVD sized ones which are built weekly.
> 
> The images are downloaded using jigdo which sources the majority of the data
> from a local Debian mirror, for this reason I have not worried about the fact
> that the i386 and amd64 tests are downloading the same thing (adding a specific
> download job would require finding up to 4GB of scratch space for each flight).
> 
> The ISOs booted using pygrub which can extract the kernel and initrd from a ISO
> image. The resulting guests are also booted with pygrub since the pv-grub-menu
> package is not available on the ISO images and we have pvgrub coverage from the
> netboot tests.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  Osstest/Debian.pm    |  7 +++++--
>  make-distros-flight  | 22 ++++++++++++++++++++
>  ts-debian-di-install | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 84 insertions(+), 2 deletions(-)
> 
> diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
> index ad478ab..b149e8d 100644
> --- a/Osstest/Debian.pm
> +++ b/Osstest/Debian.pm
> @@ -433,8 +433,6 @@ sub preseed_base ($$;@) {
>      $extra_packages ||= '';
>  
>      my $preseed= (<<END);
> -d-i mirror/suite string $suite
> -
>  d-i debian-installer/locale string en_GB
>  d-i console-keymaps-at/keymap select gb
>  d-i keyboard-configuration/xkb-keymap string en_GB
> @@ -502,6 +500,11 @@ $xopts{ExtraPreseed}
>  
>  END
>  
> +    # For CDROM the suite is part of the image
> +    $preseed .= <<END unless $xopts{CDROM};
> +d-i mirror/suite string $suite
> +END
> +
>      # deb http://ftp.debian.org/debian/ wheezy-backports main
>      $preseed .= <<END if $xopts{EnableBackports};
>  d-i apt-setup/local0/repository string http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
> diff --git a/make-distros-flight b/make-distros-flight
> index 244d582..a679c61 100755
> --- a/make-distros-flight
> +++ b/make-distros-flight
> @@ -86,6 +86,19 @@ test_do_one_netboot () {
>        all_hostflags=$most_hostflags
>  }
>  
> +test_do_one_netinst () {
> +  # Always pygrub since no pv-grub-menu on CD
> +  job_create_test test-$xenarch$kern-$dom0arch-$domU-$cd-netinst \
> +    test-debian-di xl $xenarch $dom0arch                        \
> +      kernbuildjob=${bfi}build-$dom0arch-$kernbuild             \
> +      debian_arch=$domU                                         \
> +      debian_cd=$cd                                             \
> +      debian_method=netinst                                     \
> +      debian_bootloader=pygrub                                  \
> +      all_hostflags=$most_hostflags
> +
> +}
> +
>  test_matrix_do_one () {
>    case ${xenarch} in
>    amd64) domUarches="amd64 i386";;
> @@ -104,6 +117,15 @@ test_matrix_do_one () {
>  
>      done
>  
> +    for cd in current weekly ; do
> +      case ${domU}_${dist} in
> +      armhf_*) continue;; # No iso targets for armhf
> +      *) ;;
> +      esac
> +
> +      test_do_one_netinst
> +
> +    done
>    done
>  }
>  
> diff --git a/ts-debian-di-install b/ts-debian-di-install
> index 5ed15e4..56feee6 100755
> --- a/ts-debian-di-install
> +++ b/ts-debian-di-install
> @@ -46,6 +46,53 @@ sub prep () {
>      target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
>  }
>  
> +sub setup_netinst($$$)
> +{
...
> +    # Use the MD5SUMs file as an index
> +    logm("Fetch index from $baseurl/MD5SUMS");
> +    open C, "curl $baseurl/MD5SUMS|" or die "curl: $!";

This needs to be done in a way that has a timeout.  I don't know what
curl's default is but I think it would be best to arrange to use cmd
in TestSupport.pm somehow.  (Maybe directly, in which case it needs to
be exported.)


> +    target_cmd_root($ho, <<END, 3600);
> +    cd $didir && jigdo-lite --noask $netinst_jigdo
> +END

Can you indent the script line so this is easier to read ?

> +    store_runvar("$gho->{Guest}_netinst_jigdo", $netinst_jigdo);
> +
> +    return (<<END, "\"file:$didir/$filebase.iso,xvdd:cdrom,r\",");
> +bootloader = "pygrub"
> +bootloader_args = ["--kernel=$props->{IsoPath}/vmlinuz", "--ramdisk=$props->{IsoPath}/initrd.gz"]

Could be wrapped.

Thanks,
Ian.

Re: [PATCH OSSTEST v2 12/15] distros: support PV guest install from Debian netinst media.

[Ian Campbell]

On Fri, 2014-05-02 at 15:01 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 12/15] distros: support PV guest install from Debian netinst media."):
> > +sub setup_netinst($$$)
> > +{
> ...
> > +    # Use the MD5SUMs file as an index
> > +    logm("Fetch index from $baseurl/MD5SUMS");
> > +    open C, "curl $baseurl/MD5SUMS|" or die "curl: $!";
> 
> This needs to be done in a way that has a timeout.  I don't know what
> curl's default is but I think it would be best to arrange to use cmd
> in TestSupport.pm somehow.  (Maybe directly, in which case it needs to
> be exported.)

I'll investigate.

Is doing this from the test controller OK, moving it to the test dom0
would nicely fix thistoo.

> > +    target_cmd_root($ho, <<END, 3600);
> > +    cd $didir && jigdo-lite --noask $netinst_jigdo
> > +END
> 
> Can you indent the script line so this is easier to read ?

Yes.

> > +    store_runvar("$gho->{Guest}_netinst_jigdo", $netinst_jigdo);
> > +
> > +    return (<<END, "\"file:$didir/$filebase.iso,xvdd:cdrom,r\",");
> > +bootloader = "pygrub"
> > +bootloader_args = ["--kernel=$props->{IsoPath}/vmlinuz", "--ramdisk=$props->{IsoPath}/initrd.gz"]
> 
> Could be wrapped.

Ack.

Re: [PATCH OSSTEST v2 12/15] distros: support PV guest install from Debian netinst media.

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST v2 12/15] distros: support PV guest install from Debian netinst media."):
> On Fri, 2014-05-02 at 15:01 +0100, Ian Jackson wrote:
> > This needs to be done in a way that has a timeout.  I don't know what
> > curl's default is but I think it would be best to arrange to use cmd
> > in TestSupport.pm somehow.  (Maybe directly, in which case it needs to
> > be exported.)
> 
> I'll investigate.
> 
> Is doing this from the test controller OK, moving it to the test dom0
> would nicely fix thistoo.

I think it's fine to do it from either.

Thanks,
Ian.

[PATCH OSSTEST v2 13/15] Test pygrub and pvgrub on the regular flights

[Ian Campbell]

Since we now have the ability to test these drop one of each of pygrub,
pvgrub-32 and pvgrub-64 into the standard flights. Omitting the {Guest}_diver
runvar causes ts-debian-di-install to use the d-i images in the location
configured via TftpDiVersion, so they are version controlled along with the d-i
version used for the host.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 make-flight | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/make-flight b/make-flight
index aadf58b..c4ddbc4 100755
--- a/make-flight
+++ b/make-flight
@@ -233,6 +233,42 @@ do_passthrough_tests () {
   done
 }
 
+do_pygrub_tests () {
+  if [ $xenarch != amd64 -o $dom0arch != amd64 -o "$kern" != "" ]; then
+    return
+  fi
+
+  job_create_test test-$xenarch$kern-$dom0arch-pygrub   \
+    test-debian-di xl $xenarch $dom0arch                \
+      debian_arch=amd64                                 \
+      debian_dist=$guestsuite                           \
+      debian_method=netboot                             \
+      debian_bootloader=pygrub                          \
+      all_hostflags=$most_hostflags
+}
+
+do_pvgrub_tests () {
+  if [ $xenarch != amd64 -o $dom0arch != amd64 -o "$kern" != "" ]; then
+    return
+  fi
+
+  job_create_test test-$xenarch$kern-$dom0arch-amd64-pvgrub     \
+    test-debian-di xl $xenarch $dom0arch                        \
+      debian_arch=amd64                                         \
+      debian_dist=$guestsuite                                   \
+      debian_method=netboot                                     \
+      debian_bootloader=pvgrub                                  \
+      all_hostflags=$most_hostflags                             \
+
+  job_create_test test-$xenarch$kern-$dom0arch-i386-pvgrub      \
+    test-debian-di xl $xenarch $dom0arch                        \
+      debian_arch=i386                                          \
+      debian_dist=$guestsuite                                   \
+      debian_method=netboot                                     \
+      debian_bootloader=pvgrub                                  \
+      all_hostflags=$most_hostflags
+}
+
 test_matrix_do_one () {
 
   # Basic PV Linux test with xl
@@ -315,6 +351,9 @@ test_matrix_do_one () {
   fi
 
   do_passthrough_tests
+
+  do_pygrub_tests
+  do_pvgrub_tests
 }
 
 test_matrix_iterate
-- 
1.9.0

Re: [PATCH OSSTEST v2 13/15] Test pygrub and pvgrub on the regular flights

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 13/15] Test pygrub and pvgrub on the regular flights"):
> Since we now have the ability to test these drop one of each of
> pygrub, pvgrub-32 and pvgrub-64 into the standard flights. Omitting
> the {Guest}_diver runvar causes ts-debian-di-install to use the d-i
> images in the location configured via TftpDiVersion, so they are
> version controlled along with the d-i version used for the host.

This looks plausible.  But can you show the mg-show-flight-runvars
diff ?

Thanks,
Ian.

Re: [PATCH OSSTEST v2 13/15] Test pygrub and pvgrub on the regular flights

[Ian Campbell]

On Fri, 2014-05-02 at 15:02 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 13/15] Test pygrub and pvgrub on the regular flights"):
> > Since we now have the ability to test these drop one of each of
> > pygrub, pvgrub-32 and pvgrub-64 into the standard flights. Omitting
> > the {Guest}_diver runvar causes ts-debian-di-install to use the d-i
> > images in the location configured via TftpDiVersion, so they are
> > version controlled along with the d-i version used for the host.
> 
> This looks plausible.  But can you show the mg-show-flight-runvars
> diff ?

Will do in the next post.

[PATCH OSSTEST v2 14/15] distros: add branch infrastructure

[Ian Campbell]

Since the distro nightlies are not version controlled we cannot use the usual
mechanisms for detecting regressions. Instead compare each run against its
predecessor, determined using sg-check-tested.

XXX needs testing in production environment.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ap-fetch-version     |  4 ++++
 ap-fetch-version-old |  3 +++
 ap-print-url         |  3 +++
 cr-daily-branch      | 11 ++++++++++-
 cri-common           |  1 +
 5 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/ap-fetch-version b/ap-fetch-version
index 7cc2d98..17ab7c8 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -77,6 +77,10 @@ gnulib-libvirt)
 	repo_tree_rev_fetch_git gnulib-libvirt \
 		$TREE_GNULIB_LIBVIRT master $LOCALREV_GNULIB_LIBVIRT
 	;;
+distros)
+	# None, this is just a regular test.
+	echo latest
+	;;
 osstest)
 	if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then
 	    git fetch $HOME/testing.git pretest:ap-fetch >&2
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index 3879e09..621641e 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -82,6 +82,9 @@ gnulib-libvirt)
 	# No push gate, same as ap-fetch-version
 	./ap-fetch-version $branch
 	;;
+distros)
+	echo "lastflight" # compare with previous flight
+	;;
 osstest)
 	if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then
 	    git fetch -f $HOME/testing.git incoming:ap-fetch
diff --git a/ap-print-url b/ap-print-url
index b1af0b9..9084635 100755
--- a/ap-print-url
+++ b/ap-print-url
@@ -52,6 +52,9 @@ libvirt)
 gnulib-libvirt)
 	echo $TREE_GNULIB_LIBVIRT
 	;;
+distros)
+	echo none:;
+	;;
 osstest)
 	echo none:;
 	;;
diff --git a/cr-daily-branch b/cr-daily-branch
index 0cafe39..663870d 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -187,6 +187,11 @@ if [ "x$OLD_REVISION" = xdetermine-late ]; then
 	OLD_REVISION="`./ap-fetch-version-baseline-late $branch $NEW_REVISION`"
 fi
 
+case $branch in
+distros) makeflight=./make-distros-flight ;;
+*)       makeflight=./make-flight ;;
+esac
+
 if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
         wantpush=false
 	for checkbranch in x $BRANCHES_ALWAYS; do
@@ -201,7 +206,7 @@ if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
 fi
 
 $DAILY_BRANCH_PREMAKE_HOOK
-flight=`./make-flight $branch $xenbranch $OSSTEST_BLESSING "$@"`
+flight=`$makeflight $branch $xenbranch $OSSTEST_BLESSING "$@"`
 $DAILY_BRANCH_POSTMAKE_HOOK
 
 heading=tmp/$flight.heading-info
@@ -221,6 +226,10 @@ fi
 revlog=tmp/$flight.revision-log
 
 case "$NEW_REVISION/$OLD_REVISION" in
+/lastflight)
+        sgr_args+=" --machine-readable-output=$mrof"
+	sgr_args+=" --that-flight=`check_tested`"
+	;;
 */*[^0-9a-f]* | *[^0-9a-f]*/*)
         echo >&2 "NO SGR COMPARISON badchar $NEW_REVISION/$OLD_REVISION"
         ;;
diff --git a/cri-common b/cri-common
index f472927..c02ad06 100644
--- a/cri-common
+++ b/cri-common
@@ -43,6 +43,7 @@ select_xenbranch () {
 	linux-*)		tree=linux;	xenbranch=xen-unstable ;;
 	linuxfirmware)	    tree=linuxfirmware;	xenbranch=xen-unstable ;;
 	libvirt)		tree=libvirt;	xenbranch=xen-unstable ;;
+	distros)		tree=distros;	xenbranch=xen-unstable ;;
 	osstest)		tree=osstest;	xenbranch=xen-unstable ;;
 	esac
 	if [ "x$tree" = xlinux ]; then
-- 
1.9.0

Re: [PATCH OSSTEST v2 14/15] distros: add branch infrastructure

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 14/15] distros: add branch infrastructure"):
> Since the distro nightlies are not version controlled we cannot use the usual
> mechanisms for detecting regressions. Instead compare each run against its
> predecessor, determined using sg-check-tested.
> 
> XXX needs testing in production environment.
...
> --- a/ap-fetch-version
> +++ b/ap-fetch-version
> @@ -77,6 +77,10 @@ gnulib-libvirt)
...	;;
> +distros)
> +	# None, this is just a regular test.
> +	echo latest
> +	;;

Is doing this really easier than doing it in cr-daily-branch ?

And I think perhaps ...

> diff --git a/cri-common b/cri-common
> index f472927..c02ad06 100644
> --- a/cri-common
> +++ b/cri-common
> @@ -43,6 +43,7 @@ select_xenbranch () {
>  	linux-*)		tree=linux;	xenbranch=xen-unstable ;;
>  	linuxfirmware)	    tree=linuxfirmware;	xenbranch=xen-unstable ;;
>  	libvirt)		tree=libvirt;	xenbranch=xen-unstable ;;
> +	distros)		tree=distros;	xenbranch=xen-unstable ;;

... this should be "tree=none".  What do you think ?

Ian.

Re: [PATCH OSSTEST v2 14/15] distros: add branch infrastructure

[Ian Campbell]

On Fri, 2014-05-02 at 15:04 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 14/15] distros: add branch infrastructure"):
> > Since the distro nightlies are not version controlled we cannot use the usual
> > mechanisms for detecting regressions. Instead compare each run against its
> > predecessor, determined using sg-check-tested.
> > 
> > XXX needs testing in production environment.
> ...
> > --- a/ap-fetch-version
> > +++ b/ap-fetch-version
> > @@ -77,6 +77,10 @@ gnulib-libvirt)
> ...	;;
> > +distros)
> > +	# None, this is just a regular test.
> > +	echo latest
> > +	;;
> 
> Is doing this really easier than doing it in cr-daily-branch ?

You mean avoiding calling ap-fetch-version for this branch? I didn't
try, I'd have thought it would involve a handful of special cases.
Although if you mean to handle none here (cf your second comment) then
that might work nicely.

> And I think perhaps ...
> 
> > diff --git a/cri-common b/cri-common
> > index f472927..c02ad06 100644
> > --- a/cri-common
> > +++ b/cri-common
> > @@ -43,6 +43,7 @@ select_xenbranch () {
> >  	linux-*)		tree=linux;	xenbranch=xen-unstable ;;
> >  	linuxfirmware)	    tree=linuxfirmware;	xenbranch=xen-unstable ;;
> >  	libvirt)		tree=libvirt;	xenbranch=xen-unstable ;;
> > +	distros)		tree=distros;	xenbranch=xen-unstable ;;
> 
> ... this should be "tree=none".  What do you think ?

I've no idea... The logic is that there is no pushed thing there is no
tree?

Ian.

Re: [PATCH OSSTEST v2 14/15] distros: add branch infrastructure

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST v2 14/15] distros: add branch infrastructure"):
> On Fri, 2014-05-02 at 15:04 +0100, Ian Jackson wrote:
> > Is doing this really easier than doing it in cr-daily-branch ?
> 
> You mean avoiding calling ap-fetch-version for this branch? I didn't
> try, I'd have thought it would involve a handful of special cases.
> Although if you mean to handle none here (cf your second comment) then
> that might work nicely.

ap-print-url is used only to produce $treeurl which is only used when
calling sg-check-tested and sg-report-flight - but those calls need to
be done differently anyway and no $treeurl is applicable.

ap-fetch-version-old is meaningless on "distros" and is only called
because cr-daily-branch doesn't treat it specially.

I suppose what I mean is: is ap-* the right place to know that
"distros" isn't the same kind of thing at all ?

> > And I think perhaps ...
> > > +	distros)		tree=distros;	xenbranch=xen-unstable ;;
> > ... this should be "tree=none".  What do you think ?
> 
> I've no idea... The logic is that there is no pushed thing there is no
> tree?

Well, "tree" in cr-daily-branch is:

 - The thing whose version we treat differently, taking the "to be
   tested" rather than "baseline" version.  There is no such thing
   here - at least, nothing that has a version plumbed through to
   the runvars.

 - Something we use to search the database for previous tests, looking
   for flights with specific values of built_revision_$tree (for for
   sg-check-tested and sg-report-flight).  But we want a totally
   different approach.

Ian.

Re: [PATCH OSSTEST v2 14/15] distros: add branch infrastructure

[Ian Campbell]

On Fri, 2014-05-02 at 15:30 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST v2 14/15] distros: add branch infrastructure"):
> > On Fri, 2014-05-02 at 15:04 +0100, Ian Jackson wrote:
> > > Is doing this really easier than doing it in cr-daily-branch ?
> > 
> > You mean avoiding calling ap-fetch-version for this branch? I didn't
> > try, I'd have thought it would involve a handful of special cases.
> > Although if you mean to handle none here (cf your second comment) then
> > that might work nicely.
> 
> ap-print-url is used only to produce $treeurl which is only used when
> calling sg-check-tested and sg-report-flight - but those calls need to
> be done differently anyway and no $treeurl is applicable.
> 
> ap-fetch-version-old is meaningless on "distros" and is only called
> because cr-daily-branch doesn't treat it specially.
> 
> I suppose what I mean is: is ap-* the right place to know that
> "distros" isn't the same kind of thing at all ?

I'm perfectly happy to let you decide on this. By which I mean I'll try
the "avoid ap-*" approach for next time.

> > > And I think perhaps ...
> > > > +	distros)		tree=distros;	xenbranch=xen-unstable ;;
> > > ... this should be "tree=none".  What do you think ?
> > 
> > I've no idea... The logic is that there is no pushed thing there is no
> > tree?
> 
> Well, "tree" in cr-daily-branch is:
> 
>  - The thing whose version we treat differently, taking the "to be
>    tested" rather than "baseline" version.  There is no such thing
>    here - at least, nothing that has a version plumbed through to
>    the runvars.
> 
>  - Something we use to search the database for previous tests, looking
>    for flights with specific values of built_revision_$tree (for for
>    sg-check-tested and sg-report-flight).  But we want a totally
>    different approach.

Makes sense, I think.

Ian.

[PATCH OSSTEST v2 15/15] distros: Run a flight over the weekend.

[Ian Campbell]

Once a week should be sufficient for this test. It involves quite a large
number of jobs so schedule it to start early on Saturday so it can run over the
weekend when, in theory, things should be quieter.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 crontab | 1 +
 1 file changed, 1 insertion(+)

diff --git a/crontab b/crontab
index 002876a..8e706d6 100644
--- a/crontab
+++ b/crontab
@@ -8,6 +8,7 @@ MAILTO=ian.jackson@citrix.com,ian.campbell@eu.citrix.com
 18		11	* * *		cd testing.git && BRANCHES=linux-mingo-tip-master ./cr-for-branches branches -w "./cr-daily-branch --real"
 18		12	* * *		cd testing.git && BRANCHES=linux-3.0		./cr-for-branches branches -w "./cr-daily-branch --real"
 18		13	* * *		cd testing.git && BRANCHES=libvirt		./cr-for-branches branches -w "./cr-daily-branch --real"
+46		7	* * 6		cd testing.git && BRANCHES=distros		./cr-for-branches branches -w "./cr-daily-branch --real"
 6-59/15   	*	* * *		cd testing.git && EXTRA_BRANCHES='linux-linus linux-3.0' ./cr-for-branches bisects -w "./cr-try-bisect --real"
 #8-59/5		*	* * *		cd bisects/adhoc.git &&	with-lock-ex -q data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
 3		5	* * *		savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
-- 
1.9.0

Re: [PATCH OSSTEST v2 15/15] distros: Run a flight over the weekend.

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 15/15] distros: Run a flight over the weekend."):
> Once a week should be sufficient for this test. It involves quite a large
> number of jobs so schedule it to start early on Saturday so it can run over the
> weekend when, in theory, things should be quieter.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

(subject of course to all the rest of the series going in)

Thanks,
Ian.

Re: [PATCH OSSTEST v2 00/15] add distro domU testing flight

[Fabio Fantoni]

Il 28/04/2014 12:41, Ian Campbell ha scritto:
> The following series adds a new flight type intended to test distro domU
> support, i.e. running distro installers and kernels on top of current
> Xen to ensure things work. I intend for the flight to be run on a weekly
> basis, and I've scheduled it here to run at the weekend when in theory
> things might be quieter.
>
> I've initially just done Debian PV domU support since Debian is what I
> know, but I hope others will add other distros as time goes on.
>
> The series introduces support for running the Debian Installer in a PV
> domU booting from netboot (PXE-ish) and netinst (ISO media) and adding
> jobs to test all of those for Squeeze onwards up to Sid and the
> daily/weekly builds.
>
> The flight is a bit unusual since distro installer builds are not really
> versioned in a way which is useful to osstest, so instead each flight
> will be compared against the previous run only (this stuff needs testing
> in production).
>
> The new Debian install functionality here also gives us the opportunity
> to test pygrub and pvgrub, so a couple of jobs are added to the regular
> tests using installer images versioned via TftpDiVersion in the usual
> way. This requires an early patch "mg-debian-installer-update: grab Xen
> PV domU capable images too" to go in and a patch to update TftpDiVersion
> to a refreshed version to be inserted before "Test pygrub and pvgrub on
> the regular flights" is committed.
>
> I've also pushed a branch, pull details at the end.
>
> v1 of this[0] wasn't really a proper posting, but calling this v2 to
> avoid any potential confusion.
>
> Ian.

Thanks for adding also these very useful tests, currently I use nearly 
always debian on pv domUs.
Briefly looking at your patches it seems that you use pygrub also for 
domUs installation, I use the installer kernel and initrd directly 
instead, for example:
kernel = "/boot/domu/wheezy/vmlinuz" (from .../images/netboot/xen/)
ramdisk = "/boot/domu/wheezy/initrd.gz" (from .../images/netboot/xen/)
extra = ...

Is there some "special" reason to install domUs starting with pygrub 
instead of directly?
Since last year with recents grub2 of Sid the domUs don't boot anymore 
with pygrub and I use pvgrub2 instead, I haven't tried installation from 
pygrub instead of directly but perharps it has the same problem.
Is pvgrub2 test now missing on osstest?

Thanks for any reply.

>
> [0] http://article.gmane.org/gmane.comp.emulators.xen.devel/180019
>
>
> The following changes since commit 0b5094e69de4c1a6cfb866d0cfe4d10ba655285c:
>
>    crontab: Add a daily run of the libvirt branch. (2014-04-25 10:07:56 +0100)
>
> are available in the git repository at:
>
>    git://xenbits.xen.org/people/ianc/osstest.git distro-flight-v2
>
> for you to fetch changes up to 95006fcc3c921467d85794d7f27f67232ad6597a:
>
>    distros: Run a flight over the weekend. (2014-04-26 09:47:36 +0100)
>
> ----------------------------------------------------------------
> Ian Campbell (15):
>        cr-external-linux: remove
>        Build against liblzma to support .xz compressed kernels
>        mg-debian-installer-update: grab Xen PV domU capable images too
>        TestSupport: Add helper to fetch a URL on a host
>        TestSupport: Add helper to wait for a guest to shutdown
>        TestSupport: allow overring of on_* in prepareguest_part_xencfg
>        TestSupport: allow caller of prepareguest_part_xencfg to specify viftype
>        Debian: add preseed_create_guest helper
>        distros: add support for installing Debian PV guests via d-i, flight and jobs
>        distros: support booting Debian PV (d-i installed) guests with pvgrub.
>        distros: attempt pvgrub support for PV Wheezy too.
>        distros: support PV guest install from Debian netinst media.
>        Test pygrub and pvgrub on the regular flights
>        distros: add branch infrastructure
>        distros: Run a flight over the weekend.
>
>   Osstest/Debian.pm          |  83 +++++++++++++++-
>   Osstest/TestSupport.pm     |  32 ++++++-
>   ap-fetch-version           |   4 +
>   ap-fetch-version-old       |   3 +
>   ap-print-url               |   3 +
>   cr-daily-branch            |  11 ++-
>   cr-external-linux          |  50 ----------
>   cri-common                 |   1 +
>   crontab                    |   1 +
>   make-distros-flight        | 140 +++++++++++++++++++++++++++
>   make-flight                |  39 ++++++++
>   mg-debian-installer-update |  11 ++-
>   sg-run-job                 |  11 +++
>   ts-debian-di-install       | 233 +++++++++++++++++++++++++++++++++++++++++++++
>   ts-xen-build-prep          |   2 +-
>   ts-xen-install             |   2 +-
>   16 files changed, 562 insertions(+), 64 deletions(-)
>   delete mode 100755 cr-external-linux
>   create mode 100755 make-distros-flight
>   create mode 100755 ts-debian-di-install
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [PATCH OSSTEST v2 00/15] add distro domU testing flight

[Ian Campbell]

On Mon, 2014-04-28 at 13:30 +0200, Fabio Fantoni wrote:
> Is there some "special" reason to install domUs starting with pygrub 
> instead of directly?

I'm only doing this for netinst tests which use pygrub to extract the
kernel from the ISO image in a convenient way.

For netboot tests it does indeed download the kernel and boot it
directly.

> Since last year with recents grub2 of Sid the domUs don't boot anymore 
> with pygrub and I use pvgrub2 instead, I haven't tried installation from 
> pygrub instead of directly but perharps it has the same problem.
> Is pvgrub2 test now missing on osstest?

Nobody has contributed one yet.

Ian.

Re: [PATCH OSSTEST v2 00/15] add distro domU testing flight

[Fabio Fantoni]

Il 28/04/2014 13:32, Ian Campbell ha scritto:
> On Mon, 2014-04-28 at 13:30 +0200, Fabio Fantoni wrote:
>> Is there some "special" reason to install domUs starting with pygrub
>> instead of directly?
> I'm only doing this for netinst tests which use pygrub to extract the
> kernel from the ISO image in a convenient way.
>
> For netboot tests it does indeed download the kernel and boot it
> directly.

Thanks for your reply.

>
>> Since last year with recents grub2 of Sid the domUs don't boot anymore
>> with pygrub and I use pvgrub2 instead, I haven't tried installation from
>> pygrub instead of directly but perharps it has the same problem.
>> Is pvgrub2 test now missing on osstest?
> Nobody has contributed one yet.
>
> Ian.
>

These are my notes about the preparation and tests done of pvgrub2 
(which I'm also using on the new systems in productionwith Wheezy as 
dom0), maybe it can be useful for those who want to use it:

> -------------------------
> PVGRUB2 upstream
> ---
> aptitude install autogen libdevmapper-dev libfuse-dev unifont # some 
> are optionals but useful (other requirements if I am not mistaken are 
> included in the preparation of xen and qemu from source)
> git clone git://git.sv.gnu.org/grub.git
> ./autogen.sh
> ./configure --target=x86_64 --with-platform=xen
> make
> mkdir -p boot/grub/
> cat > boot/grub/grub.cfg <<'EOF'
> insmod lvm
> insmod ext2
> insmod part_msdos
> insmod part_gpt
> insmod btrfs
> insmod xzio
>
> insmod regexp
> for dev in (*); do
>     # $device: parenthesis removed from $dev
>     regexp -s device '\((.*)\)' $dev
>     set root=$device
>     for file in /boot/vmlinuz-* /boot/linux-*; do
>         if test -f $file; then
>             set saved_root=$root
>         fi
>     done
> done
> set root=$saved_root
>
> if test -f /boot/grub2/grub.cfg ; then
>     configfile /boot/grub2/grub.cfg
> elif test -f /boot/grub/grub.cfg ; then
>     configfile /boot/grub/grub.cfg
> fi
> EOF
> pkgdatadir=. ./grub-mkstandalone --grub-mkimage=./grub-mkimage -o 
> pvgrub2.xen -O x86_64-xen -d grub-core/ boot/grub/grub.cfg
> mv pvgrub2.xen /boot
> ---
> Note: it works only with domUs that have grub2, old domUs with grub1 
> must use pygrub instead
> -------------------------

Re: [PATCH OSSTEST v2 00/15] add distro domU testing flight

[Ian Campbell]

On Mon, 2014-04-28 at 11:41 +0100, Ian Campbell wrote:
> The following series adds a new flight type intended to test distro domU

Do you think we could push
http://article.gmane.org/gmane.comp.emulators.xen.devel/197368 plus
these 3 from the start of this series:
        cr-external-linux: remove
        Build against liblzma to support .xz compressed kernels
        mg-debian-installer-update: grab Xen PV domU capable images too
? All are acked (except for the linked one which you just said thanks
too, but I'm taking that as an Ack ;-))

At the same time I'd like to change to update TftpDiVersion to something
regenerated with the above mg-d-i-u changes.

Ian.

Re: [PATCH OSSTEST v2 00/15] add distro domU testing flight

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH OSSTEST v2 00/15] add distro domU testing flight"):
> On Mon, 2014-04-28 at 11:41 +0100, Ian Campbell wrote:
> > The following series adds a new flight type intended to test distro domU
> 
> Do you think we could push
> http://article.gmane.org/gmane.comp.emulators.xen.devel/197368 plus
> these 3 from the start of this series:
>         cr-external-linux: remove
>         Build against liblzma to support .xz compressed kernels
>         mg-debian-installer-update: grab Xen PV domU capable images too
> ? All are acked (except for the linked one which you just said thanks
> too, but I'm taking that as an Ack ;-))

I'm assuming you have a branch of fixes awaiting push ?  It should go
in there, and probably be next in the queue.

> At the same time I'd like to change to update TftpDiVersion to something
> regenerated with the above mg-d-i-u changes.

Sure.

Ian.

Re: [PATCH OSSTEST v2 00/15] add distro domU testing flight

[Ian Campbell]

On Fri, 2014-05-16 at 18:08 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH OSSTEST v2 00/15] add distro domU testing flight"):
> > On Mon, 2014-04-28 at 11:41 +0100, Ian Campbell wrote:
> > > The following series adds a new flight type intended to test distro domU
> > 
> > Do you think we could push
> > http://article.gmane.org/gmane.comp.emulators.xen.devel/197368 plus
> > these 3 from the start of this series:
> >         cr-external-linux: remove
> >         Build against liblzma to support .xz compressed kernels
> >         mg-debian-installer-update: grab Xen PV domU capable images too
> > ? All are acked (except for the linked one which you just said thanks
> > too, but I'm taking that as an Ack ;-))
> 
> I'm assuming you have a branch of fixes awaiting push ?

Not a single branch, but I can construct one easily enough.

Ian.

>   It should go
> in there, and probably be next in the queue.
> 
> > At the same time I'd like to change to update TftpDiVersion to something
> > regenerated with the above mg-d-i-u changes.
> 
> Sure.
> 
> Ian.