[PATCH OSSTEST v5 01/24] TestSupport: Add helper to fetch a URL on a host

[PATCH OSSTEST v5 01/24] TestSupport: Add helper to fetch a URL on a host

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v5: Support http_proxy via $c{HttpProxy}
v3: Make sure wget is installed
---
 Osstest/Debian.pm      |  2 +-
 Osstest/TestSupport.pm | 10 +++++++++-
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 69530fb..5e6bcba 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -617,7 +617,7 @@ d-i apt-setup/another boolean false
 d-i apt-setup/non-free boolean false
 d-i apt-setup/contrib boolean false
 
-d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, chiark-utils-bin, $extra_packages
+d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, chiark-utils-bin, wget, $extra_packages
 
 $xopts{ExtraPreseed}
 
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index a674f35..fc2e406 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -55,7 +55,7 @@ BEGIN {
                       target_putfilecontents_stash
 		      target_putfilecontents_root_stash
                       target_put_guest_image target_editfile
-                      target_editfile_cancel
+                      target_editfile_cancel target_fetchurl
                       target_editfile_root target_file_exists
                       target_run_apt
                       target_install_packages target_install_packages_norec
@@ -1560,6 +1560,14 @@ END
     return $cfgpath;
 }
 
+sub target_fetchurl($$$;$) {
+    my ($ho, $url, $path, $timeo) = @_;
+    $timeo ||= 2000;
+    my $useproxy = "export http_proxy=$c{HttpProxy};" if $c{HttpProxy};
+    target_cmd_root($ho, "$useproxy wget --progress=dot:mega -O $path $url", $timeo);
+}
+
+
 sub target_put_guest_image ($$$) {
     my ($ho, $gho, $default) = @_;
     my $specimage = $r{"$gho->{Guest}_image"};
-- 
2.1.4

[PATCH OSSTEST v5 02/24] TestSupport: allow caller of prepareguest_part_xencfg to specify viftype

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
--
v3: Use CamelCase for xopts, use the correct condition
---
 Osstest/TestSupport.pm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index fc2e406..f374f9c 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1533,12 +1533,13 @@ sub prepareguest_part_xencfg ($$$$$) {
     my $onpoweroff= $xopts->{OnPowerOff} || 'destroy';
     my $oncrash= $xopts->{OnCrash} || 'preserve';
     my $vcpus= guest_var($gho, 'vcpus', $xopts->{DefVcpus} || 2);
+    my $viftype= $xopts->{VifType} ? "type=$xopts->{VifType}," : "";
     my $xoptcfg= $xopts->{ExtraConfig};
     $xoptcfg='' unless defined $xoptcfg;
     my $xencfg= <<END;
 name        = '$gho->{Name}'
 memory = ${ram_mb}
-vif         = [ 'type=ioemu,mac=$gho->{Ether}' ]
+vif         = [ '${viftype}mac=$gho->{Ether}' ]
 #
 on_poweroff = '$onpoweroff'
 on_reboot   = '$onreboot'
@@ -1634,6 +1635,7 @@ END
         $cfg .= "bios='$bios'\n";
     }
 
+    $xopts{VifType} ||= "ioemu";
     my $cfgpath= prepareguest_part_xencfg($ho, $gho, $ram_mb, \%xopts, $cfg);
     target_cmd_root($ho, <<END);
         (echo $passwd; echo $passwd) | vncpasswd $gho->{Guest}.vncpw
-- 
2.1.4

[PATCH OSSTEST v5 03/24] create_webfile: Support use with guests as well as hosts.

[Ian Campbell]

In particular make the path unique by ensuring it includes the host
and guest name in the guest case.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v4: Dropped redundant -> from $ho->{Host}->{Name}.
v3: New patch
---
 Osstest/TestSupport.pm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index f374f9c..c42c3e1 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1980,6 +1980,8 @@ sub await_webspace_fetch_byleaf ($$$$$) {
 sub create_webfile ($$$) {
     my ($ho, $tail, $contents) = @_; # $contents as for file_link_contents
     my $wf_rhs= $ho->{Name}."_".$tail;
+    # $ho->{Host} is set if $ho is a guest.
+    $wf_rhs= $ho->{Host}{Name}."_${wf_rhs}" if $ho->{Host};
     my $wf_common= $c{WebspaceCommon}.$wf_rhs;
     my $wf_url= $c{WebspaceUrl}.$wf_common;
     my $wf_file= $c{WebspaceFile}.$wf_common;
-- 
2.1.4

[PATCH OSSTEST v5 04/24] Debian: refactor code to add preseed commands to the preseed file

[Ian Campbell]

Call it from ts-debian-hvm-install.

This means that, in future, ts-debian-di-install can use
preseed_hook_command and preseed_hook_installscript.  No functional
change for now.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: longtaox.pang@intel.com
---
v5: Extended commit message
v3: New patch
---
 Osstest/Debian.pm     | 16 +++++++++++-----
 ts-debian-hvm-install |  3 +++
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 5e6bcba..ef602b1 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -37,7 +37,7 @@ BEGIN {
                       %preseed_cmds
                       preseed_base
                       preseed_create
-                      preseed_hook_command preseed_hook_installscript
+                      preseed_hook_command preseed_hook_installscript preseed_hook_cmds
                       di_installcmdline_core
                       );
     %EXPORT_TAGS = ( );
@@ -907,10 +907,7 @@ d-i partman-auto/expert_recipe string					\\
 
 END
 
-    foreach my $di_key (keys %preseed_cmds) {
-        $preseed_file .= "d-i preseed/$di_key string ".
-            (join ' && ', @{ $preseed_cmds{$di_key} }). "\n";
-    }
+    $preseed_file .= preseed_hook_cmds();
 
     if ($ho->{Flags}{'no-di-kernel'}) {
 	$preseed_file .= <<END;
@@ -954,4 +951,13 @@ chmod +x '$installer_pathname'
 END
 }
 
+sub preseed_hook_cmds () {
+    my $preseed;
+    foreach my $di_key (keys %preseed_cmds) {
+        $preseed .= "d-i preseed/$di_key string ".
+            (join ' && ', @{ $preseed_cmds{$di_key} }). "\n";
+    }
+    return $preseed;
+}
+
 1;
diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index cfd5144..95fce9a 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -77,6 +77,9 @@ d-i preseed/late_command string \\
         in-target mkdir -p /root/.ssh; \\
         in-target sh -c "echo -e '$authkeys'> /root/.ssh/authorized_keys";
 END
+
+    $preseed_file .= preseed_hook_cmds();
+
     return $preseed_file;
 }
 
-- 
2.1.4

[PATCH OSSTEST v5 05/24] Debian: refactor preseeding of .ssh directories

[Ian Campbell]

Causes known_hosts to be consistently created as well as ~osstest/.ssh
to be consistently populated (it previsouly wasn't for HVM guests).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: longtaox.pang@intel.com
---
v5: Call preseed_ssh from preseed_base
v3: New patch
---
 Osstest/Debian.pm     | 105 ++++++++++++++++++++++++++++----------------------
 ts-debian-hvm-install |   5 +--
 2 files changed, 61 insertions(+), 49 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index ef602b1..5daa19f 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -37,6 +37,7 @@ BEGIN {
                       %preseed_cmds
                       preseed_base
                       preseed_create
+                      preseed_ssh
                       preseed_hook_command preseed_hook_installscript preseed_hook_cmds
                       di_installcmdline_core
                       );
@@ -551,8 +552,64 @@ sub di_installcmdline_core ($$;@) {
     return @cl;
 }
 
-sub preseed_base ($$$;@) {
-    my ($ho,$suite,$extra_packages,%xopts) = @_;
+sub preseed_ssh ($$) {
+    my ($ho,$sfx) = @_;
+
+    my $authkeys_url= create_webfile($ho, "authkeys$sfx", authorized_keys());
+
+    my $hostkeyfile= "$c{OverlayLocal}/etc/ssh/ssh_host_rsa_key.pub";
+    my $hostkey= get_filecontents($hostkeyfile);
+    chomp($hostkey); $hostkey.="\n";
+    my $knownhosts= '';
+
+    my $hostsq= $dbh_tests->prepare(<<END);
+        SELECT val FROM runvars
+         WHERE flight=? AND name LIKE '%host'
+         GROUP BY val
+END
+    $hostsq->execute($flight);
+    while (my ($node) = $hostsq->fetchrow_array()) {
+        my $longname= "$node.$c{TestHostDomain}";
+        my (@hostent)= gethostbyname($longname);
+        if (!@hostent) {
+            logm("skipping host key for nonexistent host $longname");
+            next;
+        }
+        my $specs= join ',', $longname, $node, map {
+            join '.', unpack 'W4', $_;
+        } @hostent[4..$#hostent];
+        logm("adding host key for $specs");
+        $knownhosts.= "$specs ".$hostkey;
+    }
+    $hostsq->finish();
+
+    $knownhosts.= "localhost,127.0.0.1 ".$hostkey;
+    my $knownhosts_url= create_webfile($ho, "known_hosts$sfx", $knownhosts);
+
+    preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+r=/target/root
+cd \$r
+
+umask 022
+mkdir .ssh
+wget -O .ssh/authorized_keys '$authkeys_url'
+wget -O .ssh/known_hosts     '$knownhosts_url'
+
+u=osstest
+h=/home/\$u
+mkdir /target\$h/.ssh
+cp .ssh/authorized_keys /target\$h/.ssh
+chroot /target chown -R \$u.\$u \$h/.ssh
+END
+}
+
+sub preseed_base ($$$$;@) {
+    my ($ho,$suite,$sfx,$extra_packages,%xopts) = @_;
+
+    preseed_ssh($ho, $sfx);
 
     my $preseed = <<"END";
 d-i mirror/suite string $suite
@@ -639,43 +696,12 @@ END
 sub preseed_create ($$;@) {
     my ($ho, $sfx, %xopts) = @_;
 
-    my $authkeys_url= create_webfile($ho, "authkeys$sfx", authorized_keys());
-
-    my $hostkeyfile= "$c{OverlayLocal}/etc/ssh/ssh_host_rsa_key.pub";
-    my $hostkey= get_filecontents($hostkeyfile);
-    chomp($hostkey); $hostkey.="\n";
-    my $knownhosts= '';
-
     my $disk= $xopts{DiskDevice} || '/dev/sda';
     my $suite= $xopts{Suite} || $c{DebianSuite};
 
     my $d_i= $ho->{Tftp}{Path}.'/'.$ho->{Tftp}{DiBase}.'/'.$r{arch}.'/'.
 	$c{TftpDiVersion}.'-'.$ho->{Suite};
 
-    my $hostsq= $dbh_tests->prepare(<<END);
-        SELECT val FROM runvars
-         WHERE flight=? AND name LIKE '%host'
-         GROUP BY val
-END
-    $hostsq->execute($flight);
-    while (my ($node) = $hostsq->fetchrow_array()) {
-        my $longname= "$node.$c{TestHostDomain}";
-        my (@hostent)= gethostbyname($longname);
-        if (!@hostent) {
-            logm("skipping host key for nonexistent host $longname");
-            next;
-        }
-        my $specs= join ',', $longname, $node, map {
-            join '.', unpack 'W4', $_;
-        } @hostent[4..$#hostent];
-        logm("adding host key for $specs");
-        $knownhosts.= "$specs ".$hostkey;
-    }
-    $hostsq->finish();
-
-    $knownhosts.= "localhost,127.0.0.1 ".$hostkey;
-    my $knownhosts_url= create_webfile($ho, "known_hosts$sfx", $knownhosts);
-
     my $overlays= '';
     my $create_overlay= sub {
         my ($srcdir, $tfilename) = @_;
@@ -747,17 +773,6 @@ set -ex
 r=/target/root
 cd \$r
 
-umask 022
-mkdir .ssh
-wget -O .ssh/authorized_keys '$authkeys_url'
-wget -O .ssh/known_hosts     '$knownhosts_url'
-
-u=osstest
-h=/home/\$u
-mkdir /target\$h/.ssh
-cp .ssh/authorized_keys /target\$h/.ssh
-chroot /target chown -R \$u.\$u \$h/.ssh
-
 echo FANCYTTY=0 >> /target/etc/lsb-base-logging.sh
 
 $overlays
@@ -872,7 +887,7 @@ END
 
     my $extra_packages = join(",",@extra_packages);
 
-    my $preseed_file= preseed_base($ho,$suite,$extra_packages,%xopts);
+    my $preseed_file= preseed_base($ho,$suite,$sfx,$extra_packages,%xopts);
 
     $preseed_file .= (<<END);
 d-i partman-auto/method string lvm
diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index 95fce9a..0085d82 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -43,8 +43,7 @@ our $gho;
 
 sub preseed () {
 
-    my $preseed_file = preseed_base($gho,'wheezy','',());
-    my $authkeys = join('\\n', split(/\n/, authorized_keys()));
+    my $preseed_file = preseed_base($gho,'wheezy','','',());
 
     $preseed_file .= (<<END);
 d-i netcfg/get_hostname string $gn
@@ -74,8 +73,6 @@ d-i apt-setup/cdrom/set-first boolean false
 d-i preseed/late_command string \\
         in-target mkdir -p /boot/efi/EFI/boot; \\
         in-target cp /boot/efi/EFI/debian/grubx64.efi /boot/efi/EFI/boot/bootx64.efi ;\\
-        in-target mkdir -p /root/.ssh; \\
-        in-target sh -c "echo -e '$authkeys'> /root/.ssh/authorized_keys";
 END
 
     $preseed_file .= preseed_hook_cmds();
-- 
2.1.4

[PATCH OSSTEST v5 06/24] Debian: Refactor installation of overlays, so it can be used for guests too

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: longtaox.pang@intel.com
---
v5: OverLay local now handled preseed_base since all callers called it
v3: New patch

fixup: Move OverlayLocal to preseed_base
---
 Osstest/Debian.pm | 58 ++++++++++++++++++++++++++++---------------------------
 1 file changed, 30 insertions(+), 28 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 5daa19f..a0aedfa 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -38,7 +38,9 @@ BEGIN {
                       preseed_base
                       preseed_create
                       preseed_ssh
-                      preseed_hook_command preseed_hook_installscript preseed_hook_cmds
+                      preseed_hook_command preseed_hook_installscript
+                      preseed_hook_overlay
+                      preseed_hook_cmds
                       di_installcmdline_core
                       );
     %EXPORT_TAGS = ( );
@@ -610,6 +612,7 @@ sub preseed_base ($$$$;@) {
     my ($ho,$suite,$sfx,$extra_packages,%xopts) = @_;
 
     preseed_ssh($ho, $sfx);
+    preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
 
     my $preseed = <<"END";
 d-i mirror/suite string $suite
@@ -702,26 +705,6 @@ sub preseed_create ($$;@) {
     my $d_i= $ho->{Tftp}{Path}.'/'.$ho->{Tftp}{DiBase}.'/'.$r{arch}.'/'.
 	$c{TftpDiVersion}.'-'.$ho->{Suite};
 
-    my $overlays= '';
-    my $create_overlay= sub {
-        my ($srcdir, $tfilename) = @_;
-        my $url= create_webfile($ho, "$tfilename$sfx", sub {
-            my ($fh) = @_;
-            contents_make_cpio($fh, 'ustar', $srcdir);
-        });
-        $overlays .= <<END;
-wget -O overlay.tar '$url'
-cd /target
-tar xf \$r/overlay.tar
-cd \$r
-rm overlay.tar
-
-END
-    };
-
-    $create_overlay->('overlay',        'overlay.tar');
-    $create_overlay->($c{OverlayLocal}, 'overlay-local.tar');
-
     preseed_hook_installscript($ho, $sfx,
           '/lib/partman/init.d', '000override-parted-devices', <<END);
 #!/bin/sh
@@ -766,18 +749,13 @@ ls -l /dev/sd*
 true
 END
 
+    preseed_hook_overlay($ho, $sfx, 'overlay', 'overlay.tar');
+
     preseed_hook_command($ho, 'late_command', $sfx, <<END);
 #!/bin/sh
 set -ex
 
-r=/target/root
-cd \$r
-
 echo FANCYTTY=0 >> /target/etc/lsb-base-logging.sh
-
-$overlays
-
-echo latecmd done.
 END
 
     my $dtbs = "$d_i/dtbs.tar.gz";
@@ -966,6 +944,30 @@ chmod +x '$installer_pathname'
 END
 }
 
+sub preseed_hook_overlay ($$$$) {
+    my ($ho, $sfx, $srcdir, $tfilename) = @_;
+    my $url= create_webfile($ho, "$tfilename$sfx", sub {
+        my ($fh) = @_;
+        contents_make_cpio($fh, 'ustar', $srcdir);
+    });
+    preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+r=/target/root
+cd \$r
+
+umask 022
+
+wget -O overlay.tar '$url'
+cd /target
+tar xf \$r/overlay.tar
+cd \$r
+rm overlay.tar
+
+END
+}
+
 sub preseed_hook_cmds () {
     my $preseed;
     foreach my $di_key (keys %preseed_cmds) {
-- 
2.1.4

[PATCH OSSTEST v5 07/24] Debian: add preseed_create_guest helper

[Ian Campbell]

Creates a preseed file suitable for use in a PV guest

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v5: preseed_ssh now called from preseed_base, so not here
    OverlayLocal is now called from preseed base, so not here
    No need to set a default for $extra_packages in preseed_base,
    everyone passes something.
v4: Rebase, pass $ho to preseed_base
v3: Handle $xopts{ExtraPreseed} undefined in preseed_base

fixup: No need to default

fixup: Move OverlayLocal to preseed_base
---
 Osstest/Debian.pm | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index a0aedfa..c2a4ec5 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -37,6 +37,7 @@ BEGIN {
                       %preseed_cmds
                       preseed_base
                       preseed_create
+                      preseed_create_guest
                       preseed_ssh
                       preseed_hook_command preseed_hook_installscript
                       preseed_hook_overlay
@@ -611,6 +612,8 @@ END
 sub preseed_base ($$$$;@) {
     my ($ho,$suite,$sfx,$extra_packages,%xopts) = @_;
 
+    $xopts{ExtraPreseed} ||= '';
+
     preseed_ssh($ho, $sfx);
     preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
 
@@ -696,6 +699,28 @@ END
     return $preseed;
 }
 
+sub preseed_create_guest ($$;@) {
+    my ($ho, $sfx, %xopts) = @_;
+
+    my $suite= $xopts{Suite} || $c{DebianSuite};
+
+    my $extra_packages;
+
+    my $preseed_file= preseed_base($ho, $suite, $sfx, $extra_packages, %xopts);
+    $preseed_file.= (<<END);
+d-i     partman-auto/method             string regular
+d-i     partman-auto/choose_recipe \\
+                select All files in one partition (recommended for new users)
+
+d-i     grub-installer/bootdev          string /dev/xvda
+
+END
+
+    $preseed_file .= preseed_hook_cmds();
+
+    return create_webfile($ho, "preseed$sfx", $preseed_file);
+}
+
 sub preseed_create ($$;@) {
     my ($ho, $sfx, %xopts) = @_;
 
-- 
2.1.4

[PATCH OSSTEST v5 08/24] make-flight: Handle $BUILD_LVEXTEND_MAX in mfi-common:create_build_jobs()

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v3: New patch
---
 make-flight | 4 ----
 mfi-common  | 4 ++++
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/make-flight b/make-flight
index cc8ecdb..ec0a20c 100755
--- a/make-flight
+++ b/make-flight
@@ -36,10 +36,6 @@ defguestsuite=`getconfig GuestDebianSuite`
 
 if [ x$buildflight = x ]; then
 
-  if [ "x$BUILD_LVEXTEND_MAX" != x ]; then
-     BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
-  fi
-
   create_build_jobs
 
 else
diff --git a/mfi-common b/mfi-common
index 16fc8c5..a9e966f 100644
--- a/mfi-common
+++ b/mfi-common
@@ -63,6 +63,10 @@ create_build_jobs () {
   local enable_ovmf
   local build_hostflags
 
+  if [ "x$BUILD_LVEXTEND_MAX" != x ]; then
+     BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
+  fi
+
   for arch in ${BUILD_ARCHES- i386 amd64 armhf }; do
 
     if [ "x$arch" = xdisable ]; then continue; fi
-- 
2.1.4

[PATCH OSSTEST v5 09/24] distros: add support for installing Debian PV guests via d-i, flight and jobs

[Ian Campbell]

This patch introduces ts-debian-di-install which can install Debian
from a netboot (PXE) debian installer image. By default it installs
from the d-i image used by osstest (using the special Xen PV guest
enabled flavour where necessary) but it can also install the current
release versions from Squeeze onwards (up to and including Sid) and
the d-i daily builds. This is controlled by runvars {Guest}_diver =
osstest|current and {Guest}_dist = squeeze|...|sid.  The resulting
guests boot the distro kernel using pygrub (pvgrub will follow).

The distros flights differ substantially from the existing flights.
Introduce make-distros-flight using the functionality previously
refactored into mfi-common. The new flight tests all versions of
Debian from Squeeze onward as an amd64, i386 and armhf guests (armhf
from Jessie onwards only) using the usual smoke tests.

Test names are suffixed -pygrub pending the addition of pvgrub
variants in a future commit.

Add the new cases to sg-run-job

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: use guest create
v3: $BUILD_LVEXTEND_MAX now handled in mfi-common
    Consolidate setting of ruvars
    Include $flight and $job in tmpdir name
    Use Osstest::Debian::di_installcmdline_core
    Document the usage of get_host_property on a guest object
    Correct ARM netboot paths
    Include bootloader in test name
       Should include -pv too?
    console= repetition for Jessie onwards.
    Wait for up to an hour for the install. I'd seen timeouts right at
    the end of the install with the previous value

fixup: distros: add support for installing Debian PV guests via d-i, flight and jobs
---
 Osstest/TestSupport.pm |   3 +
 make-distros-flight    |  98 +++++++++++++++++++++++++++++
 sg-run-job             |  11 ++++
 ts-debian-di-install   | 168 +++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 280 insertions(+)
 create mode 100755 make-distros-flight
 create mode 100755 ts-debian-di-install

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index c42c3e1..bdb1662 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -906,8 +906,11 @@ sub propname_massage ($) {
     return $prop;
 }
 
+# It is fine to call this on a guest object too, in which case it will
+# always return $defval.
 sub get_host_property ($$;$) {
     my ($ho, $prop, $defval) = @_;
+    return $defval unless $ho->{Properties};
     my $val = $ho->{Properties}{propname_massage($prop)};
     return defined($val) ? $val : $defval;
 }
diff --git a/make-distros-flight b/make-distros-flight
new file mode 100755
index 0000000..5067342
--- /dev/null
+++ b/make-distros-flight
@@ -0,0 +1,98 @@
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+set -e
+
+branch=$1
+xenbranch=$2
+blessing=$3
+buildflight=$4
+
+flight=`./cs-flight-create $blessing $branch`
+
+. cri-common
+. ap-common
+. mfi-common
+
+defsuite=`getconfig DebianSuite`
+defguestsuite=`getconfig GuestDebianSuite`
+
+if [ x$buildflight = x ]; then
+
+  WANT_XEND=false REVISION_LINUX_OLD=disable
+
+  create_build_jobs
+
+else
+
+  bfi=$buildflight.
+
+fi
+
+job_create_test_filter_callback () {
+  if [ "$xenarch" = "i386" ]; then return 1; fi
+  return 0
+}
+
+test_matrix_branch_filter_callback () {
+    :
+}
+
+test_do_one_netboot () {
+  job_create_test                                               \
+   test-$xenarch$kern-$dom0arch-$domU-$dist-netboot-pygrub      \
+    test-debian-di xl $xenarch $dom0arch                        \
+      kernbuildjob=${bfi}build-$dom0arch-$kernbuild             \
+      debian_arch=$domU                                         \
+      debian_dist=$dist                                         \
+      debian_method=netboot                                     \
+      debian_diver=current                                      \
+      all_hostflags=$most_hostflags
+}
+
+test_matrix_do_one () {
+  case ${xenarch} in
+  amd64) domUarches="amd64 i386";;
+  armhf) domUarches="armhf";;
+  esac
+
+  for domU in $domUarches ; do
+    for dist in squeeze wheezy jessie sid daily ; do
+      case ${domU}_${dist} in
+      armhf_squeeze) continue;; # No armhf in Squeeze
+      armhf_wheezy) continue;; # No armhf guest support in Wheezy
+      *) ;;
+      esac
+
+      test_do_one_netboot
+
+    done
+
+  done
+}
+
+test_matrix_iterate
+
+echo $flight
+
+# Local variables:
+# mode: sh
+# sh-basic-offset: 2
+# indent-tabs-mode: nil
+# End:
diff --git a/sg-run-job b/sg-run-job
index eae159d..e4a02cf 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -258,6 +258,17 @@ proc run-job/test-debian {} {
     test-guest debian
 }
 
+proc install-guest-debian-di {} {
+    run-ts . = ts-debian-di-install
+    run-ts . = ts-guest-start + debian
+}
+
+proc need-hosts/test-debian-di {} { return host }
+proc run-job/test-debian-di {} {
+    install-guest-debian-di
+    test-guest debian
+}
+
 proc need-hosts/test-freebsd {} { return host }
 proc run-job/test-freebsd {} {
     run-ts . = ts-freebsd-install
diff --git a/ts-debian-di-install b/ts-debian-di-install
new file mode 100755
index 0000000..a829185
--- /dev/null
+++ b/ts-debian-di-install
@@ -0,0 +1,168 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost,$gn) = @ARGV;
+$whhost ||= 'host';
+$gn ||= 'debian';
+
+our $ho= selecthost($whhost);
+
+our $ram_mb=    512;
+our $disk_mb= 10000;
+
+our $guesthost= "$gn.guest.osstest";
+our $gho;
+
+sub prep () {
+    target_install_packages_norec($ho, qw(lvm2));
+
+    $gho= prepareguest($ho, $gn, $guesthost, 22,
+                       $disk_mb, 40);
+
+    prepareguest_part_lvmdisk($ho, $gho, $disk_mb);
+
+    target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+}
+
+sub setup_netboot($$$)
+{
+    my ($didir, $arch, $suite) = @_;
+
+    my $di_ver= $r{"$gho->{Guest}_diver"} || "osstest";
+
+    my ($kernel,$initrd);
+
+    if ( $di_ver eq "osstest" ) {
+	my $di_path = $c{TftpPath}.'/'.$ho->{Tftp}{DiBase}.'/'.$r{arch}.'/'.$c{TftpDiVersion}.'-'.$ho->{Suite};
+
+	$kernel = "$di_path/vmlinuz-xen";
+	$initrd = "$di_path/initrd.gz-xen";
+
+	target_putfile_root($ho, 60, $kernel, "$didir/kernel_${suite}_${arch}");
+	target_putfile_root($ho, 60, $initrd, "$didir/initrd_${suite}_${arch}");
+
+    } else {
+	my $mirror = "http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath}";
+
+	my $di_url = $suite eq "daily" ?
+	    "http://d-i.debian.org/daily-images/$arch/daily/netboot" :
+	    "$mirror/dists/$suite/main/installer-$arch/$di_ver/images/netboot";
+
+	$di_url .= "/xen" if $arch =~ m/amd64|i386/;
+	$di_url .= "/debian-installer/arm64" if $arch =~ /arm64/;
+
+	$kernel = "$di_url/vmlinuz";
+	$initrd = "$di_url/initrd.gz";
+
+	target_fetchurl($ho, $kernel, "$didir/kernel_${suite}_${arch}");
+	target_fetchurl($ho, $initrd, "$didir/initrd_${suite}_${arch}");
+    }
+
+    store_runvar("$gho->{Guest}_netboot_kernel", $kernel);
+    store_runvar("$gho->{Guest}_netboot_initrd", $initrd);
+
+    return <<END;
+kernel      = "$didir/kernel_${suite}_${arch}"
+ramdisk     = "$didir/initrd_${suite}_${arch}"
+END
+}
+sub ginstall () {
+    my $arch= $r{"$gho->{Guest}_arch"};
+    my $method= $r{"$gho->{Guest}_method"};
+
+    my $tmpdir= "/root/$flight-$job-di";
+    target_cmd_root($ho, <<END);
+rm -rf $tmpdir
+mkdir $tmpdir
+END
+
+    my ($method_cfg, $ps_url, $extra_disk);
+
+    if ( $method eq "netboot" )
+    {
+	my $suite= $r{"$gho->{Guest}_dist"};
+	logm("$method $suite/$arch");
+
+	$method_cfg = setup_netboot($tmpdir, $arch, $suite);
+
+	$suite = "sid" if $suite eq "daily";
+
+	$ps_url = preseed_create_guest($gho, '', Suite=>$suite);
+
+	$extra_disk = "";
+    }
+    else
+    {
+	die "$method";
+    }
+
+    my @cmdline = ();
+    push @cmdline, "debian-installer/exit/always_halt=true";
+    push @cmdline, "domain=$c{TestHostDomain}";
+    push @cmdline, "console=hvc0";
+    push @cmdline, di_installcmdline_core($gho, $ps_url);
+    push @cmdline, "--";
+    # See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007 for
+    # why this is repeated.
+    push @cmdline, "console=hvc0";
+
+    my $cmdline = join(" ", @cmdline);
+
+    my %install_xopts = (
+	OnPowerOff => "preserve"
+    );
+
+    prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
+$method_cfg
+extra       = "$cmdline"
+#
+disk        = [
+            $extra_disk 'phy:$gho->{Lvdev},xvda,w'
+            ]
+END
+
+    guest_create($gho);
+
+    guest_checkrunning($ho, $gho) or die "$gho->{Name} not running";
+
+    guest_await_shutdown($ho,$gho,3600);
+    guest_destroy($gho);
+
+    my $blcfg = <<END;
+bootloader = "pygrub"
+END
+
+    prepareguest_part_xencfg($ho, $gho, $ram_mb, {}, <<END);
+$blcfg
+#
+disk        = [
+            'phy:$gho->{Lvdev},xvda,w'
+            ]
+END
+    return;
+}
+
+prep();
+ginstall();
-- 
2.1.4

[PATCH OSSTEST v5 00/24] add distro domU testing flight

[Ian Campbell]

Hi,

It's been a long time since v3 of this series, which was
http://article.gmane.org/gmane.comp.emulators.xen.devel/224415. That
contains the background which I won't repeat here. v4 was
http://article.gmane.org/gmane.comp.emulators.xen.devel/238558

Since v4 I've addressed the review feedback and tweaked the slack space
for non-LVM volumes to based on running some test runs (1M wasn't enough
for raw)

As last time there are some patches in here which I think will be useful
to the Intel folks doing the nested virt testing, specifically the
refactoring of how overlays and ssh host keys are done will be useful
for installing a guest to be treated as the L1 host. LongTao, I've CCd
you only on the 3 patches which I think will be of interest instead of
the full 24 patch series. Note that some of this differs this time
around based on the review feedback.

I've run some more adhoc jobs on the Cambridge instance, the results of
the last of which are at:
http://www.chiark.greenend.org.uk/~xensrcts/logs/37076/

I've also run an adhoc xen-unstable test of the new jobs there, results
at hhttp://www.chiark.greenend.org.uk/~xensrcts/logs/37077/ the
interesting jobs are *-pvgrub, *-pygrub for the bootloader tests and
*-{qcow2,raw,vhd} for the image format stuff, LVM is tested by the
existing -{xl,libvirt} jobs.

When I originally started this work I envisioned a flight running on the
main production instance (in Cambridge at the time). Now that we have
the new colo I would still consider that the main production instance is
the best home, however given the new colo is not yet up to full capacity
we could also consider running this flight in Cambridge for the time
being (once the control VM is no longer so sick). What do you think?

Also when running the adhoc tests the sheer number of jobs which are
involved (with my hope there will be more in the future as other distros
get in on the act) I was considering splitting it into multiple
distros-debian-{squeeze,wheezy,jessie,sid,daily} flights. Thoughts?

Summary of (A)cks and (M)odified:

M        TestSupport: Add helper to fetch a URL on a host
 A       TestSupport: allow caller of prepareguest_part_xencfg to specify viftype
 A       create_webfile: Support use with guests as well as hosts.
M        Debian: refactor code to add preseed commands to the preseed file
M        Debian: refactor preseeding of .ssh directories
MA       Debian: Refactor installation of overlays, so it can be used for guests too
M        Debian: add preseed_create_guest helper
 A       make-flight: Handle $BUILD_LVEXTEND_MAX in mfi-common:create_build_jobs()
         distros: add support for installing Debian PV guests via d-i, flight and jobs
         distros: support booting Debian PV (d-i installed) guests with pvgrub.
         distros: Support pvgrub for Wheezy too.
         distros: support PV guest install from Debian netinst media.
         Test pygrub and pvgrub on the regular flights
         distros: add branch infrastructure
 A       distros: Run a flight over the weekend.
         Debian: Handle lack of bootloader support in d-i on ARM.
         standalone: propagate result of command from with_logging
         ts-debian-di-install: Refactor root_disk specification
         make-flight: refactor PV debian tests
M        Add testing of non-LVM/phy disk backends.
         mfi-common: Allow make-*flight to filter the set of build jobs to include
         make-distros-flight: don't bother building for XSM.
         distros: email only me on play flights
         ts-debian-di-install: Use ftp.debian.org directly

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[PATCH OSSTEST v5 10/24] distros: support booting Debian PV (d-i installed) guests with pvgrub.

[Ian Campbell]

This requires the use of the pv-grub-menu package which is in Jessie
onwards.  (it is in wheezy-backports which is the subject of a
subsequent patch).

The bootloader to use is specified via a runvar {Guest}_bootloader.

Adjust make-distros-flight to use pvgrub for some subset of i386 and
amd64 guests to get coverage.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: Define and use arch_debian2xen and arch_xen2debian
    Avoid pv-grub-x86_64.gz on i386 dom0, we don't built it there.
    Fiddle with py vs pv grub stripy a bit.
---
 Osstest.pm           |  7 +++++++
 Osstest/Debian.pm    |  2 +-
 make-distros-flight  | 21 ++++++++++++++++++++-
 ts-debian-di-install | 11 +++++++++--
 4 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/Osstest.pm b/Osstest.pm
index 7989129..c0e3ee7 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -38,6 +38,7 @@ BEGIN {
                       db_begin_work
                       ensuredir get_filecontents_core_quiet system_checked
                       nonempty visible_undef show_abs_time
+                      %arch_debian2xen %arch_xen2debian
                       );
     %EXPORT_TAGS = ( );
 
@@ -49,6 +50,12 @@ our $mjobdb;
 
 our $dbh_tests;
 
+our %arch_debian2xen = qw(i386 x86_32
+			  amd64 x86_64
+			  armhf armhf);
+our %arch_xen2debian;
+$arch_xen2debian{$arch_debian2xen{$_}} = $_ foreach keys %arch_debian2xen;
+
 #---------- static default config settings ----------
 
 our %c = qw(
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index c2a4ec5..53e77e2 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -704,7 +704,7 @@ sub preseed_create_guest ($$;@) {
 
     my $suite= $xopts{Suite} || $c{DebianSuite};
 
-    my $extra_packages;
+    my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
 
     my $preseed_file= preseed_base($ho, $suite, $sfx, $extra_packages, %xopts);
     $preseed_file.= (<<END);
diff --git a/make-distros-flight b/make-distros-flight
index 5067342..80df21b 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -55,14 +55,33 @@ test_matrix_branch_filter_callback () {
 }
 
 test_do_one_netboot () {
+  stripy bootloader pvgrub pygrub \
+    "$xenarch" "amd64" \
+    "$dom0arch" "i386" \
+    "$domU" "amd64" \
+    "$dist" "sid"
+
+  case ${dom0arch}_${domU}_${dist} in
+    arm*_arm*_*) bootloader="pygrub";; # no pvgrub for arm
+
+    # Needs a menu.lst, not present in Squeeze+ due to switch to grub2,
+    # workedaround in Jessie+ with pv-grub-menu package.
+    *_squeeze) bootloader="pygrub";;
+    *_wheezy) bootloader="pygrub";;
+
+    # pv-grub-x86_64.gz is not built by 32-bit dom0 userspace build.
+    i386_amd64_*) bootloader="pygrub";;
+  esac
+
   job_create_test                                               \
-   test-$xenarch$kern-$dom0arch-$domU-$dist-netboot-pygrub      \
+   test-$xenarch$kern-$dom0arch-$domU-$dist-netboot-$bootloader \
     test-debian-di xl $xenarch $dom0arch                        \
       kernbuildjob=${bfi}build-$dom0arch-$kernbuild             \
       debian_arch=$domU                                         \
       debian_dist=$dist                                         \
       debian_method=netboot                                     \
       debian_diver=current                                      \
+      debian_bootloader=$bootloader                             \
       all_hostflags=$most_hostflags
 }
 
diff --git a/ts-debian-di-install b/ts-debian-di-install
index a829185..73cb43d 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -93,6 +93,8 @@ sub ginstall () {
     my $method= $r{"$gho->{Guest}_method"};
 
     my $tmpdir= "/root/$flight-$job-di";
+    my $bl= $r{"$gho->{Guest}_bootloader"};
+
     target_cmd_root($ho, <<END);
 rm -rf $tmpdir
 mkdir $tmpdir
@@ -109,7 +111,7 @@ END
 
 	$suite = "sid" if $suite eq "daily";
 
-	$ps_url = preseed_create_guest($gho, '', Suite=>$suite);
+	$ps_url = preseed_create_guest($gho, '', Suite=>$suite, PvMenuLst=>($bl eq "pvgrub"));
 
 	$extra_disk = "";
     }
@@ -150,7 +152,12 @@ END
     guest_await_shutdown($ho,$gho,3600);
     guest_destroy($gho);
 
-    my $blcfg = <<END;
+    my $xenarch = $arch_debian2xen{$arch};
+    my $pvgrub = "/usr/local/lib/xen/boot/pv-grub-$xenarch.gz";
+    my $blcfg = $bl eq "pvgrub" ? <<END : <<END;
+kernel = "$pvgrub"
+extra = "(hd0,0)/boot/grub/menu.lst"
+END
 bootloader = "pygrub"
 END
 
-- 
2.1.4

[PATCH OSSTEST v5 11/24] distros: Support pvgrub for Wheezy too.

[Ian Campbell]

This requires us to install pv-grub-menu from backports, which we do
using a late_command.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3:
  - Remove spurious () from <<(END) (and the prexisting "" too)
  - Remove $xopts{EnableBackports} and automatically handle the need to add
    backports in preseed_base.
  - Install via late_command not apt-setup, since the former has
    issues, hence subject drops "attempt to..."
---
 Osstest/Debian.pm   | 44 +++++++++++++++++++++++++++++++++++++++++++-
 make-distros-flight |  4 ++--
 2 files changed, 45 insertions(+), 3 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 53e77e2..4dcea26 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -691,6 +691,17 @@ END
 d-i clock-setup/ntp-server string $ntpserver
 END
 
+    # deb http://ftp.debian.org/debian/ wheezy-backports main
+    $preseed .= <<END if $extra_packages =~ m#\b/$suite-backports\b#;
+d-i apt-setup/local0/repository string \\
+    http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
+d-i apt-setup/local0/comment string $suite backports
+END
+# Jessie onwards...
+#    $preseed .= <<END if $extra_packages =~ m#\b/$suite-backports\b#;
+#d-i apt-setup/services-select multiselect security, updates, backports
+#END
+
     $preseed .= <<"END";
 
 ### END OF DEBIAN PRESEED BASE
@@ -704,7 +715,38 @@ sub preseed_create_guest ($$;@) {
 
     my $suite= $xopts{Suite} || $c{DebianSuite};
 
-    my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
+    my $extra_packages = "";
+    if ($xopts{PvMenuLst}) {
+        if ($suite =~ m/wheezy/) {
+            # pv-grub-menu/wheezy-backports + using apt-setup to add
+            # backports results in iproute, ifupdown and
+            # isc-dhcp-client getting removed because tasksel's
+            # invocation of apt-get install somehow decides the
+            # iproute2 from wheezy-backports is a thing it wants to
+            # install. So instead lets fake it with a late command...
+            #
+            # This also has the bonus of working round an issue with
+            # 1.2.1~bpo70+1 which created an invalid menu.lst using
+            # "root(/dev/xvda,0)" which pvgrub cannot parse because
+            # the Grub device.map isn't present at pkgsel/include time
+            # but it is by late_command time. This was fixed by
+            # version 1.3 which is in Jessie onwards.
+            preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+(
+    echo
+    echo \\\# $suite backports
+    echo deb http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
+) >> /target/etc/apt/sources.list
+in-target apt-get update
+in-target apt-get install -y -t wheezy-backports pv-grub-menu
+END
+        } else {
+            $extra_packages = "pv-grub-menu";
+        }
+    }
 
     my $preseed_file= preseed_base($ho, $suite, $sfx, $extra_packages, %xopts);
     $preseed_file.= (<<END);
diff --git a/make-distros-flight b/make-distros-flight
index 80df21b..97df89a 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -65,9 +65,9 @@ test_do_one_netboot () {
     arm*_arm*_*) bootloader="pygrub";; # no pvgrub for arm
 
     # Needs a menu.lst, not present in Squeeze+ due to switch to grub2,
-    # workedaround in Jessie+ with pv-grub-menu package.
+    # workedaround in Wheezy+ with pv-grub-menu package (backports in Wheezy,
+    # in Jessie+ main).
     *_squeeze) bootloader="pygrub";;
-    *_wheezy) bootloader="pygrub";;
 
     # pv-grub-x86_64.gz is not built by 32-bit dom0 userspace build.
     i386_amd64_*) bootloader="pygrub";;
-- 
2.1.4

[PATCH OSSTEST v5 12/24] distros: support PV guest install from Debian netinst media.

[Ian Campbell]

The netinst media are iso images containing a base Debian install and
some (image size dependent) additional tasks.

On x86 the "multiarch" iso flavour contains a Xen capable kernel for
both i386 and amd64 so use that.

This adds support for two classes of ISO, the CD sized ones which are
built nightly and the DVD sized ones which are built weekly.

The images are downloaded using jigdo which sources the majority of
the data from a local Debian mirror, for this reason I have not
worried about the fact that the i386 and amd64 tests are downloading
the same thing (adding a specific download job would require finding
up to 4GB of scratch space for each flight).

The ISOs booted using pygrub which can extract the kernel and initrd
from a ISO image. The resulting guests are also booted with pygrub
since the pv-grub-menu package is not available on the ISO images and
we have pvgrub coverage from the netboot tests.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: Support http_proxy in jigdo download
v3:
  - Indent jigdo-lite script line to improve readability
  - Wrap bootloader_args
  - Fetch URL on target, to get a timeout. Use wget since that is arranged to
    be present.
  - include bootloader in test name.
---
 Osstest/Debian.pm    |  7 ++++--
 make-distros-flight  | 23 ++++++++++++++++++
 ts-debian-di-install | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 95 insertions(+), 2 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 4dcea26..8ecf1dd 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -618,8 +618,6 @@ sub preseed_base ($$$$;@) {
     preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
 
     my $preseed = <<"END";
-d-i mirror/suite string $suite
-
 d-i debian-installer/locale string en_GB
 d-i console-keymaps-at/keymap select gb
 d-i keyboard-configuration/xkb-keymap string en_GB
@@ -691,6 +689,11 @@ END
 d-i clock-setup/ntp-server string $ntpserver
 END
 
+    # For CDROM the suite is part of the image
+    $preseed .= <<END unless $xopts{CDROM};
+d-i mirror/suite string $suite
+END
+
     # deb http://ftp.debian.org/debian/ wheezy-backports main
     $preseed .= <<END if $extra_packages =~ m#\b/$suite-backports\b#;
 d-i apt-setup/local0/repository string \\
diff --git a/make-distros-flight b/make-distros-flight
index 97df89a..e3ec8f7 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -85,6 +85,20 @@ test_do_one_netboot () {
       all_hostflags=$most_hostflags
 }
 
+test_do_one_netinst () {
+  # Always pygrub since no pv-grub-menu on CD
+  job_create_test                                               \
+   test-$xenarch$kern-$dom0arch-$domU-$cd-netinst-pygrub        \
+    test-debian-di xl $xenarch $dom0arch                        \
+      kernbuildjob=${bfi}build-$dom0arch-$kernbuild             \
+      debian_arch=$domU                                         \
+      debian_cd=$cd                                             \
+      debian_method=netinst                                     \
+      debian_bootloader=pygrub                                  \
+      all_hostflags=$most_hostflags
+
+}
+
 test_matrix_do_one () {
   case ${xenarch} in
   amd64) domUarches="amd64 i386";;
@@ -103,6 +117,15 @@ test_matrix_do_one () {
 
     done
 
+    for cd in current weekly ; do
+      case ${domU}_${dist} in
+      armhf_*) continue;; # No iso targets for armhf
+      *) ;;
+      esac
+
+      test_do_one_netinst
+
+    done
   done
 }
 
diff --git a/ts-debian-di-install b/ts-debian-di-install
index 73cb43d..62e9bb5 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -46,6 +46,63 @@ sub prep () {
     target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
 }
 
+sub setup_netinst($$$)
+{
+    my ($didir, $arch, $cd) = @_;
+    my %arch_props = (
+	amd64 => { PathArch => "multi-arch", FileArch => "amd64-i386", IsoPath => "/install.amd/xen" },
+	i386  => { PathArch => "multi-arch", FileArch => "amd64-i386", IsoPath => "/install.386/xen" },
+	armhf => { PathArch => "armhf",      FileArch => "armhf",      IsoPath => "/install.armhf" }
+    );
+    my $props = $arch_props{$arch} or die "Unknown arch $arch";
+
+    target_install_packages($ho, qw(jigdo-file));
+
+    my $baseurl = $cd eq "current" ?
+      "http://cdimage.debian.org/debian-cd/current/$props->{PathArch}/jigdo-cd" :
+      "http://cdimage.debian.org/cdimage/weekly-builds/$props->{PathArch}/jigdo-cd";
+
+    my $filebase;
+
+    # Use the MD5SUMs file as an index
+    logm("Fetch index from $baseurl/MD5SUMS");
+    my $idx = target_cmd_output_root($ho, "wget --quiet -O - $baseurl/MD5SUMS");
+    foreach (split /\n/, $idx) {
+	m/^[0-9a-f]{32}  (debian-.*-$props->{FileArch}-netinst)\.iso$/ or next;
+	$filebase = $1;
+	last;
+    }
+
+    die unless $filebase;
+
+    logm("Downloading $baseurl/$filebase.jigdo");
+    my $netinst_jigdo = "$baseurl/$filebase.jigdo";
+    # Jigdo uses wget internally, and so obeys $http_proxy. This seems
+    # simpler than the advice at
+    # https://www.debian.org/CD/jigdo-cd/#faq which is to edit
+    # ~/.jigdo-lite and change the wgetOpts setting.
+    my $useproxy = $c{DebianMirrorProxy} // $c{HttpProxy};
+    my $cmd = '';
+    $cmd .= <<END if $useproxy;
+        export http_proxy=$useproxy
+END
+    $cmd .= <<END;
+        cd $didir && jigdo-lite --noask $netinst_jigdo
+END
+    # Jigdo seems to use /etc/apt/sources.list or something, so this
+    # just works using the already configured mirror without
+    # additional configuration, which is good because there doesn't
+    # seem to be any support for such things, at least in Squeeze.
+    target_cmd_root($ho, $cmd, 3600);
+    store_runvar("$gho->{Guest}_netinst_jigdo", $netinst_jigdo);
+
+    return (<<END, "\"file:$didir/$filebase.iso,xvdd:cdrom,r\",");
+bootloader = "pygrub"
+bootloader_args = ["--kernel=$props->{IsoPath}/vmlinuz",
+                   "--ramdisk=$props->{IsoPath}/initrd.gz"]
+END
+}
+
 sub setup_netboot($$$)
 {
     my ($didir, $arch, $suite) = @_;
@@ -115,6 +172,16 @@ END
 
 	$extra_disk = "";
     }
+    elsif ($method eq "netinst" )
+    {
+	my $cd = $r{"$gho->{Guest}_cd"};
+
+	logm("$method $cd/$arch");
+
+	($method_cfg,$extra_disk) = setup_netinst($tmpdir, $arch, $cd);
+
+	$ps_url = preseed_create_guest($gho, '', CDROM=>1);
+    }
     else
     {
 	die "$method";
-- 
2.1.4

[PATCH OSSTEST v5 13/24] Test pygrub and pvgrub on the regular flights

[Ian Campbell]

Since we now have the ability to test these drop one of each of
pygrub, pvgrub-32 and pvgrub-64 into the standard flights. Omitting
the {Guest}_diver runvar causes ts-debian-di-install to use the d-i
images in the location configured via TftpDiVersion, so they are
Version Controlled along with the d-i version used for the host.

This adds three new jobs:
test-amd64-amd64-amd64-pvgrub:
    all_hostflags             arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
    arch                      amd64
    buildjob                  build-amd64
    debian_arch               amd64
    debian_bootloader         pvgrub
    debian_dist               wheezy
    debian_method             netboot
    kernbuildjob              build-amd64-pvops
    kernkind                  pvops
    toolstack                 xl
    xenbuildjob               build-amd64
test-amd64-amd64-i386-pvgrub:
    all_hostflags             arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
    arch                      amd64
    buildjob                  build-amd64
    debian_arch               i386
    debian_bootloader         pvgrub
    debian_dist               wheezy
    debian_method             netboot
    kernbuildjob              build-amd64-pvops
    kernkind                  pvops
    toolstack                 xl
    xenbuildjob               build-amd64
test-amd64-amd64-pygrub:
    all_hostflags             arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
    arch                      amd64
    buildjob                  build-amd64
    debian_arch               amd64
    debian_bootloader         pygrub
    debian_dist               wheezy
    debian_method             netboot
    kernbuildjob              build-amd64-pvops
    kernkind                  pvops
    toolstack                 xl
    xenbuildjob               build-amd64

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: added runvar details
---
 make-flight | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/make-flight b/make-flight
index ec0a20c..ca48b81 100755
--- a/make-flight
+++ b/make-flight
@@ -303,6 +303,42 @@ do_passthrough_tests () {
   done
 }
 
+do_pygrub_tests () {
+  if [ $xenarch != amd64 -o $dom0arch != amd64 -o "$kern" != "" ]; then
+    return
+  fi
+
+  job_create_test test-$xenarch$kern-$dom0arch-pygrub   \
+    test-debian-di xl $xenarch $dom0arch                \
+      debian_arch=amd64                                 \
+      debian_dist=$guestsuite                           \
+      debian_method=netboot                             \
+      debian_bootloader=pygrub                          \
+      all_hostflags=$most_hostflags
+}
+
+do_pvgrub_tests () {
+  if [ $xenarch != amd64 -o $dom0arch != amd64 -o "$kern" != "" ]; then
+    return
+  fi
+
+  job_create_test test-$xenarch$kern-$dom0arch-amd64-pvgrub     \
+    test-debian-di xl $xenarch $dom0arch                        \
+      debian_arch=amd64                                         \
+      debian_dist=$guestsuite                                   \
+      debian_method=netboot                                     \
+      debian_bootloader=pvgrub                                  \
+      all_hostflags=$most_hostflags                             \
+
+  job_create_test test-$xenarch$kern-$dom0arch-i386-pvgrub      \
+    test-debian-di xl $xenarch $dom0arch                        \
+      debian_arch=i386                                          \
+      debian_dist=$guestsuite                                   \
+      debian_method=netboot                                     \
+      debian_bootloader=pvgrub                                  \
+      all_hostflags=$most_hostflags
+}
+
 do_pv_debian_tests () {
   xsms=$(xenbranch_xsm_variants)
 
@@ -427,6 +463,9 @@ test_matrix_do_one () {
 
   fi
   #do_passthrough_tests
+
+  do_pygrub_tests
+  do_pvgrub_tests
 }
 
 test_matrix_iterate
-- 
2.1.4

[PATCH OSSTEST v5 14/24] distros: add branch infrastructure

[Ian Campbell]

Since the distro nightlies are not version controlled we cannot use
the usual mechanisms for detecting regressions. Special case things
appropriately. We use an OLD_REVISION of "flight-NNN" to signify that
the old revision is another flight and not a tree revision.

A grep over $NEW_REVISION needed adjusting since NEW_REVISION is empty
in this mode, leading to "grep <filename>" which hangs waiting for
stdin.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3:
  Handle within cr-daily-branch, since ap-fetch-version* don't make sense for
  a branch such as this.
---
 cr-daily-branch | 36 ++++++++++++++++++++++++++++--------
 cri-common      |  1 +
 2 files changed, 29 insertions(+), 8 deletions(-)

diff --git a/cr-daily-branch b/cr-daily-branch
index c7d1071..55afadf 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -68,23 +68,34 @@ fetch_version () {
 	printf '%s\n' "$fetch_version_result"
 }
 
-treeurl=`./ap-print-url $branch`
+case $branch in
+    distros)
+	treeurl=none;;
+    *)
+	treeurl=`./ap-print-url $branch`;;
+esac
 
 force_baseline=false
 skipidentical=true
 wantpush=$OSSTEST_PUSH
 
-if [ "x$OLD_REVISION" = x ]; then
-        OLD_REVISION="`./ap-fetch-version-old $branch`"
-        export OLD_REVISION
-fi
-
 check_tested () {
 	./sg-check-tested --debug --branch=$branch \
 	  --blessings=${DAILY_BRANCH_TESTED_BLESSING:-$OSSTEST_BLESSING} \
 	  "$@"
 }
 
+if [ "x$OLD_REVISION" = x ]; then
+    case $branch in
+	distros)
+	    OSSTEST_NO_BASELINE=y
+	    OLD_REVISION=flight-`check_tested`
+	    ;;
+	*) OLD_REVISION="`./ap-fetch-version-old $branch`";;
+    esac
+    export OLD_REVISION
+fi
+
 if [ "x$OSSTEST_NO_BASELINE" != xy ] ; then
 	testedflight=`check_tested --revision-$tree="$OLD_REVISION"`
 
@@ -227,6 +238,11 @@ if [ "x$OLD_REVISION" = xdetermine-late ]; then
 	OLD_REVISION="`./ap-fetch-version-baseline-late $branch $NEW_REVISION`"
 fi
 
+case $branch in
+distros) makeflight=./make-distros-flight ;;
+*)       makeflight=./make-flight ;;
+esac
+
 if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
         wantpush=false
 	for checkbranch in x $BRANCHES_ALWAYS; do
@@ -241,7 +257,7 @@ if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
 fi
 
 $DAILY_BRANCH_PREMAKE_HOOK
-flight=`./make-flight $branch $xenbranch $OSSTEST_BLESSING "$@"`
+flight=`$makeflight $branch $xenbranch $OSSTEST_BLESSING "$@"`
 $DAILY_BRANCH_POSTMAKE_HOOK
 
 heading=tmp/$flight.heading-info
@@ -261,6 +277,10 @@ fi
 revlog=tmp/$flight.revision-log
 
 case "$NEW_REVISION/$OLD_REVISION" in
+/flight-[0-9]*)
+	echo >&2 "SGR COMPARISON AGAINST ${OLD_REVISION}"
+	sgr_args+=" --that-flight=${OLD_REVISION#flight-}"
+	;;
 */*[^0-9a-f]* | *[^0-9a-f]*/*)
         echo >&2 "NO SGR COMPARISON badchar $NEW_REVISION/$OLD_REVISION"
         ;;
@@ -321,7 +341,7 @@ start_email $flight $branch "$sgr_args" "$subject_prefix"
 push=false
 if grep '^tolerable$' $mrof >/dev/null 2>&1; then push=$wantpush; fi
 if test -f $branch.force; then push=$OSSTEST_PUSH; fi
-if grep -xF $NEW_REVISION $branch.force-rev; then push=$OSSTEST_PUSH; fi
+if test -n "$NEW_REVISION" && grep -xF $NEW_REVISION $branch.force-rev; then push=$OSSTEST_PUSH; fi
 if test -f $branch.block; then push=false; fi
 
 if test -e $mrof && test -e $tree_bisect && ! grep '^broken' $mrof; then
diff --git a/cri-common b/cri-common
index ad44546..4bf7548 100644
--- a/cri-common
+++ b/cri-common
@@ -72,6 +72,7 @@ select_xenbranch () {
 	rumpuserxen)	      tree=rumpuserxen; xenbranch=xen-unstable ;;
 	seabios)		tree=seabios;	xenbranch=xen-unstable ;;
 	ovmf)			tree=ovmf;	xenbranch=xen-unstable ;;
+	distros)		tree=none;	xenbranch=xen-unstable ;;
 	osstest)		tree=osstest;	xenbranch=xen-unstable ;;
 	esac
 	if [ "x$tree" = xlinux ]; then
-- 
2.1.4

[PATCH OSSTEST v5 15/24] distros: Run a flight over the weekend.

[Ian Campbell]

Once a week should be sufficient for this test. It involves quite a
large number of jobs so schedule it to start early on Saturday so it
can run over the weekend when, in theory, things should be quieter.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 crontab | 1 +
 1 file changed, 1 insertion(+)

diff --git a/crontab b/crontab
index 5dc8675..1917ec4 100644
--- a/crontab
+++ b/crontab
@@ -6,5 +6,6 @@ MAILTO=ian.jackson@citrix.com,ian.campbell@eu.citrix.com
 18		9	* * 1,3,5	cd testing.git && BRANCHES=linux-next		./cr-for-branches branches -w "./cr-daily-branch --real"
 18		4	* * *		cd testing.git && BRANCHES='linux-linus linux-mingo-tip-master linux-3.0 libvirt rumpuserxen' ./cr-for-branches branches -w "./cr-daily-branch --real"
 6-59/15   	*	* * *		cd testing.git && EXTRA_BRANCHES='linux-linus linux-3.0 rumpuserxen libvirt' ./cr-for-branches bisects -w "./cr-try-bisect --real"
+46		7	* * 6		cd testing.git && BRANCHES=distros		./cr-for-branches branches -w "./cr-daily-branch --real"
 #8-59/5		*	* * *		cd bisects/adhoc.git &&	with-lock-ex -q data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
 3		4	* * *		savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
-- 
2.1.4

[PATCH OSSTEST v5 16/24] Debian: Handle lack of bootloader support in d-i on ARM.

[Ian Campbell]

Debian doesn't currently know what bootloader to install in a Xen
guest on ARM. We install pv-grub-menu above which actually does what
we need, but the installer doesn't treat that as a "bootloader".

Most ARM platforms end up installing a u-boot boot.scr, based on a
platform whitelist. This doesn't seem appropriate for us. Grub is not
available for arm32. For arm64 we will eventually end up with in-guest
UEFI and therefore grub-efi and things will work normally. I'm not
sure what the answer is going to be for arm32.

This patch enables the workaround for Wheezy, Jessie and Sid,
post-Jessie should be enabled as we add them. (Pre-wheezy does not
support running as a Xen guest on ARM so we don't test them at all).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: Handle sid too
v3: New
---
 Osstest/Debian.pm    | 14 ++++++++++++--
 ts-debian-di-install |  6 ++++--
 2 files changed, 16 insertions(+), 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 8ecf1dd..5b884e0 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -713,8 +713,8 @@ END
     return $preseed;
 }
 
-sub preseed_create_guest ($$;@) {
-    my ($ho, $sfx, %xopts) = @_;
+sub preseed_create_guest ($$$;@) {
+    my ($ho, $arch, $sfx, %xopts) = @_;
 
     my $suite= $xopts{Suite} || $c{DebianSuite};
 
@@ -761,6 +761,16 @@ d-i     grub-installer/bootdev          string /dev/xvda
 
 END
 
+    # Debian doesn't currently know what bootloader to install in a
+    # Xen guest on ARM. We install pv-grub-menu above which actually
+    # does what we need, but the installer doesn't treat that as a
+    # "bootloader".
+    logm("\$arch is $arch, \$suite is $suite");
+    $preseed_file.= (<<END) if $arch =~ /^arm/ && $suite =~ /wheezy|jessie|sid/;
+d-i     nobootloader/confirmation_common boolean true
+
+END
+
     $preseed_file .= preseed_hook_cmds();
 
     return create_webfile($ho, "preseed$sfx", $preseed_file);
diff --git a/ts-debian-di-install b/ts-debian-di-install
index 62e9bb5..56b6ad8 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -168,7 +168,9 @@ END
 
 	$suite = "sid" if $suite eq "daily";
 
-	$ps_url = preseed_create_guest($gho, '', Suite=>$suite, PvMenuLst=>($bl eq "pvgrub"));
+	$ps_url = preseed_create_guest($gho, $arch, '',
+				       Suite=>$suite,
+				       PvMenuLst=>($bl eq "pvgrub"));
 
 	$extra_disk = "";
     }
@@ -180,7 +182,7 @@ END
 
 	($method_cfg,$extra_disk) = setup_netinst($tmpdir, $arch, $cd);
 
-	$ps_url = preseed_create_guest($gho, '', CDROM=>1);
+	$ps_url = preseed_create_guest($gho, $arch, '', CDROM=>1);
     }
     else
     {
-- 
2.1.4

[PATCH OSSTEST v5 17/24] standalone: propagate result of command from with_logging

[Ian Campbell]

Such that the overall script gets the return code of the test/job
which was run.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 standalone | 1 +
 1 file changed, 1 insertion(+)

diff --git a/standalone b/standalone
index d8113e4..4564cf4 100755
--- a/standalone
+++ b/standalone
@@ -159,6 +159,7 @@ with_logging() {
     if [ $rc -ne 0 ] ; then
 	echo "FAILED rc=${rc}" >&2
     fi
+    return $rc
 }
 
 # other potential ops:
-- 
2.1.4

[PATCH OSSTEST v5 18/24] ts-debian-di-install: Refactor root_disk specification

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: new patch
---
 ts-debian-di-install | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/ts-debian-di-install b/ts-debian-di-install
index 56b6ad8..281e7cb 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -205,12 +205,14 @@ END
 	OnPowerOff => "preserve"
     );
 
+    my $root_disk = "'phy:$gho->{Lvdev},xvda,w'";
+
     prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
 $method_cfg
 extra       = "$cmdline"
 #
 disk        = [
-            $extra_disk 'phy:$gho->{Lvdev},xvda,w'
+            $extra_disk $root_disk
             ]
 END
 
@@ -234,7 +236,7 @@ END
 $blcfg
 #
 disk        = [
-            'phy:$gho->{Lvdev},xvda,w'
+            $root_disk
             ]
 END
     return;
-- 
2.1.4

[PATCH OSSTEST v5 19/24] make-flight: refactor PV debian tests

[Ian Campbell]

No functional change, standalong-generate-dump-flight-runvars confirms
no change to the runvars.

Includes a hook which is not used yet, $recipe_sfx.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: new patch
---
 make-flight | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/make-flight b/make-flight
index ca48b81..909cebe 100755
--- a/make-flight
+++ b/make-flight
@@ -339,6 +339,17 @@ do_pvgrub_tests () {
       all_hostflags=$most_hostflags
 }
 
+do_pv_debian_test_one () {
+  testname=$1; shift
+  recipe_sfx=$1; shift
+  toolstack=$1; shift
+
+  job_create_test test-$xenarch$kern-$dom0arch-$testname        \
+     test-debian$recipe_sfx $toolstack                          \
+            $xenarch $dom0arch                                  \
+            $debian_runvars all_hostflags=$most_hostflags $@
+}
+
 do_pv_debian_tests () {
   xsms=$(xenbranch_xsm_variants)
 
@@ -354,20 +365,13 @@ do_pv_debian_tests () {
       suffix=${platform:+-$platform}
       hostflags=${most_hostflags}${platform:+,platform-$platform}
 
-      job_create_test test-$xenarch$kern-$dom0arch-xl$suffix   \
-                      test-debian xl                           \
-                      $xenarch $dom0arch                       \
-                      enable_xsm=$xsm                          \
-                      $debian_runvars all_hostflags=$hostflags
+      do_pv_debian_test_one xl$suffix '' xl enable_xsm=$xsm
+
     done
   done
 
   for xsm in $xsms ; do
-    job_create_test test-$xenarch$kern-$dom0arch-libvirt         \
-                    test-debian libvirt                          \
-                    $xenarch $dom0arch                           \
-                    enable_xsm=$xsm                              \
-                    $debian_runvars all_hostflags=$most_hostflags
+    do_pv_debian_test_one libvirt '' libvirt enable_xsm=$xsm
   done
 }
 
-- 
2.1.4

[PATCH OSSTEST v5 20/24] Add testing of non-LVM/phy disk backends.

[Ian Campbell]

xen-create-image makes this tricky to do since it is rather LVM
centric. Now that we have the ability to install from d-i it's
possible to arrange fairly easily that they use non-LVM disk backend
arrangements.

Here we add support to the test script and infra and create a bunch of
new jobs testing the cross product of {xl,libvirt} x {raw,qcow2,vhd}.

The test scripts are modified such that when constructing a domain
with a non-LVM diskfmt runvar:
 - the LVM device is slightly enlarged to account for file format
   headers (1M should be plenty).
 - the LVM device will have an ext3 filesystem created on it instead
   of being used as a phy device for the guest. Reusing the LVM volume
   in this way means we don't need to do more storage management in
   dom0 (i.e. arranging for / to be large enough, or managing a
   special "images" LV)
 - the relevant type of container is created within the filesystem
   using the appropriate tool.
 - New properties Disk{fmt,spec} are added to all $gho, containing
   the format used for the root disk and the xl diskspec to load it.
     - lvm backed guests use a xend/xm compatible spec, everything
       else uses the improved xl syntax which libvirt also supports.
       We won't test non-LVM on xend.
 - New properties Disk{mnt,img} are added to $gho which are not using
   LVM. These contain the mount point to use (configurable via
   OSSTEST_CONFIG and runvars) and the full path (including mount
   point) to the image itself.
 - When starting or stopping a guest we arrange for the filesystem to
   be (u)mounted.
     - The prepearation when starting a guest copes gracefully with
       the disk already being prepared.
     - Hooks are called from guest_create() and guest_destroy() to
       manipulate the disk as needed.

Using standalong-generate-dump-flight-runvars a representative set of
runvars is:

xen-unstable               test-amd64-amd64-xl-qcow2                 all_hostflags               arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
xen-unstable               test-amd64-amd64-xl-qcow2                 arch                        amd64
xen-unstable               test-amd64-amd64-xl-qcow2                 buildjob                    build-amd64
xen-unstable               test-amd64-amd64-xl-qcow2                 debian_arch                 amd64
xen-unstable               test-amd64-amd64-xl-qcow2                 debian_bootloader           pygrub
xen-unstable               test-amd64-amd64-xl-qcow2                 debian_diskfmt              qcow2
xen-unstable               test-amd64-amd64-xl-qcow2                 debian_dist                 wheezy
xen-unstable               test-amd64-amd64-xl-qcow2                 debian_method               netboot
xen-unstable               test-amd64-amd64-xl-qcow2                 kernbuildjob                build-amd64-pvops
xen-unstable               test-amd64-amd64-xl-qcow2                 kernkind                    pvops
xen-unstable               test-amd64-amd64-xl-qcow2                 toolstack                   xl
xen-unstable               test-amd64-amd64-xl-qcow2                 xenbuildjob                 build-amd64

Compared to test-amd64-amd64-pygrub (which is the most similar job) and
normalising the test name the difference is:
 xen-unstable               test-amd64-amd64-SUFFIX                   all_hostflags               arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
 xen-unstable               test-amd64-amd64-SUFFIX                   arch                        amd64
 xen-unstable               test-amd64-amd64-SUFFIX                   buildjob                    build-amd64
 xen-unstable               test-amd64-amd64-SUFFIX                   debian_arch                 amd64
 xen-unstable               test-amd64-amd64-SUFFIX                   debian_bootloader           pygrub
+xen-unstable               test-amd64-amd64-SUFFIX                   debian_diskfmt              qcow2
 xen-unstable               test-amd64-amd64-SUFFIX                   debian_dist                 wheezy
 xen-unstable               test-amd64-amd64-SUFFIX                   debian_method               netboot
 xen-unstable               test-amd64-amd64-SUFFIX                   kernbuildjob                build-amd64-pvops
 xen-unstable               test-amd64-amd64-SUFFIX                   kernkind                    pvops
 xen-unstable               test-amd64-amd64-SUFFIX                   toolstack                   xl
 xen-unstable               test-amd64-amd64-SUFFIX                   xenbuildjob                 build-amd64

These are added to the following flights:
  +libvirt
  +linux-3.0
  +linux-3.10
  +linux-3.14
  +linux-3.4
  +linux-arm-xen
  +linux-linus
  +linux-mingo-tip-master
  +linux-next
  +osstest
  +qemu-mainline
  +qemu-upstream-4.3-testing
  +qemu-upstream-4.4-testing
  +qemu-upstream-unstable
  +seabios
  +xen-4.0-testing
  +xen-4.1-testing
  +xen-4.2-testing
  +xen-4.3-testing
  +xen-4.4-testing
  +xen-unstable

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v5: Assume 100MB/s dd from /dev/zero when creating a raw disk image
    Allow 10M of slack on filesystem for raw, qcow and vhd. 1M wasn't
    enough in practice for raw.
v4: new patch

fixup!10 megabytes slack on disk
---
 Osstest/TestSupport.pm | 97 ++++++++++++++++++++++++++++++++++++++++++++++++--
 make-flight            | 16 +++++++++
 ts-debian-di-install   | 10 ++----
 ts-guest-start         |  1 -
 4 files changed, 114 insertions(+), 10 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index bdb1662..2b975eb 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -89,7 +89,8 @@ BEGIN {
                       target_var target_var_prefix
                       selectguest prepareguest more_prepareguest_hvm
                       guest_var guest_var_commalist
-                      prepareguest_part_lvmdisk prepareguest_part_xencfg
+                      prepareguest_part_lvmdisk prepareguest_part_diskimg
+                      prepareguest_part_xencfg
                       guest_umount_lv guest_await guest_await_dhcp_tcp
                       guest_checkrunning guest_check_ip guest_find_ether
                       guest_find_domid guest_check_up guest_check_up_quick
@@ -97,6 +98,7 @@ BEGIN {
                       guest_await_shutdown guest_await_destroy guest_destroy
                       guest_vncsnapshot_begin guest_vncsnapshot_stash
 		      guest_check_remus_ok guest_editconfig
+                      guest_prepare_disk guest_unprepare_disk
                       host_involves_pcipassthrough host_get_pcipassthrough_devs
                       toolstack guest_create
 
@@ -1319,6 +1321,7 @@ sub selectguest ($$) {
     }
     logm("guest: using $gn on $gho->{Host}{Name}");
     guest_find_lv($gho);
+    guest_find_diskimg($gho);
     guest_find_ether($gho);
     guest_find_tcpcheckport($gho);
     dhcp_watch_setup($ho,$gho);
@@ -1334,6 +1337,22 @@ sub guest_find_lv ($) {
         ? '/dev/'.$gho->{Vg}.'/'.$gho->{Lv} : undef;
 }
 
+sub guest_find_diskimg($)
+{
+    my ($gho) = @_;
+    $gho->{Diskfmt} = $r{"$gho->{Guest}_diskfmt"} // "lvm";
+    $gho->{Diskspec} = "phy:$gho->{Lvdev},xvda,w";
+
+    return if $gho->{Diskfmt} eq "lvm";
+
+    my $mntroot = get_host_property($gho->{Host}, "DiskImageMount",
+			    $c{DiskImageMount} // "/var/lib/xen/images");
+
+    $gho->{Diskmnt} = "$mntroot/$gho->{Guest}";
+    $gho->{Diskimg} = "$gho->{Diskmnt}/disk.$gho->{Diskfmt}";
+    $gho->{Diskspec} = "format=$gho->{Diskfmt},vdev=xvda,target=$gho->{Diskimg}";
+}
+
 sub guest_find_ether ($) {
     my ($gho) = @_;
     $gho->{Ether}= $r{"$gho->{Guest}_ether"};
@@ -1373,6 +1392,7 @@ sub guest_destroy ($) {
     my ($gho) = @_;
     my $ho = $gho->{Host};
     toolstack($ho)->destroy($gho);
+    guest_unprepare_disk($gho);
 }
 
 sub guest_await_destroy ($$) {
@@ -1384,9 +1404,32 @@ sub guest_await_destroy ($$) {
 sub guest_create ($) {
     my ($gho) = @_;
     my $ho = $gho->{Host};
+    guest_prepare_disk($gho);
     toolstack($ho)->create($gho);
 }
 
+sub guest_prepare_disk ($) {
+    my ($gho) = @_;
+
+    guest_umount_lv($gho->{Host}, $gho);
+
+    return if $gho->{Diskfmt} eq "lvm";
+
+    target_cmd_root($gho->{Host}, <<END);
+mkdir -p $gho->{Diskmnt}
+mount $gho->{Lvdev} $gho->{Diskmnt};
+END
+}
+
+sub guest_unprepare_disk ($) {
+    my ($gho) = @_;
+    return if $gho->{Diskfmt} eq "lvm";
+    target_cmd_root($gho->{Host}, <<END);
+umount $gho->{Lvdev} || :
+END
+}
+
+
 
 sub target_choose_vg ($$) {
     my ($ho, $mbneeded) = @_;
@@ -1518,6 +1561,7 @@ sub prepareguest ($$$$$$) {
     }
 
     guest_find_lv($gho);
+    guest_find_diskimg($gho);
     guest_find_ether($gho);
     guest_find_tcpcheckport($gho);
     return $gho;
@@ -1528,7 +1572,56 @@ sub prepareguest_part_lvmdisk ($$$) {
     target_cmd_root($ho, "lvremove -f $gho->{Lvdev} ||:");
     target_cmd_root($ho, "lvcreate -L ${disk_mb}M -n $gho->{Lv} $gho->{Vg}");
     target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Lvdev} count=10");
-}    
+}
+
+sub make_vhd ($$$) {
+    my ($ho, $gho, $disk_mb) = @_;
+    target_cmd_root($ho, "vhd-util create -n $gho->{Rootimg} -s $disk_mb");
+}
+sub make_qcow2 ($$$) {
+    my ($ho, $gho, $disk_mb) = @_;
+    # upstream qemu's version. Seems preferable to qemu-xen-img from qemu-trad.
+    my $qemu_img = "/usr/local/lib/xen/bin/qemu-img";
+    target_cmd_root($ho, "$qemu_img create -f qcow2 $gho->{Rootimg} ${disk_mb}M");
+}
+sub make_raw ($$$) {
+    my ($ho, $gho, $disk_mb) = @_;
+    # In local tests this reported 130MB/s, so calculate a timeout assuming 100MB/s.
+    target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Rootimg} bs=1M count=${disk_mb}",
+	${disk_mb} / 100);
+}
+
+sub prepareguest_part_diskimg ($$$) {
+    my ($ho, $gho, $disk_mb) = @_;
+
+    my $diskfmt = $gho->{Diskfmt};
+    # Allow an extra 10 megabytes for image format headers
+    my $disk_overhead = $diskfmt eq "lvm" ? 0 : 10;
+
+    logm("preparing guest disks in $diskfmt format");
+
+    target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+
+    prepareguest_part_lvmdisk($ho, $gho, $disk_mb + $disk_overhead);
+
+    if ($diskfmt ne "lvm") {
+
+	$gho->{Rootimg} = "$gho->{Diskmnt}/disk.$diskfmt";
+	$gho->{Rootcfg} = "format=$diskfmt,vdev=xvda,target=$gho->{Rootimg}";
+
+	target_cmd_root($ho, <<END);
+mkfs.ext3 $gho->{Lvdev}
+mkdir -p $gho->{Diskmnt}
+mount $gho->{Lvdev} $gho->{Diskmnt}
+END
+        no strict qw(refs);
+        &{"make_$diskfmt"}($ho, $gho, $disk_mb);
+
+	target_cmd_root($ho, <<END);
+umount $gho->{Lvdev}
+END
+    }
+}
 
 sub prepareguest_part_xencfg ($$$$$) {
     my ($ho, $gho, $ram_mb, $xopts, $cfgrest) = @_;
diff --git a/make-flight b/make-flight
index 909cebe..c923f21 100755
--- a/make-flight
+++ b/make-flight
@@ -373,6 +373,22 @@ do_pv_debian_tests () {
   for xsm in $xsms ; do
     do_pv_debian_test_one libvirt '' libvirt enable_xsm=$xsm
   done
+
+  for ts in xl libvirt ; do
+
+    for fmt in raw vhd qcow2 ; do
+
+      fmt_runvar="debian_diskfmt=$fmt"
+
+      do_pv_debian_test_one $ts-$fmt '-di' $ts  \
+          debian_arch=amd64                     \
+          debian_dist=$guestsuite               \
+          debian_method=netboot                 \
+          debian_bootloader=pygrub              \
+          $fmt_runvar
+
+    done
+  done
 }
 
 test_matrix_do_one () {
diff --git a/ts-debian-di-install b/ts-debian-di-install
index 281e7cb..839cc27 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -41,9 +41,7 @@ sub prep () {
     $gho= prepareguest($ho, $gn, $guesthost, 22,
                        $disk_mb, 40);
 
-    prepareguest_part_lvmdisk($ho, $gho, $disk_mb);
-
-    target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+    prepareguest_part_diskimg($ho, $gho, $disk_mb);
 }
 
 sub setup_netinst($$$)
@@ -205,14 +203,12 @@ END
 	OnPowerOff => "preserve"
     );
 
-    my $root_disk = "'phy:$gho->{Lvdev},xvda,w'";
-
     prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
 $method_cfg
 extra       = "$cmdline"
 #
 disk        = [
-            $extra_disk $root_disk
+            $extra_disk '$gho->{Diskspec}'
             ]
 END
 
@@ -236,7 +232,7 @@ END
 $blcfg
 #
 disk        = [
-            $root_disk
+            '$gho->{Diskspec}'
             ]
 END
     return;
diff --git a/ts-guest-start b/ts-guest-start
index 1aa9e69..a434720 100755
--- a/ts-guest-start
+++ b/ts-guest-start
@@ -25,7 +25,6 @@ tsreadconfig();
 our ($ho,$gho) = ts_get_host_guest(@ARGV);
 
 sub start () {
-    guest_umount_lv($ho, $gho);
     guest_create($gho);
 }
 
-- 
2.1.4

[PATCH OSSTEST v5 21/24] mfi-common: Allow make-*flight to filter the set of build jobs to include

[Ian Campbell]

By using the same job_create_build(_filter_callback) scheme used for
the test jobs.

Will be used in make-distros-flight.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 make-distros-flight |  4 ++++
 make-flight         |  4 ++++
 mfi-common          | 21 +++++++++++++++------
 3 files changed, 23 insertions(+), 6 deletions(-)

diff --git a/make-distros-flight b/make-distros-flight
index e3ec8f7..2724dd2 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -33,6 +33,10 @@ flight=`./cs-flight-create $blessing $branch`
 defsuite=`getconfig DebianSuite`
 defguestsuite=`getconfig GuestDebianSuite`
 
+job_create_build_filter_callback () {
+    :
+}
+
 if [ x$buildflight = x ]; then
 
   WANT_XEND=false REVISION_LINUX_OLD=disable
diff --git a/make-flight b/make-flight
index c923f21..ef1e422 100755
--- a/make-flight
+++ b/make-flight
@@ -34,6 +34,10 @@ flight=`./cs-flight-create $blessing $branch`
 defsuite=`getconfig DebianSuite`
 defguestsuite=`getconfig GuestDebianSuite`
 
+job_create_build_filter_callback () {
+    :
+}
+
 if [ x$buildflight = x ]; then
 
   create_build_jobs
diff --git a/mfi-common b/mfi-common
index a9e966f..a100afb 100644
--- a/mfi-common
+++ b/mfi-common
@@ -54,6 +54,15 @@ xenbranch_xsm_variants () {
     esac
 }
 
+job_create_build () {
+  job_create_build_filter_callback "$@" || return 0
+
+  local job=$1; shift
+  local recipe=$1; shift
+
+  ./cs-job-create $flight $job $recipe "$@"
+}
+
 create_build_jobs () {
 
   local arch
@@ -164,7 +173,7 @@ create_build_jobs () {
       else
         xsm_suffix=""
       fi
-      ./cs-job-create $flight build-$arch$xsm_suffix build                   \
+      job_create_build build-$arch$xsm_suffix build                          \
                 arch=$arch enable_xend=$build_defxend enable_ovmf=$enable_ovmf\
                 enable_xsm=$enable_xsm                                       \
         tree_qemu=$TREE_QEMU                                                 \
@@ -183,7 +192,7 @@ create_build_jobs () {
     done
 
     if [ $build_extraxend = "true" ] ; then
-    ./cs-job-create $flight build-$arch-xend build                           \
+    job_create_build build-$arch-xend build                                  \
                 arch=$arch enable_xend=true enable_ovmf=$enable_ovmf         \
         tree_qemu=$TREE_QEMU                                                 \
         tree_qemuu=$TREE_QEMU_UPSTREAM                                       \
@@ -196,7 +205,7 @@ create_build_jobs () {
                 revision_qemuu=$REVISION_QEMU_UPSTREAM
     fi
 
-    ./cs-job-create $flight build-$arch-pvops build-kern                     \
+    job_create_build build-$arch-pvops build-kern                            \
                 arch=$arch kconfighow=xen-enable-xen-config                  \
                 $RUNVARS $BUILD_RUNVARS $BUILD_LINUX_RUNVARS $arch_runvars   \
                 $suite_runvars                                               \
@@ -208,7 +217,7 @@ create_build_jobs () {
 
     if [ "x$REVISION_LIBVIRT" != xdisable ]; then
 
-    ./cs-job-create $flight build-$arch-libvirt build-libvirt                \
+    job_create_build build-$arch-libvirt build-libvirt                       \
                 arch=$arch                                                   \
         tree_xen=$TREE_XEN                                                   \
                 $RUNVARS $BUILD_RUNVARS $BUILD_LIBVIRT_RUNVARS $arch_runvars \
@@ -223,7 +232,7 @@ create_build_jobs () {
 
     case $arch in
     i386|amd64)
-    ./cs-job-create $flight build-$arch-rumpuserxen build-rumpuserxen        \
+    job_create_build build-$arch-rumpuserxen build-rumpuserxen               \
                 arch=$arch                                                   \
         tree_xen=$TREE_XEN                                                   \
                 $RUNVARS $BUILD_RUNVARS $BUILD_RUMPUSERXEN_RUNVARS $arch_runvars \
@@ -252,7 +261,7 @@ create_build_jobs () {
 
     if [ "x$REVISION_LINUX_OLD" != xdisable ]; then
 
-      ./cs-job-create $flight build-$arch-oldkern build-kern            \
+      job_create_build build-$arch-oldkern build-kern                   \
                 arch=$arch kconfighow=create-config-sh                  \
                 kimagefile=vmlinux                                      \
                 $RUNVARS $BUILD_RUNVARS $BUILD_LINUX_OLD_RUNVARS        \
-- 
2.1.4

[PATCH OSSTEST v5 22/24] make-distros-flight: don't bother building for XSM.

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 make-distros-flight | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/make-distros-flight b/make-distros-flight
index 2724dd2..62c527b 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -34,7 +34,10 @@ defsuite=`getconfig DebianSuite`
 defguestsuite=`getconfig GuestDebianSuite`
 
 job_create_build_filter_callback () {
-    :
+  case " $* " in
+    *" enable_xsm=true "*) return 1;;
+  esac
+  return 0
 }
 
 if [ x$buildflight = x ]; then
-- 
2.1.4

[PATCH OSSTEST v5 23/24] distros: email only me on play flights

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 daily-cron-email-play--distros | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 daily-cron-email-play--distros

diff --git a/daily-cron-email-play--distros b/daily-cron-email-play--distros
new file mode 100644
index 0000000..0d82e65
--- /dev/null
+++ b/daily-cron-email-play--distros
@@ -0,0 +1 @@
+To: ian.campbell@citrix.com
-- 
2.1.4

[PATCH OSSTEST v5 24/24] ts-debian-di-install: Use ftp.debian.org directly

[Ian Campbell]

The local proxy seems to serve stale packages for Jessie etc, I blame
the intercepting cache on the way out of our network, similar to
b5f15136900d "mg-debian-installer-update: workaround caching proxies",
except it is between the apt-cache and the world not the osstest vm
and the world.

Since the netboot kernel+initrd are reasonably small, these flights
are infrequent and they are intended to test the current upstream
version I think this is tollerable.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ts-debian-di-install | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/ts-debian-di-install b/ts-debian-di-install
index 839cc27..469c1b6 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -119,7 +119,9 @@ sub setup_netboot($$$)
 	target_putfile_root($ho, 60, $initrd, "$didir/initrd_${suite}_${arch}");
 
     } else {
-	my $mirror = "http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath}";
+	#my $mirror = "http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath}";
+        # XXX local mirror seems to serve up stale files.
+	my $mirror = "http://ftp.debian.org/debian";
 
 	my $di_url = $suite eq "daily" ?
 	    "http://d-i.debian.org/daily-images/$arch/daily/netboot" :
-- 
2.1.4

Re: [PATCH OSSTEST v5 04/24] Debian: refactor code to add preseed commands to the preseed file

[Ian Campbell]

On Wed, 2015-04-15 at 11:35 +0100, Ian Campbell wrote:

> diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
> index cfd5144..95fce9a 100755
> --- a/ts-debian-hvm-install
> +++ b/ts-debian-hvm-install
> @@ -77,6 +77,9 @@ d-i preseed/late_command string \\
>          in-target mkdir -p /root/.ssh; \\
>          in-target sh -c "echo -e '$authkeys'> /root/.ssh/authorized_keys";
>  END
> +
> +    $preseed_file .= preseed_hook_cmds();

Longtao reused this patch for his nestedhvm testing series and
discovered a bug. The use of preseed_hook_cmds shadows the
preseed/late_command which is just visible here in the patch context
meaning that the authorized keys and update-rc.d are not actually run.

The code snippet needs to use a "preseed_hook_command($ho,
'late_command', $sfx, <<END);" type thing instead.

I'll fix this up in the next iteration.

Ian.

Re: [PATCH OSSTEST v5 04/24] Debian: refactor code to add preseed commands to the preseed file

[Ian Campbell]

On Wed, 2015-04-29 at 03:32 +0100, Ian Campbell wrote:
> On Wed, 2015-04-15 at 11:35 +0100, Ian Campbell wrote:
> 
> > diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
> > index cfd5144..95fce9a 100755
> > --- a/ts-debian-hvm-install
> > +++ b/ts-debian-hvm-install
> > @@ -77,6 +77,9 @@ d-i preseed/late_command string \\
> >          in-target mkdir -p /root/.ssh; \\
> >          in-target sh -c "echo -e '$authkeys'> /root/.ssh/authorized_keys";
> >  END
> > +
> > +    $preseed_file .= preseed_hook_cmds();
> 
> Longtao reused this patch for his nestedhvm testing series and
> discovered a bug. The use of preseed_hook_cmds shadows the
> preseed/late_command which is just visible here in the patch context
> meaning that the authorized keys and update-rc.d are not actually run.
> 
> The code snippet needs to use a "preseed_hook_command($ho,
> 'late_command', $sfx, <<END);" type thing instead.
> 
> I'll fix this up in the next iteration.

I intend to squash the following incremental patch into "Debian:
refactor code to add preseed commands to the preseed file" in the next
posting. I've not yet tested it, I'll kick something off over the
weekend and hopefully repost next week.

commit c663ce9b76dc3b2c2e1d900d65b58f50f2ee36c5
Author: Ian Campbell <ian.campbell@citrix.com>
Date:   Fri May 1 12:01:47 2015 +0100

    squash!Debian: refactor code to add preseed commands to the preseed file
    
    v6: Use preseed_hook_command rather than an "d-i preseed/late_comman"
    entry in the preseed file in ts-debian-hvm-install. The former is
    overwritten by the result of preseed_hook_cmds.

diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index 0085d82..98e5d76 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -69,10 +69,11 @@ d-i partman-auto/expert_recipe string \\
                 .
 
 d-i apt-setup/cdrom/set-first boolean false
+END
 
-d-i preseed/late_command string \\
-        in-target mkdir -p /boot/efi/EFI/boot; \\
-        in-target cp /boot/efi/EFI/debian/grubx64.efi /boot/efi/EFI/boot/bootx64.efi ;\\
+preseed_hook_command($ho, 'late_command', '', <<END);
+        in-target mkdir -p /boot/efi/EFI/boot
+        in-target cp /boot/efi/EFI/debian/grubx64.efi /boot/efi/EFI/boot/bootx64.efi
 END
 
     $preseed_file .= preseed_hook_cmds();

Re: [PATCH OSSTEST v5 04/24] Debian: refactor code to add preseed commands to the preseed file

[Pang, LongtaoX]

> -----Original Message-----
> From: Ian Campbell [mailto:ian.campbell@citrix.com]
> Sent: Friday, May 01, 2015 7:07 PM
> To: ian.jackson@eu.citrix.com
> Cc: Hu, Robert; Pang, LongtaoX; xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] [PATCH OSSTEST v5 04/24] Debian: refactor code to add
> preseed commands to the preseed file
> 
> On Wed, 2015-04-29 at 03:32 +0100, Ian Campbell wrote:
> > On Wed, 2015-04-15 at 11:35 +0100, Ian Campbell wrote:
> >
> > > diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
> > > index cfd5144..95fce9a 100755
> > > --- a/ts-debian-hvm-install
> > > +++ b/ts-debian-hvm-install
> > > @@ -77,6 +77,9 @@ d-i preseed/late_command string \\
> > >          in-target mkdir -p /root/.ssh; \\
> > >          in-target sh -c "echo -e '$authkeys'> /root/.ssh/authorized_keys";
> > >  END
> > > +
> > > +    $preseed_file .= preseed_hook_cmds();
> >
> > Longtao reused this patch for his nestedhvm testing series and
> > discovered a bug. The use of preseed_hook_cmds shadows the
> > preseed/late_command which is just visible here in the patch context
> > meaning that the authorized keys and update-rc.d are not actually run.
> >
> > The code snippet needs to use a "preseed_hook_command($ho,
> > 'late_command', $sfx, <<END);" type thing instead.
> >
> > I'll fix this up in the next iteration.
> 
> I intend to squash the following incremental patch into "Debian:
> refactor code to add preseed commands to the preseed file" in the next
> posting. I've not yet tested it, I'll kick something off over the
> weekend and hopefully repost next week.
> 
> commit c663ce9b76dc3b2c2e1d900d65b58f50f2ee36c5
> Author: Ian Campbell <ian.campbell@citrix.com>
> Date:   Fri May 1 12:01:47 2015 +0100
> 
>     squash!Debian: refactor code to add preseed commands to the preseed file
> 
>     v6: Use preseed_hook_command rather than an "d-i preseed/late_comman"
>     entry in the preseed file in ts-debian-hvm-install. The former is
>     overwritten by the result of preseed_hook_cmds.
> 
> diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
> index 0085d82..98e5d76 100755
> --- a/ts-debian-hvm-install
> +++ b/ts-debian-hvm-install
> @@ -69,10 +69,11 @@ d-i partman-auto/expert_recipe string \\
>                  .
> 
>  d-i apt-setup/cdrom/set-first boolean false
> +END
> 
> -d-i preseed/late_command string \\
> -        in-target mkdir -p /boot/efi/EFI/boot; \\
> -        in-target cp /boot/efi/EFI/debian/grubx64.efi
> /boot/efi/EFI/boot/bootx64.efi ;\\
> +preseed_hook_command($ho, 'late_command', '', <<END);
> +        in-target mkdir -p /boot/efi/EFI/boot
> +        in-target cp /boot/efi/EFI/debian/grubx64.efi
> /boot/efi/EFI/boot/bootx64.efi
>  END
> 
>      $preseed_file .= preseed_hook_cmds();
> 
I have verified this squash patch in my environment.
After push in this patch based on your former v5_patch [04,05,06], the 'late_command' work fine. 
So, I could add ' in-target sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list;' code in late_command to disable 'cdrom' entry to make HTTP URL entry available in sources.list for L1 hvm guest.

Re: [PATCH OSSTEST v5 04/24] Debian: refactor code to add preseed commands to the preseed file

[Ian Campbell]

On Mon, 2015-05-04 at 02:08 +0000, Pang, LongtaoX wrote:
> I have verified this squash patch in my environment.
> After push in this patch based on your former v5_patch [04,05,06], the
> 'late_command' work fine. 

> So, I could add ' in-target sed -i 's/^deb
> *cdrom/#&/g' /etc/apt/sources.list;' code in late_command to disable
> 'cdrom' entry to make HTTP URL entry available in sources.list for L1
> hvm guest.

Great, thanks for letting us know.

I did some testing of v6 over the weekend and fixed a couple of minor
regressions which I caused to test-*-debianhvm-*. A final test is
running now so I hope to post v6 either this afternoon or tomorrow AM.

Ian.

Re: [PATCH OSSTEST v5 04/24] Debian: refactor code to add preseed commands to the preseed file

[Ian Campbell]

On Tue, 2015-05-05 at 10:57 +0100, Ian Campbell wrote:
> I did some testing of v6 over the weekend and fixed a couple of minor
> regressions which I caused to test-*-debianhvm-*. A final test is
> running now so I hope to post v6 either this afternoon or tomorrow AM.

Unfortunately that test showed up a further issue, so not this
afternoon, hopefully it will be an easy fix and I can post tomorrow.

Ian

[PATCH OSSTEST v5 00/24] add distro domU testing flight

[Ian Campbell]

Hi,

Since v5 I've run various adhoc tests and fixed a bunch of issues (most
noticeably in "Debian: refactor code to add preseed commands to the
preseed file" which had broken most debian hvm installs).

As last time there are some patches in here which I think will be useful
to the Intel folks doing the nested virt testing, specifically the
refactoring of how overlays and ssh host keys are done will be useful
for installing a guest to be treated as the L1 host. (Note that this has
changed a bit, again, since last time)

Since Intel are building on some of this with their nested virt patch
series I'd like to get the early part of this series in sooner than the
rest, I think up to "make-flight: Handle $BUILD_LVEXTEND_MAX in
mfi-common:create_build_jobs()" makes sense. I have run an adhoc
xen-unstable flight on just those changes and the results are at
http://osstest.xs.citrite.net/~osstest/testlogs/logs/37326/ (only
visible in Citrix, sorry). The results email is also appended. No
regressions AFAICT.

(Possibly "Debian: [rR]efactor.*" could go in in isolation, but I didn't
test that by itself)

This also builds on some changes which are in "[GIT PULL OSSTEST] Merge
up changes from Cambridge instance.". Perhaps we should push that as one
batch? I can provide a branch if needed.

Open questions from last time:

When I originally started this work I envisioned a flight running on the
main production instance (in Cambridge at the time). Now that we have
the new colo I would still consider that the main production instance is
the best home, however given the new colo is not yet up to full capacity
we could also consider running this flight in Cambridge for the time
being. What do you think?

Also when running the adhoc tests the sheer number of jobs which are
involved (with my hope there will be more in the future as other distros
get in on the act) I was considering splitting it into multiple
distros-debian-{squeeze,wheezy,jessie,sid,daily} flights. Thoughts?

Summary of (A)cks, (M)odified and (N)ew:

         TestSupport: Add helper to fetch a URL on a host
 A       TestSupport: allow caller of prepareguest_part_xencfg to specify viftype
 A       create_webfile: Support use with guests as well as hosts.
M        Debian: refactor code to add preseed commands to the preseed file
         Debian: refactor preseeding of .ssh directories
 A       Debian: Refactor installation of overlays, so it can be used for guests too
         Debian: add preseed_create_guest helper
  N      ts-debian-hvm-install: Only apply EFI workaround for Wheezy
 A       make-flight: Handle $BUILD_LVEXTEND_MAX in mfi-common:create_build_jobs()
         <------------ CONSIDER APPLYING TO HERE ------------>
M        distros: add support for installing Debian PV guests via d-i, flight and jobs
         distros: support booting Debian PV (d-i installed) guests with pvgrub.
         distros: Support pvgrub for Wheezy too.
         distros: support PV guest install from Debian netinst media.
         Test pygrub and pvgrub on the regular flights
         distros: add branch infrastructure
 A       distros: Run a flight over the weekend.
         Debian: Handle lack of bootloader support in d-i on ARM.
         standalone: propagate result of command from with_logging
         ts-debian-di-install: Refactor root_disk specification
         make-flight: refactor PV debian tests
M        Add testing of non-LVM/phy disk backends.
         mfi-common: Allow make-*flight to filter the set of build jobs to include
         make-distros-flight: don't bother building for XSM.
         distros: email only me on play flights
         ts-debian-di-install: Use ftp.debian.org directly

Ian.


"Old" tested version had not actually been tested; therefore in this
flight we test it, rather than a new candidate.  The baseline, if
any, is the most recent actually tested revision.

flight 37326 xen-unstable play [play]
http://osstest.xs.citrite.net/~osstest/testlogs/logs/37326/

Failures :-/ but no regressions.

Tests which did not succeed, but are not blocking:
 test-amd64-amd64-rumpuserxen-amd64  1 build-check(1)               blocked n/a
 test-amd64-i386-rumpuserxen-i386  1 build-check(1)               blocked  n/a
 test-amd64-i386-xl-qemuu-winxpsp3 16 guest-stop                fail never pass
 test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm 9 debian-hvm-install fail never pass
 test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stop             fail never pass
 test-armhf-armhf-xl-sedf-pin 12 migrate-support-check        fail   never pass
 test-amd64-amd64-xl-winxpsp3 16 guest-stop                   fail   never pass
 test-amd64-i386-xl-qemut-winxpsp3 16 guest-stop                fail never pass
 test-amd64-i386-xl-winxpsp3  16 guest-stop                   fail   never pass
 test-amd64-amd64-xl-qemut-win7-amd64 16 guest-stop             fail never pass
 test-amd64-amd64-libvirt     12 migrate-support-check        fail   never pass
 test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm 9 debian-hvm-install fail never pass
 test-armhf-armhf-xl-xsm       6 xen-boot                     fail   never pass
 test-armhf-armhf-xl          12 migrate-support-check        fail   never pass
 test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop              fail never pass
 test-amd64-amd64-xl-qemuu-winxpsp3 16 guest-stop               fail never pass
 test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop              fail never pass
 test-amd64-i386-libvirt      12 migrate-support-check        fail   never pass
 test-amd64-amd64-xl-pvh-amd  11 guest-start                  fail   never pass
 test-amd64-amd64-xl-pvh-intel 11 guest-start                  fail  never pass
 test-amd64-amd64-xl-qemut-winxpsp3 16 guest-stop               fail never pass
 test-amd64-i386-libvirt-xsm  11 guest-start                  fail   never pass
 test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm 9 debian-hvm-install fail never pass
 test-armhf-armhf-xl-midway   12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-sedf     12 migrate-support-check        fail   never pass
 test-amd64-i386-xl-win7-amd64 16 guest-stop                   fail  never pass
 test-armhf-armhf-xl-credit2  12 migrate-support-check        fail   never pass
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1 16 guest-stop         fail never pass
 test-amd64-i386-xl-winxpsp3-vcpus1 16 guest-stop               fail never pass
 test-amd64-i386-xl-qemut-debianhvm-amd64-xsm 9 debian-hvm-install fail never pass
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1 16 guest-stop         fail never pass
 test-amd64-amd64-xl-win7-amd64 16 guest-stop                   fail never pass
 test-armhf-armhf-xl-multivcpu 12 migrate-support-check        fail  never pass
 test-amd64-amd64-libvirt-xsm 11 guest-start                  fail   never pass
 test-amd64-amd64-xl-xsm      11 guest-start                  fail   never pass
 test-armhf-armhf-libvirt-xsm  6 xen-boot                     fail   never pass
 test-amd64-i386-xl-xsm       11 guest-start                  fail   never pass
 test-armhf-armhf-libvirt     12 migrate-support-check        fail   never pass

version targeted for testing:
 xen                  3a28f760508fb35c430edac17a9efde5aff6d1d5
baseline version:
 xen                  3a28f760508fb35c430edac17a9efde5aff6d1d5

jobs:
 test-amd64-amd64-xl                                          pass    
 test-armhf-armhf-xl                                          pass    
 test-amd64-i386-xl                                           pass    
 test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm                fail    
 test-amd64-i386-xl-qemut-debianhvm-amd64-xsm                 fail    
 test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm                fail    
 test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm                 fail    
 test-amd64-amd64-libvirt-xsm                                 fail    
 test-armhf-armhf-libvirt-xsm                                 fail    
 test-amd64-i386-libvirt-xsm                                  fail    
 test-amd64-amd64-xl-xsm                                      fail    
 test-armhf-armhf-xl-xsm                                      fail    
 test-amd64-i386-xl-xsm                                       fail    
 test-amd64-amd64-xl-pvh-amd                                  fail    
 test-amd64-i386-rhel6hvm-amd                                 pass    
 test-amd64-i386-qemut-rhel6hvm-amd                           pass    
 test-amd64-i386-qemuu-rhel6hvm-amd                           pass    
 test-amd64-amd64-xl-qemut-debianhvm-amd64                    pass    
 test-amd64-i386-xl-qemut-debianhvm-amd64                     pass    
 test-amd64-amd64-xl-qemuu-debianhvm-amd64                    pass    
 test-amd64-i386-xl-qemuu-debianhvm-amd64                     pass    
 test-amd64-i386-freebsd10-amd64                              pass    
 test-amd64-amd64-xl-qemuu-ovmf-amd64                         pass    
 test-amd64-i386-xl-qemuu-ovmf-amd64                          pass    
 test-amd64-amd64-rumpuserxen-amd64                           broken  
 test-amd64-amd64-xl-qemut-win7-amd64                         fail    
 test-amd64-i386-xl-qemut-win7-amd64                          fail    
 test-amd64-amd64-xl-qemuu-win7-amd64                         fail    
 test-amd64-i386-xl-qemuu-win7-amd64                          fail    
 test-amd64-amd64-xl-win7-amd64                               fail    
 test-amd64-i386-xl-win7-amd64                                fail    
 test-amd64-amd64-xl-credit2                                  pass    
 test-armhf-armhf-xl-credit2                                  pass    
 test-amd64-i386-freebsd10-i386                               pass    
 test-amd64-i386-rumpuserxen-i386                             broken  
 test-amd64-amd64-xl-pvh-intel                                fail    
 test-amd64-i386-rhel6hvm-intel                               pass    
 test-amd64-i386-qemut-rhel6hvm-intel                         pass    
 test-amd64-i386-qemuu-rhel6hvm-intel                         pass    
 test-amd64-amd64-libvirt                                     pass    
 test-armhf-armhf-libvirt                                     pass    
 test-amd64-i386-libvirt                                      pass    
 test-armhf-armhf-xl-midway                                   pass    
 test-amd64-amd64-xl-multivcpu                                pass    
 test-armhf-armhf-xl-multivcpu                                pass    
 test-amd64-amd64-pair                                        pass    
 test-amd64-i386-pair                                         pass    
 test-amd64-amd64-xl-sedf-pin                                 pass    
 test-armhf-armhf-xl-sedf-pin                                 pass    
 test-amd64-amd64-xl-sedf                                     pass    
 test-armhf-armhf-xl-sedf                                     pass    
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1                     fail    
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1                     fail    
 test-amd64-i386-xl-winxpsp3-vcpus1                           fail    
 test-amd64-amd64-xl-qemut-winxpsp3                           fail    
 test-amd64-i386-xl-qemut-winxpsp3                            fail    
 test-amd64-amd64-xl-qemuu-winxpsp3                           fail    
 test-amd64-i386-xl-qemuu-winxpsp3                            fail    
 test-amd64-amd64-xl-winxpsp3                                 fail    
 test-amd64-i386-xl-winxpsp3                                  fail    


------------------------------------------------------------
sg-report-flight on osstest.xs.citrite.net
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images

Logs, config files, etc. are available at
    http://osstest.xs.citrite.net/~osstest/testlogs/logs

Test harness code can be found at
    http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary


Published tested tree is already up to date.

Re: [PATCH OSSTEST v5 00/24] add distro domU testing flight

[Ian Campbell]

On Wed, 2015-05-06 at 10:46 +0100, Ian Campbell wrote:
> Hi,
> 
> Since v5

Subject should have said v6 and I accidentally sent the patches
themselves to xendevel and not xen-devel@lists. I've resent, sorry to
Ian especially for the extra spam :-(

Ian.

Re: [PATCH OSSTEST v5 00/24] add distro domU testing flight

[Ian Jackson]

Ian Campbell writes ("[Xen-devel] [PATCH OSSTEST v5 00/24] add distro domU testing flight"):
> As last time there are some patches in here which I think will be useful
> to the Intel folks doing the nested virt testing, specifically the
> refactoring of how overlays and ssh host keys are done will be useful
> for installing a guest to be treated as the L1 host. (Note that this has
> changed a bit, again, since last time)

I think I have reviewed all of those now.

> When I originally started this work I envisioned a flight running on the
> main production instance (in Cambridge at the time). Now that we have
> the new colo I would still consider that the main production instance is
> the best home, however given the new colo is not yet up to full capacity
> we could also consider running this flight in Cambridge for the time
> being. What do you think?

I think we should use the resources we have available, and that
includes the Cambridge instance (which is hopefully less broken now).
Until the Xen Project colo instance is _much_ bigger than Cambridge, I
think we should run it here in Cambridge.

> Also when running the adhoc tests the sheer number of jobs which are
> involved (with my hope there will be more in the future as other distros
> get in on the act) I was considering splitting it into multiple
> distros-debian-{squeeze,wheezy,jessie,sid,daily} flights. Thoughts?

As discussed, we think this is a good idea.  In particular, it would
allow running stable releases less often.  (As a result, this will
produce a whole lot of distros-<distro>-<revision> flights.)

Thanks,
Ian.

Re: [PATCH OSSTEST v5 00/24] add distro domU testing flight

[Ian Campbell]

On Tue, 2015-05-12 at 17:06 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[Xen-devel] [PATCH OSSTEST v5 00/24] add distro domU testing flight"):
> > As last time there are some patches in here which I think will be useful
> > to the Intel folks doing the nested virt testing, specifically the
> > refactoring of how overlays and ssh host keys are done will be useful
> > for installing a guest to be treated as the L1 host. (Note that this has
> > changed a bit, again, since last time)
> 
> I think I have reviewed all of those now.

Thanks!

I think you have acked all of the ones I identified as being an
initially commitable chunk except one "TestSupport: Add helper to fetch
a URL on a host" which I think is not used by the nested virt series and
so can wait.

Assuming that turns out to be the case are you OK with me pushing that
subset to pretest?

> > When I originally started this work I envisioned a flight running on the
> > main production instance (in Cambridge at the time). Now that we have
> > the new colo I would still consider that the main production instance is
> > the best home, however given the new colo is not yet up to full capacity
> > we could also consider running this flight in Cambridge for the time
> > being. What do you think?
> 
> I think we should use the resources we have available, and that
> includes the Cambridge instance (which is hopefully less broken now).
> Until the Xen Project colo instance is _much_ bigger than Cambridge, I
> think we should run it here in Cambridge.

OK, will do. Should I arrange for reports to go to xen-devel even though
logs will only be available internally?

> > Also when running the adhoc tests the sheer number of jobs which are
> > involved (with my hope there will be more in the future as other distros
> > get in on the act) I was considering splitting it into multiple
> > distros-debian-{squeeze,wheezy,jessie,sid,daily} flights. Thoughts?
> 
> As discussed, we think this is a good idea.  In particular, it would
> allow running stable releases less often.  (As a result, this will
> produce a whole lot of distros-<distro>-<revision> flights.)

Correct, I'll refactor along those lines.

Ian.

Re: [PATCH OSSTEST v5 00/24] add distro domU testing flight

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH OSSTEST v5 00/24] add distro domU testing flight"):
> I think you have acked all of the ones I identified as being an
> initially commitable chunk except one "TestSupport: Add helper to fetch
> a URL on a host" which I think is not used by the nested virt series and
> so can wait.

Right.

> Assuming that turns out to be the case are you OK with me pushing that
> subset to pretest?

Yes.  You might like to combine it with my LVM resize/mkfs series (if
you think the latter is ready), which I have put here for your
convenience:
  xenbits.xen.org:/home/iwj/ext/osstest.git#wip.build-prep-no-fs-resize.v1

> > I think we should use the resources we have available, and that
> > includes the Cambridge instance (which is hopefully less broken now).
> > Until the Xen Project colo instance is _much_ bigger than Cambridge, I
> > think we should run it here in Cambridge.
> 
> OK, will do. Should I arrange for reports to go to xen-devel even though
> logs will only be available internally?

Good question.  Probably, I think.

Ian.