* [PATCH OSSTEST v8 01/14] mfi-common: Allow make-*flight to filter the set of build jobs to include
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
@ 2015-07-08 12:30 ` Ian Campbell
2015-07-08 12:30 ` [PATCH OSSTEST v8 02/14] TestSupport: Add helper to fetch a URL on a host Ian Campbell
` (13 subsequent siblings)
14 siblings, 0 replies; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 12:30 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
By using the same job_create_build(_filter_callback) scheme used for
the test jobs.
Will be used in make-distros-flight.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v8: Moved to head of queue, make-distros-flight isn't introduced yet
so that hunk is dropped here and comes back later on.
---
make-flight | 4 ++++
mfi-common | 21 +++++++++++++++------
2 files changed, 19 insertions(+), 6 deletions(-)
diff --git a/make-flight b/make-flight
index c763ce9..de8393a 100755
--- a/make-flight
+++ b/make-flight
@@ -34,6 +34,10 @@ flight=`./cs-flight-create $blessing $branch`
defsuite=`getconfig DebianSuite`
defguestsuite=`getconfig GuestDebianSuite`
+job_create_build_filter_callback () {
+ :
+}
+
if [ x$buildflight = x ]; then
create_build_jobs
diff --git a/mfi-common b/mfi-common
index a9e966f..a100afb 100644
--- a/mfi-common
+++ b/mfi-common
@@ -54,6 +54,15 @@ xenbranch_xsm_variants () {
esac
}
+job_create_build () {
+ job_create_build_filter_callback "$@" || return 0
+
+ local job=$1; shift
+ local recipe=$1; shift
+
+ ./cs-job-create $flight $job $recipe "$@"
+}
+
create_build_jobs () {
local arch
@@ -164,7 +173,7 @@ create_build_jobs () {
else
xsm_suffix=""
fi
- ./cs-job-create $flight build-$arch$xsm_suffix build \
+ job_create_build build-$arch$xsm_suffix build \
arch=$arch enable_xend=$build_defxend enable_ovmf=$enable_ovmf\
enable_xsm=$enable_xsm \
tree_qemu=$TREE_QEMU \
@@ -183,7 +192,7 @@ create_build_jobs () {
done
if [ $build_extraxend = "true" ] ; then
- ./cs-job-create $flight build-$arch-xend build \
+ job_create_build build-$arch-xend build \
arch=$arch enable_xend=true enable_ovmf=$enable_ovmf \
tree_qemu=$TREE_QEMU \
tree_qemuu=$TREE_QEMU_UPSTREAM \
@@ -196,7 +205,7 @@ create_build_jobs () {
revision_qemuu=$REVISION_QEMU_UPSTREAM
fi
- ./cs-job-create $flight build-$arch-pvops build-kern \
+ job_create_build build-$arch-pvops build-kern \
arch=$arch kconfighow=xen-enable-xen-config \
$RUNVARS $BUILD_RUNVARS $BUILD_LINUX_RUNVARS $arch_runvars \
$suite_runvars \
@@ -208,7 +217,7 @@ create_build_jobs () {
if [ "x$REVISION_LIBVIRT" != xdisable ]; then
- ./cs-job-create $flight build-$arch-libvirt build-libvirt \
+ job_create_build build-$arch-libvirt build-libvirt \
arch=$arch \
tree_xen=$TREE_XEN \
$RUNVARS $BUILD_RUNVARS $BUILD_LIBVIRT_RUNVARS $arch_runvars \
@@ -223,7 +232,7 @@ create_build_jobs () {
case $arch in
i386|amd64)
- ./cs-job-create $flight build-$arch-rumpuserxen build-rumpuserxen \
+ job_create_build build-$arch-rumpuserxen build-rumpuserxen \
arch=$arch \
tree_xen=$TREE_XEN \
$RUNVARS $BUILD_RUNVARS $BUILD_RUMPUSERXEN_RUNVARS $arch_runvars \
@@ -252,7 +261,7 @@ create_build_jobs () {
if [ "x$REVISION_LINUX_OLD" != xdisable ]; then
- ./cs-job-create $flight build-$arch-oldkern build-kern \
+ job_create_build build-$arch-oldkern build-kern \
arch=$arch kconfighow=create-config-sh \
kimagefile=vmlinux \
$RUNVARS $BUILD_RUNVARS $BUILD_LINUX_OLD_RUNVARS \
--
2.1.4
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH OSSTEST v8 02/14] TestSupport: Add helper to fetch a URL on a host
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
2015-07-08 12:30 ` [PATCH OSSTEST v8 01/14] mfi-common: Allow make-*flight to filter the set of build jobs to include Ian Campbell
@ 2015-07-08 12:30 ` Ian Campbell
2015-07-08 13:29 ` Ian Jackson
2015-07-08 12:30 ` [PATCH OSSTEST v8 03/14] distros: add support for installing Debian PV guests via d-i, flight and jobs Ian Campbell
` (12 subsequent siblings)
14 siblings, 1 reply; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 12:30 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v8: Use \Q...\E to safely quote $url and $path
v7: Quote $url and $path, switch to a heredoc to avoid resulting over
long line
v5: Support http_proxy via $c{HttpProxy}
v3: Make sure wget is installed
---
Osstest/Debian.pm | 2 +-
Osstest/TestSupport.pm | 12 +++++++++++-
2 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 718a7e2..2d49ff8 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -841,7 +841,7 @@ d-i apt-setup/another boolean false
d-i apt-setup/non-free boolean false
d-i apt-setup/contrib boolean false
-d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, chiark-utils-bin, $extra_packages
+d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, chiark-utils-bin, wget, $extra_packages
d-i grub-installer/force-efi-extra-removable boolean true
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index b5994a4..1cace4f 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -55,7 +55,7 @@ BEGIN {
target_putfilecontents_stash
target_putfilecontents_root_stash
target_put_guest_image target_editfile
- target_editfile_cancel
+ target_editfile_cancel target_fetchurl
target_editfile_root target_file_exists
target_editfile_kvp_replace
target_run_apt
@@ -1595,6 +1595,16 @@ END
return $cfgpath;
}
+sub target_fetchurl($$$;$) {
+ my ($ho, $url, $path, $timeo) = @_;
+ $timeo ||= 2000;
+ my $useproxy = "export http_proxy=$c{HttpProxy};" if $c{HttpProxy};
+ target_cmd_root($ho, <<END, $timeo);
+ $useproxy wget --progress=dot:mega -O \Q$path\E \Q$url\E
+END
+}
+
+
sub target_put_guest_image ($$;$) {
my ($ho, $gho, $default) = @_;
my $specimage = $r{"$gho->{Guest}_image"};
--
2.1.4
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH OSSTEST v8 03/14] distros: add support for installing Debian PV guests via d-i, flight and jobs
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
2015-07-08 12:30 ` [PATCH OSSTEST v8 01/14] mfi-common: Allow make-*flight to filter the set of build jobs to include Ian Campbell
2015-07-08 12:30 ` [PATCH OSSTEST v8 02/14] TestSupport: Add helper to fetch a URL on a host Ian Campbell
@ 2015-07-08 12:30 ` Ian Campbell
2015-07-08 12:30 ` [PATCH OSSTEST v8 04/14] distros: support booting Debian PV (d-i installed) guests with pvgrub Ian Campbell
` (11 subsequent siblings)
14 siblings, 0 replies; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 12:30 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
This patch introduces ts-debian-di-install which can install Debian
from a netboot (PXE) debian installer image. By default it installs
from the d-i image used by osstest (using the special Xen PV guest
enabled flavour where necessary) but it can also fetch the kernel and
ramdisk from URLs specified in runvars. The resulting guests boot the
distro kernel using pygrub (pvgrub will follow).
The distros flights differ substantially from the existing flights.
Introduce make-distros-flight using the functionality previously
refactored into mfi-common. The new flight tests all versions of
Debian from Squeeze onward as an amd64, i386 and armhf guests (armhf
from Jessie onwards only) using the usual smoke tests.
Test names are suffixed -pygrub pending the addition of pvgrub
variants in a future commit.
Add the new cases to sg-run-job
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v8: "mfi-common: Allow make-*flight to filter the set of build jobs to
include" was moved to the front of the series, so the bits of that
and "make-distros-flight: don't bother building for XSM or libvirt."
have been folded in here. This patch and those two were all
already acked so I have retained the ack here.
Added description of configuraiton runvar's to
ts-debian-di-install itself. This was due to Ian's feedback on
"distros: support PV guest install from Debian netinst media."
further down the series, so I have retained the ack.
v7: Use {Guest}_suite as runvar. Also use $suite not $dist in
make-distros-flight for consistency.
Switch to a flight per Debian suite model rather than one
enourmous flight.
Switch to constructing the URLs in make-distros-flight
v6: Only apply -xen suffix to x86 images when doing a netboot using
the osstest version of d-i, since that is the only arch where we
create such files, other arches can use the bare names.
Use the guest $arch not the host $r{arch} when finding the
kernel+initrd to use for d-i install using the osstest d-i.
v4: use guest create
v3: $BUILD_LVEXTEND_MAX now handled in mfi-common
Consolidate setting of ruvars
Include $flight and $job in tmpdir name
Use Osstest::Debian::di_installcmdline_core
Document the usage of get_host_property on a guest object
Correct ARM netboot paths
Include bootloader in test name
Should include -pv too?
console= repetition for Jessie onwards.
Wait for up to an hour for the install. I'd seen timeouts right at
the end of the install with the previous value
---
Osstest/TestSupport.pm | 3 +
make-distros-flight | 138 +++++++++++++++++++++++++++++++++++++
sg-run-job | 11 +++
ts-debian-di-install | 180 +++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 332 insertions(+)
create mode 100755 make-distros-flight
create mode 100755 ts-debian-di-install
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 1cace4f..3a7a535 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -931,8 +931,11 @@ sub propname_massage ($) {
return $prop;
}
+# It is fine to call this on a guest object too, in which case it will
+# always return $defval.
sub get_host_property ($$;$) {
my ($ho, $prop, $defval) = @_;
+ return $defval unless $ho->{Properties};
my $val = $ho->{Properties}{propname_massage($prop)};
return defined($val) ? $val : $defval;
}
diff --git a/make-distros-flight b/make-distros-flight
new file mode 100755
index 0000000..bdca7d1
--- /dev/null
+++ b/make-distros-flight
@@ -0,0 +1,138 @@
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+
+set -e
+
+branch=$1
+xenbranch=$2
+blessing=$3
+buildflight=$4
+
+flight=`./cs-flight-create $blessing $branch`
+
+. cri-common
+. ap-common
+. mfi-common
+
+defsuite=`getconfig DebianSuite`
+defguestsuite=`getconfig GuestDebianSuite`
+
+case $branch in
+ distros-debian-*) debian_suite=${branch#distros-debian-} ;;
+ *) echo $branch >&2; exit 1 ;;
+esac
+
+job_create_build_filter_callback () {
+ local job=$1; shift
+
+ case "$job" in
+ build-*-libvirt) return 1;;
+ esac
+ case " $* " in
+ *" enable_xsm=true "*) return 1;;
+ esac
+ return 0
+}
+
+if [ x$buildflight = x ]; then
+
+ WANT_XEND=false REVISION_LINUX_OLD=disable
+
+ create_build_jobs
+
+else
+
+ bfi=$buildflight.
+
+fi
+
+job_create_test_filter_callback () {
+ if [ "$xenarch" = "i386" ]; then return 1; fi
+ return 0
+}
+
+test_matrix_branch_filter_callback () {
+ :
+}
+
+test_do_one_netboot () {
+ local diurl=
+
+ if [ x$gsuite = xsnapshot ] ; then
+ diurl="http://d-i.debian.org/daily-images/$domU/daily/netboot"
+ gsuite=sid
+ gver=daily
+ else
+ local mirror="http://`getconfig DebianMirrorHost`/`getconfig DebianMirrorSubpath`"
+ diurl="$mirror/dists/$gsuite/main/installer-$domU/current/images/netboot"
+ gver=$gsuite
+ fi
+
+ case ${domU}_${gsuite} in
+ armhf_squeeze) return;; # No armhf in Squeeze
+ armhf_wheezy) return;; # No armhf guest support in Wheezy
+ *) ;;
+ esac
+
+ case $domU in
+ i386|amd64)
+ diurl="$diurl/xen";;
+ arm64)
+ diurl="$diurl/debian-installer/arm64";;
+ esac
+
+ job_create_test \
+ test-$xenarch$kern-$dom0arch-$domU-$gver-netboot-pygrub \
+ test-debian-di xl $xenarch $dom0arch \
+ kernbuildjob=${bfi}build-$dom0arch-$kernbuild \
+ debian_arch=$domU \
+ debian_suite=$gsuite \
+ debian_method=netboot \
+ debian_netboot_kernel="$diurl/vmlinuz" \
+ debian_netboot_ramdisk="$diurl/initrd.gz" \
+ all_hostflags=$most_hostflags
+}
+
+test_matrix_do_one () {
+ case ${xenarch} in
+ amd64) domUarches="amd64 i386";;
+ armhf) domUarches="armhf";;
+ esac
+
+ for domU in $domUarches ; do
+
+ if [ -n $debian_suite ] ; then
+ gsuite=$debian_suite
+
+ test_do_one_netboot
+
+ fi
+
+ done
+}
+
+test_matrix_iterate
+
+echo $flight
+
+# Local variables:
+# mode: sh
+# sh-basic-offset: 2
+# indent-tabs-mode: nil
+# End:
diff --git a/sg-run-job b/sg-run-job
index d53fd83..bba52ed 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -258,6 +258,17 @@ proc run-job/test-debian {} {
test-guest debian
}
+proc install-guest-debian-di {} {
+ run-ts . = ts-debian-di-install
+ run-ts . = ts-guest-start + debian
+}
+
+proc need-hosts/test-debian-di {} { return host }
+proc run-job/test-debian-di {} {
+ install-guest-debian-di
+ test-guest debian
+}
+
proc need-hosts/test-freebsd {} { return host }
proc run-job/test-freebsd {} {
run-ts . = ts-freebsd-install
diff --git a/ts-debian-di-install b/ts-debian-di-install
new file mode 100755
index 0000000..08019a9
--- /dev/null
+++ b/ts-debian-di-install
@@ -0,0 +1,180 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Configured via the following runvars (which are mandatory unless
+# otherwise noted):
+#
+# - <gident>_arch:
+# Debian arch to install.
+# - <gident>_method:
+# Install method, currently only "netboot".
+#
+# For method="netboot":
+#
+# - <gident>_netboot_kernel:
+# URL of the kernel to boot
+# - <gident>_netboot_ramdisk:
+# URL of the ramdisk to boot
+#
+# If neither kernel nor ramdisk are specified then the current
+# TftpDiVersion of d-i will be used, and the runvars will be set to
+# the file path used.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost,$gn) = @ARGV;
+$whhost ||= 'host';
+$gn ||= 'debian';
+
+our $ho= selecthost($whhost);
+
+our $ram_mb= 512;
+our $disk_mb= 10000;
+
+our $guesthost= "$gn.guest.osstest";
+our $gho;
+
+sub prep () {
+ target_install_packages_norec($ho, qw(lvm2));
+
+ $gho= prepareguest($ho, $gn, $guesthost, 22,
+ $disk_mb, 40);
+
+ prepareguest_part_lvmdisk($ho, $gho, $disk_mb);
+
+ target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+}
+
+sub setup_netboot($$$)
+{
+ my ($didir, $arch, $suite) = @_;
+
+ my ($kernel,$ramdisk);
+
+ if ( $r{ "$gho->{Guest}_netboot_kernel" } &&
+ $r{ "$gho->{Guest}_netboot_ramdisk" } ) {
+ target_fetchurl($ho, $r{ "$gho->{Guest}_netboot_kernel" },
+ "$didir/kernel_${suite}_${arch}");
+ target_fetchurl($ho, $r{ "$gho->{Guest}_netboot_ramdisk" },
+ "$didir/ramdisk_${suite}_${arch}");
+ } else {
+ # Both or neither must be specified
+ die if $r{ "$gho->{Guest}_netboot_kernel" }
+ || $r{ "$gho->{Guest}_netboot_ramdisk" };
+
+ my $di_path = $c{TftpPath}.'/'.$ho->{Tftp}{DiBase}.'/'.${arch}.'/'.$c{TftpDiVersion}.'-'.$ho->{Suite};
+
+ my $suffix = '';
+ $suffix .= "-xen" if ${arch} =~ m/amd64|i386/;
+ $kernel = "$di_path/vmlinuz$suffix";
+ $ramdisk = "$di_path/initrd.gz$suffix";
+
+ target_putfile_root($ho, 60, $kernel, "$didir/kernel_${suite}_${arch}");
+ target_putfile_root($ho, 60, $ramdisk, "$didir/ramdisk_${suite}_${arch}");
+
+ store_runvar("$gho->{Guest}_netboot_kernel", $kernel);
+ store_runvar("$gho->{Guest}_netboot_ramdisk", $ramdisk);
+ }
+
+ return <<END;
+kernel = "$didir/kernel_${suite}_${arch}"
+ramdisk = "$didir/ramdisk_${suite}_${arch}"
+END
+}
+
+sub ginstall () {
+ my $arch= $r{"$gho->{Guest}_arch"};
+ my $method= $r{"$gho->{Guest}_method"};
+
+ my $tmpdir= "/root/$flight-$job-di";
+ target_cmd_root($ho, <<END);
+rm -rf $tmpdir
+mkdir $tmpdir
+END
+
+ my ($method_cfg, $ps_url, $extra_disk);
+
+ if ( $method eq "netboot" )
+ {
+ my $suite= $r{"$gho->{Guest}_suite"};
+ logm("$method $suite/$arch");
+
+ $method_cfg = setup_netboot($tmpdir, $arch, $suite);
+
+ $ps_url = preseed_create_guest($gho, '', Suite=>$suite);
+
+ $extra_disk = "";
+ }
+ else
+ {
+ die "$method";
+ }
+
+ my @cmdline = ();
+ push @cmdline, "debian-installer/exit/always_halt=true";
+ push @cmdline, "domain=$c{TestHostDomain}";
+ push @cmdline, "console=hvc0";
+ push @cmdline, di_installcmdline_core($gho, $ps_url);
+ push @cmdline, "--";
+ # See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007 for
+ # why this is repeated.
+ push @cmdline, "console=hvc0";
+
+ my $cmdline = join(" ", @cmdline);
+
+ my %install_xopts = (
+ OnPowerOff => "preserve"
+ );
+
+ prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
+$method_cfg
+extra = "$cmdline"
+#
+disk = [
+ $extra_disk 'phy:$gho->{Lvdev},xvda,w'
+ ]
+END
+
+ guest_create($gho);
+
+ guest_checkrunning($ho, $gho) or die "$gho->{Name} not running";
+
+ guest_await_shutdown($ho,$gho,3600);
+ guest_destroy($gho);
+
+ my $blcfg = <<END;
+bootloader = "pygrub"
+END
+
+ prepareguest_part_xencfg($ho, $gho, $ram_mb, {}, <<END);
+$blcfg
+#
+disk = [
+ 'phy:$gho->{Lvdev},xvda,w'
+ ]
+END
+ return;
+}
+
+prep();
+ginstall();
--
2.1.4
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH OSSTEST v8 04/14] distros: support booting Debian PV (d-i installed) guests with pvgrub.
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
` (2 preceding siblings ...)
2015-07-08 12:30 ` [PATCH OSSTEST v8 03/14] distros: add support for installing Debian PV guests via d-i, flight and jobs Ian Campbell
@ 2015-07-08 12:30 ` Ian Campbell
2015-07-08 12:30 ` [PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too Ian Campbell
` (10 subsequent siblings)
14 siblings, 0 replies; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 12:30 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
This requires the use of the pv-grub-menu package which is in Jessie
onwards. (it is in wheezy-backports which is the subject of a
subsequent patch).
The bootloader to use is specified via a runvar {Guest}_bootloader.
Adjust make-distros-flight to use pvgrub for some subset of i386 and
amd64 guests to get coverage.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v8: Added comment regarding new runvar. Since this was inspired by
Ian's comment on "distros: support PV guest install from Debian
netinst media I have retained the ack.
v7: Move definition of $extra_packages variable to here which is its
first usage.
Use {Guest}_suite not {Guest}_dist as runvar to choose version.
v3: Define and use arch_debian2xen and arch_xen2debian
Avoid pv-grub-x86_64.gz on i386 dom0, we don't built it there.
Fiddle with py vs pv grub stripy a bit.
---
Osstest.pm | 7 +++++++
Osstest/Debian.pm | 4 +++-
make-distros-flight | 20 +++++++++++++++++++-
ts-debian-di-install | 18 ++++++++++++++----
4 files changed, 43 insertions(+), 6 deletions(-)
diff --git a/Osstest.pm b/Osstest.pm
index 6535401..8f97dd2 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -39,6 +39,7 @@ BEGIN {
db_begin_work db_prepare
ensuredir get_filecontents_core_quiet system_checked
nonempty visible_undef show_abs_time
+ %arch_debian2xen %arch_xen2debian
);
%EXPORT_TAGS = ( );
@@ -54,6 +55,12 @@ scalar *main::DEBUG;
# declaration prevents `Name "main::DEBUG" used only once'
# scalar prevents `useless use of a variable in void context'
+our %arch_debian2xen = qw(i386 x86_32
+ amd64 x86_64
+ armhf armhf);
+our %arch_xen2debian;
+$arch_xen2debian{$arch_debian2xen{$_}} = $_ foreach keys %arch_debian2xen;
+
#---------- static default config settings ----------
our %c = qw(
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 2d49ff8..1edf49f 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -867,7 +867,9 @@ sub preseed_create_guest ($$;@) {
my $suite= $xopts{Suite} || $c{DebianSuite};
- my $preseed_file= preseed_base($ho, $suite, $sfx, '', %xopts);
+ my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
+
+ my $preseed_file= preseed_base($ho, $suite, $sfx, $extra_packages, %xopts);
$preseed_file.= (<<END);
d-i partman-auto/method string regular
d-i partman-auto/choose_recipe \\
diff --git a/make-distros-flight b/make-distros-flight
index bdca7d1..c19e3ba 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -90,6 +90,11 @@ test_do_one_netboot () {
*) ;;
esac
+ stripy bootloader pvgrub pygrub \
+ "$xenarch" "amd64" \
+ "$dom0arch" "i386" \
+ "$domU" "amd64" \
+
case $domU in
i386|amd64)
diurl="$diurl/xen";;
@@ -97,8 +102,20 @@ test_do_one_netboot () {
diurl="$diurl/debian-installer/arm64";;
esac
+ case ${dom0arch}_${domU}_${gsuite} in
+ arm*_arm*_*) bootloader="pygrub";; # no pvgrub for arm
+
+ # Needs a menu.lst, not present in Squeeze+ due to switch to grub2,
+ # workedaround in Jessie+ with pv-grub-menu package.
+ *_squeeze) bootloader="pygrub";;
+ *_wheezy) bootloader="pygrub";;
+
+ # pv-grub-x86_64.gz is not built by 32-bit dom0 userspace build.
+ i386_amd64_*) bootloader="pygrub";;
+ esac
+
job_create_test \
- test-$xenarch$kern-$dom0arch-$domU-$gver-netboot-pygrub \
+ test-$xenarch$kern-$dom0arch-$domU-$gver-netboot-$bootloader \
test-debian-di xl $xenarch $dom0arch \
kernbuildjob=${bfi}build-$dom0arch-$kernbuild \
debian_arch=$domU \
@@ -106,6 +123,7 @@ test_do_one_netboot () {
debian_method=netboot \
debian_netboot_kernel="$diurl/vmlinuz" \
debian_netboot_ramdisk="$diurl/initrd.gz" \
+ debian_bootloader=$bootloader \
all_hostflags=$most_hostflags
}
diff --git a/ts-debian-di-install b/ts-debian-di-install
index 08019a9..a59194a 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -22,13 +22,16 @@
# Debian arch to install.
# - <gident>_method:
# Install method, currently only "netboot".
+# - <gident>_bootloader:
+# The PV bootloader to use when booting the guest. One of
+# "pvgrub" or "pygrub". Default is "pygrub".
#
# For method="netboot":
#
# - <gident>_netboot_kernel:
-# URL of the kernel to boot
+# URL of the kernel to boot.
# - <gident>_netboot_ramdisk:
-# URL of the ramdisk to boot
+# URL of the ramdisk to boot.
#
# If neither kernel nor ramdisk are specified then the current
# TftpDiVersion of d-i will be used, and the runvars will be set to
@@ -107,6 +110,8 @@ sub ginstall () {
my $method= $r{"$gho->{Guest}_method"};
my $tmpdir= "/root/$flight-$job-di";
+ my $bl= $r{"$gho->{Guest}_bootloader"};
+
target_cmd_root($ho, <<END);
rm -rf $tmpdir
mkdir $tmpdir
@@ -121,7 +126,7 @@ END
$method_cfg = setup_netboot($tmpdir, $arch, $suite);
- $ps_url = preseed_create_guest($gho, '', Suite=>$suite);
+ $ps_url = preseed_create_guest($gho, '', Suite=>$suite, PvMenuLst=>($bl eq "pvgrub"));
$extra_disk = "";
}
@@ -162,7 +167,12 @@ END
guest_await_shutdown($ho,$gho,3600);
guest_destroy($gho);
- my $blcfg = <<END;
+ my $xenarch = $arch_debian2xen{$arch};
+ my $pvgrub = "/usr/local/lib/xen/boot/pv-grub-$xenarch.gz";
+ my $blcfg = $bl eq "pvgrub" ? <<END : <<END;
+kernel = "$pvgrub"
+extra = "(hd0,0)/boot/grub/menu.lst"
+END
bootloader = "pygrub"
END
--
2.1.4
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too.
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
` (3 preceding siblings ...)
2015-07-08 12:30 ` [PATCH OSSTEST v8 04/14] distros: support booting Debian PV (d-i installed) guests with pvgrub Ian Campbell
@ 2015-07-08 12:30 ` Ian Campbell
2015-07-08 13:32 ` Ian Jackson
2015-07-08 12:30 ` [PATCH OSSTEST v8 06/14] Test pygrub and pvgrub on the regular flights Ian Campbell
` (9 subsequent siblings)
14 siblings, 1 reply; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 12:30 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
This requires us to install pv-grub-menu from backports, which we do
using a late_command.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v8:
- Use a heredoc for sources.list additions, since this was suggested
by Ian I would have retained the ack apart from the second change.
- Dropped unused $cd argument to setup_netinst. (Noticed when I came
to document <gident>_cd and found it wasn't used!)
v7:
- Remove vestigial attempts to enable -backports via d-i preseeding.
v3:
- Remove spurious () from <<(END) (and the prexisting "" too)
- Remove $xopts{EnableBackports} and automatically handle the need to add
backports in preseed_base.
- Install via late_command not apt-setup, since the former has
issues, hence subject drops "attempt to..."
---
Osstest/Debian.pm | 40 ++++++++++++++++++++++++---
make-distros-flight | 58 +++++++++++++++++++++++++++++++++++++--
ts-debian-di-install | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++-
3 files changed, 168 insertions(+), 6 deletions(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 1edf49f..7c94b6c 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -777,8 +777,6 @@ sub preseed_base ($$$$;@) {
preseed_hook_overlay($ho, $sfx, 'overlay', 'overlay.tar');
my $preseed = <<"END";
-d-i mirror/suite string $suite
-
d-i debian-installer/locale string en_GB
d-i console-keymaps-at/keymap select gb
d-i keyboard-configuration/xkb-keymap string en_GB
@@ -854,6 +852,11 @@ END
d-i clock-setup/ntp-server string $ntpserver
END
+ # For CDROM the suite is part of the image
+ $preseed .= <<END unless $xopts{CDROM};
+d-i mirror/suite string $suite
+END
+
$preseed .= <<"END";
### END OF DEBIAN PRESEED BASE
@@ -867,7 +870,38 @@ sub preseed_create_guest ($$;@) {
my $suite= $xopts{Suite} || $c{DebianSuite};
- my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
+ my $extra_packages = "";
+ if ($xopts{PvMenuLst}) {
+ if ($suite =~ m/wheezy/) {
+ # pv-grub-menu/wheezy-backports + using apt-setup to add
+ # backports results in iproute, ifupdown and
+ # isc-dhcp-client getting removed because tasksel's
+ # invocation of apt-get install somehow decides the
+ # iproute2 from wheezy-backports is a thing it wants to
+ # install. So instead lets fake it with a late command...
+ #
+ # This also has the bonus of working round an issue with
+ # 1.2.1~bpo70+1 which created an invalid menu.lst using
+ # "root(/dev/xvda,0)" which pvgrub cannot parse because
+ # the Grub device.map isn't present at pkgsel/include time
+ # but it is by late_command time. This was fixed by
+ # version 1.3 which is in Jessie onwards.
+ preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+cat <<EOF >>/target/etc/apt/sources.list
+
+\# $suite backports
+deb http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
+EOF
+in-target apt-get update
+in-target apt-get install -y -t wheezy-backports pv-grub-menu
+END
+ } else {
+ $extra_packages = "pv-grub-menu";
+ }
+ }
my $preseed_file= preseed_base($ho, $suite, $sfx, $extra_packages, %xopts);
$preseed_file.= (<<END);
diff --git a/make-distros-flight b/make-distros-flight
index c19e3ba..49f4b60 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -106,9 +106,9 @@ test_do_one_netboot () {
arm*_arm*_*) bootloader="pygrub";; # no pvgrub for arm
# Needs a menu.lst, not present in Squeeze+ due to switch to grub2,
- # workedaround in Jessie+ with pv-grub-menu package.
+ # workedaround in Wheezy+ with pv-grub-menu package (backports in Wheezy,
+ # in Jessie+ main).
*_squeeze) bootloader="pygrub";;
- *_wheezy) bootloader="pygrub";;
# pv-grub-x86_64.gz is not built by 32-bit dom0 userspace build.
i386_amd64_*) bootloader="pygrub";;
@@ -127,6 +127,48 @@ test_do_one_netboot () {
all_hostflags=$most_hostflags
}
+test_do_one_netinst () {
+ local path_arch
+ case $domU in
+ amd64|i386) path_arch="multi-arch"; file_arch="amd64-i386";;
+ *) path_arch="$domU"; file_arch="$domU";;
+ esac
+ case $domU in
+ amd64) iso_path="/install.amd/xen";;
+ i386) iso_path="/install.386/xen";;
+ *) iso_path="/install.$domU";;
+ esac
+
+ local cdurl=
+ case $cd in
+ current)
+ cdurl="http://cdimage.debian.org/debian-cd/current/${path_arch}/jigdo-cd"
+ ;;
+ weekly)
+ cdurl="http://cdimage.debian.org/cdimage/weekly-builds/${path_arch}/jigdo-cd"
+ ;;
+ *)
+ echo "cd $cd?"
+ exit 1
+ ;;
+ esac
+
+ # Always pygrub since no pv-grub-menu on CD
+ job_create_test \
+ test-$xenarch$kern-$dom0arch-$domU-$cd-netinst-pygrub \
+ test-debian-di xl $xenarch $dom0arch \
+ kernbuildjob=${bfi}build-$dom0arch-$kernbuild \
+ debian_arch=$domU \
+ debian_method=netinst \
+ debian_netinst_baseurl=$cdurl \
+ debian_netinst_filere="debian-.*-${file_arch}-netinst" \
+ debian_netinst_kernel="${iso_path}/vmlinuz" \
+ debian_netinst_ramdisk="${iso_path}/initrd.gz" \
+ debian_bootloader=pygrub \
+ all_hostflags=$most_hostflags
+
+}
+
test_matrix_do_one () {
case ${xenarch} in
amd64) domUarches="amd64 i386";;
@@ -142,6 +184,18 @@ test_matrix_do_one () {
fi
+ if [ $debian_suite = "snapshot" ]; then
+ gsuite=sid
+ for cd in current weekly ; do
+ case ${domU}_${gsuite} in
+ armhf_*) continue;; # No iso targets for armhf
+ *) ;;
+ esac
+
+ test_do_one_netinst
+
+ done
+ fi
done
}
diff --git a/ts-debian-di-install b/ts-debian-di-install
index a59194a..1a7e1d0 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -21,7 +21,7 @@
# - <gident>_arch:
# Debian arch to install.
# - <gident>_method:
-# Install method, currently only "netboot".
+# Install method, one of "netboot" or "netinst".
# - <gident>_bootloader:
# The PV bootloader to use when booting the guest. One of
# "pvgrub" or "pygrub". Default is "pygrub".
@@ -36,6 +36,18 @@
# If neither kernel nor ramdisk are specified then the current
# TftpDiVersion of d-i will be used, and the runvars will be set to
# the file path used.
+#
+# For method="netinst"
+#
+# - <gident>_netinst_baseurl:
+# Base URL of directory containing the netinst jigdo images.
+# - <gident>_netinst_filere:
+# Regular expression to match actual file within baseurl to use
+# (without .jigdo suffix)
+# - <gident>_netinst_kernel:
+# Path to kernel within the netinst image.
+# - <gident>_netinst_ramdisk
+# Path to ramdisk within the netinst image.
use strict qw(vars);
use DBI;
@@ -68,6 +80,60 @@ sub prep () {
target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
}
+sub setup_netinst($$)
+{
+ my ($didir, $arch) = @_;
+
+ target_install_packages($ho, qw(jigdo-file));
+
+ my $baseurl = $r{ "$gho->{Guest}_netinst_baseurl" };
+ my $filere = $r{ "$gho->{Guest}_netinst_filere" };
+ my $kernel = $r{ "$gho->{Guest}_netinst_kernel" };
+ my $ramdisk = $r{ "$gho->{Guest}_netinst_ramdisk" };
+
+ die "params" unless $baseurl && $filere && $kernel && $ramdisk;
+
+ my $filebase;
+
+ # Use the MD5SUMs file as an index
+ logm("Fetch index from $baseurl/MD5SUMS");
+ my $idx = target_cmd_output_root($ho, "wget --quiet -O - $baseurl/MD5SUMS");
+ foreach (split /\n/, $idx) {
+ m/^[0-9a-f]{32} ($filere)\.iso$/ or next;
+ $filebase = $1;
+ last;
+ }
+
+ die unless $filebase;
+
+ logm("Downloading $baseurl/$filebase.jigdo");
+ my $netinst_jigdo = "$baseurl/$filebase.jigdo";
+ # Jigdo uses wget internally, and so obeys $http_proxy. This seems
+ # simpler than the advice at
+ # https://www.debian.org/CD/jigdo-cd/#faq which is to edit
+ # ~/.jigdo-lite and change the wgetOpts setting.
+ my $useproxy = $c{DebianMirrorProxy} // $c{HttpProxy} // '';
+ my $cmd = '';
+ $cmd .= <<END if $useproxy;
+ export http_proxy=$useproxy
+END
+ $cmd .= <<END;
+ cd $didir && jigdo-lite --noask $netinst_jigdo
+END
+ # Jigdo seems to use /etc/apt/sources.list or something, so this
+ # just works using the already configured mirror without
+ # additional configuration, which is good because there doesn't
+ # seem to be any support for such things, at least in Squeeze.
+ target_cmd_root($ho, $cmd, 3600);
+ store_runvar("$gho->{Guest}_netinst_jigdo", $netinst_jigdo);
+
+ return (<<END, "\"file:$didir/$filebase.iso,xvdd:cdrom,r\",");
+bootloader = "pygrub"
+bootloader_args = ["--kernel=$kernel",
+ "--ramdisk=$ramdisk"]
+END
+}
+
sub setup_netboot($$$)
{
my ($didir, $arch, $suite) = @_;
@@ -130,6 +196,14 @@ END
$extra_disk = "";
}
+ elsif ($method eq "netinst" )
+ {
+ logm("$method $arch");
+
+ ($method_cfg,$extra_disk) = setup_netinst($tmpdir, $arch);
+
+ $ps_url = preseed_create_guest($gho, '', CDROM=>1);
+ }
else
{
die "$method";
--
2.1.4
^ permalink raw reply related [flat|nested] 27+ messages in thread* Re: [PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too.
2015-07-08 12:30 ` [PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too Ian Campbell
@ 2015-07-08 13:32 ` Ian Jackson
2015-07-08 13:51 ` Ian Campbell
0 siblings, 1 reply; 27+ messages in thread
From: Ian Jackson @ 2015-07-08 13:32 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too."):
> This requires us to install pv-grub-menu from backports, which we do
> using a late_command.
...
> +cat <<EOF >>/target/etc/apt/sources.list
> +
> +\# $suite backports
What is that \ doing ? (Here documents do not - in either Perl no
shell - elide #-comments.)
Perhaps I have misunderstood.
Ian.
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too.
2015-07-08 13:32 ` Ian Jackson
@ 2015-07-08 13:51 ` Ian Campbell
2015-07-08 13:58 ` Ian Jackson
0 siblings, 1 reply; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 13:51 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Wed, 2015-07-08 at 14:32 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too."):
> > This requires us to install pv-grub-menu from backports, which we do
> > using a late_command.
> ...
> > +cat <<EOF >>/target/etc/apt/sources.list
> > +
> > +\# $suite backports
>
> What is that \ doing ? (Here documents do not - in either Perl no
> shell - elide #-comments.)
>
> Perhaps I have misunderstood.
Nope, that would be me I think.
I couldn't remember so I err'd on the side of caution intending to
check, but then resent before my test got as far as Wheezy.
Ian.
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too.
2015-07-08 13:51 ` Ian Campbell
@ 2015-07-08 13:58 ` Ian Jackson
2015-07-08 14:55 ` Ian Campbell
0 siblings, 1 reply; 27+ messages in thread
From: Ian Jackson @ 2015-07-08 13:58 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("Re: [PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too."):
> On Wed, 2015-07-08 at 14:32 +0100, Ian Jackson wrote:
> > Ian Campbell writes ("[PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too."):
> > > +\# $suite backports
> >
> > What is that \ doing ? (Here documents do not - in either Perl no
> > shell - elide #-comments.)
> >
> > Perhaps I have misunderstood.
>
> Nope, that would be me I think.
>
> I couldn't remember so I err'd on the side of caution intending to
> check, but then resent before my test got as far as Wheezy.
OK, well, if you remove that, then
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian.
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too.
2015-07-08 13:58 ` Ian Jackson
@ 2015-07-08 14:55 ` Ian Campbell
2015-07-08 15:25 ` Ian Jackson
0 siblings, 1 reply; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 14:55 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Wed, 2015-07-08 at 14:58 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too."):
> > On Wed, 2015-07-08 at 14:32 +0100, Ian Jackson wrote:
> > > Ian Campbell writes ("[PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too."):
> > > > +\# $suite backports
> > >
> > > What is that \ doing ? (Here documents do not - in either Perl no
> > > shell - elide #-comments.)
> > >
> > > Perhaps I have misunderstood.
> >
> > Nope, that would be me I think.
> >
> > I couldn't remember so I err'd on the side of caution intending to
> > check, but then resent before my test got as far as Wheezy.
>
> OK, well, if you remove that,
Will do.
> then
>
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Thanks. I think that makes all but the last patch acked.
The last one needs some more thought about the approach, as discussed on
v7, but perhaps this could go in without that one? The new additions to
the main xen flights should work without, it's just the debian-* flights
which will be run here in Cambridge which might suffer.
Ian.
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too.
2015-07-08 14:55 ` Ian Campbell
@ 2015-07-08 15:25 ` Ian Jackson
2015-07-08 15:51 ` Ian Campbell
0 siblings, 1 reply; 27+ messages in thread
From: Ian Jackson @ 2015-07-08 15:25 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("Re: [PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too."):
> On Wed, 2015-07-08 at 14:58 +0100, Ian Jackson wrote:
> > Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
>
> Thanks. I think that makes all but the last patch acked.
You mean
"make-distros-flight: Use ftp.debian.org directly"
?
I decided that it was justified for the reasons given in your commit
message, and have acked it already.
Ian.
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too.
2015-07-08 15:25 ` Ian Jackson
@ 2015-07-08 15:51 ` Ian Campbell
0 siblings, 0 replies; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 15:51 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Wed, 2015-07-08 at 16:25 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too."):
> > On Wed, 2015-07-08 at 14:58 +0100, Ian Jackson wrote:
> > > Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
> >
> > Thanks. I think that makes all but the last patch acked.
>
> You mean
> "make-distros-flight: Use ftp.debian.org directly"
> ?
Yes.
> I decided that it was justified for the reasons given in your commit
> message, and have acked it already.
So you have, sorry for the noise.
Ian.
^ permalink raw reply [flat|nested] 27+ messages in thread
* [PATCH OSSTEST v8 06/14] Test pygrub and pvgrub on the regular flights
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
` (4 preceding siblings ...)
2015-07-08 12:30 ` [PATCH OSSTEST v8 05/14] distros: Support pvgrub for Wheezy too Ian Campbell
@ 2015-07-08 12:30 ` Ian Campbell
2015-07-08 12:30 ` [PATCH OSSTEST v8 07/14] distros: add branch infrastructure Ian Campbell
` (8 subsequent siblings)
14 siblings, 0 replies; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 12:30 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Since we now have the ability to test these drop one of each of
pygrub, pvgrub-32 and pvgrub-64 into the standard flights. Omitting
the {Guest}_diver runvar causes ts-debian-di-install to use the d-i
images in the location configured via TftpDiVersion, so they are
Version Controlled along with the d-i version used for the host.
This adds three new jobs:
+test-amd64-amd64-amd64-pvgrub all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
+test-amd64-amd64-amd64-pvgrub arch amd64
+test-amd64-amd64-amd64-pvgrub buildjob build-amd64
+test-amd64-amd64-amd64-pvgrub debian_arch amd64
+test-amd64-amd64-amd64-pvgrub debian_bootloader pvgrub
+test-amd64-amd64-amd64-pvgrub debian_method netboot
+test-amd64-amd64-amd64-pvgrub debian_suite wheezy
+test-amd64-amd64-amd64-pvgrub kernbuildjob build-amd64-pvops
+test-amd64-amd64-amd64-pvgrub kernkind pvops
+test-amd64-amd64-amd64-pvgrub toolstack xl
+test-amd64-amd64-amd64-pvgrub xenbuildjob build-amd64
+test-amd64-amd64-i386-pvgrub all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
+test-amd64-amd64-i386-pvgrub arch amd64
+test-amd64-amd64-i386-pvgrub buildjob build-amd64
+test-amd64-amd64-i386-pvgrub debian_arch i386
+test-amd64-amd64-i386-pvgrub debian_bootloader pvgrub
+test-amd64-amd64-i386-pvgrub debian_method netboot
+test-amd64-amd64-i386-pvgrub debian_suite wheezy
+test-amd64-amd64-i386-pvgrub kernbuildjob build-amd64-pvops
+test-amd64-amd64-i386-pvgrub kernkind pvops
+test-amd64-amd64-i386-pvgrub toolstack xl
+test-amd64-amd64-i386-pvgrub xenbuildjob build-amd64
+test-amd64-amd64-pygrub all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
+test-amd64-amd64-pygrub arch amd64
+test-amd64-amd64-pygrub buildjob build-amd64
+test-amd64-amd64-pygrub debian_arch amd64
+test-amd64-amd64-pygrub debian_bootloader pygrub
+test-amd64-amd64-pygrub debian_method netboot
+test-amd64-amd64-pygrub debian_suite wheezy
+test-amd64-amd64-pygrub kernbuildjob build-amd64-pvops
+test-amd64-amd64-pygrub kernkind pvops
+test-amd64-amd64-pygrub toolstack xl
+test-amd64-amd64-pygrub xenbuildjob build-amd64
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v7: Use {Guest}_suite not {Guest}_dist as runvar to choose version.
Refreshed runvars i ncommit message.
v3: added runvar details
---
make-flight | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/make-flight b/make-flight
index de8393a..725da26 100755
--- a/make-flight
+++ b/make-flight
@@ -325,6 +325,42 @@ do_passthrough_tests () {
done
}
+do_pygrub_tests () {
+ if [ $xenarch != amd64 -o $dom0arch != amd64 -o "$kern" != "" ]; then
+ return
+ fi
+
+ job_create_test test-$xenarch$kern-$dom0arch-pygrub \
+ test-debian-di xl $xenarch $dom0arch \
+ debian_arch=amd64 \
+ debian_suite=$guestsuite \
+ debian_method=netboot \
+ debian_bootloader=pygrub \
+ all_hostflags=$most_hostflags
+}
+
+do_pvgrub_tests () {
+ if [ $xenarch != amd64 -o $dom0arch != amd64 -o "$kern" != "" ]; then
+ return
+ fi
+
+ job_create_test test-$xenarch$kern-$dom0arch-amd64-pvgrub \
+ test-debian-di xl $xenarch $dom0arch \
+ debian_arch=amd64 \
+ debian_suite=$guestsuite \
+ debian_method=netboot \
+ debian_bootloader=pvgrub \
+ all_hostflags=$most_hostflags \
+
+ job_create_test test-$xenarch$kern-$dom0arch-i386-pvgrub \
+ test-debian-di xl $xenarch $dom0arch \
+ debian_arch=i386 \
+ debian_suite=$guestsuite \
+ debian_method=netboot \
+ debian_bootloader=pvgrub \
+ all_hostflags=$most_hostflags
+}
+
do_pv_debian_tests () {
xsms=$(xenbranch_xsm_variants)
@@ -473,6 +509,9 @@ test_matrix_do_one () {
fi
#do_passthrough_tests
+
+ do_pygrub_tests
+ do_pvgrub_tests
}
test_matrix_iterate
--
2.1.4
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH OSSTEST v8 07/14] distros: add branch infrastructure
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
` (5 preceding siblings ...)
2015-07-08 12:30 ` [PATCH OSSTEST v8 06/14] Test pygrub and pvgrub on the regular flights Ian Campbell
@ 2015-07-08 12:30 ` Ian Campbell
2015-07-08 12:30 ` [PATCH OSSTEST v8 08/14] crontab-cambridge: Use hard tabs for alignment Ian Campbell
` (7 subsequent siblings)
14 siblings, 0 replies; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 12:30 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Since the distro nightlies are not version controlled we cannot use
the usual mechanisms for detecting regressions. Special case things
appropriately. We use an OLD_REVISION of "flight-NNN" to signify that
the old revision is another flight and not a tree revision.
A grep over $NEW_REVISION needed adjusting since NEW_REVISION is empty
in this mode, leading to "grep <filename>" which hangs waiting for
stdin.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v7:
Handle empty $NEW_REVISION by quoting it instead of a needless test -n
Switch to flight-per-suite model
v3:
Handle within cr-daily-branch, since ap-fetch-version* don't make sense for
a branch such as this.
---
cr-daily-branch | 36 ++++++++++++++++++++++++++++--------
cri-common | 1 +
2 files changed, 29 insertions(+), 8 deletions(-)
diff --git a/cr-daily-branch b/cr-daily-branch
index 34b6d2b..1fcfd9d 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -68,23 +68,34 @@ fetch_version () {
printf '%s\n' "$fetch_version_result"
}
-treeurl=`./ap-print-url $branch`
+case $branch in
+ distros-*)
+ treeurl=none;;
+ *)
+ treeurl=`./ap-print-url $branch`;;
+esac
force_baseline=false
skipidentical=true
wantpush=$OSSTEST_PUSH
-if [ "x$OLD_REVISION" = x ]; then
- OLD_REVISION="`./ap-fetch-version-old $branch`"
- export OLD_REVISION
-fi
-
check_tested () {
./sg-check-tested --debug --branch=$branch \
--blessings=${DAILY_BRANCH_TESTED_BLESSING:-$OSSTEST_BLESSING} \
"$@"
}
+if [ "x$OLD_REVISION" = x ]; then
+ case $branch in
+ distros-*)
+ OSSTEST_NO_BASELINE=y
+ OLD_REVISION=flight-`check_tested`
+ ;;
+ *) OLD_REVISION="`./ap-fetch-version-old $branch`";;
+ esac
+ export OLD_REVISION
+fi
+
if [ "x$OSSTEST_NO_BASELINE" != xy ] ; then
testedflight=`check_tested --revision-$tree="$OLD_REVISION"`
@@ -227,6 +238,11 @@ if [ "x$OLD_REVISION" = xdetermine-late ]; then
OLD_REVISION="`./ap-fetch-version-baseline-late $branch $NEW_REVISION`"
fi
+case $branch in
+distros-*) makeflight=./make-distros-flight ;;
+*) makeflight=./make-flight ;;
+esac
+
if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
wantpush=false
for checkbranch in x $BRANCHES_ALWAYS; do
@@ -241,7 +257,7 @@ if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
fi
$DAILY_BRANCH_PREMAKE_HOOK
-flight=`./make-flight $branch $xenbranch $OSSTEST_BLESSING "$@"`
+flight=`$makeflight $branch $xenbranch $OSSTEST_BLESSING "$@"`
$DAILY_BRANCH_POSTMAKE_HOOK
heading=tmp/$flight.heading-info
@@ -261,6 +277,10 @@ fi
revlog=tmp/$flight.revision-log
case "$NEW_REVISION/$OLD_REVISION" in
+/flight-[0-9]*)
+ echo >&2 "SGR COMPARISON AGAINST ${OLD_REVISION}"
+ sgr_args+=" --that-flight=${OLD_REVISION#flight-}"
+ ;;
*/*[^0-9a-f]* | *[^0-9a-f]*/*)
echo >&2 "NO SGR COMPARISON badchar $NEW_REVISION/$OLD_REVISION"
;;
@@ -321,7 +341,7 @@ start_email $flight $branch "$sgr_args" "$subject_prefix"
push=false
if grep '^tolerable$' $mrof >/dev/null 2>&1; then push=$wantpush; fi
if test -f $branch.force; then push=$OSSTEST_PUSH; fi
-if grep -xF $NEW_REVISION $branch.force-rev; then push=$OSSTEST_PUSH; fi
+if grep -xF "$NEW_REVISION" $branch.force-rev; then push=$OSSTEST_PUSH; fi
if test -f $branch.block; then push=false; fi
if test -e $mrof && test -e $tree_bisect && ! grep '^broken' $mrof; then
diff --git a/cri-common b/cri-common
index ad44546..58b08f2 100644
--- a/cri-common
+++ b/cri-common
@@ -72,6 +72,7 @@ select_xenbranch () {
rumpuserxen) tree=rumpuserxen; xenbranch=xen-unstable ;;
seabios) tree=seabios; xenbranch=xen-unstable ;;
ovmf) tree=ovmf; xenbranch=xen-unstable ;;
+ distros-*) tree=none; xenbranch=xen-unstable ;;
osstest) tree=osstest; xenbranch=xen-unstable ;;
esac
if [ "x$tree" = xlinux ]; then
--
2.1.4
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH OSSTEST v8 08/14] crontab-cambridge: Use hard tabs for alignment.
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
` (6 preceding siblings ...)
2015-07-08 12:30 ` [PATCH OSSTEST v8 07/14] distros: add branch infrastructure Ian Campbell
@ 2015-07-08 12:30 ` Ian Campbell
2015-07-08 13:32 ` Ian Jackson
2015-07-08 12:30 ` [PATCH OSSTEST v8 09/14] distros: Run one suite per day on a weekly basis Ian Campbell
` (6 subsequent siblings)
14 siblings, 1 reply; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 12:30 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Also quote the value of BRANCHES=.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v8: Slit out from "distros: Run one suite per day on a weekly basis"
---
crontab-cambridge | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/crontab-cambridge b/crontab-cambridge
index 60bb4fd..e0c3eff 100644
--- a/crontab-cambridge
+++ b/crontab-cambridge
@@ -1,5 +1,5 @@
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=ian.jackson@citrix.com,ian.campbell@eu.citrix.com
-# m h dom mon dow command
-4-59/30 * * * * cd testing.git && BRANCHES=osstest ./cr-for-branches branches -q "./cr-daily-branch --real"
-3 4 * * * savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
+# m h dom mon dow command
+4-59/30 * * * * cd testing.git && BRANCHES='osstest' ./cr-for-branches branches -q "./cr-daily-branch --real"
+3 4 * * * savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
--
2.1.4
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH OSSTEST v8 09/14] distros: Run one suite per day on a weekly basis
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
` (7 preceding siblings ...)
2015-07-08 12:30 ` [PATCH OSSTEST v8 08/14] crontab-cambridge: Use hard tabs for alignment Ian Campbell
@ 2015-07-08 12:30 ` Ian Campbell
2015-07-08 13:33 ` Ian Jackson
2015-07-08 12:30 ` [PATCH OSSTEST v8 10/14] Debian: Handle lack of bootloader support in d-i on ARM Ian Campbell
` (5 subsequent siblings)
14 siblings, 1 reply; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 12:30 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Once a week should be sufficient for these tests. Perhaps in the
future we will want to increase the frequency for the suites under
active development (testing, unstable)
For now run this on the Citrix Cambridge instance until the XenProject
instance has sufficient capacity.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v8: Switch to hard tabs split to previous patch.
v7: Replaces "distros: Run a flight over the weekend."
Now run in Cambridge
Run separate flight per-suite
Dropped Ack
---
crontab-cambridge | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/crontab-cambridge b/crontab-cambridge
index e0c3eff..7d3ed57 100644
--- a/crontab-cambridge
+++ b/crontab-cambridge
@@ -2,4 +2,9 @@ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=ian.jackson@citrix.com,ian.campbell@eu.citrix.com
# m h dom mon dow command
4-59/30 * * * * cd testing.git && BRANCHES='osstest' ./cr-for-branches branches -q "./cr-daily-branch --real"
+46 7 * * 6 cd testing.git && BRANCHES='distros-debian-snapshot' ./cr-for-branches branches -w "./cr-daily-branch --real"
+46 7 * * 5 cd testing.git && BRANCHES='distros-debian-sid' ./cr-for-branches branches -w "./cr-daily-branch --real"
+46 7 * * 4 cd testing.git && BRANCHES='distros-debian-jessie' ./cr-for-branches branches -w "./cr-daily-branch --real"
+46 7 * * 3 cd testing.git && BRANCHES='distros-debian-wheezy' ./cr-for-branches branches -w "./cr-daily-branch --real"
+46 7 * * 2 cd testing.git && BRANCHES='distros-debian-squeeze' ./cr-for-branches branches -w "./cr-daily-branch --real"
3 4 * * * savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
--
2.1.4
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH OSSTEST v8 10/14] Debian: Handle lack of bootloader support in d-i on ARM.
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
` (8 preceding siblings ...)
2015-07-08 12:30 ` [PATCH OSSTEST v8 09/14] distros: Run one suite per day on a weekly basis Ian Campbell
@ 2015-07-08 12:30 ` Ian Campbell
2015-07-08 12:30 ` [PATCH OSSTEST v8 11/14] ts-debian-di-install: Refactor root_disk specification Ian Campbell
` (4 subsequent siblings)
14 siblings, 0 replies; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 12:30 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Debian doesn't currently know what bootloader to install in a Xen
guest on ARM. We install pv-grub-menu above which actually does what
we need, but the installer doesn't treat that as a "bootloader".
Most ARM platforms end up installing a u-boot boot.scr, based on a
platform whitelist. This doesn't seem appropriate for us. Grub is not
available for arm32. For arm64 we will eventually end up with in-guest
UEFI and therefore grub-efi and things will work normally. I'm not
sure what the answer is going to be for arm32.
This patch enables the workaround for Wheezy, Jessie and Sid,
post-Jessie should be enabled as we add them. (Pre-wheezy does not
support running as a Xen guest on ARM so we don't test them at all).
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v4: Handle sid too
v3: New
---
Osstest/Debian.pm | 14 ++++++++++++--
ts-debian-di-install | 6 ++++--
2 files changed, 16 insertions(+), 4 deletions(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 7c94b6c..4669047 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -865,8 +865,8 @@ END
return $preseed;
}
-sub preseed_create_guest ($$;@) {
- my ($ho, $sfx, %xopts) = @_;
+sub preseed_create_guest ($$$;@) {
+ my ($ho, $arch, $sfx, %xopts) = @_;
my $suite= $xopts{Suite} || $c{DebianSuite};
@@ -913,6 +913,16 @@ d-i grub-installer/bootdev string /dev/xvda
END
+ # Debian doesn't currently know what bootloader to install in a
+ # Xen guest on ARM. We install pv-grub-menu above which actually
+ # does what we need, but the installer doesn't treat that as a
+ # "bootloader".
+ logm("\$arch is $arch, \$suite is $suite");
+ $preseed_file.= (<<END) if $arch =~ /^arm/ && $suite =~ /wheezy|jessie|sid/;
+d-i nobootloader/confirmation_common boolean true
+
+END
+
$preseed_file .= preseed_hook_cmds();
return create_webfile($ho, "preseed$sfx", $preseed_file);
diff --git a/ts-debian-di-install b/ts-debian-di-install
index 1a7e1d0..373fad1 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -192,7 +192,9 @@ END
$method_cfg = setup_netboot($tmpdir, $arch, $suite);
- $ps_url = preseed_create_guest($gho, '', Suite=>$suite, PvMenuLst=>($bl eq "pvgrub"));
+ $ps_url = preseed_create_guest($gho, $arch, '',
+ Suite=>$suite,
+ PvMenuLst=>($bl eq "pvgrub"));
$extra_disk = "";
}
@@ -202,7 +204,7 @@ END
($method_cfg,$extra_disk) = setup_netinst($tmpdir, $arch);
- $ps_url = preseed_create_guest($gho, '', CDROM=>1);
+ $ps_url = preseed_create_guest($gho, $arch, '', CDROM=>1);
}
else
{
--
2.1.4
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH OSSTEST v8 11/14] ts-debian-di-install: Refactor root_disk specification
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
` (9 preceding siblings ...)
2015-07-08 12:30 ` [PATCH OSSTEST v8 10/14] Debian: Handle lack of bootloader support in d-i on ARM Ian Campbell
@ 2015-07-08 12:30 ` Ian Campbell
2015-07-08 12:31 ` [PATCH OSSTEST v8 12/14] make-flight: refactor PV debian tests Ian Campbell
` (3 subsequent siblings)
14 siblings, 0 replies; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 12:30 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v4: new patch
---
ts-debian-di-install | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/ts-debian-di-install b/ts-debian-di-install
index 373fad1..6fafd6d 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -227,12 +227,14 @@ END
OnPowerOff => "preserve"
);
+ my $root_disk = "'phy:$gho->{Lvdev},xvda,w'";
+
prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
$method_cfg
extra = "$cmdline"
#
disk = [
- $extra_disk 'phy:$gho->{Lvdev},xvda,w'
+ $extra_disk $root_disk
]
END
@@ -256,7 +258,7 @@ END
$blcfg
#
disk = [
- 'phy:$gho->{Lvdev},xvda,w'
+ $root_disk
]
END
return;
--
2.1.4
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH OSSTEST v8 12/14] make-flight: refactor PV debian tests
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
` (10 preceding siblings ...)
2015-07-08 12:30 ` [PATCH OSSTEST v8 11/14] ts-debian-di-install: Refactor root_disk specification Ian Campbell
@ 2015-07-08 12:31 ` Ian Campbell
2015-07-08 12:31 ` [PATCH OSSTEST v8 13/14] Add testing of file backed disk formats Ian Campbell
` (2 subsequent siblings)
14 siblings, 0 replies; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 12:31 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
No functional change, standalone-generate-dump-flight-runvars confirms
no change to the runvars.
Includes a hook which is not used yet, $recipe_sfx.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v4: new patch
---
make-flight | 24 ++++++++++++++----------
1 file changed, 14 insertions(+), 10 deletions(-)
diff --git a/make-flight b/make-flight
index 725da26..2a132df 100755
--- a/make-flight
+++ b/make-flight
@@ -361,6 +361,17 @@ do_pvgrub_tests () {
all_hostflags=$most_hostflags
}
+do_pv_debian_test_one () {
+ testname=$1; shift
+ recipe_sfx=$1; shift
+ toolstack=$1; shift
+
+ job_create_test test-$xenarch$kern-$dom0arch-$testname \
+ test-debian$recipe_sfx $toolstack \
+ $xenarch $dom0arch \
+ $debian_runvars all_hostflags=$most_hostflags $@
+}
+
do_pv_debian_tests () {
xsms=$(xenbranch_xsm_variants)
@@ -376,20 +387,13 @@ do_pv_debian_tests () {
suffix=${platform:+-$platform}
hostflags=${most_hostflags}${platform:+,platform-$platform}
- job_create_test test-$xenarch$kern-$dom0arch-xl$suffix \
- test-debian xl \
- $xenarch $dom0arch \
- enable_xsm=$xsm \
- $debian_runvars all_hostflags=$hostflags
+ do_pv_debian_test_one xl$suffix '' xl enable_xsm=$xsm
+
done
done
for xsm in $xsms ; do
- job_create_test test-$xenarch$kern-$dom0arch-libvirt \
- test-debian libvirt \
- $xenarch $dom0arch \
- enable_xsm=$xsm \
- $debian_runvars all_hostflags=$most_hostflags
+ do_pv_debian_test_one libvirt '' libvirt enable_xsm=$xsm
done
}
--
2.1.4
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH OSSTEST v8 13/14] Add testing of file backed disk formats
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
` (11 preceding siblings ...)
2015-07-08 12:31 ` [PATCH OSSTEST v8 12/14] make-flight: refactor PV debian tests Ian Campbell
@ 2015-07-08 12:31 ` Ian Campbell
2015-07-08 13:34 ` Ian Jackson
2015-07-08 12:31 ` [PATCH OSSTEST v8 14/14] make-distros-flight: Use ftp.debian.org directly Ian Campbell
2015-07-23 14:40 ` [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
14 siblings, 1 reply; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 12:31 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
xen-create-image makes this tricky to do since it is rather LVM
centric. Now that we have the ability to install from d-i it's
possible to arrange fairly easily that they use something other than
a phy backend over a bare LVM device.
Here we add support to the test script and infra and create a bunch of
new jobs testing the cross product of {xl,libvirt} x {raw,qcow2,vhd}.
A disk format of "raw" means a raw backing file, where as "none" (the
default) means to continue to use the base LVM device.
The test scripts are modified such that when constructing a domain
with a diskfmt runvar specifeies a file backed disk format (i.e. not
"none"):
- the LVM device is slightly enlarged to account for file format
headers (1M should be plenty).
- the LVM device will have an ext3 filesystem created on it instead
of being used as a phy device for the guest. Reusing the LVM volume
in this way means we don't need to do more storage management in
dom0 (i.e. arranging for / to be large enough, or managing a
special "images" LV)
- the relevant type of container is created within the filesystem
using the appropriate tool.
- New properties Disk{fmt,spec} are added to all $gho, containing
the format used for the root disk and the xl diskspec to load it.
- lvm backed guests use a xend/xm compatible spec, everything
else uses the improved xl syntax which libvirt also supports.
We won't test non-LVM on xend.
- New properties Disk{mnt,img} are added to $gho which are not using
LVM. These contain the mount point to use (configurable via
OSSTEST_CONFIG and runvars) and the full path (including mount
point) to the image itself.
- When starting or stopping a guest we arrange for the filesystem to
be (u)mounted.
- The prepearation when starting a guest copes gracefully with
the disk already being prepared.
- Hooks are called from guest_create() and guest_destroy() to
manipulate the disk as needed.
Using standalong-generate-dump-flight-runvars a representative set of
runvars is:
+test-amd64-amd64-xl-qcow2 all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
+test-amd64-amd64-xl-qcow2 arch amd64
+test-amd64-amd64-xl-qcow2 buildjob build-amd64
+test-amd64-amd64-xl-qcow2 debian_arch amd64
+test-amd64-amd64-xl-qcow2 debian_bootloader pygrub
+test-amd64-amd64-xl-qcow2 debian_diskfmt qcow2
+test-amd64-amd64-xl-qcow2 debian_kernkind pvops
+test-amd64-amd64-xl-qcow2 debian_method netboot
+test-amd64-amd64-xl-qcow2 debian_suite wheezy
+test-amd64-amd64-xl-qcow2 kernbuildjob build-amd64-pvops
+test-amd64-amd64-xl-qcow2 kernkind pvops
+test-amd64-amd64-xl-qcow2 toolstack xl
+test-amd64-amd64-xl-qcow2 xenbuildjob build-amd64
Compared to test-amd64-amd64-pygrub (which is the most similar job) and
normalising the test name the difference is:
test-amd64-amd64-SUFFIX all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
test-amd64-amd64-SUFFIX arch amd64
test-amd64-amd64-SUFFIX buildjob build-amd64
test-amd64-amd64-SUFFIX debian_arch amd64
test-amd64-amd64-SUFFIX debian_bootloader pygrub
+test-amd64-amd64-SUFFIX debian_diskfmt qcow2
+test-amd64-amd64-SUFFIX debian_kernkind pvops
test-amd64-amd64-SUFFIX debian_method netboot
test-amd64-amd64-SUFFIX debian_suite wheezy
test-amd64-amd64-SUFFIX kernbuildjob build-amd64-pvops
test-amd64-amd64-SUFFIX kernkind pvops
test-amd64-amd64-SUFFIX toolstack xl
test-amd64-amd64-SUFFIX xenbuildjob build-amd64
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v8: Default diskfmt is "none" (was "lvm"), i.e. use the LVM device
directly. Reword the commit log to reflect this.
v7: Use the right arch for tests, not always amd64 (doesn't work well
on arm!)
Defer guest_find_diskimg until _vg runvar and thence Lvdev are
setup:
selectguest calls guest_find_lv then guest_find_diskimg, using
preexisting runvars.
But prepare_guest calls selectguest before setting disk_lv, so
Lvdev ends up undefined, after setting disk_lv prepare_guest
calls guest_find_lv+guest_find_diskimg again and things get
configured.
This follows how guest_find_lv only sets Lvdev iff Vg and Lv
are both set.
Use {Guest}_suite not {Guest}_dist as runvar to choose version.
Assume slower dd for raw population, since I was still seeing
timeouts, assume at worst 1/2 the speed I happened to see in a
local test
Refresh the runvars in the commit log and drop the list of flights.
v6: Use bs=1MB (=1*1000) when creating the raw images instead of bs=1M
(=1*1024), this matches the units used by lvcreate's -L option and
therefore arranges that the imaage actually fits.
v5: Assume 100MB/s dd from /dev/zero when creating a raw disk image
Allow 10M of slack on filesystem for raw, qcow and vhd. 1M wasn't
enough in practice for raw.
v4: new patch
---
Osstest/TestSupport.pm | 100 ++++++++++++++++++++++++++++++++++++++++++++++++-
make-flight | 16 ++++++++
ts-debian-di-install | 10 ++---
ts-guest-start | 1 -
4 files changed, 117 insertions(+), 10 deletions(-)
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 3a7a535..3e09e8a 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -91,7 +91,8 @@ BEGIN {
target_var target_var_prefix
selectguest prepareguest more_prepareguest_hvm
guest_var guest_var_commalist guest_var_boolean
- prepareguest_part_lvmdisk prepareguest_part_xencfg
+ prepareguest_part_lvmdisk prepareguest_part_diskimg
+ prepareguest_part_xencfg
guest_umount_lv guest_await guest_await_dhcp_tcp
guest_checkrunning guest_check_ip guest_find_ether
guest_find_domid guest_check_up guest_check_up_quick
@@ -99,6 +100,7 @@ BEGIN {
guest_await_shutdown guest_await_destroy guest_destroy
guest_vncsnapshot_begin guest_vncsnapshot_stash
guest_check_remus_ok guest_editconfig
+ guest_prepare_disk guest_unprepare_disk
host_involves_pcipassthrough host_get_pcipassthrough_devs
toolstack guest_create
@@ -1344,6 +1346,7 @@ sub selectguest ($$) {
}
logm("guest: using $gn on $gho->{Host}{Name}");
guest_find_lv($gho);
+ guest_find_diskimg($gho);
guest_find_ether($gho);
guest_find_tcpcheckport($gho);
dhcp_watch_setup($ho,$gho);
@@ -1359,6 +1362,25 @@ sub guest_find_lv ($) {
? '/dev/'.$gho->{Vg}.'/'.$gho->{Lv} : undef;
}
+sub guest_find_diskimg($)
+{
+ my ($gho) = @_;
+
+ return unless $gho->{Lvdev};
+
+ $gho->{Diskfmt} = $r{"$gho->{Guest}_diskfmt"} // "none";
+ $gho->{Diskspec} = "phy:$gho->{Lvdev},xvda,w";
+
+ return if $gho->{Diskfmt} eq "none";
+
+ my $mntroot = get_host_property($gho->{Host}, "DiskImageMount",
+ $c{DiskImageMount} // "/var/lib/xen/images");
+
+ $gho->{Diskmnt} = "$mntroot/$gho->{Guest}";
+ $gho->{Diskimg} = "$gho->{Diskmnt}/disk.$gho->{Diskfmt}";
+ $gho->{Diskspec} = "format=$gho->{Diskfmt},vdev=xvda,target=$gho->{Diskimg}";
+}
+
sub guest_find_ether ($) {
my ($gho) = @_;
$gho->{Ether}= $r{"$gho->{Guest}_ether"};
@@ -1398,6 +1420,7 @@ sub guest_destroy ($) {
my ($gho) = @_;
my $ho = $gho->{Host};
toolstack($ho)->destroy($gho);
+ guest_unprepare_disk($gho);
}
sub guest_await_destroy ($$) {
@@ -1409,9 +1432,32 @@ sub guest_await_destroy ($$) {
sub guest_create ($) {
my ($gho) = @_;
my $ho = $gho->{Host};
+ guest_prepare_disk($gho);
toolstack($ho)->create($gho);
}
+sub guest_prepare_disk ($) {
+ my ($gho) = @_;
+
+ guest_umount_lv($gho->{Host}, $gho);
+
+ return if $gho->{Diskfmt} eq "none";
+
+ target_cmd_root($gho->{Host}, <<END);
+mkdir -p $gho->{Diskmnt}
+mount $gho->{Lvdev} $gho->{Diskmnt};
+END
+}
+
+sub guest_unprepare_disk ($) {
+ my ($gho) = @_;
+ return if $gho->{Diskfmt} eq "none";
+ target_cmd_root($gho->{Host}, <<END);
+umount $gho->{Lvdev} || :
+END
+}
+
+
sub target_choose_vg ($$) {
my ($ho, $mbneeded) = @_;
@@ -1550,6 +1596,7 @@ sub prepareguest ($$$$$$) {
}
guest_find_lv($gho);
+ guest_find_diskimg($gho);
guest_find_ether($gho);
guest_find_tcpcheckport($gho);
return $gho;
@@ -1560,7 +1607,56 @@ sub prepareguest_part_lvmdisk ($$$) {
target_cmd_root($ho, "lvremove -f $gho->{Lvdev} ||:");
target_cmd_root($ho, "lvcreate -L ${disk_mb}M -n $gho->{Lv} $gho->{Vg}");
target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Lvdev} count=10");
-}
+}
+
+sub make_vhd ($$$) {
+ my ($ho, $gho, $disk_mb) = @_;
+ target_cmd_root($ho, "vhd-util create -n $gho->{Rootimg} -s $disk_mb");
+}
+sub make_qcow2 ($$$) {
+ my ($ho, $gho, $disk_mb) = @_;
+ # upstream qemu's version. Seems preferable to qemu-xen-img from qemu-trad.
+ my $qemu_img = "/usr/local/lib/xen/bin/qemu-img";
+ target_cmd_root($ho, "$qemu_img create -f qcow2 $gho->{Rootimg} ${disk_mb}M");
+}
+sub make_raw ($$$) {
+ my ($ho, $gho, $disk_mb) = @_;
+ # In local tests this reported 130MB/s, so calculate a timeout assuming 65MB/s.
+ target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Rootimg} bs=1MB count=${disk_mb}",
+ ${disk_mb} / 65);
+}
+
+sub prepareguest_part_diskimg ($$$) {
+ my ($ho, $gho, $disk_mb) = @_;
+
+ my $diskfmt = $gho->{Diskfmt};
+ # Allow an extra 10 megabytes for image format headers
+ my $disk_overhead = $diskfmt eq "none" ? 0 : 10;
+
+ logm("preparing guest disks in $diskfmt format");
+
+ target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+
+ prepareguest_part_lvmdisk($ho, $gho, $disk_mb + $disk_overhead);
+
+ if ($diskfmt ne "none") {
+
+ $gho->{Rootimg} = "$gho->{Diskmnt}/disk.$diskfmt";
+ $gho->{Rootcfg} = "format=$diskfmt,vdev=xvda,target=$gho->{Rootimg}";
+
+ target_cmd_root($ho, <<END);
+mkfs.ext3 $gho->{Lvdev}
+mkdir -p $gho->{Diskmnt}
+mount $gho->{Lvdev} $gho->{Diskmnt}
+END
+ no strict qw(refs);
+ &{"make_$diskfmt"}($ho, $gho, $disk_mb);
+
+ target_cmd_root($ho, <<END);
+umount $gho->{Lvdev}
+END
+ }
+}
sub prepareguest_part_xencfg ($$$$$) {
my ($ho, $gho, $ram_mb, $xopts, $cfgrest) = @_;
diff --git a/make-flight b/make-flight
index 2a132df..a8ed20e 100755
--- a/make-flight
+++ b/make-flight
@@ -395,6 +395,22 @@ do_pv_debian_tests () {
for xsm in $xsms ; do
do_pv_debian_test_one libvirt '' libvirt enable_xsm=$xsm
done
+
+ for ts in xl libvirt ; do
+
+ for fmt in raw vhd qcow2 ; do
+
+ fmt_runvar="debian_diskfmt=$fmt"
+
+ do_pv_debian_test_one $ts-$fmt '-di' $ts \
+ debian_arch=$dom0arch \
+ debian_suite=$guestsuite \
+ debian_method=netboot \
+ debian_bootloader=pygrub \
+ $fmt_runvar
+
+ done
+ done
}
test_matrix_do_one () {
diff --git a/ts-debian-di-install b/ts-debian-di-install
index 6fafd6d..34b8e1e 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -75,9 +75,7 @@ sub prep () {
$gho= prepareguest($ho, $gn, $guesthost, 22,
$disk_mb, 40);
- prepareguest_part_lvmdisk($ho, $gho, $disk_mb);
-
- target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+ prepareguest_part_diskimg($ho, $gho, $disk_mb);
}
sub setup_netinst($$)
@@ -227,14 +225,12 @@ END
OnPowerOff => "preserve"
);
- my $root_disk = "'phy:$gho->{Lvdev},xvda,w'";
-
prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
$method_cfg
extra = "$cmdline"
#
disk = [
- $extra_disk $root_disk
+ $extra_disk '$gho->{Diskspec}'
]
END
@@ -258,7 +254,7 @@ END
$blcfg
#
disk = [
- $root_disk
+ '$gho->{Diskspec}'
]
END
return;
diff --git a/ts-guest-start b/ts-guest-start
index 1aa9e69..a434720 100755
--- a/ts-guest-start
+++ b/ts-guest-start
@@ -25,7 +25,6 @@ tsreadconfig();
our ($ho,$gho) = ts_get_host_guest(@ARGV);
sub start () {
- guest_umount_lv($ho, $gho);
guest_create($gho);
}
--
2.1.4
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH OSSTEST v8 14/14] make-distros-flight: Use ftp.debian.org directly
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
` (12 preceding siblings ...)
2015-07-08 12:31 ` [PATCH OSSTEST v8 13/14] Add testing of file backed disk formats Ian Campbell
@ 2015-07-08 12:31 ` Ian Campbell
2015-07-08 13:35 ` Ian Jackson
2015-07-23 14:40 ` [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
14 siblings, 1 reply; 27+ messages in thread
From: Ian Campbell @ 2015-07-08 12:31 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
The local proxy seems to serve stale packages for Jessie etc, I blame
the intercepting cache on the way out of our network, similar to
b5f15136900d "mg-debian-installer-update: workaround caching proxies",
except it is between the apt-cache and the world not the osstest vm
and the world.
Since the netboot kernel+initrd are reasonably small, these flights
are infrequent and they are intended to test the current upstream
version I think this is tollerable.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
make-distros-flight | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/make-distros-flight b/make-distros-flight
index 49f4b60..d407fcb 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -79,7 +79,9 @@ test_do_one_netboot () {
gsuite=sid
gver=daily
else
- local mirror="http://`getconfig DebianMirrorHost`/`getconfig DebianMirrorSubpath`"
+ #local mirror="http://`getconfig DebianMirrorHost`/`getconfig DebianMirrorSubpath`"
+ # XXX local mirror seems to serve up stale files.
+ local mirror="http://ftp.debian.org/debian"
diurl="$mirror/dists/$gsuite/main/installer-$domU/current/images/netboot"
gver=$gsuite
fi
--
2.1.4
^ permalink raw reply related [flat|nested] 27+ messages in thread* Re: [PATCH OSSTEST v8 14/14] make-distros-flight: Use ftp.debian.org directly
2015-07-08 12:31 ` [PATCH OSSTEST v8 14/14] make-distros-flight: Use ftp.debian.org directly Ian Campbell
@ 2015-07-08 13:35 ` Ian Jackson
0 siblings, 0 replies; 27+ messages in thread
From: Ian Jackson @ 2015-07-08 13:35 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v8 14/14] make-distros-flight: Use ftp.debian.org directly"):
> The local proxy seems to serve stale packages for Jessie etc, I blame
> the intercepting cache on the way out of our network, similar to
> b5f15136900d "mg-debian-installer-update: workaround caching proxies",
> except it is between the apt-cache and the world not the osstest vm
> and the world.
>
> Since the netboot kernel+initrd are reasonably small, these flights
> are infrequent and they are intended to test the current upstream
> version I think this is tollerable.
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH OSSTEST v8 00/14] add distro domU testing flight
2015-07-08 12:30 [PATCH OSSTEST v8 00/14] add distro domU testing flight Ian Campbell
` (13 preceding siblings ...)
2015-07-08 12:31 ` [PATCH OSSTEST v8 14/14] make-distros-flight: Use ftp.debian.org directly Ian Campbell
@ 2015-07-23 14:40 ` Ian Campbell
14 siblings, 0 replies; 27+ messages in thread
From: Ian Campbell @ 2015-07-23 14:40 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Wed, 2015-07-08 at 13:30 +0100, Ian Campbell wrote:
>
> I retained acks even when changing things due to either the moving of
> the make-*flight filter or the moving of the runvar docs to the
> script, otherwise I dropped them, I hope that is ok.
>
> Summary of (A)cks, (M)odified and (N)ew (NM==Replaced something):
Ian acked the rest and after discussion with him I've pushed this to
pretest.
Cheers,
Ian.
>
> AM mfi-common: Allow make-*flight to filter the set of build
> jobs to include
> M TestSupport: Add helper to fetch a URL on a host
> AM distros: add support for installing Debian PV guests via d
> -i, flight and jobs
> AM distros: support booting Debian PV (d-i installed) guests
> with pvgrub.
> M distros: Support pvgrub for Wheezy too.
> A Test pygrub and pvgrub on the regular flights
> A distros: add branch infrastructure
> N crontab-cambridge: Use hard tabs for alignment.
> M distros: Run one suite per day on a weekly basis
> A Debian: Handle lack of bootloader support in d-i on ARM.
> A ts-debian-di-install: Refactor root_disk specification
> A make-flight: refactor PV debian tests
> M Add testing of file backed disk formats
> make-distros-flight: Use ftp.debian.org directly
>
> Results for an adhoc xen-unstable flight are at
> http://osstest.xs.citrite.net/~osstest/testlogs/logs/37711/
> And for Jessie:
> http://osstest.xs.citrite.net/~osstest/testlogs/logs/37717/
>
> Ian.
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 27+ messages in thread