[PATCH OSSTEST v2 0/8] Fixes for switching to Jessie as base OS for test hosts

[PATCH OSSTEST v2 0/8] Fixes for switching to Jessie as base OS for test hosts

[Ian Campbell]

Compared with last time there are a few new patches to fix things I noticed
while testing and the prerequisites for actually deploying (f/w fix on
arndale and git cache) are now addressed. It remains the case that the
first 7 patches are OK to go in now and the final one to do the switch
should perhaps be scheduled separately.

Ian.

[PATCH OSSTEST v2 1/8] Debian: uboot: Use di_vg_name() and lv_dev_mapper() for root=

[Ian Campbell]

root is not a "guest lv", so using target_guest_lv_name is misleading.

target_guest_lv_name also fails to properly handle the d-i vg name
correctly, which di_vg_name does. Specifically under Jessie an extra
"-vg" is appended to the hostname compared with Wheezy and Squeeze.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 153b375..9347c49 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -131,7 +131,7 @@ sub uboot_common_kernel_bootargs ($)
 {
     my ($ho) = @_;
 
-    my $root= target_guest_lv_name($ho,"root");
+    my $root= lv_dev_mapper(di_vg_name($ho),"root");
     my $rootdelay= get_host_property($ho, "rootdelay");
 
     my @bootargs;
-- 
2.5.3

[PATCH OSSTEST v2 2/8] TestSupport: Use lv_dev_mapper in guest_find_lv

[Ian Campbell]

This has the effect of switching $gho->{Lvdev} from /dev/$VG/$LV to
/dev/mapper/$VG-$LV (where $VG and $LV have an s/-/--/ transformation
applied).

The two paths point to the samedevice and most call sites don't care
about the distinction. Some places which use "kpartx $dev" to Lvdev
expect to be able to append a partition number to $dev, while kpartx
only creates the /dev/mapper form, meaning such places cannot use
Lvdev. By making this switch we allow these places (such as
ts-freebsd-install) to use kpartx.

The only other place I'm aware of which requires one form or the other
is the Debian initramfs code which expects root=/dev/mapper/vg-lv and
does not accept /dev/vg/lv. However this is already handled correctly.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: New(ish, posted in reply to v1)
---
 Osstest/TestSupport.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 09694f4..d8c5762 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1398,7 +1398,7 @@ sub guest_find_lv ($) {
     $gho->{Vg}= $r{"${gn}_vg"};
     $gho->{Lv}= $r{"${gn}_disk_lv"};
     $gho->{Lvdev}= (defined $gho->{Vg} && defined $gho->{Lv})
-        ? '/dev/'.$gho->{Vg}.'/'.$gho->{Lv} : undef;
+        ? lv_dev_mapper($gho->{Vg},$gho->{Lv}) : undef;
 }
 
 sub guest_find_diskimg($)
-- 
2.5.3

[PATCH OSSTEST v2 3/8] ts-freebsd-install: Use $gho->{Lvdev} instead of target_guest_lv_name

[Ian Campbell]

prepareguest has already assigned this so we should use it instead of
replicating (perhaps wrongly since target_guest_lv_name and
target_choose_vg can behave differently if multiple vgs are present).

$gho->{Lvdev} has been adjusted to return /dev/mapper/vg-lv paths
which is required to be able to add partition numbers since kpartx
does not create the /dev/vg/lv form.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-freebsd-install | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/ts-freebsd-install b/ts-freebsd-install
index 98dad24..d1f4bd3 100755
--- a/ts-freebsd-install
+++ b/ts-freebsd-install
@@ -49,7 +49,7 @@ sub prep () {
 
     target_put_guest_image($ho, $gho);
 
-    my $rootpartition_dev = target_guest_lv_name($ho, $gho->{Name}) . "--disk3";
+    my $root_part_nr = 3;
 
     target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
 
@@ -64,7 +64,7 @@ sub prep () {
 
             kpartx -a $gho->{Lvdev}
             mkdir -p $mnt
-            mount -t ufs -o ufstype=ufs2,rw $rootpartition_dev $mnt
+            mount -t ufs -o ufstype=ufs2,rw $gho->{Lvdev}${root_part_nr} $mnt
 END
 
     target_cmd_root($ho, <<END, 15);
-- 
2.5.3


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[PATCH OSSTEST v2 4/8] TestSupport: Remove now unused target_guest_lv_name.

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index d8c5762..69da459 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -63,7 +63,7 @@ BEGIN {
                       target_install_packages target_install_packages_norec
                       target_jobdir target_extract_jobdistpath_subdir
                       target_extract_jobdistpath
-                      lv_dev_mapper target_guest_lv_name
+                      lv_dev_mapper
 
                       poll_loop tcpconnect await_tcp
                       contents_make_cpio file_simple_write_contents
@@ -721,11 +721,6 @@ sub lv_dev_mapper ($$) {
     return "/dev/mapper/$vg-$lv";
 }    
 
-sub target_guest_lv_name($$) {
-    my ($ho, $lv) = @_;
-    return lv_dev_mapper("$ho->{Name}", $lv);
-}
-
 #---------- dhcp watching ----------
 
 sub dhcp_watch_setup ($$) {
-- 
2.5.3

[PATCH OSSTEST v2 5/8] ts-kernel-build: Explicitly enable CONFIG_CGROUPS

[Ian Campbell]

This is not enabled in the arm defconfig (it is in x86) and without it
a Debian Jessie userspace with systemd fails to boot with:

systemd[1]: Failed to mount tmpfs at /sys/fs/cgroup: No such file or directory

While we arrange to use sysvinit in the host and in d-i installed
guests we do not do so for xen-create-image guests.

Arranging to install sysvinit-core during xen-create-image looks to be
non-trivial. I hope it won't be necessary.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: New patch
---
 ts-kernel-build | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/ts-kernel-build b/ts-kernel-build
index a16527b..6d59e6e 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -549,6 +549,9 @@ setopt CONFIG_BLK_CPQ_CISS_DA m
 
 setopt CONFIG_FHANDLE y
 
+# Needed for Jessie+ userspace when systemd is used.
+setopt CONFIG_CGROUPS y
+
 # Needed for efibootmgr
 setopt CONFIG_EFI_VARS y
 
-- 
2.5.3

Re: [PATCH OSSTEST v2 5/8] ts-kernel-build: Explicitly enable CONFIG_CGROUPS

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 5/8] ts-kernel-build: Explicitly enable CONFIG_CGROUPS"):
> This is not enabled in the arm defconfig (it is in x86) and without it
> a Debian Jessie userspace with systemd fails to boot with:
> 
> systemd[1]: Failed to mount tmpfs at /sys/fs/cgroup: No such file or directory
> 
> While we arrange to use sysvinit in the host and in d-i installed
> guests we do not do so for xen-create-image guests.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

> Arranging to install sysvinit-core during xen-create-image looks to be
> non-trivial. I hope it won't be necessary.

If we do decide it's necessary, ISTR that telling x-c-i to add
specific additional packages is not too hard.

Ian.

Re: [PATCH OSSTEST v2 5/8] ts-kernel-build: Explicitly enable CONFIG_CGROUPS

[Ian Campbell]

On Tue, 2015-10-06 at 16:05 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 5/8] ts-kernel-build: Explicitly
> enable CONFIG_CGROUPS"):
> > This is not enabled in the arm defconfig (it is in x86) and without it
> > a Debian Jessie userspace with systemd fails to boot with:
> > 
> > systemd[1]: Failed to mount tmpfs at /sys/fs/cgroup: No such file or
> > directory
> > 
> > While we arrange to use sysvinit in the host and in d-i installed
> > guests we do not do so for xen-create-image guests.
> 
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Thanks.

> > Arranging to install sysvinit-core during xen-create-image looks to be
> > non-trivial. I hope it won't be necessary.
> 
> If we do decide it's necessary, ISTR that telling x-c-i to add
> specific additional packages is not too hard.

Really? I did look and the advice I saw on the Internet seemed to revolve
around adding wrappers around debootstrap and telling x-c-i to use them.
> 
> Ian.

Re: [PATCH OSSTEST v2 5/8] ts-kernel-build: Explicitly enable CONFIG_CGROUPS

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST v2 5/8] ts-kernel-build: Explicitly enable CONFIG_CGROUPS"):
> On Tue, 2015-10-06 at 16:05 +0100, Ian Jackson wrote:
> > Ian Campbell writes ("[PATCH OSSTEST v2 5/8] ts-kernel-build: Explicitly
> > enable CONFIG_CGROUPS"):
> > > Arranging to install sysvinit-core during xen-create-image looks to be
> > > non-trivial. I hope it won't be necessary.
> > 
> > If we do decide it's necessary, ISTR that telling x-c-i to add
> > specific additional packages is not too hard.
> 
> Really? I did look and the advice I saw on the Internet seemed to revolve
> around adding wrappers around debootstrap and telling x-c-i to use them.

The --role= option seems to provide a suiteable hook.  I looked at
/etc/xen-tools/role.d/builder and it seems a reasonable template.

Anyway, we can cross this bridge when we come to it.

Ian.

[PATCH OSSTEST v2 6/8] ts-debian-di-install: Use correct per-arch name for kernel.

[Ian Campbell]

The x86 and arm kernels are inconsistently named upstream, and then
renamed in mg-debian-installer-update as:

            ====== KERNEL ============  ====== INITRD ==================
            Debian	   Osstest	Debian		   Osstest
            -----------    -----------  ---------------    -------------
x86/native: linux	=> linux	initrd.gz	=> initrd.gz
x86/xen:    xen/vmlinuz	=> vmlinuz-xen	xen/initrd.gz	=> initrd.gz-xen
arm/native: vmlinuz	=> linux	initrd.gz	=> initrd.gz
arm/xen:    vmlinuz	=> linux	initrd.gz	=> initrd.gz

Cope with this here, by treating the x86/xen (PV) case as the special
case that it is (due to the requirement of a separate i686-pae
installer kernel).

This goes unnoticed in the distros-debian flights because the kernel
is downloaded at runtime via a runvar. It matters once regular
(non-distro-debian) flights run with Jessie because then the d-i which
is used by the di based tests is taken from the result of
mg-debian-installer-update (e.g. for the -qcow2 flights etc).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: New patch
---
 ts-debian-di-install | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/ts-debian-di-install b/ts-debian-di-install
index 9905691..64b5d44 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -153,10 +153,13 @@ sub setup_netboot($$$)
 
 	my $di_path = $c{TftpPath}.'/'.$ho->{Tftp}{DiBase}.'/'.${arch}.'/'.$c{TftpDiVersion}.'-'.$ho->{Suite};
 
-        my $suffix = '';
-        $suffix .= "-xen" if ${arch} =~ m/amd64|i386/;
-	$kernel = "$di_path/vmlinuz$suffix";
-	$ramdisk = "$di_path/initrd.gz$suffix";
+        if (${arch} =~ m/amd64|i386/) {
+	    $kernel = "$di_path/vmlinuz-xen";
+	    $ramdisk = "$di_path/initrd.gz-xen";
+        } else {
+	    $kernel = "$di_path/linux";
+	    $ramdisk = "$di_path/initrd.gz";
+        }
 
 	target_putfile_root($ho, 60, $kernel, "$didir/kernel_${suite}_${arch}");
 	target_putfile_root($ho, 60, $ramdisk, "$didir/ramdisk_${suite}_${arch}");
-- 
2.5.3

Re: [PATCH OSSTEST v2 6/8] ts-debian-di-install: Use correct per-arch name for kernel.

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 6/8] ts-debian-di-install: Use correct per-arch name for kernel."):
> The x86 and arm kernels are inconsistently named upstream, and then
> renamed in mg-debian-installer-update as:
> 
>             ====== KERNEL ============  ====== INITRD ==================
>             Debian	   Osstest	Debian		   Osstest
>             -----------    -----------  ---------------    -------------
> x86/native: linux	=> linux	initrd.gz	=> initrd.gz
> x86/xen:    xen/vmlinuz	=> vmlinuz-xen	xen/initrd.gz	=> initrd.gz-xen
> arm/native: vmlinuz	=> linux	initrd.gz	=> initrd.gz
> arm/xen:    vmlinuz	=> linux	initrd.gz	=> initrd.gz

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

[PATCH OSSTEST v2 7/8] ts-freebsd-install: Pass -s option to kpartx

[Ian Campbell]

This is "Sync mode. Don't return until the partitions are created",
which seems to be needed in Jessie. The option also exists in Wheezy,
according to the manpage.

Without this the following mount fails having apparently raced against
the creation of the device nodes.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: New patch
---
 ts-freebsd-install | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ts-freebsd-install b/ts-freebsd-install
index d1f4bd3..d630f2a 100755
--- a/ts-freebsd-install
+++ b/ts-freebsd-install
@@ -62,7 +62,7 @@ sub prep () {
             dd if=$rimagebase.raw of=$gho->{Lvdev} bs=1M
             rm $rimagebase.raw
 
-            kpartx -a $gho->{Lvdev}
+            kpartx -s -a $gho->{Lvdev}
             mkdir -p $mnt
             mount -t ufs -o ufstype=ufs2,rw $gho->{Lvdev}${root_part_nr} $mnt
 END
-- 
2.5.3

Re: [PATCH OSSTEST v2 7/8] ts-freebsd-install: Pass -s option to kpartx

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 7/8] ts-freebsd-install: Pass -s option to kpartx"):
> This is "Sync mode. Don't return until the partitions are created",
> which seems to be needed in Jessie. The option also exists in Wheezy,
> according to the manpage.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

> Without this the following mount fails having apparently raced against
> the creation of the device nodes.

How exciting.  (Why on earth is `sync mode' not the default?  Don't
get me started...)

Ian.

Re: [PATCH OSSTEST v2 7/8] ts-freebsd-install: Pass -s option to kpartx

[Ian Campbell]

On Tue, 2015-10-06 at 16:16 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 7/8] ts-freebsd-install: Pass -s
> > Without this the following mount fails having apparently raced against
> > the creation of the device nodes.

> How exciting.  (Why on earth is `sync mode' not the default?

It's always good to have plenty of rope. No, wait, that's mountaineering.

Ian.

[PATCH OSSTEST v2 8/8] Switch to Debian 8.0 (jessie) as OS for test hosts

[Ian Campbell]

mg-debian-installer-update-all has been run on the production instance
and TftpDiVersion is also updated to match.

The resulting binaries have also been copied to the Cambridge
instance, so update Cambridge config too.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
---
 Osstest.pm                  | 2 +-
 production-config           | 2 +-
 production-config-cambridge | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/Osstest.pm b/Osstest.pm
index 4a763c6..fc8334d 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -73,7 +73,7 @@ our %c = qw(
 
     Images images
 
-    DebianSuite wheezy
+    DebianSuite jessie
     DebianMirrorSubpath debian
 
     TestHostKeypairPath id_rsa_osstest
diff --git a/production-config b/production-config
index bb8d224..b21153b 100644
--- a/production-config
+++ b/production-config
@@ -87,7 +87,7 @@ TftpPxeTemplatesReal pxelinux.cfg/%ipaddrhex%
 
 TftpPxeGroup osstest
 # Update with ./mg-debian-installer-update(-all)
-TftpDiVersion 2015-09-07
+TftpDiVersion 2015-09-18
 
 # These should normally be the same.
 # Update with ./mg-cpu-microcode-update
diff --git a/production-config-cambridge b/production-config-cambridge
index f801303..5d68ede 100644
--- a/production-config-cambridge
+++ b/production-config-cambridge
@@ -69,7 +69,7 @@ TftpPxeTemplates %name%/pxelinux.cfg
 TftpPxeTemplatesReal pxelinux.cfg/%ipaddrhex%
 
 TftpPxeGroup osstest
-TftpDiVersion 2015-09-07
+TftpDiVersion 2015-09-18
 
 # These should normally be the same.
 MicrocodeUpdateAmd64 microcode.x86.2015-06-12.cpio
-- 
2.5.3

Re: [PATCH OSSTEST v2 0/8] Fixes for switching to Jessie as base OS for test hosts

[Ian Campbell]

On Tue, 2015-10-06 at 13:46 +0100, Ian Campbell wrote:
> Compared with last time there are a few new patches to fix things I
> noticed
> while testing and the prerequisites for actually deploying (f/w fix on
> arndale and git cache) are now addressed. It remains the case that the
> first 7 patches are OK to go in now and the final one to do the switch
> should perhaps be scheduled separately.

Here is a 9/8. This should be deferred and go in along with the final patch
to switch to Jessie, i.e. after the rest are in production with Wheezy.

8>----------

>From c5a1d26efd1eb83311933c387c1953eff575fff0 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Wed, 7 Oct 2015 14:04:52 +0100
Subject: [PATCH] make-flight: Use Jessie ISO image for Debian HVM tests

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
Deployment note: debian-8.2.0-{amd64,i386}-CD-1.iso already added to
images dir in Colo & Cambridge.
---
 make-flight | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/make-flight b/make-flight
index 0291b5f..ed4e6ce 100755
--- a/make-flight
+++ b/make-flight
@@ -284,7 +284,7 @@ do_hvm_debian_test_one () {
     test-debianhvm $toolstack $xenarch $dom0arch $qemuu_runvar \
     enable_xsm=$xsm                             \
     $stubdom_runvar $testvars                   \
-    debianhvm_image=debian-7.2.0-$arch-CD-1.iso \
+    debianhvm_image=debian-8.2.0-$arch-CD-1.iso \
     debianhvm_iso_kernel=/$iso_dir/vmlinuz \
     debianhvm_iso_ramdisk=/$iso_dir/initrd.gz \
     bios=$bios \
-- 
2.5.3