[PATCH OSSTEST v2 0/8] Use the DTB provided by the kernel we are trying to boot

[PATCH OSSTEST v2 0/8] Use the DTB provided by the kernel we are trying to boot

[Ian Campbell]

On ARM we have been booting using the DTBS provided by the host Debian
kernel package (i.e. the ones from 3.16) even when we are booting a
Xen+kernel which we have built ourselves.

Recent Linux kernels (at least linux-next and linux-linus, perhaps mingo
-tip too) have changes which are incompatible with these older kernels
which stop the kernel from booting on Cubietruck (even natively).

Despite what I thought there is apparently[0] no ABI here. In any case
testing the newer DTBs is desirable so here is a series to switch to doing
so.

This is all acked but there were some changes suggested along with the
acks, which I've made and am posting here. It was all pretty minor so I've
kept the acks, I hope that is ok.

Ian.

[0] http://lists.infradead.org/pipermail/linux-arm-kernel/2015-October/3739
22.html

[PATCH OSSTEST v2 1/8] Osstest: Add %arch_debian2linux (and inverse)

[Ian Campbell]

Modelled after %arch_debian2xen.

Will be used in ts-kernel-build.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Wrapped long line
---
 Osstest.pm | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/Osstest.pm b/Osstest.pm
index fc8334d..cbef3ff 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -40,7 +40,9 @@ BEGIN {
                       db_begin_work db_prepare
                       ensuredir get_filecontents_core_quiet system_checked
                       nonempty visible_undef show_abs_time
-                      %arch_debian2xen %arch_xen2debian $cfgvar_re
+                      %arch_debian2xen %arch_xen2debian
+                      %arch_debian2linux %arch_linux2debian
+                      $cfgvar_re
                       );
     %EXPORT_TAGS = ( );
 
@@ -62,6 +64,13 @@ our %arch_debian2xen = qw(i386 x86_32
 our %arch_xen2debian;
 $arch_xen2debian{$arch_debian2xen{$_}} = $_ foreach keys %arch_debian2xen;
 
+our %arch_debian2linux = qw(i386 x86
+			    amd64 x86
+			    armhf arm);
+our %arch_linux2debian;
+$arch_linux2debian{$arch_debian2linux{$_}} = $_
+    foreach keys %arch_debian2linux;
+
 our $cfgvar_re = '[A-Z][0-9a-zA-Z-_]*';
 
 #---------- static default config settings ----------
-- 
2.5.3

[PATCH OSSTEST v2 2/8] ts-kernel-build: Introduce global $parms

[Ian Campbell]

Which contains the relevant details from %archparms, making the use
sites simpler.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-kernel-build | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/ts-kernel-build b/ts-kernel-build
index f26eba8..3006eeb 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -46,6 +46,7 @@ my $archparms = {
     'i386'  => { DefConfig => 'defconfig',          KernelImage => 'arch/x86/boot/bzImage' },
     'amd64' => { DefConfig => 'defconfig',          KernelImage => 'arch/x86/boot/bzImage' }
 };
+my $parms = $archparms->{ $r{arch} };
 
 sub enable_xen_config ();
 
@@ -333,7 +334,7 @@ END
         ed <$edscript enable-xen-config
         chmod +x enable-xen-config
         cd linux
-        make $archparms->{ $r{arch} }->{DefConfig}
+        make $parms->{DefConfig}
         # ARM defconfig omits modules for some reason...
         sed -e "s/^# CONFIG_MODULES is not set\$/CONFIG_MODULES=y/g" -i .config
         ../enable-xen-config .config
@@ -376,7 +377,7 @@ END
 
 sub kinstall () {
     my $kernfile= $r{kimagefile};
-    $kernfile= $archparms->{$r{arch}}->{KernelImage} if !defined $kernfile;
+    $kernfile= $parms->{KernelImage} if !defined $kernfile;
 
     target_cmd_build($ho, 300, $builddir, <<END);
 	mkdir -p dist/boot dist/lib/modules
-- 
2.5.3

[PATCH OSSTEST v2 3/8] ts-kernel-build: Refactor $archparams::DefConfig selection

[Ian Campbell]

This is always either "foo_defconfig" or "defconfig". Record only
"foo" or undef and construct the name.

This makes the $archparams less verbose.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-kernel-build | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/ts-kernel-build b/ts-kernel-build
index 3006eeb..b611b9a 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -42,9 +42,9 @@ selectbuildhost(\@ARGV);
 builddirsprops();
 
 my $archparms = {
-    'armhf' => { DefConfig => 'multi_v7_defconfig', KernelImage => 'arch/arm/boot/zImage' },
-    'i386'  => { DefConfig => 'defconfig',          KernelImage => 'arch/x86/boot/bzImage' },
-    'amd64' => { DefConfig => 'defconfig',          KernelImage => 'arch/x86/boot/bzImage' }
+    'armhf' => { DefConfig => 'multi_v7', KernelImage => 'arch/arm/boot/zImage' },
+    'i386'  => {                          KernelImage => 'arch/x86/boot/bzImage' },
+    'amd64' => {                          KernelImage => 'arch/x86/boot/bzImage' }
 };
 my $parms = $archparms->{ $r{arch} };
 
@@ -330,11 +330,13 @@ END
     target_putfilecontents_stash($ho,100,enable_xen_config(),
                                  "$builddir/enable-xen-config");
 
+    my $defconfig = $parms->{DefConfig} ?
+	$parms->{DefConfig}."_defconfig" : "defconfig";
     target_cmd_build($ho, 1000, $builddir, <<END);
         ed <$edscript enable-xen-config
         chmod +x enable-xen-config
         cd linux
-        make $parms->{DefConfig}
+        make $defconfig
         # ARM defconfig omits modules for some reason...
         sed -e "s/^# CONFIG_MODULES is not set\$/CONFIG_MODULES=y/g" -i .config
         ../enable-xen-config .config
-- 
2.5.3

[PATCH OSSTEST v2 4/8] ts-kernel-build: Refactor $archparams KernelImage handling

[Ian Campbell]

This is always arch/$karch/boot/$img. Store $img in %archparms and use
%arch_debian2linux to construct the full path as needed.

This makes the $archparams less verbose.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-kernel-build | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/ts-kernel-build b/ts-kernel-build
index b611b9a..e0429a4 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -42,9 +42,9 @@ selectbuildhost(\@ARGV);
 builddirsprops();
 
 my $archparms = {
-    'armhf' => { DefConfig => 'multi_v7', KernelImage => 'arch/arm/boot/zImage' },
-    'i386'  => {                          KernelImage => 'arch/x86/boot/bzImage' },
-    'amd64' => {                          KernelImage => 'arch/x86/boot/bzImage' }
+    'armhf' => { DefConfig => 'multi_v7', Image => 'zImage' },
+    'i386'  => {                          Image => 'bzImage' },
+    'amd64' => {                          Image => 'bzImage' }
 };
 my $parms = $archparms->{ $r{arch} };
 
@@ -379,7 +379,9 @@ END
 
 sub kinstall () {
     my $kernfile= $r{kimagefile};
-    $kernfile= $parms->{KernelImage} if !defined $kernfile;
+    my $karch = $arch_debian2linux{$r{arch}};
+    $kernfile = "arch/$karch/boot/$parms->{Image}"
+	if !defined $kernfile;
 
     target_cmd_build($ho, 300, $builddir, <<END);
 	mkdir -p dist/boot dist/lib/modules
-- 
2.5.3

[PATCH OSSTEST v2 5/8] ts-kernel-build: Add arm64 support

[Ian Campbell]

And to Osstest::%arch_debian2*.

Untested (but hopefully pretty obvious).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

---
Note that this replaces a hunk in "Add arm64 build and test jobs"
---
 Osstest.pm      | 6 ++++--
 ts-kernel-build | 1 +
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/Osstest.pm b/Osstest.pm
index cbef3ff..63484d2 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -60,13 +60,15 @@ scalar *main::DEBUG;
 
 our %arch_debian2xen = qw(i386 x86_32
 			  amd64 x86_64
-			  armhf armhf);
+			  armhf armhf
+                          arm64 arm64);
 our %arch_xen2debian;
 $arch_xen2debian{$arch_debian2xen{$_}} = $_ foreach keys %arch_debian2xen;
 
 our %arch_debian2linux = qw(i386 x86
 			    amd64 x86
-			    armhf arm);
+			    armhf arm
+                            arm64 arm64);
 our %arch_linux2debian;
 $arch_linux2debian{$arch_debian2linux{$_}} = $_
     foreach keys %arch_debian2linux;
diff --git a/ts-kernel-build b/ts-kernel-build
index e0429a4..ac83ddb 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -42,6 +42,7 @@ selectbuildhost(\@ARGV);
 builddirsprops();
 
 my $archparms = {
+    'arm64' => {                          Image => 'Image' },
     'armhf' => { DefConfig => 'multi_v7', Image => 'zImage' },
     'i386'  => {                          Image => 'bzImage' },
     'amd64' => {                          Image => 'bzImage' }
-- 
2.5.3

[PATCH OSSTEST v2 6/8] ts-kernel-build: Include dtbs in dist file

[Ian Campbell]

These are installed to $(INSTALL_PATH)/dtbs/$(KERNEL_RELEASE) where
$(INSTALL_PATH) defaults to /boot but we override it to our staging
/boot.

Note that ts-host-install will install the OS dtbs directly into
/boot/dtbs without the subdirectory, so this won't clash and could be
considered a fallback hence I don't propose to move those ones.

The install_dtbs target has been available since v3.14, wherease we
only test v3.16 onwards on ARM, hence no arrangements are needed to
conditionalise this installation over and above the per-arch
arrangements made here.

Having now set $(INSTALL_PATH) I think the "install" target could now
take over the installation of System.map, vmlinux and .config into
/boot but I've not checked this with all historical kernel versions
and don't intend to make this change now.

Remove any previous dist dir on install, otherwise the kernel tends to
create dist/boot/dtbs.old with the previous contents on repeated use.
Seems like good hygiene anyway.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Move modules_install to next line so it isn't lost after the
INSTALL_MOD_PATH
---
 ts-kernel-build | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/ts-kernel-build b/ts-kernel-build
index ac83ddb..513fdca 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -42,10 +42,10 @@ selectbuildhost(\@ARGV);
 builddirsprops();
 
 my $archparms = {
-    'arm64' => {                          Image => 'Image' },
-    'armhf' => { DefConfig => 'multi_v7', Image => 'zImage' },
-    'i386'  => {                          Image => 'bzImage' },
-    'amd64' => {                          Image => 'bzImage' }
+    'arm64' => {                          Image => 'Image',  Dtbs => 1 },
+    'armhf' => { DefConfig => 'multi_v7', Image => 'zImage', Dtbs => 1 },
+    'i386'  => {                          Image => 'bzImage'           },
+    'amd64' => {                          Image => 'bzImage'           }
 };
 my $parms = $archparms->{ $r{arch} };
 
@@ -384,10 +384,17 @@ sub kinstall () {
     $kernfile = "arch/$karch/boot/$parms->{Image}"
 	if !defined $kernfile;
 
+    my $dtbs_install = '';
+    $dtbs_install = 'dtbs_install' if $parms->{Dtbs};
+
     target_cmd_build($ho, 300, $builddir, <<END);
+	rm -rf dist
 	mkdir -p dist/boot dist/lib/modules
 	cd linux
-	make $makeflags INSTALL_MOD_PATH=$builddir/dist modules_install
+	make $makeflags \\
+		INSTALL_PATH=$builddir/dist/boot \\
+		INSTALL_MOD_PATH=$builddir/dist \\
+		modules_install $dtbs_install
 	cp System.map $builddir/dist/boot/
 	cp $kernfile $builddir/dist/boot/vmlinuz
 	cp .config $builddir/dist/boot/config
-- 
2.5.3

[PATCH OSSTEST v2 7/8] Debian: Enable interpolation in uboot_scr_load_dtb here doc

[Ian Campbell]

By switching <<'END' to <<END. A future patch is going to want to put
a variable here which requires interpretation by the Perl.

Unfortunately this means lots of extra backslashes to escape things
such that they pass through Perl and Shell and end up as ${foo} in the
resulting u-boot script.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Call a backslash a backslash.
---
 Osstest/Debian.pm | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 9347c49..d56e410 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -143,11 +143,11 @@ sub uboot_common_kernel_bootargs ($)
 }
 
 sub uboot_scr_load_dtb () {
-    return <<'END';
-if test -z "\${fdt_addr}" && test -n "\${fdtfile}" ; then
-    echo Loading dtbs/\${fdtfile}
-    ext2load scsi 0 \${fdt_addr_r} dtbs/\${fdtfile}
-    setenv fdt_addr \${fdt_addr_r}
+    return <<END;
+if test -z "\\\${fdt_addr}" && test -n "\\\${fdtfile}" ; then
+    echo Loading dtbs/\\\${fdtfile}
+    ext2load scsi 0 \\\${fdt_addr_r} dtbs/\\\${fdtfile}
+    setenv fdt_addr \\\${fdt_addr_r}
 fi
 END
 }
-- 
2.5.3

[PATCH OSSTEST v2 8/8] Debian: Use dtbs from kernel dist when booting that kernel

[Ian Campbell]

The kernel dist built by ts-kernel-build puts the corresponding dtbs
into /boot/dtbs/$kvers.

The host installer's dtbs remain in /boot/dtbs and are used when
booting the native kernel.

It's possible that this change will expose bugs which exist in the
DTBs in previous kernel branches (3.18 and 4.1). I've not
investigated, I think we should accept this possibility and deal with
it via backport requests (and maybe some force pushes if appropriate)
as necessary.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Add some commentary about which dtbs is where to
uboot_scr_load_dtb.
---
 Osstest/Debian.pm | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index d56e410..6a23751 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -142,11 +142,22 @@ sub uboot_common_kernel_bootargs ($)
     return @bootargs;
 }
 
-sub uboot_scr_load_dtb () {
+# ts-host-install will install the OS-supplied dtbs directly into
+# /boot/dtbs without a subdirectory, while the kernel dist tarball
+# produced by ts-kernel-build will install the DTBS from that kernel
+# into /boot/dtbs/$version.
+#
+# Therefore the optional $kvers argument allows the selection of
+# either the OS-supplied DTBS (when undef) or those of a specific
+# kernel built by osstest.
+sub uboot_scr_load_dtb (;$) {
+    my ($kvers) = @_;
+    $kvers ||= '';
+    $kvers .= '/' if $kvers;
     return <<END;
 if test -z "\\\${fdt_addr}" && test -n "\\\${fdtfile}" ; then
-    echo Loading dtbs/\\\${fdtfile}
-    ext2load scsi 0 \\\${fdt_addr_r} dtbs/\\\${fdtfile}
+    echo Loading dtbs/$kvers\\\${fdtfile}
+    ext2load scsi 0 \\\${fdt_addr_r} dtbs/$kvers\\\${fdtfile}
     setenv fdt_addr \\\${fdt_addr_r}
 fi
 END
@@ -229,7 +240,7 @@ END
 	my $early_commands = get_host_property($ho, 'UBootScriptEarlyCommands', '');
 	my $xen_addr_r = get_host_property($ho, 'UBootSetXenAddrR', undef);
 
-	my $load_dtb = uboot_scr_load_dtb();
+	my $load_dtb = uboot_scr_load_dtb($want_kernver);
 
 	my $set_xen_addr_r =
 	    $xen_addr_r ? "setenv xen_addr_r $xen_addr_r" : "";
-- 
2.5.3

Re: [PATCH OSSTEST v2 0/8] Use the DTB provided by the kernel we are trying to boot

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 0/8] Use the DTB provided by the kernel we are trying to boot"):
> On ARM we have been booting using the DTBS provided by the host Debian
> kernel package (i.e. the ones from 3.16) even when we are booting a
> Xen+kernel which we have built ourselves.
...
> This is all acked but there were some changes suggested along with the
> acks, which I've made and am posting here. It was all pretty minor so I've
> kept the acks, I hope that is ok.

Indeed, it's what I expected, thanks.

Ian.