[PATCH OSSTEST v1 00/15] Initial support for ARM64

[PATCH OSSTEST v1 00/15] Initial support for ARM64

[Ian Campbell]

The following makes a start on support for arm64 systems.

Since arm64 was only added in Debian Jessie this requires Wei's "Debian
Jessie patches" as a base line, but without the
"mg-debian-installer-update: use new url for armhf packages" patch which
is replaced here. (Wei, I cc'd you on that one)

The main development here is support for UEFI PXE booting, which is not
particular to arm64, but is more prominent there. Hopefully this work
would be useful for UEFI only x86 systems too.

This has been tested on a Mustang board booting UEFI and build-* works
with one (slightly major) exception: The Debian kernel does not
currently reboot correctly on this system. This will likely require a
kernel from backports in order to work correctly.

I haven't looked into test-* yet (although they are added) but I expect
two major bits of work will be required: Booting via uefi chainloading
and provisioning xen.cfg instead of grub.cfg and the use of a newer
linux kernel for arm systems in order to get all the necessary device
support (Debian Jessie's kernel has some backports for Mustang).

I expect that other arm64 platforms (e.g. Cavium ThunderX, AMD Seattle)
are going to need a much newer kernel anyway.

Ian.1

[PATCH OSSTEST 01/15] mg-debian-installer-update: updates to better handle Jessie onwards.

[Ian Campbell]

In mg-debian-installer-update:

  - Expand the list of (suite,arch) combinations which don't exist and
    move it to the top.

  - Tweak the backports.org handling to allow it to be specified on a
    per (suite,arch) basis, and specify it only for (wheezy,armhf)
    since it is not currently needed for jessie. Make the
    initramfs-tools fetching conditional too.

  - Only extract backports-dtbs if they are present in the package.

  - Handle arm64.

  - Collect the correct armhf kernel for Jessie onwards (vexpress
    removed from path).

Have mg-debian-installer-update-all fetch arm64 too.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
---
 Osstest/Debian.pm              |  2 +-
 mg-debian-installer-update     | 59 ++++++++++++++++++++++++++----------------
 mg-debian-installer-update-all |  2 +-
 ts-host-install                |  4 +--
 4 files changed, 41 insertions(+), 26 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index de7c6e8..cd7dc5d 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -811,7 +811,7 @@ END
     foreach my $kp (keys %{ $ho->{Flags} }) {
 	# Backwards compatibility
 	$kp = "need-kernel-deb-wheezy-backports" if $kp eq "need-kernel-deb-armmp";
-	$kp =~ s/need-kernel-deb-$ho->{Suite}// or next;
+	$kp =~ s/need-kernel-deb-$ho->{Suite}-// or next;
 
 	my $kern = "$d_i/$kp.deb";
 	my $kurl = create_webfile($ho, "kernel", sub {
diff --git a/mg-debian-installer-update b/mg-debian-installer-update
index f21f216..f6bf962 100755
--- a/mg-debian-installer-update
+++ b/mg-debian-installer-update
@@ -38,20 +38,28 @@ fetch () {
 }
 
 case ${suite}_${arch} in
+    lenny_armhf|squeeze_armhf|lenny_arm64|squeeze_arm64|wheezy_arm64)
+        # No such thing.
+        exit 0
+        ;;
     *_i386|*_amd64)
         src=$sbase/main/installer-$arch/current/images/netboot/
         files="debian-installer/$arch/initrd.gz:initrd.gz debian-installer/$arch/linux:linux xen/vmlinuz:vmlinuz-xen xen/initrd.gz:initrd.gz-xen"
         ;;
-    lenny_armhf|squeeze_armhf)
-        # No such thing.
-        exit 0
+    *_arm64)
+        src=$sbase/main/installer-$arch/current/images/netboot/
+        files="debian-installer/$arch/initrd.gz:initrd.gz debian-installer/$arch/linux:linux"
         ;;
-    *_armhf)
+    wheezy_armhf)
         # Use the vexpress image, we are going to bash in our own
         # kernel+modules anyway.
         src=$sbase/main/installer-$arch/current/images/vexpress/netboot
         files='initrd.gz' # no kernel -- needs a newer kernel
         ;;
+    *_armhf)
+        src=$sbase/main/installer-$arch/current/images/netboot
+        files='vmlinuz initrd.gz'
+	;;
 esac
 
 pfile=$sbase/non-free/binary-$arch/Packages.gz
@@ -89,28 +97,33 @@ for p in $packages; do
         rm -rf x
 done
 
-# armhf requires a newer kernel than was in Wheezy for most
-# platforms. Construct something suitable from the latest kernel in
-# wheezy-backports.
-if [ $arch = armhf ]; then
+# Some platforms require a newer kernel than is in Debian. Construct
+# something suitable from the latest kernel ($bpok=flavour) in
+# backports.
+case $suite_$arch in
+    wheezy_armhf) bpok=armmp; need_initramfs=y;;
+esac
+if [ x$bpok != x ]; then
     bp="$sbase-backports"
-    pfile=$bp/main/binary-armhf/Packages.gz
+    pfile=$bp/main/binary-$arch/Packages.gz
 
     fetch $pfile >Packages.gz
 
-    # Newer kernel often needs a newer initramfs-tools. Make that available
-    echo >&2 "collecting backports initramfs-tools"
-    pkgfile=`zcat Packages.gz | grep-dctrl -PX initramfs-tools -nsFilename | sort -n -r | head -n1`
-    rc=$?
-    set -e
-    if [ $rc != 0 ]; then fail "initramfs-tools package not found"; fi
-    fetch "$site/$pkgfile" >initramfs-tools.deb
+    if [ x$need_initramfs = xy ]; then
+        # Newer kernel often needs a newer initramfs-tools. Make that available
+        echo >&2 "collecting backports initramfs-tools"
+        pkgfile=`zcat Packages.gz | grep-dctrl -PX initramfs-tools -nsFilename | sort -n -r | head -n1`
+        rc=$?
+        set -e
+        if [ $rc -ne 0 ] || [ x$pkgfile = x ]; then fail "initramfs-tools package not found"; fi
+        fetch "$site/$pkgfile" >initramfs-tools.deb
+    fi
 
     set +e
     echo >&2 "collecting backports kernel"
     # Be careful to pickup the actual kernel package from the 'linux'
     # source and not a meta package from 'linux-latest'
-    pkgfile=`zcat Packages.gz | grep-dctrl -S linux | grep-dctrl -Pe ^linux-image-.*-armmp$ -nsFilename | sort -n -r | head -n1`
+    pkgfile=`zcat Packages.gz | grep-dctrl -S linux | grep-dctrl -Pe ^linux-image-.*-${bpok}$ -nsFilename | sort -n -r | head -n1`
     rc=$?
     set -e
     if [ $rc != 0 ]; then fail "backports kernel package not found"; fi
@@ -143,11 +156,13 @@ if [ $arch = armhf ]; then
           |pax -x sv4cpio -s '%lib%/lib%' -d -w -M norm >../cpio; cd ..
     gzip -9nf cpio
     mv cpio.gz backports.cpio.gz
-    rm -rf backports-dtbs/
-    mkdir backports-dtbs/
-    mv x/usr/lib/linux-image-*-armmp/*.dtb backports-dtbs/
-    tar --mtime=./x/usr/lib -cf backports-dtbs.tar backports-dtbs
-    gzip -9nf backports-dtbs.tar
+    if [ -d x/usr/lib/linux-image-*-${bpok}/ ] ; then
+        rm -rf backports-dtbs/
+        mkdir backports-dtbs/
+        mv x/usr/lib/linux-image-*-${bpok}/*.dtb backports-dtbs/
+        tar --mtime=./x/usr/lib -cf backports-dtbs.tar backports-dtbs
+        gzip -9nf backports-dtbs.tar
+    fi
     rm -rf x
 fi
 
diff --git a/mg-debian-installer-update-all b/mg-debian-installer-update-all
index eca4a5f..241e2e3 100755
--- a/mg-debian-installer-update-all
+++ b/mg-debian-installer-update-all
@@ -24,7 +24,7 @@ set -e
 
 suite=`getconfig DebianSuite`
 fws=`getconfig DebianNonfreeFirmware`
-arches="armhf amd64 i386"
+arches="arm64 armhf amd64 i386"
 
 for arch in $arches ; do
     ./mg-debian-installer-update $suite $arch $fws
diff --git a/ts-host-install b/ts-host-install
index f7bc454..98bcd62 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -202,7 +202,7 @@ sub setup_pxeboot_firstboot($) {
     foreach my $kp (keys %{ $ho->{Flags} }) {
         # Backwards compatibility
         $kp = "need-kernel-deb-wheezy-backports" if $kp eq "need-kernel-deb-armmp";
-        $kp =~ s/need-kernel-deb-$ho->{Suite}// or next;
+        $kp =~ s/need-kernel-deb-$ho->{Suite}-// or next;
         my $kern= "$ho->{Tftp}{Path}/$d_i/linux.$kp";
         if (stat $kern) {
             logm("using kernel from: $kern");
@@ -251,7 +251,7 @@ END
     foreach my $kp (keys %{ $ho->{Flags} }) {
 	# Backwards compatibility
 	$kp = "need-kernel-deb-wheezy-backports" if $kp eq "need-kernel-deb-armmp";
-	$kp =~ s/need-kernel-deb-$ho->{Suite}// or next;
+	$kp =~ s/need-kernel-deb-$ho->{Suite}-// or next;
 
 	$dtbs = "fdtdir /$d_i/$kp-dtbs"
 	    if -e "$ho->{Tftp}{Path}/$d_i/$kp-dtbs";
-- 
2.1.4

[PATCH OSSTEST 02/15] mg-debian-installer-update: Also collect DTBs for standard kernels

[Ian Campbell]

For armhf and arm64 for Jessie we will need these in the normal case
as well as in the backports case. Arrange to download.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 mg-debian-installer-update | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/mg-debian-installer-update b/mg-debian-installer-update
index f6bf962..3dd036a 100755
--- a/mg-debian-installer-update
+++ b/mg-debian-installer-update
@@ -49,6 +49,7 @@ case ${suite}_${arch} in
     *_arm64)
         src=$sbase/main/installer-$arch/current/images/netboot/
         files="debian-installer/$arch/initrd.gz:initrd.gz debian-installer/$arch/linux:linux"
+        dtbs=$sbase/main/installer-$arch/current/images/device-tree/
         ;;
     wheezy_armhf)
         # Use the vexpress image, we are going to bash in our own
@@ -59,7 +60,8 @@ case ${suite}_${arch} in
     *_armhf)
         src=$sbase/main/installer-$arch/current/images/netboot
         files='vmlinuz initrd.gz'
-	;;
+        dtbs=$sbase/main/installer-$arch/current/images/device-tree/
+        ;;
 esac
 
 pfile=$sbase/non-free/binary-$arch/Packages.gz
@@ -79,6 +81,15 @@ for f in $files; do
         fetch $src/$s >$d.new
 done
 
+if [ "x$dtbs" != "x" ] ; then
+    # Can't seem to get curl to globs.
+    rm -rf dtbs
+    mkdir dtbs
+    ( cd dtbs && wget -nv -A README,\*.dtb -nd -nH -np -m  $dtbs )
+    tar --mtime=./dtbs/README -cf dtbs.tar dtbs
+    gzip -9nf dtbs.tar
+fi
+
 fetch $pfile >Packages.gz
 
 for p in $packages; do
-- 
2.1.4

[PATCH OSSTEST 03/15] Introduce mg-pxe-loader-update

[Ian Campbell]

The story for PXE booting via UEFI (at least on arm64) is not so
straightforward as with pxelinux on x86. There seems to no good
bootloader to launch via UEFI+pxe, in fact all I could find was grub
(syslinux, and by extension pxelinux.efi, is x86 only).

Add mg-pxe-loader-update modelled on mg-debian-installer-update which
will download the necessary grub binaries and produce a grub image
which can be used to pxe boot.

grub lacks the convenient ability to search for config file based on
(substrings of) the MAC or IP address. So we arrange for the grub.cfg
in TftpGrubBase to chain load another config file from
TftpTmpDir/'$net_default_mac' where $net_default_mac is a grub
variable which is substituted at boot time.

Actually using this requires that bootp/dhcp provide a next-file so
UEFI knows what to boot (usually this would be pxelinux.0 which we
can't use here). Locally we have configured this as $name/pxe.img, so
we can use different loaders.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest.pm           |  3 ++
 README               |  4 ++-
 mg-pxe-loader-update | 89 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 production-config    |  4 +++
 4 files changed, 99 insertions(+), 1 deletion(-)
 create mode 100755 mg-pxe-loader-update

diff --git a/Osstest.pm b/Osstest.pm
index e8bd77b..e668b3c 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -199,6 +199,9 @@ sub readglobalconfig () {
     $c{TftpDiBase} ||= "$c{TftpPlayDir}debian-installer";
     $c{TftpDiVersion} ||= 'current';
 
+    $c{TftpGrubBase} ||= "$c{TftpPlayDir}grub";
+    $c{TftpGrubVersion} ||= 'current';
+
     $c{WebspaceFile} ||= "$ENV{'HOME'}/public_html/";
     $c{WebspaceUrl} ||= "http://$myfqdn/~$whoami/";
     $c{WebspaceCommon} ||= 'osstest/';
diff --git a/README b/README
index 44e2989..503d15d 100644
--- a/README
+++ b/README
@@ -206,7 +206,9 @@ To run osstest in standalone mode:
      netcat
      chiark-utils-bin
 
- - Optional: ipmitool
+ - Optional:
+     ipmitool -- for hosts which use IPMI for power control
+     grub-common -- for mg-pxe-loader-update
 
  - Write a config file
     ~/.xen-osstest/config
diff --git a/mg-pxe-loader-update b/mg-pxe-loader-update
new file mode 100755
index 0000000..b336fdb
--- /dev/null
+++ b/mg-pxe-loader-update
@@ -0,0 +1,89 @@
+#!/bin/bash
+# usage
+#   ./mg-pxe-loader-update jessie
+#
+# Requires grub-mkimage (Debian package: grub-common)
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+set -e
+
+. cri-getconfig
+
+suite=$1
+
+fail () { echo >&2 "$0: $1"; exit 1; }
+
+site=http://ftp.debian.org/debian/
+sbase=$site/dists/$suite
+
+archs="amd64 i386 arm64"
+
+dstroot=`getconfig TftpPath`/`getconfig TftpGrubBase`/
+date=`date +%Y-%m-%d`
+dst=$date
+
+grubpfx=`getconfig TftpGrubBase`/$date/grub
+grubcfg=`getconfig TftpTmpDir`/grub.cfg-'$net_default_mac'
+
+mkdir -p $dstroot
+cd $dstroot
+mkdir -p $dst
+cd $dst
+rm -rf grub
+mkdir grub
+
+cat >grub/grub.cfg <<EOF
+set stage1=yes
+configfile $grubcfg
+EOF
+for arch in $archs ; do
+    case $arch in
+	amd64) grubdeb=grub-efi-amd64-bin; platform=x86_64-efi;;
+	i386)  grubdeb=grub-efi-ia32-bin;  platform=i386-efi;;
+	arm64) grubdeb=grub-efi-arm64-bin; platform=arm64-efi;;
+	*) echo "No grub on $arch" >&2; exit 0;;
+    esac
+
+    pfile=$sbase/main/binary-$arch/Packages.gz
+
+    curl -s $pfile >Packages.gz
+
+    echo >&2 "collecting $grubdeb"
+    pkgfile=`zcat Packages.gz | grep-dctrl -PX $grubdeb -nsFilename | sort -n -r | head -n1`
+    rc=$?
+    set -e
+    if [ $rc -ne 0 ] || [ x$pkgfile = x ]; then fail "$grubdeb package not found"; fi
+    curl -s "$site/$pkgfile" >$grubdeb.deb
+
+    dpkg-deb -x $grubdeb.deb x
+
+    mv x/usr/lib/grub/* grub/
+
+    rm -rf x
+
+    rm Packages.gz
+
+    grub-mkimage -O "$platform" \
+		 -d ./grub/$platform \
+		 -o pxegrub-$arch.efi -p "$grubpfx" \
+		 search configfile normal efinet tftp net
+done
+
+echo $date
+echo >&2 "downloaded $dstroot/$date"
diff --git a/production-config b/production-config
index 3a0b768..344d7dd 100644
--- a/production-config
+++ b/production-config
@@ -85,8 +85,12 @@ TftpPxeTemplates %name%/pxelinux.cfg
 TftpPxeTemplatesReal pxelinux.cfg/%ipaddrhex%
 
 TftpPxeGroup osstest
+# Update with /mg-debian-installer-update(-all)
 TftpDiVersion 2015-01-10
 
+# Update with /mg-pxe-loader-update
+TftpGrubVersion XXXX-XX-XX
+
 XenUsePath /usr/groups/xencore/systems/bin/xenuse
 XenUseUser osstest
 
-- 
2.1.4

[PATCH OSSTEST 04/15] Debian: Preseed a EFI system partition during host install

[Ian Campbell]

AIUI the runes used will only result in an ESP if the system was
booted via UEFI. IOW I don't think there should be any change for
existing systems.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest.pm        | 1 +
 Osstest/Debian.pm | 6 ++++++
 2 files changed, 7 insertions(+)

diff --git a/Osstest.pm b/Osstest.pm
index e668b3c..172f904 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -67,6 +67,7 @@ our %c = qw(
     TestHostKeypairPath id_rsa_osstest
     HostProp_GenEtherPrefixBase 5e:36:0e:f5
 
+    HostDiskESP    300
     HostDiskBoot   300
     HostDiskRoot 10000
     HostDiskSwap  2000
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index cd7dc5d..cd2ad9b 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -913,6 +913,12 @@ d-i partman-basicmethods/method_only boolean false
 
 d-i partman-auto/expert_recipe string					\\
 	boot-root ::							\\
+		$c{HostDiskESP} 60 $c{HostDiskESP} free			\\
+			\$iflabel{ gpt }				\\
+			reusemethod{ }					\\
+			method{ efi }					\\
+			format{ }					\\
+		.							\\
 		$c{HostDiskBoot} 50 $c{HostDiskBoot} ext3		\\
 			\$primary{ } \$bootable{ }			\\
 			method{ format } format{ }			\\
-- 
2.1.4

[PATCH OSSTEST 05/15] ts-host-install: Split initrd out of @installcmdline

[Ian Campbell]

Other bootloaders handle this with an explicit separate option rather
than parsing it out of the command line as pxelinux does. Prepare for
supporting these.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ts-host-install | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/ts-host-install b/ts-host-install
index 98bcd62..412b1a8 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -268,9 +268,7 @@ END
     my $initrd= "$ho->{Tftp}{TmpDir}$ho->{Name}--initrd.gz";
     system_checked("cat -- @initrds >$ho->{Tftp}{Path}$initrd");
     
-    push @installcmdline, ("initrd=/$initrd",
-                           "domain=$c{TestHostDomain}",
-                           );
+    push @installcmdline, "domain=$c{TestHostDomain}";
     push @installcmdline,
         get_host_property($ho, "install-append $ho->{Suite}", ''),
         get_host_property($ho, "install-append $ho->{Suite} $r{arch}", '');
@@ -298,7 +296,7 @@ label overwrite
 	menu label ^Overwrite
 	menu default
 	kernel $kernel
-	append $installcmdline
+	append initrd=/$initrd $installcmdline
 	ipappend $ipappend
 	$dtbs
 default overwrite
-- 
2.1.4

[PATCH OSSTEST 06/15] ts-host-install: split the "di" from the "host" command line

[Ian Campbell]

(i.e. the bit before/after the -- marker). When abstracting over
different bootloaders in a future patch this will be convenient since
it allows the code to add to either.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ts-host-install | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/ts-host-install b/ts-host-install
index 412b1a8..ea92db3 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -178,8 +178,8 @@ sub setup_pxeboot_firstboot($) {
     
     my $d_i= $ho->{Tftp}{DiBase}.'/'.$r{arch}.'/'.$c{TftpDiVersion}.'-'.$ho->{Suite};
     
-    my @installcmdline= qw(vga=normal);
-    push @installcmdline, di_installcmdline_core($ho, $ps_url, %xopts);
+    my @dicmdline= qw(vga=normal);
+    push @dicmdline, di_installcmdline_core($ho, $ps_url, %xopts);
 
     my $src_initrd= "$d_i/initrd.gz";
     my @initrds= "$ho->{Tftp}{Path}/$src_initrd";
@@ -268,26 +268,27 @@ END
     my $initrd= "$ho->{Tftp}{TmpDir}$ho->{Name}--initrd.gz";
     system_checked("cat -- @initrds >$ho->{Tftp}{Path}$initrd");
     
-    push @installcmdline, "domain=$c{TestHostDomain}";
-    push @installcmdline,
+    push @dicmdline, "domain=$c{TestHostDomain}";
+    push @dicmdline,
         get_host_property($ho, "install-append $ho->{Suite}", ''),
         get_host_property($ho, "install-append $ho->{Suite} $r{arch}", '');
 
     my $console = get_host_native_linux_console($ho);
 
-    push @installcmdline, "console=$console" unless $console eq "NONE";
+    push @dicmdline, "console=$console" unless $console eq "NONE";
 
-    push @installcmdline, qw(--);
+    my @hocmdline;
 
     # See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007 for
     # why this is repeated.
-    push @installcmdline, "console=$console" unless $console eq "NONE";
+    push @hocmdline, "console=$console" unless $console eq "NONE";
 
-    push @installcmdline,
+    push @hocmdline,
         get_host_property($ho, "linux-boot-append $ho->{Suite}", ''),
         get_host_property($ho, "linux-boot-append $ho->{Suite} $r{arch}", '');
 
-    my $installcmdline= join ' ', @installcmdline;
+    my $dicmd= join ' ', @dicmdline;
+    my $hocmd= join ' ', @hocmdline;
 
     setup_pxeboot($ho, <<END);
 serial 0 $c{Baud}
@@ -296,7 +297,7 @@ label overwrite
 	menu label ^Overwrite
 	menu default
 	kernel $kernel
-	append initrd=/$initrd $installcmdline
+	append initrd=/$initrd $dicmd -- $hocmd
 	ipappend $ipappend
 	$dtbs
 default overwrite
-- 
2.1.4

[PATCH OSSTEST 07/15] Refactor pxelinux configuration

[Ian Campbell]

The mechanism used to PXE boot can differ depending on the firmware
type. Therefore refactor into Osstest::TestSupport and key off a new
host property "firmware".

Currently supported is "bios" (the default) and "uboot", both of which
use pxelinux.cfg style files.

The default for the firmware property is "bios", hence no change for
any existing system (including those which use uboot, regardless of
whether they are now configured with the firmware proprty or not)

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/TestSupport.pm | 44 +++++++++++++++++++++++++++++++++++++++++---
 ts-host-install        | 19 +++++--------------
 2 files changed, 46 insertions(+), 17 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index abb3195..da881ec 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -2078,16 +2078,34 @@ sub host_pxefile ($;$) {
         (join ",", sort keys %v)." ?";
 }
 
-sub setup_pxeboot ($$) {
+sub setup_pxelinux_bootcfg ($$) {
     my ($ho, $bootfile) = @_;
     my $f= host_pxefile($ho);
     file_link_contents("$ho->{Tftp}{Path}$ho->{Tftp}{PxeDir}$f", $bootfile,
 	"$ho->{Name}-pxelinux.cfg");
 }
 
-sub setup_pxeboot_local ($) {
+# Systems using BIOS are configured to use pxelinux
+sub setup_bios_pxeboot ($$$$$;%) {
+    my ($ho,$kern,$initrd,$dicmd,$hocmd,%xopts) = @_;
+    my $dtbs = "fdtdir $xopts{dtbs}" if $xopts{dtbs};
+    setup_pxelinux_bootcfg($ho, <<END);
+    serial 0 $c{Baud}
+timeout 5
+label overwrite
+	menu label ^Overwrite
+	menu default
+	kernel $kern
+	append $dicmd initrd=$initrd -- $hocmd
+	ipappend $xopts{ipappend}
+	$dtbs
+default overwrite
+END
+}
+
+sub setup_bios_pxeboot_local ($) {
     my ($ho) = @_;
-    setup_pxeboot($ho, <<END);
+    setup_pxelinux_bootcfg($ho, <<END);
 serial 0 $c{Baud}
 timeout 5
 label local
@@ -2096,6 +2114,26 @@ default local
 END
 }
 
+# uboot emulates pxelinux, so reuse BIOS stuff
+sub setup_uboot_pxeboot ($$$$$;%) { return &setup_bios_pxeboot; }
+sub setup_uboot_pxeboot_local ($) { return &setup_bios_pxeboot_local; }
+
+sub setup_pxeboot_local ($) {
+    my ($ho) = @_;
+    my $firmware = get_host_property($ho, "firmware", "bios");
+    $firmware =~ s/-/_/g;
+    no strict qw(refs);
+    return &{"setup_${firmware}_pxeboot_local"}($ho);
+}
+
+sub setup_pxeboot ($$$$$;%) {
+    my ($ho,$kern,$initrd,$dicmd,$hocmd,%xopts) = @_;
+    my $firmware = get_host_property($ho, "firmware", "bios");
+    $firmware =~ s/-/_/g;
+    no strict qw(refs);
+    return &{"setup_${firmware}_pxeboot"}($ho,$kern,$initrd,$dicmd,$hocmd,%xopts);
+}
+
 #---------- ISO images ----------
 sub iso_create_genisoimage ($$$$;@) {
     my ($ho,$iso,$dir,$isotimeout,@xopts) = @_;
diff --git a/ts-host-install b/ts-host-install
index ea92db3..aab5241 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -247,13 +247,14 @@ SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="$ho->{Ether}", A
 END
     }
 
-    my $dtbs = "";
+    my %xopts;
+
     foreach my $kp (keys %{ $ho->{Flags} }) {
 	# Backwards compatibility
 	$kp = "need-kernel-deb-wheezy-backports" if $kp eq "need-kernel-deb-armmp";
 	$kp =~ s/need-kernel-deb-$ho->{Suite}-// or next;
 
-	$dtbs = "fdtdir /$d_i/$kp-dtbs"
+	$xopts{dtbs} = "/$d_i/$kp-dtbs"
 	    if -e "$ho->{Tftp}{Path}/$d_i/$kp-dtbs";
     }
 
@@ -290,18 +291,8 @@ END
     my $dicmd= join ' ', @dicmdline;
     my $hocmd= join ' ', @hocmdline;
 
-    setup_pxeboot($ho, <<END);
-serial 0 $c{Baud}
-timeout 5
-label overwrite
-	menu label ^Overwrite
-	menu default
-	kernel $kernel
-	append initrd=/$initrd $dicmd -- $hocmd
-	ipappend $ipappend
-	$dtbs
-default overwrite
-END
+    $xopts{ipappend} = $ipappend;
+    setup_pxeboot($ho, $kernel, "/$initrd", $dicmd, $hocmd, %xopts);
 }
 
 install();
-- 
2.1.4

[PATCH OSSTEST 08/15] ts-host-install: Support UEFI PXE boot using grub.efi

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/TestSupport.pm | 37 ++++++++++++++++++++++++++++++++++++-
 README                 |  1 +
 2 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index da881ec..6450ee6 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -894,7 +894,7 @@ sub selecthost ($) {
     $ho->{Tftp} = { };
     $ho->{Tftp}{$_} = $c{"Tftp${_}_${tftpscope}"} || $c{"Tftp${_}"}
         foreach qw(Path TmpDir PxeDir PxeGroup PxeTemplates PxeTemplatesReal
-                   DiBase);
+                   DiBase GrubBase);
 
     #----- finalise -----
 
@@ -2118,6 +2118,41 @@ END
 sub setup_uboot_pxeboot ($$$$$;%) { return &setup_bios_pxeboot; }
 sub setup_uboot_pxeboot_local ($) { return &setup_bios_pxeboot_local; }
 
+sub setup_grub_efi_bootcfg ($$) {
+    my ($ho, $bootfile) = @_;
+    my $f = "grub.cfg-$ho->{Ether}";
+    my $grub= $ho->{Tftp}{Path}.'/'.$ho->{Tftp}{GrubBase}.'/'.$c{TftpGrubVersion}.
+	"/pxegrub-$r{arch}.efi";
+    my $pxe=$ho->{Tftp}{Path}.'/'.$ho->{Name}.'/pxe.img';
+
+    logm("Copy $grub => $pxe");
+    copy($grub, $pxe) or die "Copy $grub to $pxe failed: $!";
+
+    logm("grub_efi bootcfg into $f");
+    file_link_contents("$ho->{Tftp}{Path}$ho->{Tftp}{TmpDir}$f", $bootfile,
+		       "$ho->{Name}-pxegrub.cfg");
+}
+
+# UEFI systems PXE boot using grub.efi
+sub setup_uefi_pxeboot ($$$$$;%) {
+    my ($ho,$kern,$initrd,$dicmd,$hocmd,%xopts) = @_;
+    setup_grub_efi_bootcfg($ho, <<END);
+set default=0
+set timeout=5
+menuentry 'overwrite' {
+  linux $kern $dicmd -- $hocmd
+  initrd $initrd
+}
+END
+}
+
+sub setup_uefi_pxeboot_local ($) {
+    my ($ho) = @_;
+    setup_grub_efi_bootcfg($ho, <<END);
+echo Should do a local boot here, somehow
+END
+}
+
 sub setup_pxeboot_local ($) {
     my ($ho) = @_;
     my $firmware = get_host_property($ho, "firmware", "bios");
diff --git a/README b/README
index 503d15d..1351f49 100644
--- a/README
+++ b/README
@@ -455,6 +455,7 @@ TftpFoo_<scope> and TftpFoo
 
         DiBase        The path under `Path' to the root of the debian installer
                       images.
+        GrubBase      The path under `Path' to the root of the grub EFI PXE images.
 
     For hosts in scope "default", TftpFoo_default (if set) takes
     precedence over TftpFoo.  TftpFoo is used when the setting Foo is
-- 
2.1.4

[PATCH OSSTEST 09/15] Enable chain loading to local disk for UEFI PXE systems.

[Ian Campbell]

First arrange for bootloader to be installed to removable media path,
by using a new in Jessie preseed option. Then use that to chainload a
bootloader from the disk.

The removable media path is well known (part of the UEFI spec) which
saves us having to worry about which OS is on the host (so long as the
OS can be configured to populate the removable media path)

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm      |  2 ++
 Osstest/TestSupport.pm | 18 +++++++++++++++++-
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index cd2ad9b..e89ca6c 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -689,6 +689,8 @@ d-i apt-setup/contrib boolean false
 
 d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, chiark-utils-bin, $extra_packages
 
+d-i grub-installer/force-efi-extra-removable boolean true
+
 $xopts{ExtraPreseed}
 
 END
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 6450ee6..fe06e87 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -2148,8 +2148,24 @@ END
 
 sub setup_uefi_pxeboot_local ($) {
     my ($ho) = @_;
+    my %efi_archs = qw(amd64 X64
+                       arm32 ARM
+                       arm64 AARCH64
+                       i386  IA32);
+    die "EFI arch" unless $efi_archs{ $r{arch} };
+    my $efi = $efi_archs{ $r{arch} };
     setup_grub_efi_bootcfg($ho, <<END);
-echo Should do a local boot here, somehow
+set default=0
+set timeout=5
+menuentry 'local' {
+  insmod chain
+  insmod part_gpt
+  insmod part_msdos
+  set root=(hd0,gpt1)
+  echo "Chainloading (\${root})/EFI/BOOT/BOOTAA64.EFI"
+  chainloader (\${root})/EFI/BOOT/BOOTAA64.EFI
+  boot
+}
 END
 }
 
-- 
2.1.4

[PATCH OSSTEST 10/15] Add arm64 build and test jobs

[Ian Campbell]

Runvars:
	build-arm64                                   arch                        arm64
	build-arm64                                   build_lvextend_max          50
	build-arm64                                   enable_ovmf                 true
	build-arm64                                   enable_xend                 false
	build-arm64                                   enable_xsm                  false
	build-arm64                                   host_hostflags              share-build-jessie-arm64,arch-arm64,suite-jessie,purpose-build
	build-arm64                                   revision_ovmf
	build-arm64                                   revision_qemu
	build-arm64                                   revision_qemuu              b2da824bc5ad35fb9f1e74a203d7be96a7b0345e
	build-arm64                                   revision_seabios
	build-arm64                                   revision_xen                e13013dbf1d5997915548a3b5f1c39594d8c1d7b
	build-arm64                                   tree_ovmf
	build-arm64                                   tree_qemu                   git://xenbits.xen.org/staging/qemu-xen-unstable.git
	build-arm64                                   tree_qemuu                  git://xenbits.xen.org/staging/qemu-upstream-unstable.git
	build-arm64                                   tree_seabios
	build-arm64                                   tree_xen                    git://xenbits.xen.org/xen.git
	build-arm64-libvirt                           arch                        arm64
	build-arm64-libvirt                           build_lvextend_max          50
	build-arm64-libvirt                           buildjob                    build-arm64
	build-arm64-libvirt                           host_hostflags              share-build-jessie-arm64,arch-arm64,suite-jessie,purpose-build
	build-arm64-libvirt                           revision_libvirt            fd74e231751334b64af0934b680c5cc62f652453
	build-arm64-libvirt                           tree_libvirt                git://xenbits.xen.org/libvirt.git
	build-arm64-libvirt                           tree_xen                    git://xenbits.xen.org/xen.git
	build-arm64-pvops                             arch                        arm64
	build-arm64-pvops                             build_lvextend_max          50
	build-arm64-pvops                             host_hostflags              share-build-jessie-arm64,arch-arm64,suite-jessie,purpose-build
	build-arm64-pvops                             kconfig_override_y          CONFIG_EXT4_FS
	build-arm64-pvops                             kconfighow                  xen-enable-xen-config
	build-arm64-pvops                             revision_linux              ec18c9fc039971041d854e0d58551f1f1a32ff8f
	build-arm64-pvops                             revision_linuxfirmware      c530a75c1e6a472b0eb9558310b518f0dfcd8860
	build-arm64-pvops                             tree_linux                  git://xenbits.xen.org/linux-pvops.git
	build-arm64-pvops                             tree_linuxfirmware          git://xenbits.xen.org/osstest/linux-firmware.git
	build-arm64-pvops                             treevcs_linux               git
	build-arm64-xsm                               arch                        arm64
	build-arm64-xsm                               build_lvextend_max          50
	build-arm64-xsm                               enable_ovmf                 true
	build-arm64-xsm                               enable_xend                 false
	build-arm64-xsm                               enable_xsm                  true
	build-arm64-xsm                               host_hostflags              share-build-jessie-arm64,arch-arm64,suite-jessie,purpose-build
	build-arm64-xsm                               revision_ovmf
	build-arm64-xsm                               revision_qemu
	build-arm64-xsm                               revision_qemuu              b2da824bc5ad35fb9f1e74a203d7be96a7b0345e
	build-arm64-xsm                               revision_seabios
	build-arm64-xsm                               revision_xen                e13013dbf1d5997915548a3b5f1c39594d8c1d7b
	build-arm64-xsm                               tree_ovmf
	build-arm64-xsm                               tree_qemu                   git://xenbits.xen.org/staging/qemu-xen-unstable.git
	build-arm64-xsm                               tree_qemuu                  git://xenbits.xen.org/staging/qemu-upstream-unstable.git
	build-arm64-xsm                               tree_seabios
	build-arm64-xsm                               tree_xen                    git://xenbits.xen.org/xen.git
	test-arm64-arm64-libvirt                      all_hostflags               arch-arm64,arch-xen-arm64,suite-jessie,purpose-test
	test-arm64-arm64-libvirt                      arch                        arm64
	test-arm64-arm64-libvirt                      buildjob                    build-arm64
	test-arm64-arm64-libvirt                      debian_arch                 arm64
	test-arm64-arm64-libvirt                      debian_kernkind             pvops
	test-arm64-arm64-libvirt                      enable_xsm                  false
	test-arm64-arm64-libvirt                      kernbuildjob                build-arm64-pvops
	test-arm64-arm64-libvirt                      kernkind                    pvops
	test-arm64-arm64-libvirt                      libvirtbuildjob             build-arm64-libvirt
	test-arm64-arm64-libvirt                      toolstack                   libvirt
	test-arm64-arm64-libvirt                      xenbuildjob                 build-arm64
	test-arm64-arm64-libvirt-xsm                  all_hostflags               arch-arm64,arch-xen-arm64,suite-jessie,purpose-test
	test-arm64-arm64-libvirt-xsm                  arch                        arm64
	test-arm64-arm64-libvirt-xsm                  buildjob                    build-arm64-xsm
	test-arm64-arm64-libvirt-xsm                  debian_arch                 arm64
	test-arm64-arm64-libvirt-xsm                  debian_kernkind             pvops
	test-arm64-arm64-libvirt-xsm                  enable_xsm                  true
	test-arm64-arm64-libvirt-xsm                  kernbuildjob                build-arm64-pvops
	test-arm64-arm64-libvirt-xsm                  kernkind                    pvops
	test-arm64-arm64-libvirt-xsm                  libvirtbuildjob             build-arm64-libvirt
	test-arm64-arm64-libvirt-xsm                  toolstack                   libvirt
	test-arm64-arm64-libvirt-xsm                  xenbuildjob                 build-arm64-xsm
	test-arm64-arm64-xl                           all_hostflags               arch-arm64,arch-xen-arm64,suite-jessie,purpose-test
	test-arm64-arm64-xl                           arch                        arm64
	test-arm64-arm64-xl                           buildjob                    build-arm64
	test-arm64-arm64-xl                           debian_arch                 arm64
	test-arm64-arm64-xl                           debian_kernkind             pvops
	test-arm64-arm64-xl                           enable_xsm                  false
	test-arm64-arm64-xl                           kernbuildjob                build-arm64-pvops
	test-arm64-arm64-xl                           kernkind                    pvops
	test-arm64-arm64-xl                           toolstack                   xl
	test-arm64-arm64-xl                           xenbuildjob                 build-arm64
	test-arm64-arm64-xl-credit2                   all_hostflags               arch-arm64,arch-xen-arm64,suite-jessie,purpose-test
	test-arm64-arm64-xl-credit2                   arch                        arm64
	test-arm64-arm64-xl-credit2                   buildjob                    build-arm64
	test-arm64-arm64-xl-credit2                   debian_arch                 arm64
	test-arm64-arm64-xl-credit2                   debian_kernkind             pvops
	test-arm64-arm64-xl-credit2                   guests_vcpus                4
	test-arm64-arm64-xl-credit2                   kernbuildjob                build-arm64-pvops
	test-arm64-arm64-xl-credit2                   kernkind                    pvops
	test-arm64-arm64-xl-credit2                   toolstack                   xl
	test-arm64-arm64-xl-credit2                   xen_boot_append             sched=credit2
	test-arm64-arm64-xl-credit2                   xenbuildjob                 build-arm64
	test-arm64-arm64-xl-multivcpu                 all_hostflags               arch-arm64,arch-xen-arm64,suite-jessie,purpose-test
	test-arm64-arm64-xl-multivcpu                 arch                        arm64
	test-arm64-arm64-xl-multivcpu                 buildjob                    build-arm64
	test-arm64-arm64-xl-multivcpu                 debian_arch                 arm64
	test-arm64-arm64-xl-multivcpu                 debian_kernkind             pvops
	test-arm64-arm64-xl-multivcpu                 guests_vcpus                4
	test-arm64-arm64-xl-multivcpu                 kernbuildjob                build-arm64-pvops
	test-arm64-arm64-xl-multivcpu                 kernkind                    pvops
	test-arm64-arm64-xl-multivcpu                 toolstack                   xl
	test-arm64-arm64-xl-multivcpu                 xenbuildjob                 build-arm64
	test-arm64-arm64-xl-sedf                      all_hostflags               arch-arm64,arch-xen-arm64,suite-jessie,purpose-test
	test-arm64-arm64-xl-sedf                      arch                        arm64
	test-arm64-arm64-xl-sedf                      buildjob                    build-arm64
	test-arm64-arm64-xl-sedf                      debian_arch                 arm64
	test-arm64-arm64-xl-sedf                      debian_kernkind             pvops
	test-arm64-arm64-xl-sedf                      guests_vcpus                4
	test-arm64-arm64-xl-sedf                      kernbuildjob                build-arm64-pvops
	test-arm64-arm64-xl-sedf                      kernkind                    pvops
	test-arm64-arm64-xl-sedf                      linux_boot_append           loglevel=9 debug
	test-arm64-arm64-xl-sedf                      toolstack                   xl
	test-arm64-arm64-xl-sedf                      xen_boot_append             sched=sedf loglvl=all
	test-arm64-arm64-xl-sedf                      xenbuildjob                 build-arm64
	test-arm64-arm64-xl-sedf-pin                  all_hostflags               arch-arm64,arch-xen-arm64,suite-jessie,purpose-test
	test-arm64-arm64-xl-sedf-pin                  arch                        arm64
	test-arm64-arm64-xl-sedf-pin                  buildjob                    build-arm64
	test-arm64-arm64-xl-sedf-pin                  debian_arch                 arm64
	test-arm64-arm64-xl-sedf-pin                  debian_kernkind             pvops
	test-arm64-arm64-xl-sedf-pin                  guests_vcpus                4
	test-arm64-arm64-xl-sedf-pin                  kernbuildjob                build-arm64-pvops
	test-arm64-arm64-xl-sedf-pin                  kernkind                    pvops
	test-arm64-arm64-xl-sedf-pin                  linux_boot_append           loglevel=9 debug
	test-arm64-arm64-xl-sedf-pin                  toolstack                   xl
	test-arm64-arm64-xl-sedf-pin                  xen_boot_append             sched=sedf loglvl=all dom0_vcpus_pin
	test-arm64-arm64-xl-sedf-pin                  xenbuildjob                 build-arm64
	test-arm64-arm64-xl-xsm                       all_hostflags               arch-arm64,arch-xen-arm64,suite-jessie,purpose-test
	test-arm64-arm64-xl-xsm                       arch                        arm64
	test-arm64-arm64-xl-xsm                       buildjob                    build-arm64-xsm
	test-arm64-arm64-xl-xsm                       debian_arch                 arm64
	test-arm64-arm64-xl-xsm                       debian_kernkind             pvops
	test-arm64-arm64-xl-xsm                       enable_xsm                  true
	test-arm64-arm64-xl-xsm                       kernbuildjob                build-arm64-pvops
	test-arm64-arm64-xl-xsm                       kernkind                    pvops
	test-arm64-arm64-xl-xsm                       toolstack                   xl
	test-arm64-arm64-xl-xsm                       xenbuildjob                 build-arm64-xsm

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
---
 make-flight     |  4 ++--
 mfi-common      | 23 ++++++++++++++++++-----
 ts-kernel-build |  1 +
 3 files changed, 21 insertions(+), 7 deletions(-)

diff --git a/make-flight b/make-flight
index 8a1fceb..3cf640b 100755
--- a/make-flight
+++ b/make-flight
@@ -107,7 +107,7 @@ job_create_test_filter_callback () {
 
 test_matrix_branch_filter_callback () {
   case "$xenarch" in
-  armhf)
+  armhf|arm64)
         case "$branch" in
         linux-3.0) return 1;;
         linux-3.4) return 1;;
@@ -354,7 +354,7 @@ test_matrix_do_one () {
   do_credit2_tests
 
   # No further arm tests at the moment
-  if [ $dom0arch = armhf ]; then
+  if [ $dom0arch = armhf ] || [ $dom0arch = arm64 ]; then
       return
   fi
 
diff --git a/mfi-common b/mfi-common
index a9e966f..ffb6775 100644
--- a/mfi-common
+++ b/mfi-common
@@ -67,12 +67,12 @@ create_build_jobs () {
      BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
   fi
 
-  for arch in ${BUILD_ARCHES- i386 amd64 armhf }; do
+  for arch in ${BUILD_ARCHES- i386 amd64 armhf arm64 }; do
 
     if [ "x$arch" = xdisable ]; then continue; fi
 
     case "$arch" in
-    armhf)
+    armhf|arm64)
       case "$branch" in
       linux-3.0) continue;;
       linux-3.4) continue;;
@@ -247,7 +247,7 @@ create_build_jobs () {
     fi
 
     case "$arch" in
-    armhf) continue;; # don't do any other kernel builds
+    armhf|arm64) continue;; # don't do any other kernel builds
     esac
 
     if [ "x$REVISION_LINUX_OLD" != xdisable ]; then
@@ -319,13 +319,25 @@ test_matrix_iterate () {
   *)                    onetoolstack=xl ;;
   esac
 
-  for xenarch in ${TEST_ARCHES- i386 amd64 armhf } ; do
+  for xenarch in ${TEST_ARCHES- i386 amd64 armhf arm64 } ; do
 
     if [ "x$xenarch" = xdisable ]; then continue; fi
 
     test_matrix_branch_filter_callback || continue
 
     case "$xenarch" in
+    arm64)
+          # Arm64 from 4.5 onwards only
+          case "$xenbranch" in
+          xen-3.*-testing) continue;;
+          xen-4.0-testing) continue;;
+          xen-4.1-testing) continue;;
+          xen-4.2-testing) continue;;
+          xen-4.3-testing) continue;;
+          xen-4.4-testing) continue;;
+          *) ;;
+          esac
+          ;;
     armhf)
           # Arm from 4.3 onwards only
           case "$xenbranch" in
@@ -370,13 +382,14 @@ test_matrix_iterate () {
       *)          echo >&2 "kernkind ?  $kern"; exit 1 ;;
       esac
 
-      for dom0arch in i386 amd64 armhf; do
+      for dom0arch in i386 amd64 armhf arm64; do
 
         case ${xenarch}_${dom0arch} in
             amd64_amd64) ;;
             amd64_i386) ;;
             i386_i386) ;;
             armhf_armhf) ;;
+            arm64_arm64) ;;
             *) continue ;;
         esac
 
diff --git a/ts-kernel-build b/ts-kernel-build
index 629f0cc..c746584 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -27,6 +27,7 @@ builddirsprops();
 
 my $archparms = {
     'armhf' => { DefConfig => 'multi_v7_defconfig', KernelImage => 'arch/arm/boot/zImage' },
+    'arm64' => { DefConfig => 'defconfig',          KernelImage => 'arch/arm64/boot/Image' },
     'i386'  => { DefConfig => 'defconfig',          KernelImage => 'arch/x86/boot/bzImage' },
     'amd64' => { DefConfig => 'defconfig',          KernelImage => 'arch/x86/boot/bzImage' }
 };
-- 
2.1.4

[PATCH OSSTEST 11/15] ts-kernel-build: Additional kernel options for Mustang

[Ian Campbell]

XXX We probably need a newer kernel to actually be useful.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ts-kernel-build | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/ts-kernel-build b/ts-kernel-build
index c746584..cdef97c 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -525,6 +525,16 @@ setopt CONFIG_FUSION_LOGGING n
 
 setopt CONFIG_BLK_CPQ_CISS_DA m
 
+# APM XGENE
+setopt CONFIG_PHY_XGENE y
+#setopt CONFIG_NET_XGENE y
+setopt CONFIG_AHCI_XGENE y
+setopt CONFIG_POWER_RESET_XGENE y
+setopt CONFIG_RTC_DRV_XGENE y
+
+#arm64 crashes without -- determine why
+setopt CONFIG_SPARSEMEM_VMEMMAP y
+
 case ${XEN_TARGET_ARCH} in
     x86_32) setopt CONFIG_64BIT n ;;
     x86_64)
-- 
2.1.4

[PATCH OSSTEST 12/15] ts-host-install: Set dtbs in the non-special kernel case too.

[Ian Campbell]

XXX fold somewhere?
---
 ts-host-install | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/ts-host-install b/ts-host-install
index aab5241..2dad9f1 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -258,6 +258,9 @@ END
 	    if -e "$ho->{Tftp}{Path}/$d_i/$kp-dtbs";
     }
 
+    $xopts{dtbs} = "/$d_i/dtbs"
+	if !$xopts{dtbs} && -e "$ho->{Tftp}{Path}/$d_i/dtbs";
+
     file_simple_write_contents("$initrd_overlay.cpio", sub {
         contents_make_cpio($_[0], 'newc', "$initrd_overlay.d");
     });
-- 
2.1.4

[PATCH OSSTEST 13/15] Debian: Fixup UEFI boot order during install

[Ian Campbell]

Debian inserts itself before any existing entries, including the PXE
one, meaning we otherwise cannot remotely regroove the box. Preseed
some commands to reset the boot order to BootCurrent i.e. how we
booted (so the PXE entry).

There is still a window between the Debian entry being added (by
grub-installer.udeb) and us removing it, a failure (e.g. power cycle)
in that interval would require manual action to fixup.

This is triggered by a new quirk-uefi-bootorder-reset how flag.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---

This seems to just be normal/expected behaviour for UEFI installs (not
just of Debian), so perhaps the quirk is unneeded.
---
 Osstest/Debian.pm | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index e89ca6c..a46ffd8 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -942,6 +942,21 @@ d-i partman-auto/expert_recipe string					\\
 
 END
 
+    if (get_host_property($ho, "firmware") eq "uefi" &&
+        $ho->{Flags}{'quirk-uefi-bootorder-reset'}) {
+	# Disable any new Debian boot entry, so we reboot from PXE.
+        preseed_hook_command($ho, 'late_command', $sfx, <<'END');
+#!/bin/sh
+set -ex
+
+entry=`in-target --pass-stdout bash -c 'efibootmgr -v | sed -ne "s/BootCurrent: \([0-9]\+\)/\1/p"'` #/
+in-target efibootmgr -o $entry
+
+#in-target bash -c 'efibootmgr -o `efibootmgr -v | sed -ne "s/BootCurrent: \([0-9]\+\)/\1/p"`' #/
+
+END
+    }
+
     $preseed_file .= preseed_hook_cmds();
 
     if ($ho->{Flags}{'no-di-kernel'}) {
-- 
2.1.4

[PATCH OSSTEST 14/15] ts-kernel-build: Support --reuse to keep same build tree

[Ian Campbell]

This is very useful when iterating over kernel configurations, since
it avoids blowing away the build tree and all the existing built
objects. The Linux build system does the right thing when .config
changes and only rebuilds the affected bits.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ts-kernel-build | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/ts-kernel-build b/ts-kernel-build
index cdef97c..84cbd15 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -22,6 +22,18 @@ use Osstest::TestSupport;
 use Osstest::BuildSupport;
 
 tsreadconfig();
+
+our $reuse = 0;
+while (@ARGV and $ARGV[0] =~ m/^-/) {
+    $_= shift @ARGV;
+    last if m/^--$/;
+    if (m/^--reuse$/) {
+	$reuse = 1;
+    } else {
+	die "$_ ?";
+    }
+}
+
 selectbuildhost(\@ARGV);
 builddirsprops();
 
@@ -35,6 +47,8 @@ my $archparms = {
 sub enable_xen_config ();
 
 sub checkout () {
+    return if $reuse;
+
     prepbuilddirs();
 
     build_clone($ho, 'linux', $builddir, 'linux');
-- 
2.1.4

[PATCH OSSTEST 15/15] standalone: Prefer ./standalone.config to $HOME/.xen-osstest/config

[Ian Campbell]

OSSTEST_CONFIG still trumps both.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 standalone | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/standalone b/standalone
index 17fa40c..ad12bad 100755
--- a/standalone
+++ b/standalone
@@ -68,7 +68,13 @@ TEMP=$(getopt -o c:f:h:rRs --long config:,flight:,host:,reuse,noreuse,reinstall,
 
 eval set -- "$TEMP"
 
-config=${OSSTEST_CONFIG-$HOME/.xen-osstest/config}
+if [ -n "${OSSTEST_CONFIG}" ]; then
+    config=${OSSTEST_CONFIG}
+elif [ -f standalone.config ]; then
+    config=standalone.config
+else
+    config=$HOME/.xen-osstest/config
+fi
 flight="standalone"
 host=
 reuse=1 # Don't blow away machines by default
-- 
2.1.4

Re: [PATCH OSSTEST 01/15] mg-debian-installer-update: updates to better handle Jessie onwards.

[Wei Liu]

On Fri, May 22, 2015 at 01:29:38PM +0100, Ian Campbell wrote:
> In mg-debian-installer-update:
> 
>   - Expand the list of (suite,arch) combinations which don't exist and
>     move it to the top.
> 
>   - Tweak the backports.org handling to allow it to be specified on a
>     per (suite,arch) basis, and specify it only for (wheezy,armhf)
>     since it is not currently needed for jessie. Make the
>     initramfs-tools fetching conditional too.
> 
>   - Only extract backports-dtbs if they are present in the package.
> 
>   - Handle arm64.
> 
>   - Collect the correct armhf kernel for Jessie onwards (vexpress
>     removed from path).
> 
> Have mg-debian-installer-update-all fetch arm64 too.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> Cc: Wei Liu <wei.liu2@citrix.com>

Thanks. I will replace my patch with this one.

Wei.

Re: [PATCH OSSTEST 01/15] mg-debian-installer-update: updates to better handle Jessie onwards.

[Wei Liu]

On Mon, May 25, 2015 at 09:36:06AM +0100, Wei Liu wrote:
> On Fri, May 22, 2015 at 01:29:38PM +0100, Ian Campbell wrote:
> > In mg-debian-installer-update:
> > 
> >   - Expand the list of (suite,arch) combinations which don't exist and
> >     move it to the top.
> > 
> >   - Tweak the backports.org handling to allow it to be specified on a
> >     per (suite,arch) basis, and specify it only for (wheezy,armhf)
> >     since it is not currently needed for jessie. Make the
> >     initramfs-tools fetching conditional too.
> > 
> >   - Only extract backports-dtbs if they are present in the package.
> > 
> >   - Handle arm64.
> > 
> >   - Collect the correct armhf kernel for Jessie onwards (vexpress
> >     removed from path).
> > 
> > Have mg-debian-installer-update-all fetch arm64 too.
> > 
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > Cc: Wei Liu <wei.liu2@citrix.com>
> 
> Thanks. I will replace my patch with this one.
> 

Actually when I tried to do "git am" for this patch, it got rejected.
Which changeset should I base it on?

Wei.

> Wei.

Re: [PATCH OSSTEST 01/15] mg-debian-installer-update: updates to better handle Jessie onwards.

[Ian Campbell]

On Mon, 2015-05-25 at 09:57 +0100, Wei Liu wrote:
> On Mon, May 25, 2015 at 09:36:06AM +0100, Wei Liu wrote:
> > On Fri, May 22, 2015 at 01:29:38PM +0100, Ian Campbell wrote:
> > > In mg-debian-installer-update:
> > > 
> > >   - Expand the list of (suite,arch) combinations which don't exist and
> > >     move it to the top.
> > > 
> > >   - Tweak the backports.org handling to allow it to be specified on a
> > >     per (suite,arch) basis, and specify it only for (wheezy,armhf)
> > >     since it is not currently needed for jessie. Make the
> > >     initramfs-tools fetching conditional too.
> > > 
> > >   - Only extract backports-dtbs if they are present in the package.
> > > 
> > >   - Handle arm64.
> > > 
> > >   - Collect the correct armhf kernel for Jessie onwards (vexpress
> > >     removed from path).
> > > 
> > > Have mg-debian-installer-update-all fetch arm64 too.
> > > 
> > > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > > Cc: Wei Liu <wei.liu2@citrix.com>
> > 
> > Thanks. I will replace my patch with this one.
> > 
> 
> Actually when I tried to do "git am" for this patch, it got rejected.
> Which changeset should I base it on?

ecf4cb7 Revert "cs-bisection-step: allow -bisect blessed flights for basis pass"

plus the other 4 patches from your series, I pushed the whole lot to
xenbits per below.

Ian.


The following changes since commit ecf4cb76b15396c85fed5a2eb0a6a60e381225c5:

  Revert "cs-bisection-step: allow -bisect blessed flights for basis pass" (2015-05-20 14:09:33 +0100)

are available in the git repository at:

  git://xenbits.xen.org/people/ianc/osstest.git arm64-v1

for you to fetch changes up to 33871e796487f016f69ed40c3f8a06e472a34664:

  standalone: Prefer ./standalone.config to $HOME/.xen-osstest/config (2015-05-22 13:26:55 +0100)

----------------------------------------------------------------
Ian Campbell (16):
      More flexible handling of need-kernel-deb-$flavour host flag
      mg-debian-installer-update: updates to better handle Jessie onwards.
      mg-debian-installer-update: Also collect DTBs for standard kernels
      Introduce mg-pxe-loader-update
      Debian: Preseed a EFI system partition during host install
      ts-host-install: Split initrd out of @installcmdline
      ts-host-install: split the "di" from the "host" command line
      Refactor pxelinux configuration
      ts-host-install: Support UEFI PXE boot using grub.efi
      Enable chain loading to local disk for UEFI PXE systems.
      Add arm64 build and test jobs
      ts-kernel-build: Additional kernel options for Mustang
      ts-host-install: Set dtbs in the non-special kernel case too.
      Debian: Fixup UEFI boot order during install
      ts-kernel-build: Support --reuse to keep same build tree
      standalone: Prefer ./standalone.config to $HOME/.xen-osstest/config

Wei Liu (4):
      Debian.pm: install grub to default device
      Cope with Jessie's d-i vg name
      ts-xen-build-prep: install libnl-route-3-dev on Jessie
      ts-xen-build-prep: install libc6-dev-i386 for jessie

 Osstest.pm                     |  4 ++++
 Osstest/Debian.pm              | 29 ++++++++++++++++++++++++++++-
 Osstest/TestSupport.pm         | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----
 README                         |  5 ++++-
 make-flight                    |  4 ++--
 mfi-common                     | 23 ++++++++++++++++++-----
 mg-debian-installer-update     | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++++++----------------------------
 mg-debian-installer-update-all |  2 +-
 mg-pxe-loader-update           | 89 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 production-config              |  4 ++++
 standalone                     |  8 +++++++-
 ts-host-install                | 61 ++++++++++++++++++++++++++++++++-----------------------------
 ts-kernel-build                | 25 +++++++++++++++++++++++++
 ts-xen-build-prep              | 37 +++++++++++++++++++++----------------
 14 files changed, 382 insertions(+), 88 deletions(-)
 create mode 100755 mg-pxe-loader-update

Re: [PATCH OSSTEST v1 00/15] Initial support for ARM64

[Ian Campbell]

On Fri, 2015-05-22 at 13:29 +0100, Ian Campbell wrote:
> 
> This has been tested on a Mustang board booting UEFI and build-* works
> with one (slightly major) exception: The Debian kernel does not
> currently reboot correctly on this system. This will likely require a
> kernel from backports in order to work correctly.

In fact this will be fixed by a new kernel in 8.1 (first Jessie point
release), which is due mid June -- that's likely before we will have any
h/w...

Ian.