[CI 00/17] Gitlab CI and test framework improvements

[CI 00/17] Gitlab CI and test framework improvements

[Glenn Washburn]

This patch series is divided into roughly three sections: modifications of
tests, improvements to grub-shell, and the CI patch itself. I have other
improvements to testing coming, but I consider these more essential to the
quality of the CI tests, or highly desirable. Aside from modification of
qemu firmware paths to make them more generic, the changes should not be
visible to current testers.

The CI change is the addition of a .gitlab-ci.yml file which is the GitLab
specific CI config file and a .ci directory with scripts used by the CI
config. The scripts in the .ci dir are meant to be generic and thus usable
by other CI platforms. There is probably more that can be pulled out of
the .gitlab-ci.yml file for use in common with other CI configs, but I
decided those can be pulled out as needed.

The successful CI run can be viewed here:
  https://gitlab.com/gnu-grub/grub/-/pipelines/258485608

Glenn

Glenn Washburn (17):
  tests: Make sure LANG is set properly for iso9660_test
  tests: Fix partmap_test for arm*-efi, disk numbering has changed
  tests: When checking squashfs fstime, use superblock last modified
    time
  tests: Fail immediately when grub-shell fails and do not occlude the
    error code
  grub-shell: Allow specifying non-default trim line contents
  grub-shell: Trim line should always be matched from the beginning of
    the line
  grub-shell: Only show grub-mkrescue output if it returns an error
  grub-shell: Allow setting default timeout via
    GRUB_SHELL_DEFAULT_TIMEOUT envvar
  grub-shell: Put all generated files into working dir and use better
    file names
  grub-shell: Add grub output logfile with grub-shell --debug
  grub-shell: Set exit status to qemu exit status
  tests: Allow turning on shell tracing from environment variables
  grub-shell: Add --verbose to mkrescue when $debug is greater than 2
  grub-shell: Use malta qemu-mips machine type instead off non-existant
    indy
  grub-shell: Update qemu UEFI firmware names to be more generic
  ci: Add .gitlab-ci.yml for continuous integration with Gitlab CI
  keep: remove a lot of cruft that we probably want to reuse at some
    point

 .ci/build.sh                   |  56 +++
 .ci/functions.gitlab.sh        |  30 ++
 .ci/functions.sh               |  30 ++
 .ci/make-images.sh             |  71 ++++
 .ci/process-tests.sh           |  97 +++++
 .ci/test.sh                    |  98 +++++
 .gitlab-ci.yml                 | 632 +++++++++++++++++++++++++++++++++
 tests/ahci_test.in             |   6 +-
 tests/cdboot_test.in           |   3 +-
 tests/core_compress_test.in    |   6 +-
 tests/ehci_test.in             |   6 +-
 tests/fddboot_test.in          |   3 +-
 tests/grub_cmd_date.in         |   3 +-
 tests/grub_cmd_test.in         |   1 +
 tests/grub_script_blockarg.in  |   2 +-
 tests/grub_script_expansion.in |   3 +-
 tests/gzcompress_test.in       |   3 +-
 tests/hddboot_test.in          |   3 +-
 tests/iso9660_test.in          |   6 +
 tests/lzocompress_test.in      |   3 +-
 tests/netboot_test.in          |   3 +-
 tests/ohci_test.in             |   6 +-
 tests/partmap_test.in          |   6 +-
 tests/pata_test.in             |   3 +-
 tests/test_sha512sum.in        |   1 +
 tests/uhci_test.in             |   6 +-
 tests/util/grub-fs-tester.in   |   8 +
 tests/util/grub-shell.in       |  87 +++--
 tests/xzcompress_test.in       |   3 +-
 29 files changed, 1138 insertions(+), 47 deletions(-)
 create mode 100755 .ci/build.sh
 create mode 100644 .ci/functions.gitlab.sh
 create mode 100644 .ci/functions.sh
 create mode 100755 .ci/make-images.sh
 create mode 100755 .ci/process-tests.sh
 create mode 100755 .ci/test.sh
 create mode 100644 .gitlab-ci.yml

-- 
2.27.0

[CI 01/17] tests: Make sure LANG is set properly for iso9660_test

[Glenn Washburn]

LANG must be set to something that supports international characters,
otherwise xorriso will refuse to include the file with name having
international characters, causing the test to fail. So if LANG is not set,
set it to en_US.UTF-8, a very common UTF-8 locale. And if it is set, but
does not look like a UTF-8 locale, print a warning so the user will have a
clue as to why the iso9660_test might be failing.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/iso9660_test.in | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/tests/iso9660_test.in b/tests/iso9660_test.in
index 571b938d7..37b1d30af 100644
--- a/tests/iso9660_test.in
+++ b/tests/iso9660_test.in
@@ -7,6 +7,12 @@ if ! which xorriso >/dev/null 2>&1; then
    exit 77
 fi
 
+if [ -z "$LANG" ]; then
+   export LANG=en_US.UTF-8
+elif [ -n "${LANG##*UTF*}" ]; then
+   echo "WARNING: LANG=$LANG appears to not be unicode, international file test may fail."
+fi
+
 "@builddir@/grub-fs-tester" joliet
 "@builddir@/grub-fs-tester" rockridge
 "@builddir@/grub-fs-tester" rockridge_joliet
-- 
2.27.0

[CI 02/17] tests: Fix partmap_test for arm*-efi, disk numbering has changed

[Glenn Washburn]

Perhaps using a newer UEFI firmware is the reason for the created test disk
showing up as hd2 instead of hd3.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/partmap_test.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/partmap_test.in b/tests/partmap_test.in
index 6ef518b0a..7353dc70e 100644
--- a/tests/partmap_test.in
+++ b/tests/partmap_test.in
@@ -89,7 +89,7 @@ case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in
 	disk=arc/scsi0/disk0/rdisk0
 	;;
     arm*-efi)
-	disk=hd3
+	disk=hd2
 	;;
     *)
 	disk=hd0
-- 
2.27.0

[CI 03/17] tests: When checking squashfs fstime, use superblock last modified time

[Glenn Washburn]

Currently, the filesystem timestamp check in grub-fs-tester uses the
squashfs image file's last modified timestamp and checks to see if that
time stamp is within 3 seconds of the superblock timestamp as determined by
grub. The image file's timestamp could be more than 3 seconds off if
mksquashfs takes more than 3 seconds to generate the image, as is the case
on a low power machine. Instead use squashfs tools to get the filesystem
timestamp directly.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-fs-tester.in | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/tests/util/grub-fs-tester.in b/tests/util/grub-fs-tester.in
index bfc425e1f..4213b7bfc 100644
--- a/tests/util/grub-fs-tester.in
+++ b/tests/util/grub-fs-tester.in
@@ -1351,6 +1351,12 @@ for LOGSECSIZE in $(range "$MINLOGSECSIZE" "$MAXLOGSECSIZE" 1); do
 			# With some abstractions like mdraid flushing to disk
 			# may be delayed for a long time.
 			FSTIME="$UMOUNT_TIME";;
+		    xsquash*)
+			# Creating the squash image may take more than a few
+			# seconds. Use the more accurate timestamp from the
+			# superblock.
+			FSTIME="$(unsquashfs -s "${FSIMAGEP}0.img" | grep ^Creation | awk '{print $6 " " $7 " " $8 " " $9 " " $10; }')"
+			FSTIME="$(date -d "$FSTIME" -u '+%Y-%m-%d %H:%M:%S')";;
 		    *)
 			FSTIME="$(TZ=UTC ls --time-style="+%Y-%m-%d_%H:%M:%S" -l -d "${FSIMAGEP}0.img"|awk '{print $6; }'|sed 's,_, ,g')";;
 		esac
-- 
2.27.0

[CI 04/17] tests: Fail immediately when grub-shell fails and do not occlude the error code

[Glenn Washburn]

If grub-shell fails, that means that whatever was being tested was not
actually tested. So fail immediately. Sometimes grub-shell is not the last
command in a pipeline of several commands, and in this case the failed error
code can be hidden by a later failing command or hidden when 'set -e' is not
set and there are subsequent successful commands. When the test script fails
because of a failure in grub-shell, then test script should exit with the
failed exit code of grub-shell.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/ahci_test.in             | 6 +++++-
 tests/cdboot_test.in           | 3 ++-
 tests/core_compress_test.in    | 6 ++++--
 tests/ehci_test.in             | 6 +++++-
 tests/fddboot_test.in          | 3 ++-
 tests/grub_cmd_date.in         | 3 ++-
 tests/grub_cmd_test.in         | 1 +
 tests/grub_script_blockarg.in  | 2 +-
 tests/grub_script_expansion.in | 3 ++-
 tests/gzcompress_test.in       | 3 ++-
 tests/hddboot_test.in          | 3 ++-
 tests/lzocompress_test.in      | 3 ++-
 tests/netboot_test.in          | 3 ++-
 tests/ohci_test.in             | 6 +++++-
 tests/partmap_test.in          | 4 ++--
 tests/pata_test.in             | 3 ++-
 tests/test_sha512sum.in        | 1 +
 tests/uhci_test.in             | 6 +++++-
 tests/xzcompress_test.in       | 3 ++-
 19 files changed, 49 insertions(+), 19 deletions(-)

diff --git a/tests/ahci_test.in b/tests/ahci_test.in
index 7df560462..3fb7f54a2 100644
--- a/tests/ahci_test.in
+++ b/tests/ahci_test.in
@@ -41,7 +41,11 @@ echo "hello" > "$outfile"
 
 tar cf "$imgfile" "$outfile"
 
-if [ "$(echo "nativedisk; source '(ahci0)/$outfile';" | "${grubshell}" --qemu-opts="-drive id=disk,file=$imgfile,if=none -device ahci,id=ahci -device ide-drive,drive=disk,bus=ahci.0 " | tail -n 1)" != "Hello World" ]; then
+echo "nativedisk; source '(ahci0)/$outfile';" |
+    "${grubshell}" --qemu-opts="-drive id=disk,file=$imgfile,if=none
+				-device ahci,id=ahci
+				-device ide-drive,drive=disk,bus=ahci.0" >$outfile
+if [ "$(cat "$outfile" | tail -n 1)" != "Hello World" ]; then
    rm "$imgfile"
    rm "$outfile"
    exit 1
diff --git a/tests/cdboot_test.in b/tests/cdboot_test.in
index 75acdfedb..7229f79fb 100644
--- a/tests/cdboot_test.in
+++ b/tests/cdboot_test.in
@@ -34,6 +34,7 @@ case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in
 	exit 0;;
 esac
 
-if [ "$(echo hello | "${grubshell}" --boot=cd)" != "Hello World" ]; then
+v=`echo hello | "${grubshell}" --boot=cd`
+if [ "$v" != "Hello World" ]; then
    exit 1
 fi
diff --git a/tests/core_compress_test.in b/tests/core_compress_test.in
index 9d216ebcf..90dd00607 100644
--- a/tests/core_compress_test.in
+++ b/tests/core_compress_test.in
@@ -27,10 +27,12 @@ case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in
 esac
 
 
-if [ "$(echo hello | "${grubshell}" --grub-mkimage-extra=--compress=xz)" != "Hello World" ]; then
+v=`echo hello | "${grubshell}" --grub-mkimage-extra=--compress=xz`
+if [ "$v" != "Hello World" ]; then
    exit 1
 fi
 
-if [ "$(echo hello | "${grubshell}" --grub-mkimage-extra=--compress=none)" != "Hello World" ]; then
+v=`echo hello | "${grubshell}" --grub-mkimage-extra=--compress=none`
+if [ "$v" != "Hello World" ]; then
    exit 1
 fi
diff --git a/tests/ehci_test.in b/tests/ehci_test.in
index b197f8cdc..64bd80070 100644
--- a/tests/ehci_test.in
+++ b/tests/ehci_test.in
@@ -41,7 +41,11 @@ echo "hello" > "$outfile"
 
 tar cf "$imgfile" "$outfile"
 
-if [ "$(echo "nativedisk; source '(usb0)/$outfile';" | "${grubshell}" --qemu-opts="-device ich9-usb-ehci1 -drive id=my_usb_disk,file=$imgfile,if=none -device usb-storage,drive=my_usb_disk" | tail -n 1)" != "Hello World" ]; then
+echo "nativedisk; source '(usb0)/$outfile';" |
+    "${grubshell}" --qemu-opts="-device ich9-usb-ehci1
+				-drive id=my_usb_disk,file=$imgfile,if=none
+				-device usb-storage,drive=my_usb_disk" >$outfile
+if [ "$(cat "$outfile" | tail -n 1)" != "Hello World" ]; then
    rm "$imgfile"
    rm "$outfile"
    exit 1
diff --git a/tests/fddboot_test.in b/tests/fddboot_test.in
index 2d7dfc889..1bbe60ee5 100644
--- a/tests/fddboot_test.in
+++ b/tests/fddboot_test.in
@@ -46,6 +46,7 @@ case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in
 	exit 0;;
 esac
 
-if [ "$(echo hello | "${grubshell}" --boot=fd --mkrescue-arg="--compress=xz --fonts= --locales= --themes= -no-pad")" != "Hello World" ]; then
+v=`echo hello | "${grubshell}" --boot=fd --mkrescue-arg="--compress=xz --fonts= --locales= --themes= -no-pad"`
+if [ "$v" != "Hello World" ]; then
    exit 1
 fi
diff --git a/tests/grub_cmd_date.in b/tests/grub_cmd_date.in
index f7c9ca004..f9156691e 100644
--- a/tests/grub_cmd_date.in
+++ b/tests/grub_cmd_date.in
@@ -9,7 +9,8 @@ if [ "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" = sparc64-ieee1275 ];
 fi
 
 pdt="$(date -u +%s)"
-dt=`echo date | @builddir@/grub-shell | sed 's, [A-Z][a-z]*$,,'`
+dt=`echo date | @builddir@/grub-shell`
+dt=`echo "$dt" | sed 's, [A-Z][a-z]*$,,'`
 dtg="$(date -u -d "$dt" +%s)"
 ndt="$(date -u +%s)"
 
diff --git a/tests/grub_cmd_test.in b/tests/grub_cmd_test.in
index 3399eb292..dac6f7d6a 100644
--- a/tests/grub_cmd_test.in
+++ b/tests/grub_cmd_test.in
@@ -1,4 +1,5 @@
 #! @BUILD_SHEBANG@
+set -e
 
 # create a randome file
 empty="`mktemp "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX"`" || exit 1
diff --git a/tests/grub_script_blockarg.in b/tests/grub_script_blockarg.in
index 6ea9b8c3d..6152cc141 100644
--- a/tests/grub_script_blockarg.in
+++ b/tests/grub_script_blockarg.in
@@ -1,4 +1,4 @@
-#! @BUILD_SHEBANG@
+#! @BUILD_SHEBANG@ -e
 
 # Run GRUB script in a Qemu instance
 # Copyright (C) 2010  Free Software Foundation, Inc.
diff --git a/tests/grub_script_expansion.in b/tests/grub_script_expansion.in
index 9d0dcdd29..98d5a9068 100644
--- a/tests/grub_script_expansion.in
+++ b/tests/grub_script_expansion.in
@@ -17,7 +17,8 @@ set -e
 # You should have received a copy of the GNU General Public License
 # along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
 
-disks=`echo ls | @builddir@/grub-shell| grep -av '^Network protocols:$'| grep -av '^tftp http $'`
+disks=`echo ls | @builddir@/grub-shell`
+disks=`echo "$disks"| grep -av '^Network protocols:$'| grep -av '^tftp http $'`
 other=`echo insmod regexp\; echo \* | @builddir@/grub-shell`
 for d in $disks; do
     if echo "$d" |grep ',' >/dev/null; then
diff --git a/tests/gzcompress_test.in b/tests/gzcompress_test.in
index 42c8fe7c4..572b2a1a5 100644
--- a/tests/gzcompress_test.in
+++ b/tests/gzcompress_test.in
@@ -24,6 +24,7 @@ if ! which gzip >/dev/null 2>&1; then
    exit 77
 fi
 
-if [ "$(echo hello | "${grubshell}" --mkrescue-arg=--compress=gz)" != "Hello World" ]; then
+v=`echo hello | "${grubshell}" --mkrescue-arg=--compress=gz`
+if [ "$v" != "Hello World" ]; then
    exit 1
 fi
diff --git a/tests/hddboot_test.in b/tests/hddboot_test.in
index 6d70847a5..abe3e0ae1 100644
--- a/tests/hddboot_test.in
+++ b/tests/hddboot_test.in
@@ -31,7 +31,8 @@ case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in
 	exit 0;;
 esac
 
-if [ "$(echo hello | "${grubshell}" --boot=hd)" != "Hello World" ]; then
+v=`echo hello | "${grubshell}" --boot=hd`
+if [ "$v" != "Hello World" ]; then
    exit 1
 fi
 
diff --git a/tests/lzocompress_test.in b/tests/lzocompress_test.in
index 4e5f7e078..8f08a8046 100644
--- a/tests/lzocompress_test.in
+++ b/tests/lzocompress_test.in
@@ -24,6 +24,7 @@ if ! which lzop >/dev/null 2>&1; then
    exit 77
 fi
 
-if [ "$(echo hello | "${grubshell}" --mkrescue-arg=--compress=lzo)" != "Hello World" ]; then
+v=`echo hello | "${grubshell}" --mkrescue-arg=--compress=lzo`
+if [ "$v" != "Hello World" ]; then
    exit 1
 fi
diff --git a/tests/netboot_test.in b/tests/netboot_test.in
index 9f71e3d88..6a1d95a22 100644
--- a/tests/netboot_test.in
+++ b/tests/netboot_test.in
@@ -40,6 +40,7 @@ case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in
 	exit 0;;
 esac
 
-if [ "$(echo hello | "${grubshell}" --boot=net)" != "Hello World" ]; then
+v=`echo hello | "${grubshell}" --boot=net`
+if [ "$v" != "Hello World" ]; then
    exit 1
 fi
diff --git a/tests/ohci_test.in b/tests/ohci_test.in
index 8693f8c47..cc35f67a9 100644
--- a/tests/ohci_test.in
+++ b/tests/ohci_test.in
@@ -41,7 +41,11 @@ echo "hello" > "$outfile"
 
 tar cf "$imgfile" "$outfile"
 
-if [ "$(echo "nativedisk; source '(usb0)/$outfile';" | "${grubshell}" --qemu-opts="-device pci-ohci -drive id=my_usb_disk,file=$imgfile,if=none -device usb-storage,drive=my_usb_disk" | tail -n 1)" != "Hello World" ]; then
+echo "nativedisk; source '(usb0)/$outfile';" |
+    "${grubshell}" --qemu-opts="-device pci-ohci
+				-drive id=my_usb_disk,file=$imgfile,if=none
+				-device usb-storage,drive=my_usb_disk" >$outfile
+if [ "$(cat "$outfile" | tail -n 1)" != "Hello World" ]; then
    rm "$imgfile"
    rm "$outfile"
    exit 1
diff --git a/tests/partmap_test.in b/tests/partmap_test.in
index 7353dc70e..5f18ab51c 100644
--- a/tests/partmap_test.in
+++ b/tests/partmap_test.in
@@ -58,8 +58,8 @@ list_parts () {
     shift
 
     echo ls | "${grubshell}" --disk="${imgfile}" \
-	--modules=$mod | tr -d "\n\r" > "${outfile}"
-    cat "${outfile}"
+	--modules=$mod > "${outfile}"
+    cat "${outfile}" | tr -d "\n\r"
     echo
 }
 
diff --git a/tests/pata_test.in b/tests/pata_test.in
index 4b18fdef3..3d19cecde 100644
--- a/tests/pata_test.in
+++ b/tests/pata_test.in
@@ -45,7 +45,8 @@ echo "hello" > "$outfile"
 
 tar cf "$imgfile" "$outfile"
 
-if [ "$(echo "nativedisk; source '($indisk)/$outfile';" | "${grubshell}" --qemu-opts="-$disk $imgfile")" != "Hello World" ]; then
+v=`echo "nativedisk; source '($indisk)/$outfile';" | "${grubshell}" --qemu-opts="-$disk $imgfile"`
+if [ "$v" != "Hello World" ]; then
    rm "$imgfile"
    rm "$outfile"
    exit 1
diff --git a/tests/test_sha512sum.in b/tests/test_sha512sum.in
index 027092a8b..d97b7ae2c 100644
--- a/tests/test_sha512sum.in
+++ b/tests/test_sha512sum.in
@@ -1,4 +1,5 @@
 #! @BUILD_SHEBANG@
+set -e
 
 # create a randome file
 file="`mktemp "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX"`" || exit 1
diff --git a/tests/uhci_test.in b/tests/uhci_test.in
index f0eec5032..6aab10ba6 100644
--- a/tests/uhci_test.in
+++ b/tests/uhci_test.in
@@ -41,7 +41,11 @@ echo "hello" > "$outfile"
 
 tar cf "$imgfile" "$outfile"
 
-if [ "$(echo "nativedisk; source '(usb0)/$outfile';" | "${grubshell}" --qemu-opts="-device ich9-usb-uhci1 -drive id=my_usb_disk,file=$imgfile,if=none -device usb-storage,drive=my_usb_disk" | tail -n 1)" != "Hello World" ]; then
+echo "nativedisk; source '(usb0)/$outfile';" |
+    "${grubshell}" --qemu-opts="-device ich9-usb-uhci1
+				-drive id=my_usb_disk,file=$imgfile,if=none
+				-device usb-storage,drive=my_usb_disk" >$outfile
+if [ "$(cat "$outfile" | tail -n 1)" != "Hello World" ]; then
    rm "$imgfile"
    rm "$outfile"
    exit 1
diff --git a/tests/xzcompress_test.in b/tests/xzcompress_test.in
index 03bfb5e95..61acba33b 100644
--- a/tests/xzcompress_test.in
+++ b/tests/xzcompress_test.in
@@ -24,6 +24,7 @@ if ! which xz >/dev/null 2>&1; then
    exit 77
 fi
 
-if [ "$(echo hello | "${grubshell}" --mkrescue-arg=--compress=xz)" != "Hello World" ]; then
+v=`echo hello | "${grubshell}" --mkrescue-arg=--compress=xz`
+if [ "$v" != "Hello World" ]; then
    exit 1
 fi
-- 
2.27.0

[CI 05/17] grub-shell: Allow specifying non-default trim line contents

[Glenn Washburn]

This will be useful for tests that have unwanted output from setup. This is
not documented because its only intended to be internal at the moment. Also,
--no-trim is allowed to explicitly turn off trim.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-shell.in | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index 93e9f5148..b409962f1 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -32,6 +32,7 @@ PATH="${builddir}:$PATH"
 export PATH
 
 trim=0
+trim_head=664cbea8-132f-4770-8aa4-1696d59ac35c
 
 # Usage: usage
 # Print the usage.
@@ -212,8 +213,13 @@ for option in "$@"; do
 	echo "$0 (GNU GRUB ${PACKAGE_VERSION})"
 	exit 0 ;;
     --trim)
-	trim=1
+	trim=1 ;;
+    --trim=*)
+	trim=2
+	trim_head=`echo "$option" | sed -e 's/--trim=//' -e 's/,/ /g'`
 	;;
+    --no-trim)
+	trim=0 ;;
     --debug)
         debug=1 ;;
     --modules=*)
@@ -336,8 +342,6 @@ terminal_input ${term}
 terminal_output ${term}
 EOF
 
-trim_head=664cbea8-132f-4770-8aa4-1696d59ac35c
-
 if [ $trim = 1 ]; then
     echo "echo $trim_head" >>${cfgfile}
 fi
@@ -452,8 +456,8 @@ fi
 
 do_trim ()
 {
-    if [ $trim = 1 ]; then
-	awk '{ if (have_head == 1) print $0; } /664cbea8-132f-4770-8aa4-1696d59ac35c/ { have_head=1; }'
+    if [ $trim = 1 ] || [ $trim = 2 ]; then
+	awk '{ if (have_head == 1) print $0; } /'"$trim_head"'/ { have_head=1; }'
     else
 	cat
     fi
-- 
2.27.0

[CI 06/17] grub-shell: Trim line should always be matched from the beginning of the line

[Glenn Washburn]

When turning on shell tracing the trim line will be output before we
actually want to start the trim. However, in this case the trim line never
starts from the beginning of the line. So start trimming from the correct
line by matching from the beginning of the line.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-shell.in | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index b409962f1..602b16f3e 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -343,7 +343,7 @@ terminal_output ${term}
 EOF
 
 if [ $trim = 1 ]; then
-    echo "echo $trim_head" >>${cfgfile}
+    echo "echo; echo $trim_head" >>${cfgfile}
 fi
 
 rom_directory=`mktemp -d "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX"` || exit 1
@@ -457,7 +457,7 @@ fi
 do_trim ()
 {
     if [ $trim = 1 ] || [ $trim = 2 ]; then
-	awk '{ if (have_head == 1) print $0; } /'"$trim_head"'/ { have_head=1; }'
+	awk '{ if (have_head == 1) print $0; } /^'"$trim_head"'/ { have_head=1; }'
     else
 	cat
     fi
-- 
2.27.0

[CI 07/17] grub-shell: Only show grub-mkrescue output if it returns an error

[Glenn Washburn]

The previous behavior ignored an error and the output from grub-mkrescue.
This made it a pain to discover that grub-mkrescue was the reason that tests
which rely on grub-shell were failing.  Even after discovering grub-mkrescue
was the culprit, there was no output to indicate why it was failing.  It
turns out that grub-mkrescue is a thin wrapper around xorriso. So if you do
not have xorriso installed it will fail with an error message about not
being able to find xorriso.

This change will allow grub-mkrescue output to be written to stderr, only if
grub-mkrescue fails.  If grub-mkrescue succeeds, there will be no output
from grub-mkrescue so as not to interfere with the functioning of tests.
This change should have no effect on the running of tests or other uses of
grub-shell as it only modifies the error path.

Also, if grub-mkrescue fails, the script exits early.  Since grub-shell
needs the iso image created by grub-mkresue to boot the qemu instance, a
failure here should be considered fatal.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-shell.in | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index 602b16f3e..9d8c417da 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -60,6 +60,17 @@ Report bugs to <bug-grub@gnu.org>.
 EOF
 }
 
+# Exec given argv and only show its output on STDERR if it returns an
+# error status.
+exec_show_error () {
+    v=`$@ 2>&1`
+    ret=$?
+    if [ "$ret" != 0 ]; then
+        echo "$v" >&2
+        exit $ret
+    fi
+}
+
 . "${builddir}/grub-core/modinfo.sh"
 qemuopts="${GRUB_QEMU_OPTS}"
 serial_port=com0
@@ -383,13 +394,15 @@ if test -z "$debug"; then
 fi
 
 if [ x$boot != xnet ] && [ x$boot != xemu ]; then
-    pkgdatadir="@builddir@" "@builddir@/grub-mkrescue" "--output=${isofile}" "--override-directory=${builddir}/grub-core" \
+    pkgdatadir="@builddir@" \
+    exec_show_error "@builddir@/grub-mkrescue" "--output=${isofile}" \
+	"--override-directory=${builddir}/grub-core" \
 	--rom-directory="${rom_directory}" \
 	--locale-directory="@srcdir@/po" \
 	--themes-directory="@srcdir@/themes" \
 	$mkimage_extra_arg ${mkrescue_args} \
 	"/boot/grub/grub.cfg=${cfgfile}" "/boot/grub/testcase.cfg=${source}" \
-	${files} >/dev/null 2>&1
+	${files} || exit $?
 fi
 if [ x$boot = xhd ]; then
     if [ "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" = arm64-efi ] || [ "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" = arm-efi ]; then
-- 
2.27.0

[CI 08/17] grub-shell: Allow setting default timeout via GRUB_SHELL_DEFAULT_TIMEOUT envvar

[Glenn Washburn]

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-shell.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index 9d8c417da..e80471126 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -211,7 +211,7 @@ case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in
 	console=console;;
 esac
 
-timeout=60
+timeout=${GRUB_SHELL_DEFAULT_TIMEOUT:-60}
 mkimage_extra_arg=
 
 # Check the arguments.
-- 
2.27.0

[CI 09/17] grub-shell: Put all generated files into working dir and use better file names

[Glenn Washburn]

When running tests there are many invocations of grub-shell, and because the
output files are all random names in the same tmp directory, it becomes more
work to figure out which files went with which grub-shell invocations. So
all generated files from one invocation of grub-shell are put into a
randomly named directory, so as not to collide with other grub-shell
invocations. And now that the generated files can be put in a location where
they will not get stepped on, and they can be named sensible names.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-shell.in | 22 +++++++++++++---------
 1 file changed, 13 insertions(+), 9 deletions(-)

diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index e80471126..8c6ed76d7 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -71,6 +71,8 @@ exec_show_error () {
     fi
 }
 
+work_directory=${WORKDIR:-`mktemp -d "${TMPDIR:-/tmp}/grub-shell.XXXXXXXXXX"`} || exit 1
+
 . "${builddir}/grub-core/modinfo.sh"
 qemuopts="${GRUB_QEMU_OPTS}"
 serial_port=com0
@@ -80,7 +82,7 @@ pseries=n
 disk="hda "
 case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in
     *-emu)
-	device_map=`mktemp "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX"` || exit 1
+	device_map="$work_directory/device.map"
 	boot=emu
 	console=console
 	disk=0
@@ -318,14 +320,14 @@ for option in "$@"; do
 done
 
 if [ "x${source}" = x ] ; then
-    tmpfile=`mktemp "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX"` || exit 1
+    tmpfile="$work_directory/testcase.cfg"
     while read REPLY; do
 	echo "$REPLY" >> ${tmpfile}
     done
     source=${tmpfile}
 fi
 
-cfgfile=`mktemp "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX"` || exit 1
+cfgfile="$work_directory/grub.cfg"
 cat <<EOF >${cfgfile}
 grubshell=yes
 enable_progress_indicator=0
@@ -357,7 +359,8 @@ if [ $trim = 1 ]; then
     echo "echo; echo $trim_head" >>${cfgfile}
 fi
 
-rom_directory=`mktemp -d "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX"` || exit 1
+rom_directory="$work_directory/rom_directory"
+mkdir -p "$rom_directory"
 
 for mod in ${modules}
 do
@@ -378,7 +381,7 @@ test -z "$debug" || echo "GRUB script: ${cfgfile}" >&2
 test -z "$debug" || echo "GRUB testcase script: ${tmpfile}" >&2
 test -z "$debug" || echo "Boot device: ${boot}" >&2
 
-isofile=`mktemp "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX"` || exit 1
+isofile="$work_directory/grub.iso"
 test -z "$debug" || echo "GRUB ISO file: ${isofile}" >&2
 test -z "$debug" || echo "GRUB ROM directory: ${rom_directory}" >&2
 
@@ -450,7 +453,7 @@ if [ x$boot = xmips_qemu ]; then
 fi
 
 if [ x$boot = xcoreboot ]; then
-    imgfile=`mktemp "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX"` || exit 1
+    imgfile="$work_directory/coreboot.img"
     cp "${GRUB_COREBOOT_ROM}" "${imgfile}"
     "${GRUB_CBFSTOOL}" "${imgfile}" add-payload -f "${rom_directory}/coreboot.elf" -n fallback/payload
     bootdev="-bios ${imgfile}"
@@ -493,14 +496,15 @@ copy_extra_files() {
 }
 
 if [ x$boot = xnet ]; then
-    netdir=`mktemp -d "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX"` || exit 1
+    netdir="$work_directory/netdir"
+    mkdir -p "$netdir"
     pkgdatadir="@builddir@" "@builddir@/grub-mknetdir" "--grub-mkimage=${builddir}/grub-mkimage" "--directory=${builddir}/grub-core" "--net-directory=$netdir" ${mkrescue_args} > /dev/null
     cp "${cfgfile}" "$netdir/boot/grub/grub.cfg"
     cp "${source}" "$netdir/boot/grub/testcase.cfg"
     [ -z "$files" ] || copy_extra_files "$netdir" $files
     timeout -s KILL $timeout "${qemu}" ${qemuopts} ${serial_null} -serial file:/dev/stdout -boot n -net "user,tftp=$netdir,bootfile=/boot/grub/${grub_modinfo_target_cpu}-${grub_modinfo_platform}/core.$netbootext"  -net nic  | cat | tr -d "\r" | do_trim
 elif [ x$boot = xemu ]; then
-    rootdir="$(mktemp -d "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX")"
+    rootdir="$work_directory/rootdir"
     grubdir="$rootdir/boot/grub"
     mkdir -p "$grubdir/fonts"
     mkdir -p "$grubdir/themes"
@@ -515,7 +519,7 @@ elif [ x$boot = xemu ]; then
     cp "${cfgfile}" "$grubdir/grub.cfg"
     cp "${source}" "$grubdir/testcase.cfg"
     [ -z "$files" ] || copy_extra_files "$rootdir" $files
-    roottar="$(mktemp "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX")"
+    roottar="$work_directory/root.tar"
     (cd "$rootdir"; tar cf "$roottar" .)
     @builddir@/grub-core/grub-emu -m "$device_map" --memdisk "$roottar" -r memdisk -d "/boot/grub" | tr -d "\r" | do_trim
     test -n "$debug" || rm -rf "$rootdir"
-- 
2.27.0

[CI 10/17] grub-shell: Add grub output logfile with grub-shell --debug

[Glenn Washburn]

This allows seeing full qemu output of grub-shell, which can be invaluable
when debugging failing tests.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-shell.in | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index 8c6ed76d7..c6d7860e9 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -379,6 +379,9 @@ echo "${halt_cmd}" >>${cfgfile}
 
 test -z "$debug" || echo "GRUB script: ${cfgfile}" >&2
 test -z "$debug" || echo "GRUB testcase script: ${tmpfile}" >&2
+
+goutfile="$work_directory/grub-qemu.log"
+test -z "$debug" || echo "GRUB output log: ${goutfile}" >&2
 test -z "$debug" || echo "Boot device: ${boot}" >&2
 
 isofile="$work_directory/grub.iso"
@@ -502,7 +505,7 @@ if [ x$boot = xnet ]; then
     cp "${cfgfile}" "$netdir/boot/grub/grub.cfg"
     cp "${source}" "$netdir/boot/grub/testcase.cfg"
     [ -z "$files" ] || copy_extra_files "$netdir" $files
-    timeout -s KILL $timeout "${qemu}" ${qemuopts} ${serial_null} -serial file:/dev/stdout -boot n -net "user,tftp=$netdir,bootfile=/boot/grub/${grub_modinfo_target_cpu}-${grub_modinfo_platform}/core.$netbootext"  -net nic  | cat | tr -d "\r" | do_trim
+    timeout -s KILL $timeout "${qemu}" ${qemuopts} ${serial_null} -serial file:/dev/stdout -boot n -net "user,tftp=$netdir,bootfile=/boot/grub/${grub_modinfo_target_cpu}-${grub_modinfo_platform}/core.$netbootext"  -net nic  | cat | tr -d "\r" | tee "${goutfile}" | do_trim
 elif [ x$boot = xemu ]; then
     rootdir="$work_directory/rootdir"
     grubdir="$rootdir/boot/grub"
@@ -521,11 +524,11 @@ elif [ x$boot = xemu ]; then
     [ -z "$files" ] || copy_extra_files "$rootdir" $files
     roottar="$work_directory/root.tar"
     (cd "$rootdir"; tar cf "$roottar" .)
-    @builddir@/grub-core/grub-emu -m "$device_map" --memdisk "$roottar" -r memdisk -d "/boot/grub" | tr -d "\r" | do_trim
+    @builddir@/grub-core/grub-emu -m "$device_map" --memdisk "$roottar" -r memdisk -d "/boot/grub" | tr -d "\r" | tee "${goutfile}" | do_trim
     test -n "$debug" || rm -rf "$rootdir"
     test -n "$debug" || rm -f "$roottar"
 else
-    timeout -s KILL $timeout "${qemu}" ${qemuopts} ${serial_null} -serial file:/dev/stdout -${device}"${isofile}" ${bootdev} | cat | tr -d "\r" | do_trim
+    timeout -s KILL $timeout "${qemu}" ${qemuopts} ${serial_null} -serial file:/dev/stdout -${device}"${isofile}" ${bootdev} | cat | tr -d "\r" | tee "${goutfile}" | do_trim
 fi
 if [ x$boot = xcoreboot ]; then
     test -n "$debug" || rm -f "${imgfile}"
-- 
2.27.0

[CI 11/17] grub-shell: Set exit status to qemu exit status

[Glenn Washburn]

This allows us to test if unexpected output in test scripts is because of a
bug in grub, because there was an error in qemu, or qemu was killed due to a
timeout.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-shell.in | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index c6d7860e9..40c62b9fd 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -498,6 +498,9 @@ copy_extra_files() {
     done
 }
 
+ret=0
+mkfifo "$work_directory/qemu-pipe"
+cat < "$work_directory/qemu-pipe" | tr -d "\r" | tee "${goutfile}" | do_trim &
 if [ x$boot = xnet ]; then
     netdir="$work_directory/netdir"
     mkdir -p "$netdir"
@@ -505,7 +508,7 @@ if [ x$boot = xnet ]; then
     cp "${cfgfile}" "$netdir/boot/grub/grub.cfg"
     cp "${source}" "$netdir/boot/grub/testcase.cfg"
     [ -z "$files" ] || copy_extra_files "$netdir" $files
-    timeout -s KILL $timeout "${qemu}" ${qemuopts} ${serial_null} -serial file:/dev/stdout -boot n -net "user,tftp=$netdir,bootfile=/boot/grub/${grub_modinfo_target_cpu}-${grub_modinfo_platform}/core.$netbootext"  -net nic  | cat | tr -d "\r" | tee "${goutfile}" | do_trim
+    timeout -s KILL $timeout "${qemu}" ${qemuopts} ${serial_null} -serial file:/dev/stdout -boot n -net "user,tftp=$netdir,bootfile=/boot/grub/${grub_modinfo_target_cpu}-${grub_modinfo_platform}/core.$netbootext"  -net nic > "$work_directory/qemu-pipe" || ret=$?
 elif [ x$boot = xemu ]; then
     rootdir="$work_directory/rootdir"
     grubdir="$rootdir/boot/grub"
@@ -524,11 +527,11 @@ elif [ x$boot = xemu ]; then
     [ -z "$files" ] || copy_extra_files "$rootdir" $files
     roottar="$work_directory/root.tar"
     (cd "$rootdir"; tar cf "$roottar" .)
-    @builddir@/grub-core/grub-emu -m "$device_map" --memdisk "$roottar" -r memdisk -d "/boot/grub" | tr -d "\r" | tee "${goutfile}" | do_trim
+    @builddir@/grub-core/grub-emu -m "$device_map" --memdisk "$roottar" -r memdisk -d "/boot/grub" > "$work_directory/qemu-pipe" || ret=$?
     test -n "$debug" || rm -rf "$rootdir"
     test -n "$debug" || rm -f "$roottar"
 else
-    timeout -s KILL $timeout "${qemu}" ${qemuopts} ${serial_null} -serial file:/dev/stdout -${device}"${isofile}" ${bootdev} | cat | tr -d "\r" | tee "${goutfile}" | do_trim
+    timeout -s KILL $timeout "${qemu}" ${qemuopts} ${serial_null} -serial file:/dev/stdout -${device}"${isofile}" ${bootdev} > "$work_directory/qemu-pipe" || ret=$?
 fi
 if [ x$boot = xcoreboot ]; then
     test -n "$debug" || rm -f "${imgfile}"
@@ -536,6 +539,7 @@ fi
 test -n "$debug" || rm -f "${isofile}"
 test -n "$debug" || rm -rf "${rom_directory}"
 test -n "$debug" || rm -f "${tmpfile}" "${cfgfile}"
-exit 0
+rm -f "$work_directory/qemu-pipe"
+exit $ret
 
 
-- 
2.27.0

[CI 12/17] tests: Allow turning on shell tracing from environment variables

[Glenn Washburn]

This allows turning on shell tracing when its not practical or not possible
to use commandline arguments. Turn on tracing when the envvar is an integer
greater than 1, since these can log a lot of messages.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-fs-tester.in | 2 ++
 tests/util/grub-shell.in     | 5 ++++-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/tests/util/grub-fs-tester.in b/tests/util/grub-fs-tester.in
index 4213b7bfc..9a5f409e9 100644
--- a/tests/util/grub-fs-tester.in
+++ b/tests/util/grub-fs-tester.in
@@ -2,6 +2,8 @@
 
 set -e
 
+[ "${GRUB_TEST_DEFAULT_DEBUG:-0}" -gt 1 ] && set -x
+
 fs="$1"
 
 GRUBFSTEST="@builddir@/grub-fstest"
diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index 40c62b9fd..3b6bce4c2 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -215,6 +215,7 @@ esac
 
 timeout=${GRUB_SHELL_DEFAULT_TIMEOUT:-60}
 mkimage_extra_arg=
+debug=${GRUB_SHELL_DEFAULT_DEBUG}
 
 # Check the arguments.
 for option in "$@"; do
@@ -234,7 +235,7 @@ for option in "$@"; do
     --no-trim)
 	trim=0 ;;
     --debug)
-        debug=1 ;;
+        debug=$((debug+1)) ;;
     --modules=*)
 	ms=`echo "$option" | sed -e 's/--modules=//' -e 's/,/ /g'`
 	modules="$modules $ms" ;;
@@ -319,6 +320,8 @@ for option in "$@"; do
     esac
 done
 
+[ "${debug:-0}" -gt 1 ] && set -x
+
 if [ "x${source}" = x ] ; then
     tmpfile="$work_directory/testcase.cfg"
     while read REPLY; do
-- 
2.27.0

[CI 13/17] grub-shell: Add --verbose to mkrescue when $debug is greater than 2

[Glenn Washburn]

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-shell.in | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index 3b6bce4c2..211bfdb95 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -404,7 +404,9 @@ fi
 
 if [ x$boot != xnet ] && [ x$boot != xemu ]; then
     pkgdatadir="@builddir@" \
-    exec_show_error "@builddir@/grub-mkrescue" "--output=${isofile}" \
+    exec_show_error "@builddir@/grub-mkrescue" \
+	${debug:+$([ "$debug" -gt 2 ] && echo -n "--verbose")} \
+	"--output=${isofile}" \
 	"--override-directory=${builddir}/grub-core" \
 	--rom-directory="${rom_directory}" \
 	--locale-directory="@srcdir@/po" \
-- 
2.27.0

[CI 14/17] grub-shell: Use malta qemu-mips machine type instead off non-existant indy

[Glenn Washburn]

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-shell.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index 211bfdb95..332fae704 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -119,7 +119,7 @@ case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in
     mips-arc)
 	boot=cd
 	qemu=qemu-system-mips64
-	qemuopts="-M indy $qemuopts"
+	qemuopts="-M malta $qemuopts"
 	serial_port=arc/serial0/line0
 	console=
 	trim=1
-- 
2.27.0

[CI 15/17] grub-shell: Update qemu UEFI firmware names to be more generic

[Glenn Washburn]

The current qemu firmware paths for arm-efi and arm64-efi are hardcoded
and not available on Ubuntu/Debian. Make these paths relative and basenames
generic. The idea is that regardless where or what names firmware images
are installed under they can be copied and renamed into a temporary
directory. Then use the -L argument to qemu to point to that directory.

For convenience, the GRUB source directory which build grub-shell will be
added to the list of firmware paths that qemu will search. So all that is
needed is to copy the firmwares to the source directory with potential
renames.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-shell.in | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index 332fae704..33d39d4dd 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -74,7 +74,7 @@ exec_show_error () {
 work_directory=${WORKDIR:-`mktemp -d "${TMPDIR:-/tmp}/grub-shell.XXXXXXXXXX"`} || exit 1
 
 . "${builddir}/grub-core/modinfo.sh"
-qemuopts="${GRUB_QEMU_OPTS}"
+qemuopts="${GRUB_QEMU_OPTS} -L ${srcdir}"
 serial_port=com0
 serial_null=
 halt_cmd=halt
@@ -194,7 +194,7 @@ case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in
 	boot=hd
 	console=console
 	trim=1
-	qemuopts="-machine virt -cpu cortex-a57 -bios /usr/share/qemu-efi/QEMU_EFI.fd $qemuopts"
+	qemuopts="-machine virt -cpu cortex-a57 -bios OVMF-aarch64.fd $qemuopts"
 	disk="device virtio-blk-device,drive=hd1 -drive if=none,id=hd1,file="
 	serial_port=
 	;;
@@ -203,7 +203,7 @@ case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in
 	boot=hd
 	console=console
 	trim=1
-	qemuopts="-machine virt -bios /usr/share/ovmf-arm/QEMU_EFI.fd $qemuopts"
+	qemuopts="-machine virt -bios OVMF-arm.fd $qemuopts"
 	disk="device virtio-blk-device,drive=hd1 -drive if=none,id=hd1,file="
 	serial_port=efi0
 	;;
-- 
2.27.0

[CI 16/17] ci: Add .gitlab-ci.yml for continuous integration with Gitlab CI

[Glenn Washburn]

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 .ci/build.sh            |   66 +++
 .ci/functions.gitlab.sh |   30 +
 .ci/functions.sh        |   30 +
 .ci/make-images.sh      |   71 +++
 .ci/process-tests.sh    |  108 ++++
 .ci/test.sh             |  106 ++++
 .gitlab-ci.yml          | 1184 +++++++++++++++++++++++++++++++++++++++
 7 files changed, 1595 insertions(+)
 create mode 100755 .ci/build.sh
 create mode 100644 .ci/functions.gitlab.sh
 create mode 100644 .ci/functions.sh
 create mode 100755 .ci/make-images.sh
 create mode 100755 .ci/process-tests.sh
 create mode 100755 .ci/test.sh
 create mode 100644 .gitlab-ci.yml

diff --git a/.ci/build.sh b/.ci/build.sh
new file mode 100755
index 000000000..14dd7f7c5
--- /dev/null
+++ b/.ci/build.sh
@@ -0,0 +1,66 @@
+#!/bin/bash
+
+set -eo pipefail
+
+# Environment variables
+# JOBS: Number of concurrent jobs while building, defaults to number of
+#   processors plus 1, unless number of processors is 1 in which case its 1.
+# SRCDIR: Path to source files
+# BUILDDIR: Directory in which to place the build
+# INSTALLDIR: Directory to install binaries
+# ARCH: Architecture to build for
+# PLATFORM: Platform to build for
+# CONFIGURE_OPTS: Extra configure options
+# SHELL_TRACE: Set to 'y' to enable shell tracing
+
+[ "x${SHELL_TRACE}" = "xy" ] && set -x
+
+[ -f "$(dirname "$0")/functions.sh" ] &&
+. "$(dirname "$0")/functions.sh"
+
+[ -f "$(dirname "$0")/functions.$CI_TYPE.sh" ] &&
+. "$(dirname "$0")/functions.$CI_TYPE.sh"
+
+JOBS=${JOBS:-`getconf _NPROCESSORS_ONLN 2> /dev/null || echo 1`};
+[ "$JOBS" == 1 ] || JOBS=$(($JOBS + 1));
+
+TARGET="${ARCH}-${PLATFORM}"
+
+mkdir -pv ${BUILDDIR};
+
+RET=0;
+cd ${BUILDDIR};
+
+# echo -e "#!$SHELL\nREAL_SHELL=\${REAL_SHELL:-\$(readlink /proc/\$\$/exe)}\nexec \$REAL_SHELL \$SHELL_OPTS \"\$@\"" >${BUILDDIR}/shell-wrapper.sh;
+# cat <<EOF >${BUILDDIR}/shell-wrapper.sh
+# #!$SHELL
+# REAL_SHELL=\${REAL_SHELL:-$(readlink /proc/\$\$/exe)}
+# exec \$REAL_SHELL \$SHELL_OPTS \"\$@\"
+# EOF
+
+# chmod +x ${BUILDDIR}/shell-wrapper.sh;
+# export CONFIG_SHELL=${BUILDDIR}/shell-wrapper.sh;
+
+start_log -c -n "configure-$TARGET" "Configuring $TARGET";
+ls -la $SRCDIR;
+[ -x "$SRCDIR/configure" ] && $SRCDIR/configure --help;
+$SRCDIR/configure --target=$ARCH --with-platform=$PLATFORM \
+  --prefix=${INSTALLDIR} $CONFIGURE_OPTS || RET=$?;
+end_log -n "configure-$TARGET";
+
+if [ "$RET" -ne 0 ]; then
+  start_log -c -n "showlogs-$TARGET" "Configuring $TARGET failed, showing logs";
+  cat ${BUILDDIR}/config.log;
+  end_log -n "showlogs-$TARGET";
+  exit $RET;
+fi;
+
+start_log -c -n "build-$TARGET" "Building $TARGET";
+make -j$JOBS || RET=$?;
+end_log -n "build-$TARGET";
+[ "$RET" -ne 0 ] && exit $RET;
+
+start_log -c -n "install-$TARGET" "Installing $TARGET";
+make -j$JOBS install || RET=$?;
+end_log -n "install-$TARGET";
+exit $RET;
\ No newline at end of file
diff --git a/.ci/functions.gitlab.sh b/.ci/functions.gitlab.sh
new file mode 100644
index 000000000..e9cd2f9be
--- /dev/null
+++ b/.ci/functions.gitlab.sh
@@ -0,0 +1,30 @@
+#!/bin/bash
+
+TXT_RED="\e[31m"
+TXT_YELLOW="\e[33m"
+TXT_CLEAR="\e[0m"
+
+function start_log() {
+  while [ "$#" -gt 0 ]; do
+    case "$1" in
+      -c) LOG_COLLAPSE=1; shift;;
+      -n) LOG_NAME="$2"; shift;;
+      *) [ "$#" -eq 1 ] && LOG_MSG="$1"; shift;;
+    esac
+  done
+
+  echo -e "section_start:`date +%s`:${LOG_NAME}${LOG_COLLAPSE:+[collapsed=true]}\r\e[0K${LOG_MSG}"
+}
+
+function end_log() {
+  while [ "$#" -gt 0 ]; do
+    case "$1" in
+      -n) LOG_NAME=$2; shift;;
+      *) shift;;
+    esac
+  done
+
+  echo -e "section_end:`date +%s`:${LOG_NAME}\r\e[0K"
+}
+
+:;
diff --git a/.ci/functions.sh b/.ci/functions.sh
new file mode 100644
index 000000000..bba33fb5f
--- /dev/null
+++ b/.ci/functions.sh
@@ -0,0 +1,30 @@
+#!/bin/bash
+
+TXT_RED="\e[31m"
+TXT_YELLOW="\e[33m"
+TXT_CLEAR="\e[0m"
+
+function start_log() {
+  while [ "$#" -gt 0 ]; do
+    case "$1" in
+      -c) LOG_COLLAPSE=1; shift;;
+      -n) LOG_NAME="$2"; shift;;
+      *) [ "$#" -eq 1 ] && LOG_MSG="$1"; shift;;
+    esac
+  done
+
+  echo -e "Start:${LOG_NAME} ${LOG_MSG}"
+}
+
+function end_log() {
+  while [ "$#" -gt 0 ]; do
+    case "$1" in
+      -n) LOG_NAME=$2; shift;;
+      *) shift;;
+    esac
+  done
+
+  echo -e "End:${LOG_NAME}"
+}
+
+:;
diff --git a/.ci/make-images.sh b/.ci/make-images.sh
new file mode 100755
index 000000000..3c7d2f88c
--- /dev/null
+++ b/.ci/make-images.sh
@@ -0,0 +1,71 @@
+#!/bin/bash
+
+set -eo pipefail
+
+# Environment variables
+# SRCDIR: Path to source files
+# BUILDDIR: Directory in which to place the build
+# TARGET: Target to test
+# MAKE_ALL_IMAGE_TARGETS: If set to 'y', then all image targets, even disabled
+#   ones, will be run.
+# DISABLED_IMAGES: String of target formats to disable when building grub
+#   images delimited by new lines.
+# SHELL_TRACE: Set to 'y' to enable shell tracing
+
+if [ "x${SHELL_TRACE}" = "xy" ]; then
+  # If we export SHELL_OPTS, then all shells will be traced, most of which we do not want
+  SHELL_OPTS="-x";
+  set -x;
+fi;
+
+[ -f "$(dirname "$0")/functions.sh" ] &&
+. "$(dirname "$0")/functions.sh"
+
+[ -f "$(dirname "$0")/functions.$CI_TYPE.sh" ] &&
+. "$(dirname "$0")/functions.$CI_TYPE.sh"
+
+start_log -c -n "images-$TARGET" "Making images for $TARGET";
+
+function build_tformat() {
+  TARGET=$1;
+  FORMATS=$2;
+  for fmt in $FORMATS; do
+    if [ -z "${fmt%~*}" ]; then
+      echo "${TARGET}";
+    else
+      echo "${TARGET}-${fmt}";
+    fi;
+  done;
+}
+
+tformats="~";
+case "$TARGET" in
+  mipsel-loongson)
+    tformats="mipsel-loongson-elf mipsel-yeeloong-flash mipsel-fuloong2f-flash" ;;
+  *)
+    case "$TARGET" in
+      i386-pc) tformats="~ pxe eltorito" ;;
+      sparc64-ieee1275) tformats="aout cdcore raw" ;;
+      mips-qemu_mips | \
+      mipsel-qemu_mips) tformats="elf flash" ;;
+      arm-coreboot) tformats="vexpress veyron" ;;
+    esac;
+    tformats=$(build_tformat "${TARGET}" "$tformats");
+    ;;
+esac;
+
+RET=0;
+mkdir -pv "${BUILDDIR}/images";
+for tfmt in $tformats; do
+  if [ "x${MAKE_ALL_IMAGE_TARGETS}" != "xy" ] && ( echo "${DISABLED_IMAGES}" | grep -q "^${tfmt}$" ); then
+    echo "Image build disabled for target format $tfmt";
+    continue;
+  fi;
+
+  echo "Making image for target format: ${tfmt}";
+  ${BUILDDIR}/grub-mkimage -v -p / -O "${tfmt}" -o "${BUILDDIR}/images/grub-${tfmt}.img" echo reboot normal || _RET=$?;
+  [ "${_RET:-0}" -ne 0 ] && RET=$_RET;
+done;
+
+end_log -n "images-$TARGET";
+exit $RET;
diff --git a/.ci/process-tests.sh b/.ci/process-tests.sh
new file mode 100755
index 000000000..ab030cfc2
--- /dev/null
+++ b/.ci/process-tests.sh
@@ -0,0 +1,108 @@
+#!/bin/bash
+
+set -eo pipefail
+
+# Environment variables
+# SRCDIR: Path to source files
+# BUILDDIR: Directory in which to place the build
+# TARGET: Target to test
+# TEST_ALL_TARGETS: If set to 'y', then all tests, even disabled ones will be
+#   run.
+# DISABLED_TESTS: String of disabled tests delimited by new lines.
+# TESTS_TIMEOUT: Maximum timeout to allow for 'make check'. Set this slightly
+#   below total job timeout to allow for cleanup and log packaging code.
+# TEST_VERBOSITY: Number to set verbosity level to
+# TEST_DATA_PREFIX: Filename prefix to use when saving test data. If not set,
+#   test data will not be saved.
+# STRACE_TESTS: Set to 'y' to strace every tests for extreme debugging
+# SHELL_TRACE: Set to 'y' to enable shell tracing
+
+[ "x${SHELL_TRACE}" = "xy" ] && set -x
+
+[ -f "$(dirname "$0")/functions.sh" ] &&
+. "$(dirname "$0")/functions.sh"
+
+[ -f "$(dirname "$0")/functions.$CI_TYPE.sh" ] &&
+. "$(dirname "$0")/functions.$CI_TYPE.sh"
+
+# TARGET="${ARCH}-${PLATFORM}"
+
+test "x${SHELL_TRACE}" = "xy" && set -x;
+if [ -f ${BUILDDIR}/test.success ]; then
+  echo "Not processing test logs because make check ran successfully";
+  exit 0;
+elif [ "x${TEST_ALL_TARGETS}" != "xy" ] && ( echo "${DISABLED_TESTS}" | grep -q "^${TARGET}$" ); then
+  echo "No test output processing because testing was disabled";
+  exit 0;
+fi;
+
+start_log -c -n "process-test-$TARGET" "Processing test output of $TARGET";
+if [ -f ${BUILDDIR}/test-suite.log ]; then
+  ( grep -E "^(FAIL|ERROR|SKIP):" ${BUILDDIR}/test-suite.log || ':' ) |
+  while read STATUS TESTNAME; do
+    STATUS=${STATUS%%:};
+
+    if [ "$STATUS" = "SKIP" ]; then
+      TYPE=skip;
+    elif [ "$STATUS" = "ERROR" ]; then
+      TYPE=error;
+    elif echo "${EXPECTED_FAILURES}" | grep -qE "^(${TESTNAME}|${TARGET}:${TESTNAME})$"; then
+      echo 'Expected failed test:' "$TESTNAME";
+      TYPE=expected;
+    # If any unexpected failures in the functional tests, count a failure in
+    # grub_func_test as a true failure.
+    elif [ "$TESTNAME" = "grub_func_test" ]; then
+      grep -E 'test:'' FAIL' ${BUILDDIR}/${TESTNAME}.log | sort -u |
+      while read FTESTNAME STATUS; do
+        FTESTNAME=${FTESTNAME%:*};
+        if echo "${EXPECTED_FUNCTIONAL_FAILURES}" | grep -qE "^(${FTESTNAME}|${TARGET}:${FTESTNAME})$"; then
+          echo 'Expected failed functional test:' "$FTESTNAME";
+          TYPE=expected;
+        else
+          echo -e "${TXT_RED}"'Unexpected failed functional test:' "${FTESTNAME}$TXT_CLEAR";
+          TYPE=unexpected;
+        fi;
+        echo "${TARGET}:${TYPE}:${TESTNAME}:${FTESTNAME}" >> ${BUILDDIR}/test.failures.log;
+      done;
+      continue;
+    elif [ "x${IGNORE_TIMEDOUT_TEST_FAILURE}" = "xy" ] &&
+        tail -n1 "${BUILDDIR}/${TESTNAME}.log" | grep -q 'exit status:'' 137'; then
+      echo -e "${TXT_RED}"'Ignoring Timed-out test:' "${TESTNAME}$TXT_CLEAR";
+      echo -e -n "\e[97;100m";
+      echo "Test failed due to a timeout. This is likely due to";
+      echo "insufficient runner resources, and NOT a real failure.";
+      echo -e -n "$TXT_CLEAR";
+      TYPE=timeout;
+    else
+      echo -e "${TXT_RED}"'Unexpected failed test:' "${TESTNAME}$TXT_CLEAR";
+      echo -e -n "\e[97;100m";
+      echo "Last 100 lines of ${BUILDDIR}/${TESTNAME}.log";
+      tail -n 100 ${BUILDDIR}/${TESTNAME}.log;
+      echo -e -n "$TXT_CLEAR";
+      TYPE=unexpected;
+    fi;
+    echo "${TARGET}:${TYPE}:${TESTNAME}" >> ${BUILDDIR}/test.failures.log;
+  done;
+else
+  echo "No success canary and no test-suite.log, perhaps a timeout";
+fi;
+
+# If there are any unexpected errors return exit with error
+FAILCOND="${TARGET}:unexpected";
+if [ "x${FAIL_ON_HARD_ERRORS}" = "xy" ]; then
+  FAILCOND="${TARGET}:(unexpected|error)";
+fi;
+ if grep -E -q "${FAILCOND}" ${BUILDDIR}/test.failures.log; then
+  RET=1;
+fi;
+end_log -n "process-test-$TARGET";
+# if [ -f $CI_PROJECT_DIR/${TARGET}.timedout-failures ]; then
+#   sed 's/^/${TARGET}:/' $CI_PROJECT_DIR/${TARGET}.timedout-failures
+#     >> $CI_PROJECT_DIR/build/test.timedout.log;
+# fi;
+# if [ -f $CI_PROJECT_DIR/${TARGET}.unexpected-failures ]; then
+#   sed 's/^/${TARGET}:/' $CI_PROJECT_DIR/${TARGET}.unexpected-failures
+#     >> $CI_PROJECT_DIR/build/test.failed.log;
+#   exit 1;
+# fi;
+exit ${RET:-0};
diff --git a/.ci/test.sh b/.ci/test.sh
new file mode 100755
index 000000000..3f51de58b
--- /dev/null
+++ b/.ci/test.sh
@@ -0,0 +1,106 @@
+#!/bin/bash
+
+set -eo pipefail
+
+# Environment variables
+# SRCDIR: Path to source files
+# BUILDDIR: Directory in which to place the build
+# TARGET: Target to test
+# TEST_ALL_TARGETS: If set to 'y', then all tests, even disabled ones will be
+#   run.
+# DISABLED_TESTS: String of disabled tests delimited by new lines.
+# TESTS_TIMEOUT: Maximum timeout to allow for 'make check'. Set this slightly
+#   below total job timeout to allow for cleanup and log packaging code.
+# TEST_VERBOSITY: Number to set verbosity level to
+# TEST_DATA_PREFIX: Filename prefix to use when saving test data. If not set,
+#   test data will not be saved.
+# STRACE_TESTS: Set to 'y' to strace every tests for extreme debugging
+# SHELL_TRACE: Set to 'y' to enable shell tracing
+
+if [ "x${SHELL_TRACE}" = "xy" ]; then
+  # If we export SHELL_OPTS, then all shells will be traced, most of which we do not want
+  SHELL_OPTS="-x";
+  set -x;
+fi;
+
+[ -f "$(dirname "$0")/functions.sh" ] &&
+. "$(dirname "$0")/functions.sh"
+
+[ -f "$(dirname "$0")/functions.$CI_TYPE.sh" ] &&
+. "$(dirname "$0")/functions.$CI_TYPE.sh"
+
+if [ "x${TEST_ALL_TARGETS}" != "xy" ] && ( echo "${DISABLED_TESTS}" | grep -q "^${TARGET}$" ); then
+  echo "Tests are disabled for target $TARGET";
+  exit 0;
+fi;
+
+start_log -c -n "test-$TARGET" "Testing $TARGET";
+
+TESTTMPDIR="${TESTTMPDIR:-${TMPDIR:-/tmp}/grub-test-$TARGET}";
+COMP=xz;
+STRACE_LOG="${BUILDDIR}/xxx.strace.$COMP";
+
+if [ "x${STRACE_TESTS}" = "xy" ]; then
+  STRACE="strace -y -yy -f -v -s4096 -o >($COMP -9 > $STRACE_LOG)";
+fi;
+
+cat >${BUILDDIR}/log-tester.sh <<EOF
+#!$SHELL $SHELL_OPTS
+
+STRACE_LOG="$STRACE_LOG";
+STRACE="$STRACE";
+TESTNAME=\$(basename "\$1");
+export SHELL_OPTS="$SHELL_OPTS";
+export TMPDIR=$TESTTMPDIR/\$TESTNAME;
+mkdir -p "\$TMPDIR";
+
+# if not a shell script, run normally
+if [ "\$(head -c2 \$1)" == "#!" ]; then
+  TEST_SHELL="$(head -n1 "${BUILDDIR}/grub-shell" | tail -c+3 | tr -d ' ')";
+  # Only turn on tracing if the shell if the one used by grub-shell
+  if head -n1 \$1 | grep -q \$TEST_SHELL; then
+    if [ "${TEST_VERBOSITY:-0}" -gt 0 ] || [ "x${SHELL_TRACE}" = "xy" ]; then
+      TEST_SHELL="\$TEST_SHELL -x";
+    fi;
+  fi;
+fi;
+eval \${STRACE/xxx/\$TESTNAME} \$TEST_SHELL "\$@" && { rm -rf \${STRACE_LOG/xxx/\$TESTNAME} \$TMPDIR; }
+EOF
+export LOG_COMPILER="${BUILDDIR}/log-tester.sh";
+chmod +x ${BUILDDIR}/log-tester.sh;
+echo -n -e "${TXT_YELLOW}";
+cat ${BUILDDIR}/log-tester.sh;
+echo -n -e "${TXT_CLEAR}";
+
+if [ "${TEST_VERBOSITY:-0}" -gt 0 ]; then
+  export V=${TEST_VERBOSITY};
+  export GRUB_SHELL_DEFAULT_DEBUG=${TEST_VERBOSITY};
+  export GRUB_FSTEST_DEFAULT_DEBUG=${TEST_VERBOSITY};
+fi;
+
+if [ -n "$GRUB_SHELL_DEFAULT_DEBUG" ]; then
+  ### Turning on grub-shell debugging, turns on a graphical qemu window.
+  ### But we want to run in headless environments, so turn off graphics.
+  export GRUB_QEMU_OPTS="-nographic -monitor file:/dev/null -fw_cfg name=etc/sercon-port,string=0";
+fi;
+TIMEOUT=${TESTS_TIMEOUT};
+
+function func_strace () {
+  strace -y -yy -f -v -s4096 -o >($COMP -9 > ${BUILDDIR}/test.strace.$RANDOM.$COMP) "$@";
+};
+':' STRACE="func_strace";
+
+':' $STRACE ${TIMEOUT:+timeout $TIMEOUT} make -C ${BUILDDIR} -j$JOBS SUBDIRS= check || RET=$?;
+${TIMEOUT:+timeout $TIMEOUT} make -C ${BUILDDIR} -j$JOBS SUBDIRS= check || RET=$?;
+':' timeout $TIMEOUT $STRACE make -C ${BUILDDIR} -j$JOBS SUBDIRS= LOG_COMPILER="${BUILDDIR}/log-tester.sh" check || RET=$?;
+[ "$RET" -eq 124 -o "$RET" -eq 137 ] && echo "ERROR":" make check timed out";
+':' gzip -9 ${BUILDDIR}/test.strace.*;
+
+end_log -n "test-$TARGET";
+
+[ "$RET" -eq 0 ] && touch ${BUILDDIR}/test.success || :;
+if [ -n "$TEST_DATA_PREFIX" ]; then
+  tar -S -cJf "${TEST_DATA_PREFIX}.tar.xz" -C $(dirname "$TESTTMPDIR") $(basename "$TESTTMPDIR");
+fi
+
+exit $RET;
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
new file mode 100644
index 000000000..d4fc2337e
--- /dev/null
+++ b/.gitlab-ci.yml
@@ -0,0 +1,1184 @@
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright Glenn Washburn <development@efficientek.com>
+#
+# Build GRUB on GitLab CI - https://www.gitlab.org/
+#
+
+workflow:
+  rules:
+    - if: '$CI_PIPELINE_SOURCE == "web"'
+    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
+    - if: $CI_MERGE_REQUEST_IID
+    - if: $CI_COMMIT_TAG
+    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
+    - when: never
+
+# This allows running CI/CD pipelines for merge requests
+# include:
+#  - template: 'Workflows/MergeRequest-Pipelines.gitlab-ci.yml'
+
+# Variables that can be set by runners:
+# CONFIGURE_OPTS - Extra configure options
+# TESTS_TIMEOUT - Set timeout for make check tests (see man timeout)
+# FAIL_ON_HARD_ERRORS - If set, Hard errors will cause a failure
+# DISABLE_ALL_TESTS - If set, make check tests will be disabled
+# TEST_ALL_TARGETS - If set, failing targets which have been marked disabled will be enabled
+# TEST_VERBOSITY - Verbosity level when running tests: 1-3
+# STRACE_TESTS - If set, strace individual tests. WARNING: This will cause testing to take much longer
+#                which can cause some test to fail by timing out or having incorrect timing
+# SHELL_TRACE - If set, turn on shell tracing of everything. NOTE: TEST_VERBOSITY=3 turns on more targeted
+#               shell tracing
+
+variables:
+  CI_TYPE: gitlab
+  # CI_DEBUG_TRACE: 'true'
+  # GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/grub-$CI_COMMIT_SHORT_SHA
+  GIT_CLONE_PATH: $CI_BUILDS_DIR/grub-${CI_COMMIT_SHORT_SHA}.git
+  GIT_STRATEGY: fetch
+
+  # Include all cross toolchain paths, so we can just call them later down.
+  #PATH: "/tmp/qemu-install/bin:/usr/bin:/bin:$CROSS_DIR/gcc-8.1.0-nolibc/aarch64-linux/bin:$CROSS_DIR/gcc-8.1.0-nolibc/arm-linux-gnueabi/bin:$CROSS_DIR/gcc-8.1.0-nolibc/ia64-linux/bin:$CROSS_DIR/gcc-8.1.0-nolibc/mips64-linux/bin:$CROSS_DIR/gcc-8.1.0-nolibc/powerpc64-linux/bin:$CROSS_DIR/gcc-8.1.0-nolibc/riscv32-linux/bin:$CROSS_DIR/gcc-8.1.0-nolibc/riscv64-linux/bin:$CROSS_DIR/gcc-8.1.0-nolibc/sparc64-linux/bin"
+#  PACKAGE_CACHE: $CI_BUILDS_DIR/pkgs
+  PACKAGE_CACHE: $CI_PROJECT_DIR/pkgs
+  CROSS_DIR: $CI_PROJECT_DIR/cross
+  SRCDIR: $CI_PROJECT_DIR
+
+  ### Below are values that are meant to be configurable
+  CROSS_VERSION:
+    # value: "4.8.0"
+    # value: "4.8.5"
+    # value: "4.9.0"
+    # value: "4.9.4"
+    # value: "5.5.0"
+    # Start having riscv32/64 builds
+    # value: "7.3.0"
+    # value: "7.5.0"
+    # value: "8.1.0"
+    # value: "9.3.0"
+    value: "10.1.0"
+    description: "Version of cross-compiler to use (suggested: 10.1.0, 9.3.0, 8.1.0, 7.5.0 [riscv32/64 builds fail before 7.3.0])"
+
+  PACKAGES: |
+    libsdl1.2-dev
+    xz-utils
+    lzop
+    ovmf
+    python
+    qemu-system
+    unifont
+    wget
+    libfreetype6-dev
+    libdevmapper-dev
+    liblzma-dev
+    libfuse-dev
+    libzfslinux-dev
+    make
+    autoconf
+    automake
+    autopoint
+    git
+    bison
+    flex
+    gettext
+    pkg-config
+#    gcc-9-multilib
+
+  CONFIGURE_OPTS:
+    value: "--enable-boot-time"
+    description: "Extra options to pass to configure"
+
+  BUILD_ALL_TARGETS:
+    value: "n"
+    description: "If set 'y', all targets defined in the build matrix will be built unconditionally."
+
+  DISABLED_BUILDS:
+    value: |
+      ### qemu-system-mips64: Could not load MIPS bios 'mips_bios.bin',
+      ### and no -kernel argument was specified
+      mipsel-arc
+      mipsel-qemu_mips
+      mipsel-loongson
+    description: >
+      A list of targets for which builds are disabled, one per line. These
+      default targets are disabled generally because they do not work yet.
+      Lines beginning with '#' may be used as comments and are ignored. Some
+      hints as to why the tests fail are included in comment lines. Patches
+      which get these tests working are very welcome.
+
+  TESTS_TIMEOUT:
+    value: ""
+    description: >
+      Set timeout for completion of all make check tests (see man timeout
+      duration argument). This is most useful when the make check job is
+      taking longer than the job timeout configured by the runner. In this
+      case, set this to 5-10 minutes less than runner timeout so that there
+      is time to package up the logs for debugging.
+
+  IGNORE_TIMEDOUT_TEST_FAILURE:
+    value: "y"
+    description: >
+      If set to 'y', tests which fail due to a timeout in grub-shell of qemu
+      will not be counted as a failure. These failures are almost always the
+      result of insufficient runner resources to complete the execution of qemu
+      within the timeout period.
+
+  FAIL_ON_HARD_ERRORS:
+    value: "n"
+    description: >
+      If set to 'y', hard errors will cause a failure. A hard error indicates
+      that the test was not able to be run (eg. dependencies not found), and
+      as such a hard error does not constitute a true failure of the test.
+
+  DISABLE_ALL_TESTS:
+    value: "n"
+    description: >
+      If set to 'y', make check tests will be disabled. Effectively only test
+      whether the building is successful for configured platforms.
+
+  TEST_ALL_TARGETS:
+    value: "n"
+    description: "If set 'y', failing targets which have been marked disabled will be enabled"
+
+  MAKE_ALL_IMAGE_TARGETS:
+    value: "n"
+    description: "If set 'y', all target image formats will be built unconditionally"
+
+  TEST_VERBOSITY:
+    value: "2"
+    description: "Verbosity level when running tests [supported values: '', 1, 2]"
+
+  STRACE_TESTS:
+    value: "n"
+    description: >
+      If set, strace individual tests. WARNING: This will cause testing to take
+      much longer which can cause some test to fail by timing out or having
+      incorrect timing
+
+  SHELL_TRACE:
+    value: "n"
+    description: >
+      If set, turn on shell tracing of everything. NOTE: TEST_VERBOSITY=3 turns
+      on more targeted shell tracing
+
+  GRUB_QEMU_OPTS:
+    value: ""
+    description: "Add options to qemu when used in tests (ex: -m size=64M)"
+
+  GRUB_SHELL_DEFAULT_TIMEOUT:
+    value: "600"
+    description: >
+      Set grub-shell timeout for qemu instance (see man timeout duration
+      argument). The timeout has been raised from the default of 60 seconds
+      in the grub-shell script to 600 seconds because CI runners may run in
+      a slow virtual machine. This may need to be raised further if grub-shell
+      is failing with a timeout (exit code 137).
+
+  EXPECTED_FAILURES:
+    value: |
+      ### Usually this test passes, but occasionally the virtual machine is
+      ### too slow and the time delta threshold is exceeded. So allow failures.
+      grub_cmd_sleep
+      ### Need version of mkfs.minix which has -B option, where to find it?
+      # SKIP: minixfs_test
+      ### TODO: Gitlab shared runners use coreos kernel 4.19.78, perhaps we
+      ### can compile needed modules and load at runtime, or use our own
+      ### dedicated runner with these modules loaded.
+      ### Need mac-roman kernel module
+      # SKIP: hfs_test
+      ### Can not load the zfs modules in gitlab shared runners
+      zfs_test
+      reiserfs_test
+      f2fs_test
+      nilfs2_test
+      hfsplus_test
+      jfs_test
+      ### This should not fail, but ignoring for until it gets fixed
+      ###   ./grub-fstest: error: cannot open `(loop0)/sym': invalid symlink.
+      udf_test
+      ### sparc64-ieee1275 fails these tests, but the rest should succeed
+      ### https://marc.info/?i=20201219002902.4490b844%20()%20crass-HP-ZBook-15-G2
+      sparc64-ieee1275:grub_script_expansion
+      sparc64-ieee1275:gzcompress_test
+      sparc64-ieee1275:grub_func_test
+      sparc64-ieee1275:file_filter_test
+      ### Not sure why this is failing, but not qemu output
+      powerpc-ieee1275:pseries_test
+      ### This test is skipped by sparc64-ieee1275 due to OpenBIOS not
+      ### implementing RTC, could this be the same for powerpc-ieee1275?
+      powerpc-ieee1275:grub_cmd_date
+    description: >
+      A list of make check tests allowed to fail, one per line. Test may be
+      prefixed with "$ARCHITECTURE-$PLATFORM:" to only apply to certain
+      targets. Lines beginning with '#' may be used as comments and are
+      ignored.
+
+  EXPECTED_FUNCTIONAL_FAILURES:
+    value: |
+      ### These have not worked for a very long time
+      videotest_checksum
+      gfxterm_menu
+      cmdline_cat_test
+      ### Sometimes the machine the test are running on are slow causing
+      ### timing tests to fail.
+      sleep_test
+    description: >
+      A list of functional tests from the grub_func_test make check that are
+      allowed to fail, one per line. Lines beginning with '#' may be used as
+      comments and are ignored.
+
+  DISABLED_TESTS:
+    value: |
+      ### Failing on unreachable target error when invoking grub-mkimage
+      riscv32-efi
+      ### Need to get coreboot ROM and cbfstool
+      i386-coreboot
+      ### qemu-system-mips64: Could not load MIPS bios 'mips_bios.bin',
+      ### and no -kernel argument was specified
+      mips-arc
+      mips-qemu_mips
+      mipsel-arc
+      mipsel-qemu_mips
+      ### Need an i386 built open firmware image, otherwise uses regular bios
+      ### and stalls with "Booting from Hard Disk..."
+      i386-ieee1275
+      ### Qemu logs "Boot failed: Could not read from CDROM (code 0004)", which
+      ### according to https://lists.gnu.org/archive/html/qemu-devel/2008-06/msg00476.html
+      ### means "cd is not eltorito (BRVD)"
+      i386-multiboot
+      ### TODO: These have no support in grub-shell and use the default
+      ### qemu-system-i386, which is clearly not correct
+      arm-coreboot
+      arm-uboot
+      ia64-efi
+      riscv64-efi
+      ### Also not supported in grub-shell
+      i386-xen
+      i386-xen_pvh
+      x86_64-xen
+    description: >
+      A list of targets for which make check is not run, one per line. These
+      targets are disabled because tests mostly do not work. Lines beginning
+      with '#' may be used as comments and are ignored. Some hints as to why
+      the tests fail are included in comment lines. Patches which get these
+      tests working are very welcome.
+
+  DISABLED_IMAGES:
+    value: |
+      ### Failing on unreachable target error when invoking grub-mkimage
+      riscv32-efi
+      ### FIXME: There is a bug in grub-mkimage which causes the modules
+      ### directory for these target formats to be not found
+      arm-coreboot-vexpress
+      arm-coreboot-veyron
+    description: >
+      A list of target formats as accepted by grub-mkimage for which images
+      are not built. These target formats are disabled generally because they
+      do not work. Lines beginning with '#' may be used as comments and are
+      ignored. Some hints as to why the tests fail are included in comment
+      lines. Patches which get these tests working are very welcome.
+
+default:
+  image: ubuntu:xenial
+  # image: ubuntu:focal
+#   cache:
+#     # cache all untracked files
+#     untracked: true
+# #    policy: pull
+#     # Each branch will use the same cache
+#     key: ${CI_COMMIT_REF_SLUG}-${CI_JOB_IMAGE}
+#     paths:
+#       - ccache/
+# #      - gnulib
+
+stages:
+  - cache
+  - setup
+  - build
+  - install
+  - test
+
+.needs-cross-path:
+  variables:
+#    PATH: "$CI_BUILDS_DIR/pkgroot/sbin:$CI_BUILDS_DIR/pkgroot/usr/sbin:$CI_BUILDS_DIR/pkgroot/bin:$CI_BUILDS_DIR/pkgroot/usr/bin:$CROSS_DIR/gcc-$CROSS_VERSION-nolibc/$CROSS_TARGETS/bin:/usr/sbin:/sbin:/usr/bin:/bin"
+    PATH: "$CROSS_DIR/gcc-$CROSS_VERSION-nolibc/$CROSS_TARGETS/bin:/usr/sbin:/sbin:/usr/bin:/bin"
+
+.print_debug_info:
+  before_script: &print_debug_info-before_script
+    - echo -e "section_start:`date +%s`:debug_info[collapsed=true]\r\e[0KDebug Information"
+    - uname -a
+    - ls -l /proc/$$/exe
+    - export
+    - set
+    - dpkg-query -l
+    - find /lib/modules || ':'
+    - '`which xorriso || echo ":"` --version'
+    - find /usr/lib/locale || ':'
+    - ls -lR /usr/share/OVMF* || ':'
+    - echo -e "section_end:`date +%s`:debug_info\r\e[0K"
+
+.needs-packages: &needs-packages
+  before_script:
+    - add-apt-repository -yu ppa:ubuntu-toolchain-r/test
+#    - apt-get update -yqq
+    - apt-get -o Dir::Cache::archives=$PACKAGE_CACHE install -yqq $PACKAGES
+
+.build:ccache: &build-ccache
+  # The ccache dir needs to be cached else its pointless, but set the path in
+  # the global cache, because merging cache path entries isn't possible
+  before_script: &build-ccache-before_script
+    - apt-get update -yqq
+    - apt-get -o Dir::Cache::archives=$PACKAGE_CACHE install -yqq ccache
+    # CCache Config
+    - mkdir -p ccache/bin
+    - export CCACHE_BASEDIR=${PWD}
+    - export CCACHE_DIR=${PWD}/ccache
+    - export CCACHE_CONFIGPATH=${CCACHE_DIR}/ccache.conf
+    - export CCACHE_HARDLINK=true
+    - export CCACHE_TEMPDIR=/tmp/.ccache
+    - export CCACHE_LOGFILE="${BUILDDIR}/ccache.log"
+    # - export CCACHE_PATH="$CROSS_DIR/gcc-$CROSS_VERSION-nolibc/$CROSS_TARGETS/bin"
+    - export CC="ccache gcc"
+    # - export CC="strace -y -yy -v -s4096 -o "${BUILDDIR}/ccache.strace.log" ccache gcc"
+    # - export CC="ccache ${CROSS_TARGETS}-gcc"
+    # - export CC=/usr/lib/ccache/gcc
+    - ccache --show-stats
+    # - ln -s /usr/bin/ccache /usr/local/bin/gcc
+    #   ln -s /usr/bin/ccache /usr/local/bin/g++
+    #   ln -s /usr/bin/ccache /usr/local/bin/cc
+    #   ln -s /usr/bin/ccache /usr/local/bin/c++
+  after_script: &build-ccache-after_script
+    # The cache should be large enough to be useful but it shouldn't take
+    # too long to restore+save each run.
+    # NOTE: Envvars appear not to be preserved in after_script, so we must
+    # set the CCACHE_DIR
+    - export CCACHE_DIR=${PWD}/ccache
+    - ccache --show-stats
+    - ccache --max-size $( du --summarize --block-size=1M "$CI_PROJECT_DIR/build" | awk '{printf ("%dM", $1 * 1.5)}' )
+
+
+.install-packages:
+  before_script: &install-packages-before_script
+    # This is needed because tzdata is requesting user input to set the timezone
+    # and stalling the script
+    - TZ=UTC; ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
+    - mkdir -vp $PACKAGE_CACHE/partial
+    - start_log -c -n "install_env" "Install pkgs log"
+    - apt-get update -y
+    - PACKAGES=$(grep -v '^#' <<<"$PACKAGES");
+      echo -e 'Installing packages:' "$PACKAGES";
+      apt-get -o Dir::Cache::archives=$PACKAGE_CACHE install -yqq $PACKAGES;
+    - end_log -n "install_env"
+
+# These are packages or versions which we can not find in the default repo
+.install-non-repo-packages:
+  before_script: &install-non-repo-packages-before_script
+    - start_log -c -n "install_non_repo_env" "Install non-repo pkgs log"
+    # We need to get at least bionic's version of e2fsprogs because before that the
+    # version is < 1.43, which does not have the encrypt option
+    # - apt-get -t bionic -o Dir::Cache::archives=$PACKAGE_CACHE install -yqq e2fsprogs
+    - if dpkg --compare-versions "$(dpkg-query -l e2fsprogs |tail -n1| awk '{print $3;}')" '<=' 1.43; then
+        wget -nv -P $PACKAGE_CACHE/archives/e2fsprogs
+          http://security.ubuntu.com/ubuntu/pool/main/e/e2fsprogs/e2fsprogs_1.44.1-1ubuntu1.3_amd64.deb
+          http://security.ubuntu.com/ubuntu/pool/main/e/e2fsprogs/e2fslibs_1.44.1-1ubuntu1.3_amd64.deb
+          http://security.ubuntu.com/ubuntu/pool/main/e/e2fsprogs/libext2fs2_1.44.1-1ubuntu1.3_amd64.deb
+          http://security.ubuntu.com/ubuntu/pool/main/e/e2fsprogs/libcom-err2_1.44.1-1ubuntu1.3_amd64.deb
+          http://security.ubuntu.com/ubuntu/pool/main/e/e2fsprogs/libcomerr2_1.44.1-1ubuntu1.3_amd64.deb;
+        apt-get install -y "$PACKAGE_CACHE"/archives/e2fsprogs/*.deb;
+      fi
+    - end_log -n "install_non_repo_env"
+
+# These are extra qemu packages that we can't get through the installer
+.install-extra-qemu-packages:
+  variables: &install-extra-qemu-packages-vars
+    DEB_URLS: |
+      http://mirrors.kernel.org/ubuntu/pool/main/e/edk2/ovmf-ia32_2020.11-2_all.deb
+      http://mirrors.kernel.org/ubuntu/pool/main/e/edk2/qemu-efi-arm_2020.11-2_all.deb
+      http://mirrors.kernel.org/ubuntu/pool/main/e/edk2/qemu-efi-aarch64_2020.11-2_all.deb
+  before_script: &install-extra-qemu-packages-before_script
+    - for DEB_URL in $DEB_URLS; do
+        DEB_PATH="$PACKAGE_CACHE/archives/$(basename "$DEB_URL")";
+        if [ ! -f "$DEB_PATH" ]; then
+          wget -nv -P $PACKAGE_CACHE/archives $DEB_URL;
+        fi;
+        dpkg -i $DEB_PATH;
+      done
+    # TODO: i386-efi: Unfortunately the non-Ubuntu package ovmf-ia32_2020.11-2_all.deb
+    # package installs OVMF images that are not usable by qemu. This is the
+    # most official looking build found. Perhaps we should build our own?
+    - wget -nv -t 3 -O "${SRCDIR}/OVMF-ia32.fd"
+        https://github.com/retrage/edk2-nightly/raw/master/bin/RELEASEIa32_OVMF.fd
+    - cp -va /usr/share/qemu-efi-aarch64/QEMU_EFI.fd "${SRCDIR}/OVMF-aarch64.fd"
+    - cp -va /usr/share/AAVMF/AAVMF32_CODE.fd "${SRCDIR}/OVMF-arm.fd"
+
+.pkg-cache:
+  cache:
+    # Each branch will use the same cache
+    key: ${CI_COMMIT_REF_SLUG}-${CI_JOB_IMAGE}
+    # Set to push pull so that changes update the cache
+    policy: pull-push
+    paths:
+      - $PACKAGE_CACHE
+
+.setup-loopback:
+  before_script: &setup-loopback-before_script
+    # Gitlab docker runners do not setup the loopback devices, but it runs
+    # our code in a privileged mode so we can set it up ourselves
+    - if [ ! -e /dev/loop-control ]; then
+        mknod -m 0660 /dev/loop-control c 10 237;
+      fi
+    - for i in $(seq 0 64); do
+        mknod -m 0660 "/dev/loop$i" b 7 "$i";
+      done
+
+.all-targets:
+  parallel: &target_matrix
+    matrix:
+      - ARCH: x86_64
+        GRUB_TARGETS: [x86_64-efi, x86_64-xen]
+        CROSS_TARGETS: [x86_64-linux]
+      - ARCH: i386
+        GRUB_TARGETS: [i386-coreboot, i386-efi, i386-ieee1275, i386-multiboot, i386-pc, i386-qemu, i386-xen, i386-xen_pvh]
+        CROSS_TARGETS: [i386-linux]
+      - ARCH: powerpc
+        GRUB_TARGETS: [powerpc-ieee1275]
+        CROSS_TARGETS: [powerpc64-linux]
+      - ARCH: sparc64
+        GRUB_TARGETS: [sparc64-ieee1275]
+        CROSS_TARGETS: [sparc64-linux]
+      - ARCH: ia64
+        GRUB_TARGETS: [ia64-efi]
+        CROSS_TARGETS: [ia64-linux]
+      - ARCH: mips
+        GRUB_TARGETS: [mips-arc, mips-qemu_mips, mipsel-arc, mipsel-qemu_mips, mipsel-loongson]
+        CROSS_TARGETS: [mips64-linux]
+      - ARCH: arm
+        GRUB_TARGETS: [arm-coreboot, arm-efi, arm-uboot]
+        CROSS_TARGETS: [arm-linux-gnueabi]
+      - ARCH: arm64
+        GRUB_TARGETS: [arm64-efi]
+        CROSS_TARGETS: [aarch64-linux]
+      - ARCH: riscv32
+        GRUB_TARGETS: [riscv32-efi]
+        CROSS_TARGETS: [riscv32-linux]
+      - ARCH: riscv64
+        GRUB_TARGETS: [riscv64-efi]
+        CROSS_TARGETS: [riscv64-linux]
+
+.cache:packages:
+  stage: cache
+  extends: .pkg-cache
+  script:
+    # Install required package dependencies with package manager apt
+#    - apt-get install -o Dir::Cache::archives=$PACKAGE_CACHE -yqq software-properties-common
+#    - add-apt-repository -yu ppa:ubuntu-toolchain-r/test
+    - apt-get update -yqq
+    - mkdir -vp $PACKAGE_CACHE/partial
+#    - apt-get -o Dir::Cache::archives=$PACKAGE_CACHE install --download-only -yqq $PACKAGES
+
+.process-tests:
+  script: &process-tests-script
+    - export target=$GRUB_TARGETS
+    - test "x${SHELL_TRACE}" = "xy" && set -x;
+      if [ -f ${BUILDDIR}/test.success ]; then
+        exit 0;
+      elif [ "x${DISABLE_ALL_TESTS}" = "xy" ] || [ "x${TEST_ALL_TARGETS}" != "xy" -a -z "${DISABLED_TESTS##*$'\n'${target}$'\n'*}" ]; then
+        echo "No test output processing because testing was disabled";
+        exit 0;
+      fi;
+
+      echo -e "section_start:`date +%s`:process-test-$target[collapsed=true]\r\e[0KProcessing test output of $target";
+      if [ -f ${BUILDDIR}/test-suite.log ]; then
+        FAILURE_COND='^FAIL:';
+        if [ "x${FAIL_ON_HARD_ERRORS}" = "xy" ]; then
+          FAILURE_COND='^(FAIL|ERROR):';
+        fi;
+
+        ( grep -E "$FAILURE_COND" ${BUILDDIR}/test-suite.log || ':' ) |
+        while read _ TESTNAME; do
+          if echo "${EXPECTED_FAILURES}" | grep -qE "^(${TESTNAME}|${target}:${TESTNAME})$"; then
+            echo 'Expected failed test:' "$TESTNAME";
+            continue;
+          fi;
+
+          if [ "$TESTNAME" = "grub_func_test" ]; then
+            FAILURE=0;
+            grep -E 'test:'' FAIL' ${BUILDDIR}/${TESTNAME}.log | sort -u |
+            while read FTESTNAME STATUS; do
+              FTESTNAME=${FTESTNAME%:*};
+              if echo "${EXPECTED_FUNCTIONAL_FAILURES}" | grep -qE "^(${FTESTNAME}|${target}:${FTESTNAME})$"; then
+                echo 'Expected failed functional test:' "$FTESTNAME";
+              else
+                echo -e "${TXT_RED}"'Unexpected failed functional test:' "${FTESTNAME}${TXT_CLEAR}";
+                FAILURE=1;
+              fi;
+            done;
+            [ "${FAILURE}" -eq 0 ] && continue;
+          fi;
+
+          if [ "x${IGNORE_TIMEDOUT_TEST_FAILURE}" = "xy" ] &&
+             tail -n1 "${BUILDDIR}/${TESTNAME}.log" | grep -q 'exit status:'' 137'; then
+            echo -e "${TXT_RED}"'Ignoring Timed-out test:' "${TESTNAME}${TXT_CLEAR}";
+            echo -e -n "\e[97;100m";
+            echo "Test failed due to a timeout. This is likely due to"
+                 "insufficient runner resources, and NOT a real failure.";
+            echo -e -n "${TXT_CLEAR}";
+            echo "${TESTNAME}" >> $CI_PROJECT_DIR/${target}.timedout-failures;
+          else
+            echo -e "${TXT_RED}"'Unexpected failed test:' "${TESTNAME}${TXT_CLEAR}";
+            echo -e -n "\e[97;100m";
+            echo "Last 100 lines of ${BUILDDIR}/${TESTNAME}.log";
+            tail -n 100 ${BUILDDIR}/${TESTNAME}.log;
+            echo -e -n "${TXT_CLEAR}";
+            echo "${TESTNAME}" >> $CI_PROJECT_DIR/${target}.unexpected-failures;
+          fi;
+        done;
+      else
+        echo "No success canary and no test-suite.log, perhaps a timeout";
+      fi;
+      echo -e "section_end:`date +%s`:process-test-$target\r\e[0K";
+      if [ -f $CI_PROJECT_DIR/${target}.timedout-failures ]; then
+        sed 's/^/${target}:/' $CI_PROJECT_DIR/${target}.timedout-failures
+          >> $CI_PROJECT_DIR/build/test.timedout.log;
+      fi;
+      if [ -f $CI_PROJECT_DIR/${target}.unexpected-failures ]; then
+        sed 's/^/${target}:/' $CI_PROJECT_DIR/${target}.unexpected-failures
+          >> $CI_PROJECT_DIR/build/test.failed.log;
+        exit 1;
+      fi;
+      exit 0;
+
+.cross-compilers:
+  variables:
+    PACKAGES: |
+      wget
+      xz-utils
+    CROSS_TARGETS: &cross_targets >
+      x86_64-linux
+      i386-linux
+      powerpc64-linux
+      sparc64-linux
+      ia64-linux
+      mips64-linux
+      arm-linux-gnueabi
+      aarch64-linux
+      riscv32-linux
+      riscv64-linux
+  before_script: &cross_compilers-setup_script
+    - mkdir -pv $CROSS_DIR
+    - export HOST_ARCH=`uname -m`
+    # These give us binaries like $CROSS_DIR/gcc-8.1.0-nolibc/ia64-linux/bin/ia64-linux-gcc
+    - for i in $CROSS_TARGETS; do
+        [ -d "${CROSS_BIN_DIR}" ] ||
+        wget -nv -t 3 -O - https://mirrors.kernel.org/pub/tools/crosstool/files/bin/$HOST_ARCH/$CROSS_VERSION/$HOST_ARCH-gcc-$CROSS_VERSION-nolibc-$i.tar.xz | tar xJ -C "$CROSS_DIR";
+        export CROSS_BIN_DIR="$CROSS_DIR/gcc-$CROSS_VERSION-nolibc/$CROSS_TARGETS/bin";
+        export PATH="${CROSS_BIN_DIR}:${PATH}";
+        test -d "$CROSS_BIN_DIR" || exit 1;
+      done
+
+.setup:cross-compilers:
+  stage: setup
+  when: manual
+  extends:
+    - .pkg-cache
+    - .install-packages
+    - .cross-compilers
+  artifacts:
+    name: "${CI_JOB_NAME}-${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}-${GRUB_TARGETS}"
+    paths:
+      - $CROSS_DIR
+    expire_in: 4 hours
+
+
+setup:bootstrap:
+  stage: setup
+  extends:
+    - .pkg-cache
+  artifacts:
+    name: "${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}-bootstrap"
+    untracked: true
+    expire_in: 1 week
+    exclude:
+      - $PACKAGE_CACHE
+#  needs:
+#    - cache:packages
+  variables:
+    PACKAGES: |
+      python
+      autoconf
+      automake
+      autopoint
+      git
+      gettext
+      pkg-config
+      locales
+      language-pack-en
+      kmod
+      wget
+  before_script:
+    - . ${SRCDIR}/.ci/functions.gitlab.sh
+    - *install-packages-before_script
+  script:
+    - *print_debug_info-before_script
+    - cat /proc/filesystems
+    # - modprobe hfsplus
+    # - modprobe f2fs
+    # - modprobe reiserfs
+    - start_log -c -n "bootstrap" "Bootstrap log";
+    # We cache the bootstrap, so if the bootstrap canary exists, don't run the bootstrap
+    - '[ -f .bootstrap.finished ] || ( ./bootstrap && touch .bootstrap.finished )'
+    - end_log -n "bootstrap";
+
+.build:compile:
+  stage: build
+  when: manual
+  extends:
+    - .needs-cross-path
+    - .pkg-cache
+    - .install-packages
+  artifacts:
+    name: "${CI_JOB_NAME}-${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}-${GRUB_TARGETS}"
+    paths:
+      - $CI_PROJECT_DIR/build
+      - $CI_PROJECT_DIR/install
+      - $CROSS_DIR
+    exclude:
+      - $CI_PROJECT_DIR/build/**/*.o
+    expire_in: 4 hours
+  needs:
+    - setup:cross-compilers
+    - setup:bootstrap
+  variables:
+#    CONFIGURE_OPTS: "--disable-silent-rules"
+    PACKAGES: |
+      unifont
+      libfreetype6-dev
+      libdevmapper-dev
+      liblzma-dev
+      libfuse-dev
+      libzfslinux-dev
+      python
+      make
+      automake
+      bison
+      flex
+      gettext
+      pkg-config
+#      libsdl1.2-dev
+#      gcc-9-multilib
+  script:
+    - echo -e "section_start:`date +%s`:debug_info\r\e[0KDebugging info"
+    - 'echo "pwd: `pwd`"'
+    - 'echo "PATH: $PATH"'
+    - 'echo "CI_BUILDS_DIR: $CI_BUILDS_DIR"'
+    - 'echo "CI_PROJECT_DIR: $CI_PROJECT_DIR"'
+    - echo -e "section_start:`date +%s`:vars[collapsed=true]\r\e[0KEnvironment Variables"
+    - export
+    - echo -e "section_end:`date +%s`:vars\r\e[0K"
+    - echo -e "section_end:`date +%s`:debug_info\r\e[0K"
+    # Build all selected GRUB targets.
+    - for target in $GRUB_TARGETS; do
+        set -x;
+        while [ -z ${PATH%%*\$*} ]; do
+          eval "PATH=$PATH";
+        done;
+        plat=${target#*-};
+        arch=${target%-*};
+        case "$arch" in
+          arm64) arch=aarch64-linux;;
+          arm) arch=arm-linux-gnueabi;;
+          mipsel) arch=mips64-linux;;
+          powerpc) arch=powerpc64-linux;;
+          ia64|riscv32|riscv64|sparc64) arch=$arch-linux;;
+          i386|x86_64|mips) ;;
+          *) echo "Unkown architecture":" $arch"; exit 1;;
+        esac;
+        case "$target" in
+          sparc64-ieee1275) tformats="aout cdcore raw" ;;
+          Xmipsel-qemu_mips) tformats="elf flash" ;;
+          Xarm-coreboot) tformats="vexpress veyron" ;;
+          *) tformats='~' ;;
+        esac;
+        echo -e "section_start:`date +%s`:build-$target\r\e[0KBuilding $target";
+        builddir=$CI_PROJECT_DIR/build/obj-$target;
+        installdir=$CI_PROJECT_DIR/install/grub-$target;
+        mkdir -pv $builddir $installdir;
+        JOBS=`getconf _NPROCESSORS_ONLN 2> /dev/null || echo 1`;
+        [ "$JOBS" == 1 ] || JOBS=$(($JOBS + 1));
+        (
+          cd $builddir &&
+          $CI_PROJECT_DIR/configure --target=$arch --with-platform=$plat --prefix=$installdir $CONFIGURE_OPTS &&
+          make -j$JOBS &&
+          make -j$JOBS install;
+          for tfmt in $tformats; do
+            [ -z "${tfmt%~*}" ] ||
+            $builddir/grub-mkimage -p / -O $target${tfmt:+-$tfmt} -o /tmp/grub-$target${tfmt:+-$tfmt} echo reboot normal;
+          done;
+        ) || (
+          cd $builddir;
+          echo -e "section_start:`date +%s`:config_log-$target[collapsed=true]\r\e[0KBuild fail logs $target";
+          cat config.log;
+          echo -e "section_end:`date +%s`:config_log-$target\r\e[0K";
+          false;
+        );
+        echo -e "section_end:`date +%s`:build-$target\r\e[0K";
+      done
+  parallel:
+    <<: *target_matrix
+
+..build:install:
+  stage: install
+  allow_failure: true
+  when: manual
+  extends:
+    - .needs-cross-path
+    - .pkg-cache
+    - .install-packages
+  artifacts:
+    name: "${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}-${GRUB_TARGETS}"
+    paths:
+      - $CI_PROJECT_DIR/install
+    expire_in: 4 hours
+  needs:
+    - setup:bootstrap
+    - build:compile
+  variables:
+    PACKAGES: |
+      python
+      make
+      automake
+      pkg-config
+#      unifont
+  script:
+    - echo -e "section_start:`date +%s`:vars[collapsed=true]\r\e[0KEnvironment Variables"
+    - export
+    - echo -e "section_end:`date +%s`:vars\r\e[0K"
+    # Build all selected GRUB targets.
+    - for target in $GRUB_TARGETS; do
+        while [ -z ${PATH%%*\$*} ]; do
+          eval "PATH=$PATH";
+        done;
+        echo -e "section_start:`date +%s`:install-$target\r\e[0KInstalling $target";
+        builddir=$CI_PROJECT_DIR/build/obj-$target;
+        installdir=$CI_PROJECT_DIR/install/grub-$target;
+        mkdir -pv $builddir $installdir;
+        JOBS=`getconf _NPROCESSORS_ONLN 2> /dev/null || echo 1`;
+        [ "$JOBS" == 1 ] || JOBS=$(($JOBS + 1));
+        (
+          cd $builddir &&
+          make -j$JOBS install
+        );
+        echo -e "section_end:`date +%s`:install-$target\r\e[0K";
+      done
+  parallel:
+    <<: *target_matrix
+
+.build:rescue:
+  stage: install
+  allow_failure: true
+  when: manual
+  extends:
+    - .install-packages
+  artifacts:
+    name: "${CI_JOB_NAME}-${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}"
+    paths:
+      - $CI_PROJECT_DIR/rescue
+    expire_in: 4 hours
+  needs:
+    - build:compile
+  variables:
+    # EFI platforms use mformat from mtools when building the rescue image
+    PACKAGES: |
+      xorriso
+      mtools
+  script:
+    - echo -e "section_start:`date +%s`:vars[collapsed=true]\r\e[0KEnvironment Variables"
+    - export
+    - echo -e "section_end:`date +%s`:vars\r\e[0K"
+    # Build all selected GRUB targets.
+    - for target in $GRUB_TARGETS; do
+        echo -e "section_start:`date +%s`:mkrescue-$target\r\e[0KInstalling $target";
+        builddir=$CI_PROJECT_DIR/build/obj-$target;
+        rescuedir=$CI_PROJECT_DIR/rescue;
+        mkdir -pv $rescuedir;
+        (
+          echo -e "insmod normal; normal" > grub.cfg;
+          $builddir/grub-mkrescue -v -o $rescuedir/grub-rescue-$target.iso boot/grub/grub.cfg=grub.cfg;
+        );
+        echo -e "section_end:`date +%s`:mkrescue-$target\r\e[0K";
+      done
+  parallel:
+    <<: *target_matrix
+
+.build:docs:
+  stage: build
+  allow_failure: true
+  extends:
+    - .install-packages
+  variables:
+    # Use tex packages to build documentation
+    PACKAGES: |
+      texinfo
+      texlive
+  script:
+    - make html dvi pdf info ps man
+
+.test-all:
+#  image: ubuntu:focal
+  stage: test
+  when: manual
+  extends:
+    - .needs-cross-path
+    - .install-packages
+  artifacts:
+    name: test-artifacts
+    when: on_failure
+    paths:
+      - $CI_PROJECT_DIR/build/obj-*/*.{log,trs}
+  needs:
+    - setup:bootstrap
+    - build:compile
+  variables:
+    PACKAGES: |
+      ovmf
+      qemu-system
+      python
+      make
+      automake
+      pkg-config
+      bison
+      flex
+      wamerican
+      tar
+      cpio
+      gzip
+      lzop
+      xz-utils
+      parted
+      xorriso
+      util-linux
+      squashfs-tools
+      zfsutils-linux
+      dosfstools
+      exfat-utils
+      ntfs-3g
+      e2fsprogs
+      btrfs-progs
+      xfsprogs
+      hfsprogs
+      jfsutils
+      reiserfsprogs
+      udftools
+      nilfs-tools
+      f2fs-tools
+      genromfs
+      attr
+#      libsdl1.2-dev
+#      unifont
+#      libfreetype6-dev
+#      libdevmapper-dev
+#      liblzma-dev
+#      libfuse-dev
+#      libzfslinux-dev
+#      autoconf
+#      autopoint
+#      git
+#      gettext
+  script:
+    - for target in $GRUB_TARGETS; do
+        set -x;
+        while [ -z ${PATH%%*\$*} ]; do
+          eval "PATH=$PATH";
+        done;
+        plat=${target#*-};
+        plat=${plat%-*};
+        arch=${target%%-*};
+        echo -e "section_start:`date +%s`:test-$target[collapsed=true]\r\e[0KTesting $target";
+        builddir=$CI_PROJECT_DIR/build/obj-$target;
+        installdir=$CI_PROJECT_DIR/install/grub-$target;
+        mkdir -pv $builddir $installdir;
+        ls $builddir;
+        JOBS=`getconf _NPROCESSORS_ONLN 2> /dev/null || echo 1`;
+        [ "$JOBS" == 1 ] || JOBS=$(($JOBS + 1));
+        make -C $builddir -j$JOBS TMPDIR=/tmp SUBDIRS= V=3 check || (
+          ls $builddir;
+          false;
+        );
+        echo -e "section_end:`date +%s`:test-$target\r\e[0K";
+      done
+  parallel:
+    <<: *target_matrix
+
+# Build and test everything in one (matrix) job
+build:allinone:
+  stage: build
+  extends:
+    - .all-targets
+    - .install-packages
+  cache:
+    # Each architecture will use the same ccache
+    key: ${ARCH}-${CI_JOB_IMAGE}
+    paths:
+      - ccache/
+      - $PACKAGE_CACHE
+  artifacts:
+    name: "${CI_JOB_NAME}-${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}-${GRUB_TARGETS}"
+    # when: on_failure
+    when: always
+    paths: &target-archive-paths
+      - $CI_PROJECT_DIR/build/obj-$GRUB_TARGETS/config.h
+      - $CI_PROJECT_DIR/build/obj-$GRUB_TARGETS/*.{log,trs}
+      - $CI_PROJECT_DIR/build/obj-$GRUB_TARGETS/*.strace*
+      - $CI_PROJECT_DIR/build/obj-$GRUB_TARGETS/test-data.tar.*
+      - $CI_PROJECT_DIR/build/obj-$GRUB_TARGETS/test.success
+      - $CI_PROJECT_DIR/build/obj-$GRUB_TARGETS/images
+      # - $CI_PROJECT_DIR/build/*.log
+      # - $CROSS_DIR
+    exclude:
+      - $CI_PROJECT_DIR/build/**/*.o
+    expire_in: 5 days
+  needs:
+    - setup:bootstrap
+  variables:
+    <<: *install-extra-qemu-packages-vars
+    BUILDDIR: $CI_PROJECT_DIR/build/obj-$GRUB_TARGETS
+    INSTALLDIR: $CI_PROJECT_DIR/install/grub-$GRUB_TARGETS
+    # CONFIGURE_OPTS: "--disable-silent-rules"
+    # TMPDIR: "/tmp"
+    TEST_DATA_PREFIX: $CI_PROJECT_DIR/build/obj-${GRUB_TARGETS}/test-data
+    # GRUB_QEMU_OPTS: -m size=64M
+    PACKAGES: |
+      wget
+      ### Grub build requirements
+      unifont
+      libfreetype6-dev
+      libdevmapper-dev
+      liblzma-dev
+      libfuse-dev
+      libzfslinux-dev
+      python
+      make
+      automake
+      bison
+      flex
+      gettext
+      pkg-config
+      ### Grub testing requirements
+      strace
+      qemu-system
+      # qemu-efi-aarch64
+      # qemu-efi-arm
+      # qemu-efi
+      ovmf
+      # ovmf-ia32
+      openbios-ppc
+      openbios-sparc
+      ### Used by grub-fs-tester
+      wamerican
+      ### needed by grub-mkrescue
+      xorriso
+      ### needed by xorriso to handle utf8 chars
+      locales
+      language-pack-en
+      ### needed by grub-mkrescue for efi platforms
+      mtools
+      ### Requirements for specific tests
+      tar
+      cpio
+      gzip
+      lzop
+      xz-utils
+      parted
+      util-linux
+      squashfs-tools
+      zfsutils-linux
+      dosfstools
+      exfat-utils
+      ntfs-3g
+      e2fsprogs
+      btrfs-progs
+      xfsprogs
+      hfsprogs
+      jfsutils
+      reiserfsprogs
+      udftools
+      nilfs-tools
+      f2fs-tools
+      genromfs
+      attr
+      ### Needed for mac-roman module for hfs_test
+      # linux-image-extra
+  before_script:
+    - mkdir -pv ${BUILDDIR} ${INSTALLDIR};
+    - export > ${BUILDDIR}/environment.log
+    - . ${SRCDIR}/.ci/functions.gitlab.sh
+    - *setup-loopback-before_script
+    - *install-packages-before_script
+    - *install-non-repo-packages-before_script
+    - *install-extra-qemu-packages-before_script
+    - *cross_compilers-setup_script
+    - *build-ccache-before_script
+    - *print_debug_info-before_script
+    # Need to keep evaling PATH until there are no more variables in it
+    # because gitlab CI does not do recursive variable expansion.
+    - while [ -z ${PATH%%*\$*} ]; do
+        eval "export PATH=$PATH";
+      done;
+    - pwd
+    - ls -la
+  script:
+    # Build all selected GRUB targets.
+    - test "x${SHELL_TRACE}" = "xy" && set -x;
+    - export TARGET=${GRUB_TARGETS};
+    - export ARCH=${TARGET%-*};
+    - export PLATFORM=${TARGET#*-};
+    - if [ "x${BUILD_ALL_TARGETS}" != "xy" ] && ( echo "${DISABLED_BUILDS}" | grep -q "^${TARGET}$" ); then
+        echo -e "${TXT_CLEAR}${TXT_YELLOW}Building $TARGET has been disabled, skipping.${TXT_CLEAR}";
+        exit 0;
+      fi;
+    - (
+        case "$ARCH" in
+          arm64) ARCH=aarch64-linux;;
+          arm) ARCH=arm-linux-gnueabi;;
+          mips) ARCH=mips64-linux;;
+          powerpc) ARCH=powerpc64-linux;;
+          ia64|riscv32|riscv64|sparc64) ARCH=$ARCH-linux;;
+          i386|x86_64) ;;
+          *) echo "Unknown architecture":" $ARCH"; exit 1;;
+        esac;
+        case "$PLATFORM" in
+          coreboot)
+            export GRUB_CBFSTOOL="/where/is/cbfstool";
+            export GRUB_COREBOOT_ROM="/where/is/coreboot/rom";;
+        esac;
+
+        echo -e "#!$SHELL\nREAL_SHELL=\${REAL_SHELL:-\$(readlink /proc/\$\$/exe)}\nexec \$REAL_SHELL \$SHELL_OPTS \"\$@\"" >${BUILDDIR}/shell-wrapper.sh;
+        chmod +x ${BUILDDIR}/shell-wrapper.sh;
+        export CONFIG_SHELL=${BUILDDIR}/shell-wrapper.sh;
+        export CONFIGURE_OPTS+=" CONFIG_SHELL=${BUILDDIR}/shell-wrapper.sh";
+
+        $SRCDIR/.ci/build.sh 2>&1 | tee ${BUILDDIR}/build.log;
+
+        if [ "x${DISABLE_ALL_TESTS}" = "xy" ]; then
+          echo "All tests are disabled";
+          exit 0;
+        fi;
+
+        export TESTTMPDIR="${TMPDIR:-/tmp}/grub-test-$TARGET-${CI_COMMIT_SHORT_SHA}";
+        $SRCDIR/.ci/test.sh 2>&1 | tee ${BUILDDIR}/test.log || :;
+      ) 2>&1 | tee "${BUILDDIR}/${CI_JOB_NAME}.log";
+    # Do processing of testing output
+    #- *process-tests-script
+    - TARGET=${GRUB_TARGETS} $SRCDIR/.ci/process-tests.sh
+    # Make images for all formats
+    - TARGET=${GRUB_TARGETS} $SRCDIR/.ci/make-images.sh
+  after_script:
+    - *build-ccache-after_script
+
+.test:allinone:
+  stage: test
+  # Run always, even if build fails so we can see pass/fail of the
+  # individual matrice runs.
+  when: always
+  artifacts:
+    name: "${CI_JOB_NAME}-${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}-${GRUB_TARGETS}"
+#    when: on_failure
+    when: always
+    paths:
+      - $CI_PROJECT_DIR/build/obj-$GRUB_TARGETS/*.{log,trs}
+      - $CI_PROJECT_DIR/build/*.log
+#    expire_in: 1 hour
+  dependencies:
+#  needs:
+    - build:allinone
+  script: &test_allinone-script
+    - *process-tests-script
+  parallel:
+    <<: *target_matrix
+
+package:allinone:
+  stage: .post
+  when: always
+  artifacts:
+    name: "${CI_JOB_NAME}-${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}"
+    when: always
+    expire_in: 1 month
+    paths:
+      # - *target-archive-paths
+      # - $CI_PROJECT_DIR/build/obj-*/*.{log,trs}
+      # - $CI_PROJECT_DIR/build/obj-*/*.strace*
+      # - $CI_PROJECT_DIR/build/obj-*/test-data.tar.*
+      # - $CI_PROJECT_DIR/build/obj-*/test.success
+      - $CI_PROJECT_DIR/build/obj-*
+      - $CI_PROJECT_DIR/build/*.log
+    exclude:
+      - $CI_PROJECT_DIR/build/**/*.o
+      - $CI_PROJECT_DIR/build/**/.deps-util
+  needs:
+    - build:allinone
+    # - test:allinone
+  before_script:
+    - . ${SRCDIR}/.ci/functions.gitlab.sh
+    - SUMLOG=$CI_PROJECT_DIR/build/test-summary.log
+    # - TXT_RED="\e[31m"; TXT_YELLOW="\e[33m"; TXT_CLEAR="\e[0m";
+    - . $CI_PROJECT_DIR/.ci/functions.gitlab.sh
+    - find $CI_PROJECT_DIR/build > $CI_PROJECT_DIR/build/build-paths.log
+  script:
+    - for F in $CI_PROJECT_DIR/build/obj-*/test-suite.log; do
+        [ ! -e "$F" ] && continue;
+        TARGETDIR=$(dirname "$F");
+        TARGET=$(cut -d- -f2- <<<$(basename "${TARGETDIR}"));
+        echo 'Ran test suite for target:' "${TARGET}";
+        start_log -n "test-$TARGET" "Ran test suite for target:"" ${TARGET}" >>${SUMLOG};
+        if [ -f "${TARGETDIR}/test.failures.log" ]; then
+          grep "^${TARGET}" "${TARGETDIR}/test.failures.log" | sort |
+          ( IFS=:; while read TARGET FTYPE TESTNAME; do
+              case "$FTYPE" in
+                skip)
+                  echo -e "${TXT_CLEAR}${TXT_YELLOW}"'  Skipped test:' "${TESTNAME}${TXT_CLEAR}" >>${SUMLOG};;
+                error)
+                  echo -e "${TXT_CLEAR}${TXT_YELLOW}"'  Hard error in test:' "${TESTNAME}${TXT_CLEAR}" >>${SUMLOG};;
+                expected)
+                  echo -e "${TXT_CLEAR}${TXT_YELLOW}"'  Expected failed test:' "${TESTNAME}${TXT_CLEAR}" >>${SUMLOG};;
+                timeout)
+                  echo -e "${TXT_CLEAR}${TXT_YELLOW}"'  Ignoring timed-out test:' "${TESTNAME}${TXT_CLEAR}" >>${SUMLOG};;
+                unexpected)
+                  echo -e "${TXT_CLEAR}${TXT_RED}"'  Failed test:' "${TESTNAME}${TXT_CLEAR}" >>${SUMLOG};;
+                *)
+                  echo -e "${TXT_CLEAR}${TXT_RED}"'  Unhandled test type for '"${TESTNAME} (type=$FTYPE)${TXT_CLEAR}" >>${SUMLOG};;
+              esac;
+            done );
+            echo -e -n "${TXT_CLEAR}" >>${SUMLOG};
+        fi;
+        end_log -n "test-$TARGET" >>${SUMLOG};
+      done
+    - if [ -f "${SUMLOG}" ]; then
+        cat "${SUMLOG}";
+      fi
+
+# mergereq:package:allinone:
+#   extends:
+#     - package:allinone
+#   rules:
+#     - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
+#     - if: $CI_MERGE_REQUEST_IID
+#     - when: always
+
+#deploy-prod:
+#  stage: deploy
+#  script:
+#    - echo "This job deploys something from the $CI_COMMIT_BRANCH branch."
+#    - echo "Hello, $GITLAB_USER_LOGIN!"
+
+# TODO: put generated docs in gitlab pages
+# pages:
+#   stage: deploy
+#   needs: build:docs
+#   script:
+#     - mkdir .public
+#     - cp -r * .public
+#     - mv .public public
+#   artifacts:
+#     paths:
+#       - public
+#   only:
+#     - master
-- 
2.27.0

[CI 17/17] keep: remove a lot of cruft that we probably want to reuse at some point

[Glenn Washburn]

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 .ci/build.sh         |  10 -
 .ci/process-tests.sh |  11 -
 .ci/test.sh          |   8 -
 .gitlab-ci.yml       | 552 -------------------------------------------
 4 files changed, 581 deletions(-)

diff --git a/.ci/build.sh b/.ci/build.sh
index 14dd7f7c5..55af74a1b 100755
--- a/.ci/build.sh
+++ b/.ci/build.sh
@@ -31,16 +31,6 @@ mkdir -pv ${BUILDDIR};
 RET=0;
 cd ${BUILDDIR};
 
-# echo -e "#!$SHELL\nREAL_SHELL=\${REAL_SHELL:-\$(readlink /proc/\$\$/exe)}\nexec \$REAL_SHELL \$SHELL_OPTS \"\$@\"" >${BUILDDIR}/shell-wrapper.sh;
-# cat <<EOF >${BUILDDIR}/shell-wrapper.sh
-# #!$SHELL
-# REAL_SHELL=\${REAL_SHELL:-$(readlink /proc/\$\$/exe)}
-# exec \$REAL_SHELL \$SHELL_OPTS \"\$@\"
-# EOF
-
-# chmod +x ${BUILDDIR}/shell-wrapper.sh;
-# export CONFIG_SHELL=${BUILDDIR}/shell-wrapper.sh;
-
 start_log -c -n "configure-$TARGET" "Configuring $TARGET";
 ls -la $SRCDIR;
 [ -x "$SRCDIR/configure" ] && $SRCDIR/configure --help;
diff --git a/.ci/process-tests.sh b/.ci/process-tests.sh
index ab030cfc2..635436e39 100755
--- a/.ci/process-tests.sh
+++ b/.ci/process-tests.sh
@@ -25,8 +25,6 @@ set -eo pipefail
 [ -f "$(dirname "$0")/functions.$CI_TYPE.sh" ] &&
 . "$(dirname "$0")/functions.$CI_TYPE.sh"
 
-# TARGET="${ARCH}-${PLATFORM}"
-
 test "x${SHELL_TRACE}" = "xy" && set -x;
 if [ -f ${BUILDDIR}/test.success ]; then
   echo "Not processing test logs because make check ran successfully";
@@ -96,13 +94,4 @@ fi;
   RET=1;
 fi;
 end_log -n "process-test-$TARGET";
-# if [ -f $CI_PROJECT_DIR/${TARGET}.timedout-failures ]; then
-#   sed 's/^/${TARGET}:/' $CI_PROJECT_DIR/${TARGET}.timedout-failures
-#     >> $CI_PROJECT_DIR/build/test.timedout.log;
-# fi;
-# if [ -f $CI_PROJECT_DIR/${TARGET}.unexpected-failures ]; then
-#   sed 's/^/${TARGET}:/' $CI_PROJECT_DIR/${TARGET}.unexpected-failures
-#     >> $CI_PROJECT_DIR/build/test.failed.log;
-#   exit 1;
-# fi;
 exit ${RET:-0};
diff --git a/.ci/test.sh b/.ci/test.sh
index 3f51de58b..b780ae51f 100755
--- a/.ci/test.sh
+++ b/.ci/test.sh
@@ -85,16 +85,8 @@ if [ -n "$GRUB_SHELL_DEFAULT_DEBUG" ]; then
 fi;
 TIMEOUT=${TESTS_TIMEOUT};
 
-function func_strace () {
-  strace -y -yy -f -v -s4096 -o >($COMP -9 > ${BUILDDIR}/test.strace.$RANDOM.$COMP) "$@";
-};
-':' STRACE="func_strace";
-
-':' $STRACE ${TIMEOUT:+timeout $TIMEOUT} make -C ${BUILDDIR} -j$JOBS SUBDIRS= check || RET=$?;
 ${TIMEOUT:+timeout $TIMEOUT} make -C ${BUILDDIR} -j$JOBS SUBDIRS= check || RET=$?;
-':' timeout $TIMEOUT $STRACE make -C ${BUILDDIR} -j$JOBS SUBDIRS= LOG_COMPILER="${BUILDDIR}/log-tester.sh" check || RET=$?;
 [ "$RET" -eq 124 -o "$RET" -eq 137 ] && echo "ERROR":" make check timed out";
-':' gzip -9 ${BUILDDIR}/test.strace.*;
 
 end_log -n "test-$TARGET";
 
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d4fc2337e..b3f854f61 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -13,76 +13,20 @@ workflow:
     - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
     - when: never
 
-# This allows running CI/CD pipelines for merge requests
-# include:
-#  - template: 'Workflows/MergeRequest-Pipelines.gitlab-ci.yml'
-
-# Variables that can be set by runners:
-# CONFIGURE_OPTS - Extra configure options
-# TESTS_TIMEOUT - Set timeout for make check tests (see man timeout)
-# FAIL_ON_HARD_ERRORS - If set, Hard errors will cause a failure
-# DISABLE_ALL_TESTS - If set, make check tests will be disabled
-# TEST_ALL_TARGETS - If set, failing targets which have been marked disabled will be enabled
-# TEST_VERBOSITY - Verbosity level when running tests: 1-3
-# STRACE_TESTS - If set, strace individual tests. WARNING: This will cause testing to take much longer
-#                which can cause some test to fail by timing out or having incorrect timing
-# SHELL_TRACE - If set, turn on shell tracing of everything. NOTE: TEST_VERBOSITY=3 turns on more targeted
-#               shell tracing
-
 variables:
   CI_TYPE: gitlab
-  # CI_DEBUG_TRACE: 'true'
-  # GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/grub-$CI_COMMIT_SHORT_SHA
   GIT_CLONE_PATH: $CI_BUILDS_DIR/grub-${CI_COMMIT_SHORT_SHA}.git
   GIT_STRATEGY: fetch
 
-  # Include all cross toolchain paths, so we can just call them later down.
-  #PATH: "/tmp/qemu-install/bin:/usr/bin:/bin:$CROSS_DIR/gcc-8.1.0-nolibc/aarch64-linux/bin:$CROSS_DIR/gcc-8.1.0-nolibc/arm-linux-gnueabi/bin:$CROSS_DIR/gcc-8.1.0-nolibc/ia64-linux/bin:$CROSS_DIR/gcc-8.1.0-nolibc/mips64-linux/bin:$CROSS_DIR/gcc-8.1.0-nolibc/powerpc64-linux/bin:$CROSS_DIR/gcc-8.1.0-nolibc/riscv32-linux/bin:$CROSS_DIR/gcc-8.1.0-nolibc/riscv64-linux/bin:$CROSS_DIR/gcc-8.1.0-nolibc/sparc64-linux/bin"
-#  PACKAGE_CACHE: $CI_BUILDS_DIR/pkgs
   PACKAGE_CACHE: $CI_PROJECT_DIR/pkgs
   CROSS_DIR: $CI_PROJECT_DIR/cross
   SRCDIR: $CI_PROJECT_DIR
 
   ### Below are values that are meant to be configurable
   CROSS_VERSION:
-    # value: "4.8.0"
-    # value: "4.8.5"
-    # value: "4.9.0"
-    # value: "4.9.4"
-    # value: "5.5.0"
-    # Start having riscv32/64 builds
-    # value: "7.3.0"
-    # value: "7.5.0"
-    # value: "8.1.0"
-    # value: "9.3.0"
     value: "10.1.0"
     description: "Version of cross-compiler to use (suggested: 10.1.0, 9.3.0, 8.1.0, 7.5.0 [riscv32/64 builds fail before 7.3.0])"
 
-  PACKAGES: |
-    libsdl1.2-dev
-    xz-utils
-    lzop
-    ovmf
-    python
-    qemu-system
-    unifont
-    wget
-    libfreetype6-dev
-    libdevmapper-dev
-    liblzma-dev
-    libfuse-dev
-    libzfslinux-dev
-    make
-    autoconf
-    automake
-    autopoint
-    git
-    bison
-    flex
-    gettext
-    pkg-config
-#    gcc-9-multilib
-
   CONFIGURE_OPTS:
     value: "--enable-boot-time"
     description: "Extra options to pass to configure"
@@ -279,49 +223,12 @@ variables:
 
 default:
   image: ubuntu:xenial
-  # image: ubuntu:focal
-#   cache:
-#     # cache all untracked files
-#     untracked: true
-# #    policy: pull
-#     # Each branch will use the same cache
-#     key: ${CI_COMMIT_REF_SLUG}-${CI_JOB_IMAGE}
-#     paths:
-#       - ccache/
-# #      - gnulib
 
 stages:
-  - cache
   - setup
   - build
-  - install
   - test
 
-.needs-cross-path:
-  variables:
-#    PATH: "$CI_BUILDS_DIR/pkgroot/sbin:$CI_BUILDS_DIR/pkgroot/usr/sbin:$CI_BUILDS_DIR/pkgroot/bin:$CI_BUILDS_DIR/pkgroot/usr/bin:$CROSS_DIR/gcc-$CROSS_VERSION-nolibc/$CROSS_TARGETS/bin:/usr/sbin:/sbin:/usr/bin:/bin"
-    PATH: "$CROSS_DIR/gcc-$CROSS_VERSION-nolibc/$CROSS_TARGETS/bin:/usr/sbin:/sbin:/usr/bin:/bin"
-
-.print_debug_info:
-  before_script: &print_debug_info-before_script
-    - echo -e "section_start:`date +%s`:debug_info[collapsed=true]\r\e[0KDebug Information"
-    - uname -a
-    - ls -l /proc/$$/exe
-    - export
-    - set
-    - dpkg-query -l
-    - find /lib/modules || ':'
-    - '`which xorriso || echo ":"` --version'
-    - find /usr/lib/locale || ':'
-    - ls -lR /usr/share/OVMF* || ':'
-    - echo -e "section_end:`date +%s`:debug_info\r\e[0K"
-
-.needs-packages: &needs-packages
-  before_script:
-    - add-apt-repository -yu ppa:ubuntu-toolchain-r/test
-#    - apt-get update -yqq
-    - apt-get -o Dir::Cache::archives=$PACKAGE_CACHE install -yqq $PACKAGES
-
 .build:ccache: &build-ccache
   # The ccache dir needs to be cached else its pointless, but set the path in
   # the global cache, because merging cache path entries isn't possible
@@ -336,16 +243,8 @@ stages:
     - export CCACHE_HARDLINK=true
     - export CCACHE_TEMPDIR=/tmp/.ccache
     - export CCACHE_LOGFILE="${BUILDDIR}/ccache.log"
-    # - export CCACHE_PATH="$CROSS_DIR/gcc-$CROSS_VERSION-nolibc/$CROSS_TARGETS/bin"
     - export CC="ccache gcc"
-    # - export CC="strace -y -yy -v -s4096 -o "${BUILDDIR}/ccache.strace.log" ccache gcc"
-    # - export CC="ccache ${CROSS_TARGETS}-gcc"
-    # - export CC=/usr/lib/ccache/gcc
     - ccache --show-stats
-    # - ln -s /usr/bin/ccache /usr/local/bin/gcc
-    #   ln -s /usr/bin/ccache /usr/local/bin/g++
-    #   ln -s /usr/bin/ccache /usr/local/bin/cc
-    #   ln -s /usr/bin/ccache /usr/local/bin/c++
   after_script: &build-ccache-after_script
     # The cache should be large enough to be useful but it shouldn't take
     # too long to restore+save each run.
@@ -375,7 +274,6 @@ stages:
     - start_log -c -n "install_non_repo_env" "Install non-repo pkgs log"
     # We need to get at least bionic's version of e2fsprogs because before that the
     # version is < 1.43, which does not have the encrypt option
-    # - apt-get -t bionic -o Dir::Cache::archives=$PACKAGE_CACHE install -yqq e2fsprogs
     - if dpkg --compare-versions "$(dpkg-query -l e2fsprogs |tail -n1| awk '{print $3;}')" '<=' 1.43; then
         wget -nv -P $PACKAGE_CACHE/archives/e2fsprogs
           http://security.ubuntu.com/ubuntu/pool/main/e/e2fsprogs/e2fsprogs_1.44.1-1ubuntu1.3_amd64.deb
@@ -464,89 +362,6 @@ stages:
         GRUB_TARGETS: [riscv64-efi]
         CROSS_TARGETS: [riscv64-linux]
 
-.cache:packages:
-  stage: cache
-  extends: .pkg-cache
-  script:
-    # Install required package dependencies with package manager apt
-#    - apt-get install -o Dir::Cache::archives=$PACKAGE_CACHE -yqq software-properties-common
-#    - add-apt-repository -yu ppa:ubuntu-toolchain-r/test
-    - apt-get update -yqq
-    - mkdir -vp $PACKAGE_CACHE/partial
-#    - apt-get -o Dir::Cache::archives=$PACKAGE_CACHE install --download-only -yqq $PACKAGES
-
-.process-tests:
-  script: &process-tests-script
-    - export target=$GRUB_TARGETS
-    - test "x${SHELL_TRACE}" = "xy" && set -x;
-      if [ -f ${BUILDDIR}/test.success ]; then
-        exit 0;
-      elif [ "x${DISABLE_ALL_TESTS}" = "xy" ] || [ "x${TEST_ALL_TARGETS}" != "xy" -a -z "${DISABLED_TESTS##*$'\n'${target}$'\n'*}" ]; then
-        echo "No test output processing because testing was disabled";
-        exit 0;
-      fi;
-
-      echo -e "section_start:`date +%s`:process-test-$target[collapsed=true]\r\e[0KProcessing test output of $target";
-      if [ -f ${BUILDDIR}/test-suite.log ]; then
-        FAILURE_COND='^FAIL:';
-        if [ "x${FAIL_ON_HARD_ERRORS}" = "xy" ]; then
-          FAILURE_COND='^(FAIL|ERROR):';
-        fi;
-
-        ( grep -E "$FAILURE_COND" ${BUILDDIR}/test-suite.log || ':' ) |
-        while read _ TESTNAME; do
-          if echo "${EXPECTED_FAILURES}" | grep -qE "^(${TESTNAME}|${target}:${TESTNAME})$"; then
-            echo 'Expected failed test:' "$TESTNAME";
-            continue;
-          fi;
-
-          if [ "$TESTNAME" = "grub_func_test" ]; then
-            FAILURE=0;
-            grep -E 'test:'' FAIL' ${BUILDDIR}/${TESTNAME}.log | sort -u |
-            while read FTESTNAME STATUS; do
-              FTESTNAME=${FTESTNAME%:*};
-              if echo "${EXPECTED_FUNCTIONAL_FAILURES}" | grep -qE "^(${FTESTNAME}|${target}:${FTESTNAME})$"; then
-                echo 'Expected failed functional test:' "$FTESTNAME";
-              else
-                echo -e "${TXT_RED}"'Unexpected failed functional test:' "${FTESTNAME}${TXT_CLEAR}";
-                FAILURE=1;
-              fi;
-            done;
-            [ "${FAILURE}" -eq 0 ] && continue;
-          fi;
-
-          if [ "x${IGNORE_TIMEDOUT_TEST_FAILURE}" = "xy" ] &&
-             tail -n1 "${BUILDDIR}/${TESTNAME}.log" | grep -q 'exit status:'' 137'; then
-            echo -e "${TXT_RED}"'Ignoring Timed-out test:' "${TESTNAME}${TXT_CLEAR}";
-            echo -e -n "\e[97;100m";
-            echo "Test failed due to a timeout. This is likely due to"
-                 "insufficient runner resources, and NOT a real failure.";
-            echo -e -n "${TXT_CLEAR}";
-            echo "${TESTNAME}" >> $CI_PROJECT_DIR/${target}.timedout-failures;
-          else
-            echo -e "${TXT_RED}"'Unexpected failed test:' "${TESTNAME}${TXT_CLEAR}";
-            echo -e -n "\e[97;100m";
-            echo "Last 100 lines of ${BUILDDIR}/${TESTNAME}.log";
-            tail -n 100 ${BUILDDIR}/${TESTNAME}.log;
-            echo -e -n "${TXT_CLEAR}";
-            echo "${TESTNAME}" >> $CI_PROJECT_DIR/${target}.unexpected-failures;
-          fi;
-        done;
-      else
-        echo "No success canary and no test-suite.log, perhaps a timeout";
-      fi;
-      echo -e "section_end:`date +%s`:process-test-$target\r\e[0K";
-      if [ -f $CI_PROJECT_DIR/${target}.timedout-failures ]; then
-        sed 's/^/${target}:/' $CI_PROJECT_DIR/${target}.timedout-failures
-          >> $CI_PROJECT_DIR/build/test.timedout.log;
-      fi;
-      if [ -f $CI_PROJECT_DIR/${target}.unexpected-failures ]; then
-        sed 's/^/${target}:/' $CI_PROJECT_DIR/${target}.unexpected-failures
-          >> $CI_PROJECT_DIR/build/test.failed.log;
-        exit 1;
-      fi;
-      exit 0;
-
 .cross-compilers:
   variables:
     PACKAGES: |
@@ -575,20 +390,6 @@ stages:
         test -d "$CROSS_BIN_DIR" || exit 1;
       done
 
-.setup:cross-compilers:
-  stage: setup
-  when: manual
-  extends:
-    - .pkg-cache
-    - .install-packages
-    - .cross-compilers
-  artifacts:
-    name: "${CI_JOB_NAME}-${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}-${GRUB_TARGETS}"
-    paths:
-      - $CROSS_DIR
-    expire_in: 4 hours
-
-
 setup:bootstrap:
   stage: setup
   extends:
@@ -599,8 +400,6 @@ setup:bootstrap:
     expire_in: 1 week
     exclude:
       - $PACKAGE_CACHE
-#  needs:
-#    - cache:packages
   variables:
     PACKAGES: |
       python
@@ -618,296 +417,11 @@ setup:bootstrap:
     - . ${SRCDIR}/.ci/functions.gitlab.sh
     - *install-packages-before_script
   script:
-    - *print_debug_info-before_script
-    - cat /proc/filesystems
-    # - modprobe hfsplus
-    # - modprobe f2fs
-    # - modprobe reiserfs
     - start_log -c -n "bootstrap" "Bootstrap log";
     # We cache the bootstrap, so if the bootstrap canary exists, don't run the bootstrap
     - '[ -f .bootstrap.finished ] || ( ./bootstrap && touch .bootstrap.finished )'
     - end_log -n "bootstrap";
 
-.build:compile:
-  stage: build
-  when: manual
-  extends:
-    - .needs-cross-path
-    - .pkg-cache
-    - .install-packages
-  artifacts:
-    name: "${CI_JOB_NAME}-${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}-${GRUB_TARGETS}"
-    paths:
-      - $CI_PROJECT_DIR/build
-      - $CI_PROJECT_DIR/install
-      - $CROSS_DIR
-    exclude:
-      - $CI_PROJECT_DIR/build/**/*.o
-    expire_in: 4 hours
-  needs:
-    - setup:cross-compilers
-    - setup:bootstrap
-  variables:
-#    CONFIGURE_OPTS: "--disable-silent-rules"
-    PACKAGES: |
-      unifont
-      libfreetype6-dev
-      libdevmapper-dev
-      liblzma-dev
-      libfuse-dev
-      libzfslinux-dev
-      python
-      make
-      automake
-      bison
-      flex
-      gettext
-      pkg-config
-#      libsdl1.2-dev
-#      gcc-9-multilib
-  script:
-    - echo -e "section_start:`date +%s`:debug_info\r\e[0KDebugging info"
-    - 'echo "pwd: `pwd`"'
-    - 'echo "PATH: $PATH"'
-    - 'echo "CI_BUILDS_DIR: $CI_BUILDS_DIR"'
-    - 'echo "CI_PROJECT_DIR: $CI_PROJECT_DIR"'
-    - echo -e "section_start:`date +%s`:vars[collapsed=true]\r\e[0KEnvironment Variables"
-    - export
-    - echo -e "section_end:`date +%s`:vars\r\e[0K"
-    - echo -e "section_end:`date +%s`:debug_info\r\e[0K"
-    # Build all selected GRUB targets.
-    - for target in $GRUB_TARGETS; do
-        set -x;
-        while [ -z ${PATH%%*\$*} ]; do
-          eval "PATH=$PATH";
-        done;
-        plat=${target#*-};
-        arch=${target%-*};
-        case "$arch" in
-          arm64) arch=aarch64-linux;;
-          arm) arch=arm-linux-gnueabi;;
-          mipsel) arch=mips64-linux;;
-          powerpc) arch=powerpc64-linux;;
-          ia64|riscv32|riscv64|sparc64) arch=$arch-linux;;
-          i386|x86_64|mips) ;;
-          *) echo "Unkown architecture":" $arch"; exit 1;;
-        esac;
-        case "$target" in
-          sparc64-ieee1275) tformats="aout cdcore raw" ;;
-          Xmipsel-qemu_mips) tformats="elf flash" ;;
-          Xarm-coreboot) tformats="vexpress veyron" ;;
-          *) tformats='~' ;;
-        esac;
-        echo -e "section_start:`date +%s`:build-$target\r\e[0KBuilding $target";
-        builddir=$CI_PROJECT_DIR/build/obj-$target;
-        installdir=$CI_PROJECT_DIR/install/grub-$target;
-        mkdir -pv $builddir $installdir;
-        JOBS=`getconf _NPROCESSORS_ONLN 2> /dev/null || echo 1`;
-        [ "$JOBS" == 1 ] || JOBS=$(($JOBS + 1));
-        (
-          cd $builddir &&
-          $CI_PROJECT_DIR/configure --target=$arch --with-platform=$plat --prefix=$installdir $CONFIGURE_OPTS &&
-          make -j$JOBS &&
-          make -j$JOBS install;
-          for tfmt in $tformats; do
-            [ -z "${tfmt%~*}" ] ||
-            $builddir/grub-mkimage -p / -O $target${tfmt:+-$tfmt} -o /tmp/grub-$target${tfmt:+-$tfmt} echo reboot normal;
-          done;
-        ) || (
-          cd $builddir;
-          echo -e "section_start:`date +%s`:config_log-$target[collapsed=true]\r\e[0KBuild fail logs $target";
-          cat config.log;
-          echo -e "section_end:`date +%s`:config_log-$target\r\e[0K";
-          false;
-        );
-        echo -e "section_end:`date +%s`:build-$target\r\e[0K";
-      done
-  parallel:
-    <<: *target_matrix
-
-..build:install:
-  stage: install
-  allow_failure: true
-  when: manual
-  extends:
-    - .needs-cross-path
-    - .pkg-cache
-    - .install-packages
-  artifacts:
-    name: "${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}-${GRUB_TARGETS}"
-    paths:
-      - $CI_PROJECT_DIR/install
-    expire_in: 4 hours
-  needs:
-    - setup:bootstrap
-    - build:compile
-  variables:
-    PACKAGES: |
-      python
-      make
-      automake
-      pkg-config
-#      unifont
-  script:
-    - echo -e "section_start:`date +%s`:vars[collapsed=true]\r\e[0KEnvironment Variables"
-    - export
-    - echo -e "section_end:`date +%s`:vars\r\e[0K"
-    # Build all selected GRUB targets.
-    - for target in $GRUB_TARGETS; do
-        while [ -z ${PATH%%*\$*} ]; do
-          eval "PATH=$PATH";
-        done;
-        echo -e "section_start:`date +%s`:install-$target\r\e[0KInstalling $target";
-        builddir=$CI_PROJECT_DIR/build/obj-$target;
-        installdir=$CI_PROJECT_DIR/install/grub-$target;
-        mkdir -pv $builddir $installdir;
-        JOBS=`getconf _NPROCESSORS_ONLN 2> /dev/null || echo 1`;
-        [ "$JOBS" == 1 ] || JOBS=$(($JOBS + 1));
-        (
-          cd $builddir &&
-          make -j$JOBS install
-        );
-        echo -e "section_end:`date +%s`:install-$target\r\e[0K";
-      done
-  parallel:
-    <<: *target_matrix
-
-.build:rescue:
-  stage: install
-  allow_failure: true
-  when: manual
-  extends:
-    - .install-packages
-  artifacts:
-    name: "${CI_JOB_NAME}-${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}"
-    paths:
-      - $CI_PROJECT_DIR/rescue
-    expire_in: 4 hours
-  needs:
-    - build:compile
-  variables:
-    # EFI platforms use mformat from mtools when building the rescue image
-    PACKAGES: |
-      xorriso
-      mtools
-  script:
-    - echo -e "section_start:`date +%s`:vars[collapsed=true]\r\e[0KEnvironment Variables"
-    - export
-    - echo -e "section_end:`date +%s`:vars\r\e[0K"
-    # Build all selected GRUB targets.
-    - for target in $GRUB_TARGETS; do
-        echo -e "section_start:`date +%s`:mkrescue-$target\r\e[0KInstalling $target";
-        builddir=$CI_PROJECT_DIR/build/obj-$target;
-        rescuedir=$CI_PROJECT_DIR/rescue;
-        mkdir -pv $rescuedir;
-        (
-          echo -e "insmod normal; normal" > grub.cfg;
-          $builddir/grub-mkrescue -v -o $rescuedir/grub-rescue-$target.iso boot/grub/grub.cfg=grub.cfg;
-        );
-        echo -e "section_end:`date +%s`:mkrescue-$target\r\e[0K";
-      done
-  parallel:
-    <<: *target_matrix
-
-.build:docs:
-  stage: build
-  allow_failure: true
-  extends:
-    - .install-packages
-  variables:
-    # Use tex packages to build documentation
-    PACKAGES: |
-      texinfo
-      texlive
-  script:
-    - make html dvi pdf info ps man
-
-.test-all:
-#  image: ubuntu:focal
-  stage: test
-  when: manual
-  extends:
-    - .needs-cross-path
-    - .install-packages
-  artifacts:
-    name: test-artifacts
-    when: on_failure
-    paths:
-      - $CI_PROJECT_DIR/build/obj-*/*.{log,trs}
-  needs:
-    - setup:bootstrap
-    - build:compile
-  variables:
-    PACKAGES: |
-      ovmf
-      qemu-system
-      python
-      make
-      automake
-      pkg-config
-      bison
-      flex
-      wamerican
-      tar
-      cpio
-      gzip
-      lzop
-      xz-utils
-      parted
-      xorriso
-      util-linux
-      squashfs-tools
-      zfsutils-linux
-      dosfstools
-      exfat-utils
-      ntfs-3g
-      e2fsprogs
-      btrfs-progs
-      xfsprogs
-      hfsprogs
-      jfsutils
-      reiserfsprogs
-      udftools
-      nilfs-tools
-      f2fs-tools
-      genromfs
-      attr
-#      libsdl1.2-dev
-#      unifont
-#      libfreetype6-dev
-#      libdevmapper-dev
-#      liblzma-dev
-#      libfuse-dev
-#      libzfslinux-dev
-#      autoconf
-#      autopoint
-#      git
-#      gettext
-  script:
-    - for target in $GRUB_TARGETS; do
-        set -x;
-        while [ -z ${PATH%%*\$*} ]; do
-          eval "PATH=$PATH";
-        done;
-        plat=${target#*-};
-        plat=${plat%-*};
-        arch=${target%%-*};
-        echo -e "section_start:`date +%s`:test-$target[collapsed=true]\r\e[0KTesting $target";
-        builddir=$CI_PROJECT_DIR/build/obj-$target;
-        installdir=$CI_PROJECT_DIR/install/grub-$target;
-        mkdir -pv $builddir $installdir;
-        ls $builddir;
-        JOBS=`getconf _NPROCESSORS_ONLN 2> /dev/null || echo 1`;
-        [ "$JOBS" == 1 ] || JOBS=$(($JOBS + 1));
-        make -C $builddir -j$JOBS TMPDIR=/tmp SUBDIRS= V=3 check || (
-          ls $builddir;
-          false;
-        );
-        echo -e "section_end:`date +%s`:test-$target\r\e[0K";
-      done
-  parallel:
-    <<: *target_matrix
-
 # Build and test everything in one (matrix) job
 build:allinone:
   stage: build
@@ -922,7 +436,6 @@ build:allinone:
       - $PACKAGE_CACHE
   artifacts:
     name: "${CI_JOB_NAME}-${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}-${GRUB_TARGETS}"
-    # when: on_failure
     when: always
     paths: &target-archive-paths
       - $CI_PROJECT_DIR/build/obj-$GRUB_TARGETS/config.h
@@ -931,8 +444,6 @@ build:allinone:
       - $CI_PROJECT_DIR/build/obj-$GRUB_TARGETS/test-data.tar.*
       - $CI_PROJECT_DIR/build/obj-$GRUB_TARGETS/test.success
       - $CI_PROJECT_DIR/build/obj-$GRUB_TARGETS/images
-      # - $CI_PROJECT_DIR/build/*.log
-      # - $CROSS_DIR
     exclude:
       - $CI_PROJECT_DIR/build/**/*.o
     expire_in: 5 days
@@ -942,10 +453,7 @@ build:allinone:
     <<: *install-extra-qemu-packages-vars
     BUILDDIR: $CI_PROJECT_DIR/build/obj-$GRUB_TARGETS
     INSTALLDIR: $CI_PROJECT_DIR/install/grub-$GRUB_TARGETS
-    # CONFIGURE_OPTS: "--disable-silent-rules"
-    # TMPDIR: "/tmp"
     TEST_DATA_PREFIX: $CI_PROJECT_DIR/build/obj-${GRUB_TARGETS}/test-data
-    # GRUB_QEMU_OPTS: -m size=64M
     PACKAGES: |
       wget
       ### Grub build requirements
@@ -1017,14 +525,11 @@ build:allinone:
     - *install-extra-qemu-packages-before_script
     - *cross_compilers-setup_script
     - *build-ccache-before_script
-    - *print_debug_info-before_script
     # Need to keep evaling PATH until there are no more variables in it
     # because gitlab CI does not do recursive variable expansion.
     - while [ -z ${PATH%%*\$*} ]; do
         eval "export PATH=$PATH";
       done;
-    - pwd
-    - ls -la
   script:
     # Build all selected GRUB targets.
     - test "x${SHELL_TRACE}" = "xy" && set -x;
@@ -1067,34 +572,12 @@ build:allinone:
         $SRCDIR/.ci/test.sh 2>&1 | tee ${BUILDDIR}/test.log || :;
       ) 2>&1 | tee "${BUILDDIR}/${CI_JOB_NAME}.log";
     # Do processing of testing output
-    #- *process-tests-script
     - TARGET=${GRUB_TARGETS} $SRCDIR/.ci/process-tests.sh
     # Make images for all formats
     - TARGET=${GRUB_TARGETS} $SRCDIR/.ci/make-images.sh
   after_script:
     - *build-ccache-after_script
 
-.test:allinone:
-  stage: test
-  # Run always, even if build fails so we can see pass/fail of the
-  # individual matrice runs.
-  when: always
-  artifacts:
-    name: "${CI_JOB_NAME}-${CI_COMMIT_REF_SLUG}-${CI_COMMIT_SHORT_SHA}-${GRUB_TARGETS}"
-#    when: on_failure
-    when: always
-    paths:
-      - $CI_PROJECT_DIR/build/obj-$GRUB_TARGETS/*.{log,trs}
-      - $CI_PROJECT_DIR/build/*.log
-#    expire_in: 1 hour
-  dependencies:
-#  needs:
-    - build:allinone
-  script: &test_allinone-script
-    - *process-tests-script
-  parallel:
-    <<: *target_matrix
-
 package:allinone:
   stage: .post
   when: always
@@ -1103,11 +586,6 @@ package:allinone:
     when: always
     expire_in: 1 month
     paths:
-      # - *target-archive-paths
-      # - $CI_PROJECT_DIR/build/obj-*/*.{log,trs}
-      # - $CI_PROJECT_DIR/build/obj-*/*.strace*
-      # - $CI_PROJECT_DIR/build/obj-*/test-data.tar.*
-      # - $CI_PROJECT_DIR/build/obj-*/test.success
       - $CI_PROJECT_DIR/build/obj-*
       - $CI_PROJECT_DIR/build/*.log
     exclude:
@@ -1115,11 +593,9 @@ package:allinone:
       - $CI_PROJECT_DIR/build/**/.deps-util
   needs:
     - build:allinone
-    # - test:allinone
   before_script:
     - . ${SRCDIR}/.ci/functions.gitlab.sh
     - SUMLOG=$CI_PROJECT_DIR/build/test-summary.log
-    # - TXT_RED="\e[31m"; TXT_YELLOW="\e[33m"; TXT_CLEAR="\e[0m";
     - . $CI_PROJECT_DIR/.ci/functions.gitlab.sh
     - find $CI_PROJECT_DIR/build > $CI_PROJECT_DIR/build/build-paths.log
   script:
@@ -1154,31 +630,3 @@ package:allinone:
     - if [ -f "${SUMLOG}" ]; then
         cat "${SUMLOG}";
       fi
-
-# mergereq:package:allinone:
-#   extends:
-#     - package:allinone
-#   rules:
-#     - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
-#     - if: $CI_MERGE_REQUEST_IID
-#     - when: always
-
-#deploy-prod:
-#  stage: deploy
-#  script:
-#    - echo "This job deploys something from the $CI_COMMIT_BRANCH branch."
-#    - echo "Hello, $GITLAB_USER_LOGIN!"
-
-# TODO: put generated docs in gitlab pages
-# pages:
-#   stage: deploy
-#   needs: build:docs
-#   script:
-#     - mkdir .public
-#     - cp -r * .public
-#     - mv .public public
-#   artifacts:
-#     paths:
-#       - public
-#   only:
-#     - master
-- 
2.27.0