[PATCH 0/3] Allow easy re-running of grub-shell from failed tests

[PATCH 0/3] Allow easy re-running of grub-shell from failed tests

[Glenn Washburn]

When a non-native (eg. grub-shell/qemu) test fails it can be useful to be
able to re-run the grub-shell invocation in at least two circumstances.

1. The tester wants to re-run the grub-shell invocation with extra args so
   that QEMU starts in GDB attach mode for using GDB to debug the test case
2. For a tester who wants to send a failed test case to another developer for
   additional debugging. It would be nice to be able to just zip/tar just
   the failed test case test output directory and allow the extracted
   directory to easily run the failed test case again. The results in this
   case still might not be the same if the sender and reciever do not have
   the same QEMU. Though when the QMEU versions are different, I expect the
   emulated execution on sender and reciever to be close enough to produce
   the same execution path (maybe not with super old QEMUs, but I don't think
   its something to be concerned about).

The first patch is needed to make the other two useful. Without it, I know of
no easy way to enable grub-shell's debug mode when run via `make check`. And
if grub-shell is not in debug mode then it will remove most/all of the
generated files, which would prevent debugging.

Glenn

Glenn Washburn (3):
  tests: Allow turning on shell tracing from environment variables
  grub-shell: Create run.sh in working directory for easily running test
    again
  grub-shell: Add $GRUB_QEMU_OPTS to run.sh to easily see unofficial
    QEMU arguments

 tests/util/grub-fs-tester.in |  2 ++
 tests/util/grub-shell.in     | 37 +++++++++++++++++++++++++++++-------
 2 files changed, 32 insertions(+), 7 deletions(-)

-- 
2.34.1

[PATCH 1/3] tests: Allow turning on shell tracing from environment variables

[Glenn Washburn]

This allows turning on shell tracing for grub-shell and grub-fs-tester when
its not practical or not possible to use commandline arguments (eg. from
`make check`). Turn on tracing when the envvar is an integer greater than 1,
since these can generate a lot of output. Since this change uses the
environment variables to set the default value for debug in grub-shell, this
allows enabling grub-shell's debug mode which will preserve various
generated output files that are helpful for debugging tests.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-fs-tester.in | 2 ++
 tests/util/grub-shell.in     | 5 ++++-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/tests/util/grub-fs-tester.in b/tests/util/grub-fs-tester.in
index cd63941a5f..c9c2d79f22 100644
--- a/tests/util/grub-fs-tester.in
+++ b/tests/util/grub-fs-tester.in
@@ -6,6 +6,8 @@ export BLKID_FILE=/dev/null
 # We can't have this set, or filesystem tests will fail.
 unset SOURCE_DATE_EPOCH
 
+[ "${GRUB_TEST_DEFAULT_DEBUG:-0}" -gt 1 ] && set -x
+
 fs="$1"
 
 GRUBFSTEST="@builddir@/grub-fstest"
diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index f5a6e822fd..60494bcf1d 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -216,6 +216,7 @@ esac
 
 timeout=60
 mkimage_extra_arg=
+debug=${GRUB_SHELL_DEFAULT_DEBUG:-$GRUB_TEST_DEFAULT_DEBUG}
 
 # Check the arguments.
 for option in "$@"; do
@@ -235,7 +236,7 @@ for option in "$@"; do
     --no-trim)
 	trim=0 ;;
     --debug)
-        debug=1 ;;
+        debug=$((debug+1)) ;;
     --modules=*)
 	ms=`echo "$option" | sed -e 's/--modules=//' -e 's/,/ /g'`
 	modules="$modules $ms" ;;
@@ -320,6 +321,8 @@ for option in "$@"; do
     esac
 done
 
+[ "${debug:-0}" -gt 1 ] && set -x
+
 if [ "x${source}" = x ] ; then
     tmpfile="$work_directory/testcase.cfg"
     while read REPLY; do
-- 
2.34.1

[PATCH 2/3] grub-shell: Create run.sh in working directory for easily running test again

[Glenn Washburn]

Now it becomes trivial to re-run a test from the output in its working
directory. This also makes it easy to send a reproducible failing test to
the mailing list. This has allowed a refactor so that the duplicated code
to call QEMU has be condensed (eg. the use of timeout and file descriptor
redirection). The run.sh script will pass any arguments given to QEMU.
This allows QEMU to be easily started in a state ready for GDB to be
attached.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-shell.in | 28 +++++++++++++++++++++++-----
 1 file changed, 23 insertions(+), 5 deletions(-)

diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index 60494bcf1d..658485f7ea 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -521,8 +521,11 @@ if [ x$boot = xnet ]; then
     cp "${cfgfile}" "$netdir/boot/grub/grub.cfg"
     cp "${source}" "$netdir/boot/grub/testcase.cfg"
     [ -z "$files" ] || copy_extra_files "$netdir" $files
-    setup_qemu_logger
-    timeout -s KILL $timeout "${qemu}" ${qemuopts} ${serial_null} -serial file:/dev/stdout -boot n -net "user,tftp=$netdir,bootfile=/boot/grub/${grub_modinfo_target_cpu}-${grub_modinfo_platform}/core.$netbootext"  -net nic > "$work_directory/qemu-pipe" || ret=$?
+    cat >"$work_directory/run.sh" <<EOF
+#! @BUILD_SHEBANG@
+qemuopts="${qemuopts}"
+exec "${qemu}" \${qemuopts} ${serial_null} -serial file:/dev/stdout -boot n -net "user,tftp=$netdir,bootfile=/boot/grub/${grub_modinfo_target_cpu}-${grub_modinfo_platform}/core.$netbootext"  -net nic "\$@"
+EOF
 elif [ x$boot = xemu ]; then
     rootdir="$work_directory/rootdir"
     grubdir="$rootdir/boot/grub"
@@ -541,11 +544,25 @@ elif [ x$boot = xemu ]; then
     [ -z "$files" ] || copy_extra_files "$rootdir" $files
     roottar="$work_directory/root.tar"
     (cd "$rootdir"; tar cf "$roottar" .)
-    setup_qemu_logger
-    "${builddir}/grub-core/grub-emu" -m "$device_map" --memdisk "$roottar" -r memdisk -d "/boot/grub" > "$work_directory/qemu-pipe" || ret=$?
+    cat >"$work_directory/run.sh" <<EOF
+#! @BUILD_SHEBANG@
+SDIR=\$(realpath -e \${0%/*})
+exec "$(realpath -e "${builddir}")/grub-core/grub-emu" -m "\$SDIR/${device_map##*/}" --memdisk "\$SDIR/${roottar##*/}" -r memdisk -d "/boot/grub"
+EOF
 else
+    cat >"$work_directory/run.sh" <<EOF
+#! @BUILD_SHEBANG@
+SDIR=\$(realpath -e \${0%/*})
+qemuopts="${qemuopts}"
+cd "\$SDIR"
+exec "${qemu}" \${qemuopts} ${serial_null} -serial file:/dev/stdout -${device}"\${SDIR}/${isofile##*/}" ${bootdev} "\$@"
+EOF
+fi
+
+if [ -f "$work_directory/run.sh" ]; then
     setup_qemu_logger
-    timeout -s KILL $timeout "${qemu}" ${qemuopts} ${serial_null} -serial file:/dev/stdout -${device}"${isofile}" ${bootdev} > "$work_directory/qemu-pipe" || ret=$?
+    chmod +x "$work_directory/run.sh"
+    timeout -s KILL $timeout "$work_directory/run.sh" > "$work_directory/qemu-pipe" || ret=$?
 fi
 
 wait
@@ -565,6 +582,7 @@ fi
 test -n "$debug" || rm -f "${isofile}"
 test -n "$debug" || rm -rf "${rom_directory}"
 test -n "$debug" || rm -f "${tmpfile}" "${cfgfile}" "${goutfile}"
+test -n "$debug" || rm -f "$work_directory/run.sh"
 
 exit $ret
 
-- 
2.34.1

[PATCH 3/3] grub-shell: Add $GRUB_QEMU_OPTS to run.sh to easily see unofficial QEMU arguments

[Glenn Washburn]

When re-running a failed test, even the non-standard grub-shell QEMU
arguments should be preserved in the run.sh to more precisely replay the
failed test run.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
 tests/util/grub-shell.in | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/tests/util/grub-shell.in b/tests/util/grub-shell.in
index 658485f7ea..75f71dc1a2 100644
--- a/tests/util/grub-shell.in
+++ b/tests/util/grub-shell.in
@@ -74,7 +74,7 @@ exec_show_error () {
 work_directory=${WORKDIR:-`mktemp -d "${TMPDIR:-/tmp}/grub-shell.XXXXXXXXXX"`} || exit 1
 
 . "${builddir}/grub-core/modinfo.sh"
-qemuopts="${GRUB_QEMU_OPTS}"
+qemuopts=
 serial_port=com0
 serial_null=
 halt_cmd=halt
@@ -523,8 +523,9 @@ if [ x$boot = xnet ]; then
     [ -z "$files" ] || copy_extra_files "$netdir" $files
     cat >"$work_directory/run.sh" <<EOF
 #! @BUILD_SHEBANG@
+GRUB_QEMU_OPTS=\${GRUB_QEMU_OPTS:-"$GRUB_QEMU_OPTS"}
 qemuopts="${qemuopts}"
-exec "${qemu}" \${qemuopts} ${serial_null} -serial file:/dev/stdout -boot n -net "user,tftp=$netdir,bootfile=/boot/grub/${grub_modinfo_target_cpu}-${grub_modinfo_platform}/core.$netbootext"  -net nic "\$@"
+exec "${qemu}" \${qemuopts} \${GRUB_QEMU_OPTS} ${serial_null} -serial file:/dev/stdout -boot n -net "user,tftp=$netdir,bootfile=/boot/grub/${grub_modinfo_target_cpu}-${grub_modinfo_platform}/core.$netbootext"  -net nic "\$@"
 EOF
 elif [ x$boot = xemu ]; then
     rootdir="$work_directory/rootdir"
@@ -553,9 +554,10 @@ else
     cat >"$work_directory/run.sh" <<EOF
 #! @BUILD_SHEBANG@
 SDIR=\$(realpath -e \${0%/*})
+GRUB_QEMU_OPTS=\${GRUB_QEMU_OPTS:-"$GRUB_QEMU_OPTS"}
 qemuopts="${qemuopts}"
 cd "\$SDIR"
-exec "${qemu}" \${qemuopts} ${serial_null} -serial file:/dev/stdout -${device}"\${SDIR}/${isofile##*/}" ${bootdev} "\$@"
+exec "${qemu}" \${qemuopts} \${GRUB_QEMU_OPTS} ${serial_null} -serial file:/dev/stdout -${device}"\${SDIR}/${isofile##*/}" ${bootdev} "\$@"
 EOF
 fi
 
-- 
2.34.1

Re: [PATCH 0/3] Allow easy re-running of grub-shell from failed tests

[Daniel Kiper]

On Sat, Jan 21, 2023 at 12:10:43AM -0600, Glenn Washburn wrote:
> When a non-native (eg. grub-shell/qemu) test fails it can be useful to be
> able to re-run the grub-shell invocation in at least two circumstances.
>
> 1. The tester wants to re-run the grub-shell invocation with extra args so
>    that QEMU starts in GDB attach mode for using GDB to debug the test case
> 2. For a tester who wants to send a failed test case to another developer for
>    additional debugging. It would be nice to be able to just zip/tar just
>    the failed test case test output directory and allow the extracted
>    directory to easily run the failed test case again. The results in this
>    case still might not be the same if the sender and reciever do not have
>    the same QEMU. Though when the QMEU versions are different, I expect the
>    emulated execution on sender and reciever to be close enough to produce
>    the same execution path (maybe not with super old QEMUs, but I don't think
>    its something to be concerned about).
>
> The first patch is needed to make the other two useful. Without it, I know of
> no easy way to enable grub-shell's debug mode when run via `make check`. And
> if grub-shell is not in debug mode then it will remove most/all of the
> generated files, which would prevent debugging.
>
> Glenn
>
> Glenn Washburn (3):
>   tests: Allow turning on shell tracing from environment variables
>   grub-shell: Create run.sh in working directory for easily running test
>     again
>   grub-shell: Add $GRUB_QEMU_OPTS to run.sh to easily see unofficial
>     QEMU arguments

For all the patches Reviewed-by: Daniel Kiper <daniel.kiper@oracle.com>...

Daniel