[kvm-unit-tests RFC PATCH 0/1] Detecting crashes in TAP output

[kvm-unit-tests RFC PATCH 0/1] Detecting crashes in TAP output

[Nico Boehr]

Hi,

I recently had a test on s390x which caused an exception in the guest. As
expected, run_tests.sh reported it as failing:

    FAIL sometest (1 tests)

However, when I turn on the TAP output everything looks like it is fine:

    TAP version 13
    ok 1 - sometest: sometest: some report
    1..1

There is no way to see from the TAP output the test actually had an unexpected
exception. Our internal scripts rely on the TAP output and will thus believe
everything is fine, even though it really isn't.

In the logfile, one can at least see the exception backtrace, but if something
exits silently, there is no indication something went wrong there either.

TAP provides the test plan (the "1..1") as a solution to this problem. It
gives the expected number of test lines if all tests would run. The harness can
count test lines in TAP output and compare this to the number of tests in the
plan and report an error if it doesn't match.

This won't work with kvm-unit-tests, since there really isn't any way to know
how many report()s and thus test lines you will have without actually running
the tests.

That's why I came up with an alternative approach in the patch below. It
adds an additional test line to the TAP output and the logfiles which
states the overall result of a test.

With the attached patch, the TAP output of the test above will now look like
this:

    TAP version 13
    ok 1 - sometest: sometest: some report
    not ok 2 - sometest: (1 tests)
    1..2

This shows something went wrong.

Your feedback or alternative solutions to this problem are welcome.

Nico Boehr (1):
  scripts/runtime: add test result to log and TAP output

 scripts/runtime.bash | 2 ++
 1 file changed, 2 insertions(+)

-- 
2.31.1

[kvm-unit-tests RFC PATCH 1/1] scripts/runtime: add test result to log and TAP output

[Nico Boehr]

Currently, when a test exits prematurely for whatever reason (crash, guest
exception, ...) the test is correctly reported as FAIL by run_tests.sh - even
when all report()s up to that point passed.

The failure is reported by print_result which checks (among others) the exit
code of the arch-specific run script.

But, as soon as one enables the TAP output in run_tests.sh, there is no way to
see that any more, as the print_result is discarded and only the run script's
output is converted to TAP.

External test runners relying on TAP output will thus believe everything is
fine even though we got a crash.

The same also applies to the logfiles.

As a simple fix, have print_result also print to RUNTIME_log_stderr.  For each
test, we will then get an additional test line in the TAP which reports the
test's result:

  not ok 36 - css: (35 tests, 2 unexpected failures)

The log files will also contain the result:

  FAIL: (35 tests, 2 unexpected failures)

This makes it easy to see whether we had a premature exit in the test. The
disadvantage being the number of test lines in the TAP will no longer match the
number of report()s in a test.

Signed-off-by: Nico Boehr <nrb@linux.ibm.com>
---
 scripts/runtime.bash | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/runtime.bash b/scripts/runtime.bash
index 6d5fced94246..7bb70d50012a 100644
--- a/scripts/runtime.bash
+++ b/scripts/runtime.bash
@@ -60,8 +60,10 @@ function print_result()
 
     if [ -z "$reason" ]; then
         echo "`$status` $testname $summary"
+        RUNTIME_log_stderr "$testname" <<< "$status: $summary"
     else
         echo "`$status` $testname ($reason)"
+        RUNTIME_log_stderr "$testname" <<< "$status: $reason"
     fi
 }
 
-- 
2.31.1