[kvm-unit-tests PATCH v3 0/1] runtime: indicate failure on crash/timeout/abort in TAP

[kvm-unit-tests PATCH v3 0/1] runtime: indicate failure on crash/timeout/abort in TAP

[Nico Boehr]

Sorry for the noise, I sent my v2 a little too early and forgot to include 
some more feedback.

Changelog v3->v2:
---
* I forgot to include Marc's suggestion to make the else if ... an elif,
  now included.
* Thomas actually suggested to quote $tap_output, which I missed, now quoted.

Changelog v2->v1:
---
* Change [[ to [
* Remove double negation

Nico Boehr (1):
  runtime: indicate failure on crash/timeout/abort in TAP

 scripts/runtime.bash | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

-- 
2.31.1

[kvm-unit-tests PATCH v3 1/1] runtime: indicate failure on crash/timeout/abort in TAP

[Nico Boehr]

When we have crashes, timeouts or aborts, there is currently no indication for
this in the TAP output. When all reports() up to this point succeeded, this
might result in a TAP file looking completely fine even though things went
terribly wrong.

For example, when I set the timeout for the diag288 test on s390x to 1 second,
it fails because it takes quite long, which is properly indicated in the
normal output:

$ ./run_tests.sh diag288
FAIL diag288 (timeout; duration=1s)

But, when I enable TAP output, I get this:

$ ./run_tests.sh -t diag288
TAP version 13
ok 1 - diag288: diag288: privileged: Program interrupt: expected(2) == received(2)
ok 2 - diag288: diag288: specification: uneven: Program interrupt: expected(6) == received(6)
ok 3 - diag288: diag288: specification: unsupported action: Program interrupt: expected(6) == received(6)
ok 4 - diag288: diag288: specification: unsupported function: Program interrupt: expected(6) == received(6)
ok 5 - diag288: diag288: specification: no init: Program interrupt: expected(6) == received(6)
ok 6 - diag288: diag288: specification: min timer: Program interrupt: expected(6) == received(6)
1..6

Which looks like a completely fine TAP file, but actually we ran into a timeout
and didn't even run all tests.

With this patch, we get an additional line at the end which properly shows
something went wrong:

not ok 7 - diag288: timeout; duration=1s

Signed-off-by: Nico Boehr <nrb@linux.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
---
 scripts/runtime.bash | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/scripts/runtime.bash b/scripts/runtime.bash
index 6d5fced94246..86405604522d 100644
--- a/scripts/runtime.bash
+++ b/scripts/runtime.bash
@@ -163,8 +163,17 @@ function run()
         print_result "SKIP" $testname "$summary"
     elif [ $ret -eq 124 ]; then
         print_result "FAIL" $testname "" "timeout; duration=$timeout"
+        if [ "$tap_output" = "yes" ]; then
+            echo "not ok TEST_NUMBER - ${testname}: timeout; duration=$timeout" >&3
+        fi
     elif [ $ret -gt 127 ]; then
-        print_result "FAIL" $testname "" "terminated on SIG$(kill -l $(($ret - 128)))"
+        signame="SIG"$(kill -l $(($ret - 128)))
+        print_result "FAIL" $testname "" "terminated on $signame"
+        if [ "$tap_output" = "yes" ]; then
+            echo "not ok TEST_NUMBER - ${testname}: terminated on $signame" >&3
+        fi
+    elif [ $ret -eq 127 ] && [ "$tap_output" = "yes" ]; then
+        echo "not ok TEST_NUMBER - ${testname}: aborted" >&3
     else
         print_result "FAIL" $testname "$summary"
     fi
-- 
2.31.1

Re: [kvm-unit-tests PATCH v3 1/1] runtime: indicate failure on crash/timeout/abort in TAP

[Thomas Huth]

On 22/03/2022 14.44, Nico Boehr wrote:
> When we have crashes, timeouts or aborts, there is currently no indication for
> this in the TAP output. When all reports() up to this point succeeded, this
> might result in a TAP file looking completely fine even though things went
> terribly wrong.
> 
> For example, when I set the timeout for the diag288 test on s390x to 1 second,
> it fails because it takes quite long, which is properly indicated in the
> normal output:
> 
> $ ./run_tests.sh diag288
> FAIL diag288 (timeout; duration=1s)
> 
> But, when I enable TAP output, I get this:
> 
> $ ./run_tests.sh -t diag288
> TAP version 13
> ok 1 - diag288: diag288: privileged: Program interrupt: expected(2) == received(2)
> ok 2 - diag288: diag288: specification: uneven: Program interrupt: expected(6) == received(6)
> ok 3 - diag288: diag288: specification: unsupported action: Program interrupt: expected(6) == received(6)
> ok 4 - diag288: diag288: specification: unsupported function: Program interrupt: expected(6) == received(6)
> ok 5 - diag288: diag288: specification: no init: Program interrupt: expected(6) == received(6)
> ok 6 - diag288: diag288: specification: min timer: Program interrupt: expected(6) == received(6)
> 1..6
> 
> Which looks like a completely fine TAP file, but actually we ran into a timeout
> and didn't even run all tests.
> 
> With this patch, we get an additional line at the end which properly shows
> something went wrong:
> 
> not ok 7 - diag288: timeout; duration=1s
> 
> Signed-off-by: Nico Boehr <nrb@linux.ibm.com>
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> ---
>   scripts/runtime.bash | 11 ++++++++++-
>   1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/runtime.bash b/scripts/runtime.bash
> index 6d5fced94246..86405604522d 100644
> --- a/scripts/runtime.bash
> +++ b/scripts/runtime.bash
> @@ -163,8 +163,17 @@ function run()
>           print_result "SKIP" $testname "$summary"
>       elif [ $ret -eq 124 ]; then
>           print_result "FAIL" $testname "" "timeout; duration=$timeout"
> +        if [ "$tap_output" = "yes" ]; then
> +            echo "not ok TEST_NUMBER - ${testname}: timeout; duration=$timeout" >&3
> +        fi
>       elif [ $ret -gt 127 ]; then
> -        print_result "FAIL" $testname "" "terminated on SIG$(kill -l $(($ret - 128)))"
> +        signame="SIG"$(kill -l $(($ret - 128)))
> +        print_result "FAIL" $testname "" "terminated on $signame"
> +        if [ "$tap_output" = "yes" ]; then
> +            echo "not ok TEST_NUMBER - ${testname}: terminated on $signame" >&3
> +        fi
> +    elif [ $ret -eq 127 ] && [ "$tap_output" = "yes" ]; then
> +        echo "not ok TEST_NUMBER - ${testname}: aborted" >&3
>       else
>           print_result "FAIL" $testname "$summary"
>       fi

Thanks, I've pushed it now to the repository!

  Thomas