[PATCH ] check: add CLI option to repeat and stop tests in case of failure

[PATCH ] check: add CLI option to repeat and stop tests in case of failure

[Ritesh Harjani]

Currently with -i <n> option the test can run for many iterations,
but in case if we want to stop the iteration in case of a failure,
it is much easier to have such an option which could check the failed
status and stop the test from further proceeding.

This patch adds such an option thereby extending the -i <n> option
functionality.

Signed-off-by: Ritesh Harjani <riteshh@linux.ibm.com>
---
 check | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/check b/check
index c6ad1d6c0733..6f3a5d47e212 100755
--- a/check
+++ b/check
@@ -15,6 +15,7 @@ sum_bad=0
 bad=""
 n_notrun=0
 notrun=""
+tc_status=""
 interrupt=true
 diff="diff -u"
 showme=false
@@ -27,6 +28,7 @@ brief_test_summary=false
 do_report=false
 DUMP_OUTPUT=false
 iterations=1
+istop=false

 # This is a global variable used to pass test failure text to reporting gunk
 _err_msg=""
@@ -68,6 +70,7 @@ check options
     -T			output timestamps
     -r			randomize test order
     -i <n>		iterate the test list <n> times
+    -istop <n>		iterate the test list <n> times, but stops iterating further in case of any test failure
     -d			dump test output to stdout
     -b			brief test summary
     -R fmt[,fmt]	generate report in formats specified. Supported format: [xunit]
@@ -300,6 +303,7 @@ while [ $# -gt 0 ]; do
 	-n)	showme=true ;;
         -r)	randomize=true ;;
 	-i)	iterations=$2; shift ;;
+	-istop) iterations=$2; istop=true; shift ;;
 	-T)	timestamp=true ;;
 	-d)	DUMP_OUTPUT=true ;;
 	-b)	brief_test_summary=true;;
@@ -926,6 +930,11 @@ function run_section()
 for ((iters = 0; iters < $iterations; iters++)) do
 	for section in $HOST_OPTIONS_SECTIONS; do
 		run_section $section
+		if [ "$tc_status" = "fail" ] && [ "$istop" = true ]; then
+			interrupt=false
+			status=`expr $sum_bad != 0`
+			exit
+		fi
 	done
 done

--
2.26.2

Re: [PATCH ] check: add CLI option to repeat and stop tests in case of failure

[Eryu Guan]

On Thu, Jan 28, 2021 at 02:19:41PM +0530, Ritesh Harjani wrote:
> Currently with -i <n> option the test can run for many iterations,
> but in case if we want to stop the iteration in case of a failure,
> it is much easier to have such an option which could check the failed
> status and stop the test from further proceeding.
> 
> This patch adds such an option thereby extending the -i <n> option
> functionality.
> 
> Signed-off-by: Ritesh Harjani <riteshh@linux.ibm.com>
> ---
>  check | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/check b/check
> index c6ad1d6c0733..6f3a5d47e212 100755
> --- a/check
> +++ b/check
> @@ -15,6 +15,7 @@ sum_bad=0
>  bad=""
>  n_notrun=0
>  notrun=""
> +tc_status=""
>  interrupt=true
>  diff="diff -u"
>  showme=false
> @@ -27,6 +28,7 @@ brief_test_summary=false
>  do_report=false
>  DUMP_OUTPUT=false
>  iterations=1
> +istop=false
> 
>  # This is a global variable used to pass test failure text to reporting gunk
>  _err_msg=""
> @@ -68,6 +70,7 @@ check options
>      -T			output timestamps
>      -r			randomize test order
>      -i <n>		iterate the test list <n> times
> +    -istop <n>		iterate the test list <n> times, but stops iterating further in case of any test failure

Perhaps '-I' is a better option?

>      -d			dump test output to stdout
>      -b			brief test summary
>      -R fmt[,fmt]	generate report in formats specified. Supported format: [xunit]
> @@ -300,6 +303,7 @@ while [ $# -gt 0 ]; do
>  	-n)	showme=true ;;
>          -r)	randomize=true ;;
>  	-i)	iterations=$2; shift ;;
> +	-istop) iterations=$2; istop=true; shift ;;

Same here.

>  	-T)	timestamp=true ;;
>  	-d)	DUMP_OUTPUT=true ;;
>  	-b)	brief_test_summary=true;;
> @@ -926,6 +930,11 @@ function run_section()
>  for ((iters = 0; iters < $iterations; iters++)) do
>  	for section in $HOST_OPTIONS_SECTIONS; do
>  		run_section $section
> +		if [ "$tc_status" = "fail" ] && [ "$istop" = true ]; then

$tc_status only records the status of last test in the test list, and if
the last test passed, the iteration continues. e.g. I tested
generic/00[12], and generic/001 fails while generic/002 passes

  ./check -i 2 generic/001 generic/002

check still run 2 iterations as "tc_status" of generic/002 is not
"fail".

I think we could check $sum_bad is 0 or not.

Thanks,
Eryu

> +			interrupt=false
> +			status=`expr $sum_bad != 0`
> +			exit
> +		fi
>  	done
>  done
> 
> --
> 2.26.2

Re: [PATCH ] check: add CLI option to repeat and stop tests in case of failure

[Ritesh Harjani]

On 1/31/21 8:33 PM, Eryu Guan wrote:
> On Thu, Jan 28, 2021 at 02:19:41PM +0530, Ritesh Harjani wrote:
>> Currently with -i <n> option the test can run for many iterations,
>> but in case if we want to stop the iteration in case of a failure,
>> it is much easier to have such an option which could check the failed
>> status and stop the test from further proceeding.
>>
>> This patch adds such an option thereby extending the -i <n> option
>> functionality.
>>
>> Signed-off-by: Ritesh Harjani <riteshh@linux.ibm.com>
>> ---
>>   check | 9 +++++++++
>>   1 file changed, 9 insertions(+)
>>
>> diff --git a/check b/check
>> index c6ad1d6c0733..6f3a5d47e212 100755
>> --- a/check
>> +++ b/check
>> @@ -15,6 +15,7 @@ sum_bad=0
>>   bad=""
>>   n_notrun=0
>>   notrun=""
>> +tc_status=""
>>   interrupt=true
>>   diff="diff -u"
>>   showme=false
>> @@ -27,6 +28,7 @@ brief_test_summary=false
>>   do_report=false
>>   DUMP_OUTPUT=false
>>   iterations=1
>> +istop=false
>>
>>   # This is a global variable used to pass test failure text to reporting gunk
>>   _err_msg=""
>> @@ -68,6 +70,7 @@ check options
>>       -T			output timestamps
>>       -r			randomize test order
>>       -i <n>		iterate the test list <n> times
>> +    -istop <n>		iterate the test list <n> times, but stops iterating further in case of any test failure
> 
> Perhaps '-I' is a better option?

Sure, will change it.

> 
>>       -d			dump test output to stdout
>>       -b			brief test summary
>>       -R fmt[,fmt]	generate report in formats specified. Supported format: [xunit]
>> @@ -300,6 +303,7 @@ while [ $# -gt 0 ]; do
>>   	-n)	showme=true ;;
>>           -r)	randomize=true ;;
>>   	-i)	iterations=$2; shift ;;
>> +	-istop) iterations=$2; istop=true; shift ;;
> 
> Same here.

yes.

> 
>>   	-T)	timestamp=true ;;
>>   	-d)	DUMP_OUTPUT=true ;;
>>   	-b)	brief_test_summary=true;;
>> @@ -926,6 +930,11 @@ function run_section()
>>   for ((iters = 0; iters < $iterations; iters++)) do
>>   	for section in $HOST_OPTIONS_SECTIONS; do
>>   		run_section $section
>> +		if [ "$tc_status" = "fail" ] && [ "$istop" = true ]; then
> 
> $tc_status only records the status of last test in the test list, and if
> the last test passed, the iteration continues. e.g. I tested
> generic/00[12], and generic/001 fails while generic/002 passes
> 
>    ./check -i 2 generic/001 generic/002
> 
> check still run 2 iterations as "tc_status" of generic/002 is not
> "fail".
> 
> I think we could check $sum_bad is 0 or not.

Yes, right. Thanks for pointing it out.

I will address these comments and spin a v2.

-ritesh