Re: [PATCH ] check: add CLI option to repeat and stop tests in case of failure

[Eryu Guan <guan@eryu.me>]

On Thu, Jan 28, 2021 at 02:19:41PM +0530, Ritesh Harjani wrote:
> Currently with -i <n> option the test can run for many iterations,
> but in case if we want to stop the iteration in case of a failure,
> it is much easier to have such an option which could check the failed
> status and stop the test from further proceeding.
> 
> This patch adds such an option thereby extending the -i <n> option
> functionality.
> 
> Signed-off-by: Ritesh Harjani <riteshh@linux.ibm.com>
> ---
>  check | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/check b/check
> index c6ad1d6c0733..6f3a5d47e212 100755
> --- a/check
> +++ b/check
> @@ -15,6 +15,7 @@ sum_bad=0
>  bad=""
>  n_notrun=0
>  notrun=""
> +tc_status=""
>  interrupt=true
>  diff="diff -u"
>  showme=false
> @@ -27,6 +28,7 @@ brief_test_summary=false
>  do_report=false
>  DUMP_OUTPUT=false
>  iterations=1
> +istop=false
> 
>  # This is a global variable used to pass test failure text to reporting gunk
>  _err_msg=""
> @@ -68,6 +70,7 @@ check options
>      -T			output timestamps
>      -r			randomize test order
>      -i <n>		iterate the test list <n> times
> +    -istop <n>		iterate the test list <n> times, but stops iterating further in case of any test failure

Perhaps '-I' is a better option?

>      -d			dump test output to stdout
>      -b			brief test summary
>      -R fmt[,fmt]	generate report in formats specified. Supported format: [xunit]
> @@ -300,6 +303,7 @@ while [ $# -gt 0 ]; do
>  	-n)	showme=true ;;
>          -r)	randomize=true ;;
>  	-i)	iterations=$2; shift ;;
> +	-istop) iterations=$2; istop=true; shift ;;

Same here.

>  	-T)	timestamp=true ;;
>  	-d)	DUMP_OUTPUT=true ;;
>  	-b)	brief_test_summary=true;;
> @@ -926,6 +930,11 @@ function run_section()
>  for ((iters = 0; iters < $iterations; iters++)) do
>  	for section in $HOST_OPTIONS_SECTIONS; do
>  		run_section $section
> +		if [ "$tc_status" = "fail" ] && [ "$istop" = true ]; then

$tc_status only records the status of last test in the test list, and if
the last test passed, the iteration continues. e.g. I tested
generic/00[12], and generic/001 fails while generic/002 passes

  ./check -i 2 generic/001 generic/002

check still run 2 iterations as "tc_status" of generic/002 is not
"fail".

I think we could check $sum_bad is 0 or not.

Thanks,
Eryu

> +			interrupt=false
> +			status=`expr $sum_bad != 0`
> +			exit
> +		fi
>  	done
>  done
> 
> --
> 2.26.2