[PATCH RFC] check: add new option "--loop <n>" which runs each test multiple times

[PATCH RFC] check: add new option "--loop <n>" which runs each test multiple times

[Theodore Ts'o]

Teach the check script a new option --loop, which re-run each test
multiple times.  This works very similarly to to -L, which will retry
a particular test after it first fails, except that the test is rerun
unconditionally.

This differs from the "-i <n>" option, which iterates each set of
tests <n> times instead of each test.  The -i option is problematic in
two ways.  First, doesn't save the test artifacts from each test run.
This is unfortunate because when the developer is trying to debug a
flaky test failure, running "check -i 100" will run a test 100 times,
but if only the 42nd test fails, the NNN.out.bad file for that failing
test run is not preserved.  The second difference between --loop and
-i is the result.xml file is rewritten after each test set, so we do
not have the cumulative statistics of the 100 test runs in the junit
XML file.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
---

Note: This commit adds a new command-line option instead of changing the
behavior of -i because it's possible that *someone* actually likes the
current behavior of the -i option, and changing how -i works might
break their test runner infrastructure.

Speaking personally, I find the current -i option completely useless
for the needs of xfstests-bld, and I would be happy to just change how
the -i option works.  This would also require changing support for -I,
but I was planning on adding an --loop-while-successful option
eventually, since it would be faster for bisection (although I
normally don't care about the junit XML file for NNN.out.bad files
when bisecting, so I find -I much less objectionable than -i).

 check | 28 ++++++++++++++++++++++------
 1 file changed, 22 insertions(+), 6 deletions(-)

diff --git a/check b/check
index cd7a79347..923d81a28 100755
--- a/check
+++ b/check
@@ -27,6 +27,8 @@ DUMP_OUTPUT=false
 iterations=1
 istop=false
 loop_on_fail=0
+loop_always=0
+loop_count=0
 exclude_tests=()
 
 # This is a global variable used to pass test failure text to reporting gunk
@@ -85,6 +87,7 @@ check options
     -s section		run only specified section from config file
     -S section		exclude the specified section from the config file
     -L <n>		loop tests <n> times following a failure, measuring aggregate pass/fail metrics
+    --loop=<n>		loop tests <n> times, measuring aggregate pass/fail metrics
 
 testlist options
     -g group[,group...]	include tests from these groups
@@ -339,6 +342,12 @@ while [ $# -gt 0 ]; do
 	--extra-space=*) export SCRATCH_DEV_EMPTY_SPACE=${r#*=} ;;
 	-L)	[[ $2 =~ ^[0-9]+$ ]] || usage
 		loop_on_fail=$2; shift
+		loop_count=$loop_on_fail
+		;;
+	--loop=*) loop_always=${1#*=}
+		[[ $loop_always =~ ^[0-9]+$ ]] || usage
+		loop_count=$(( loop_always - 1))
+		set +vx
 		;;
 
 	-*)	usage ;;
@@ -604,7 +613,7 @@ _expunge_test()
 }
 
 # retain files which would be overwritten in subsequent reruns of the same test
-_stash_fail_loop_files() {
+_stash_loop_files() {
 	local seq_prefix="${REPORT_DIR}/${1}"
 	local cp_suffix="$2"
 
@@ -629,9 +638,9 @@ _stash_test_status() {
 
 	if ((${#loop_status[*]} > 0)); then
 		# continuing or completing rerun-on-failure loop
-		_stash_fail_loop_files "$test_seq" ".rerun${#loop_status[*]}"
+		_stash_loop_files "$test_seq" ".rerun${#loop_status[*]}"
 		loop_status+=("$test_status")
-		if ((${#loop_status[*]} > loop_on_fail)); then
+		if ((${#loop_status[*]} > loop_count)); then
 			printf "%s aggregate results across %d runs: " \
 				"$test_seq" "${#loop_status[*]}"
 			awk "BEGIN {
@@ -651,9 +660,9 @@ _stash_test_status() {
 
 	case "$test_status" in
 	fail)
-		if ((loop_on_fail > 0)); then
+		if ((loop_on_fail > 0 || loop_always > 0 )); then
 			# initial failure, start rerun-on-failure loop
-			_stash_fail_loop_files "$test_seq" ".rerun0"
+			_stash_loop_files "$test_seq" ".rerun0"
 			loop_status+=("$test_status")
 		fi
 		bad+=("$test_seq")
@@ -661,7 +670,14 @@ _stash_test_status() {
 	list|notrun)
 		notrun+=("$test_seq")
 		;;
-	pass|expunge)
+	pass)
+		if (( loop_always > 0 )); then
+			# start rerun loop
+			_stash_loop_files "$test_seq" ".rerun0"
+			loop_status+=("$test_status")
+		fi
+	        ;;
+	expunge)
 		;;
 	*)
 		echo "Unexpected test $test_seq status: $test_status"
-- 
2.51.0

Re: [PATCH RFC] check: add new option "--loop <n>" which runs each test multiple times

[Theodore Tso]

On Wed, Apr 15, 2026 at 05:32:48PM -0400, Theodore Ts'o wrote:
> Teach the check script a new option --loop, which re-run each test
> multiple times.  This works very similarly to to -L, which will retry
> a particular test after it first fails, except that the test is rerun
> unconditionally.

Ping?  Does anyone have a preference between adding a new option,
--loop, or changing the heaviour of the -i option?

> This differs from the "-i <n>" option, which iterates each set of
> tests <n> times instead of each test.  The -i option is problematic in
> two ways.  First, it doesn't save the test artifacts from each test run.
> This is unfortunate because when the developer is trying to debug a
> flaky test failure, running "check -i 100" will run a test 100 times,
> but if only the 42nd test fails, the NNN.out.bad file for that failing
> test run is not preserved.  The second difference between --loop and
> -i is the result.xml file is rewritten after each test set, so we do
> not have the cumulative statistics of the 100 test runs in the junit
> XML file.

						- Ted

Re: [PATCH RFC] check: add new option "--loop <n>" which runs each test multiple times

[Zorro Lang]

On Tue, May 05, 2026 at 03:34:09PM +0200, Theodore Tso wrote:
> On Wed, Apr 15, 2026 at 05:32:48PM -0400, Theodore Ts'o wrote:
> > Teach the check script a new option --loop, which re-run each test
> > multiple times.  This works very similarly to to -L, which will retry
> > a particular test after it first fails, except that the test is rerun
> > unconditionally.
> 
> Ping?  Does anyone have a preference between adding a new option,
> --loop, or changing the heaviour of the -i option?

Hi Ted,

Personally, I feel that once we have --loop, the existing -i starts to lose
its value. --loop is far more effective at catching specific, flaky bugs due
to its stashing logic.

Furthermore, if we eventually add something like --loop-while-successful, then
-I also becomes redundant. It might be worth considering a more unified naming
convention in the long run, perhaps renaming -L to --loop-on-fail and keeping
everything under the --loop-* family. This would be much clearer for other users
than the current mix of single-letter flags.

Welcome any further feedback. Also, feel free to weigh in if you have concerns
about the potential impact of renaming these option names.

Thanks,
Zorro

> 
> > This differs from the "-i <n>" option, which iterates each set of
> > tests <n> times instead of each test.  The -i option is problematic in
> > two ways.  First, it doesn't save the test artifacts from each test run.
> > This is unfortunate because when the developer is trying to debug a
> > flaky test failure, running "check -i 100" will run a test 100 times,
> > but if only the 42nd test fails, the NNN.out.bad file for that failing
> > test run is not preserved.  The second difference between --loop and
> > -i is the result.xml file is rewritten after each test set, so we do
> > not have the cumulative statistics of the 100 test runs in the junit
> > XML file.
> 
> 						- Ted
> 

Re: [PATCH RFC] check: add new option "--loop <n>" which runs each test multiple times

[Zorro Lang]

On Wed, Apr 15, 2026 at 05:32:48PM -0400, Theodore Ts'o wrote:
> Teach the check script a new option --loop, which re-run each test
> multiple times.  This works very similarly to to -L, which will retry
> a particular test after it first fails, except that the test is rerun
> unconditionally.
> 
> This differs from the "-i <n>" option, which iterates each set of
> tests <n> times instead of each test.  The -i option is problematic in
> two ways.  First, doesn't save the test artifacts from each test run.
> This is unfortunate because when the developer is trying to debug a
> flaky test failure, running "check -i 100" will run a test 100 times,
> but if only the 42nd test fails, the NNN.out.bad file for that failing
> test run is not preserved.  The second difference between --loop and
> -i is the result.xml file is rewritten after each test set, so we do
> not have the cumulative statistics of the 100 test runs in the junit
> XML file.
> 
> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
> ---

Hi Ted,

> 
> Note: This commit adds a new command-line option instead of changing the
> behavior of -i because it's possible that *someone* actually likes the
> current behavior of the -i option, and changing how -i works might
> break their test runner infrastructure.
> 
> Speaking personally, I find the current -i option completely useless
> for the needs of xfstests-bld, and I would be happy to just change how
> the -i option works.  This would also require changing support for -I,
> but I was planning on adding an --loop-while-successful option
> eventually, since it would be faster for bisection (although I
> normally don't care about the junit XML file for NNN.out.bad files
> when bisecting, so I find -I much less objectionable than -i).
> 
>  check | 28 ++++++++++++++++++++++------
>  1 file changed, 22 insertions(+), 6 deletions(-)
> 
> diff --git a/check b/check
> index cd7a79347..923d81a28 100755
> --- a/check
> +++ b/check
> @@ -27,6 +27,8 @@ DUMP_OUTPUT=false
>  iterations=1
>  istop=false
>  loop_on_fail=0
> +loop_always=0
> +loop_count=0
>  exclude_tests=()
>  
>  # This is a global variable used to pass test failure text to reporting gunk
> @@ -85,6 +87,7 @@ check options
>      -s section		run only specified section from config file
>      -S section		exclude the specified section from the config file
>      -L <n>		loop tests <n> times following a failure, measuring aggregate pass/fail metrics
> +    --loop=<n>		loop tests <n> times, measuring aggregate pass/fail metrics
>  
>  testlist options
>      -g group[,group...]	include tests from these groups
> @@ -339,6 +342,12 @@ while [ $# -gt 0 ]; do
>  	--extra-space=*) export SCRATCH_DEV_EMPTY_SPACE=${r#*=} ;;
>  	-L)	[[ $2 =~ ^[0-9]+$ ]] || usage
>  		loop_on_fail=$2; shift
> +		loop_count=$loop_on_fail
> +		;;
> +	--loop=*) loop_always=${1#*=}

"${1#*=}", we're doing it the hard way... I really hope to rewrite the whole
arguments processing part with getopt or any other good way.

> +		[[ $loop_always =~ ^[0-9]+$ ]] || usage
> +		loop_count=$(( loop_always - 1))

OK, if --loop=0, loop_count=-1, then the test will be run once. So looks like
--loop=<n> is "loop tests an *additional* <n> times", right?

> +		set +vx
        ^^^^^^^

It seems a debug ghost is still haunting the code :)

Others look good to me.

Thanks,
Zorro

>  		;;
>  
>  	-*)	usage ;;
> @@ -604,7 +613,7 @@ _expunge_test()
>  }
>  
>  # retain files which would be overwritten in subsequent reruns of the same test
> -_stash_fail_loop_files() {
> +_stash_loop_files() {
>  	local seq_prefix="${REPORT_DIR}/${1}"
>  	local cp_suffix="$2"
>  
> @@ -629,9 +638,9 @@ _stash_test_status() {
>  
>  	if ((${#loop_status[*]} > 0)); then
>  		# continuing or completing rerun-on-failure loop
> -		_stash_fail_loop_files "$test_seq" ".rerun${#loop_status[*]}"
> +		_stash_loop_files "$test_seq" ".rerun${#loop_status[*]}"
>  		loop_status+=("$test_status")
> -		if ((${#loop_status[*]} > loop_on_fail)); then
> +		if ((${#loop_status[*]} > loop_count)); then
>  			printf "%s aggregate results across %d runs: " \
>  				"$test_seq" "${#loop_status[*]}"
>  			awk "BEGIN {
> @@ -651,9 +660,9 @@ _stash_test_status() {
>  
>  	case "$test_status" in
>  	fail)
> -		if ((loop_on_fail > 0)); then
> +		if ((loop_on_fail > 0 || loop_always > 0 )); then
>  			# initial failure, start rerun-on-failure loop
> -			_stash_fail_loop_files "$test_seq" ".rerun0"
> +			_stash_loop_files "$test_seq" ".rerun0"
>  			loop_status+=("$test_status")
>  		fi
>  		bad+=("$test_seq")
> @@ -661,7 +670,14 @@ _stash_test_status() {
>  	list|notrun)
>  		notrun+=("$test_seq")
>  		;;
> -	pass|expunge)
> +	pass)
> +		if (( loop_always > 0 )); then
> +			# start rerun loop
> +			_stash_loop_files "$test_seq" ".rerun0"
> +			loop_status+=("$test_status")
> +		fi
> +	        ;;
> +	expunge)
>  		;;
>  	*)
>  		echo "Unexpected test $test_seq status: $test_status"
> -- 
> 2.51.0
> 
>