Re: [RFC PATCH 0/2] add option to rerun failed tests

["Theodore Ts'o" <tytso@mit.edu>]

On Tue, Jun 21, 2022 at 06:01:51PM +0200, David Disseldorp wrote:
> This RFC patchset adds support to loop on failed tests, as proposed by
> Ted Ts'o in https://lwn.net/Articles/897061/:
>   add a mode that will immediately rerun a failed test 25 or 100 times
>   to establish a failure percentage.
> 
> There are a couple of things which I'd like to resolve before dropping
> the RFC flag, but would appreciate early feedback on the approach here.

This is really exciting!  I was hoping to try it out, but the first
patch doesn't apply to origin/master on xfstests-dev.

For example this patch hunk:

@@ -729,9 +750,7 @@ function run_section()
 	prev_seq=""
 	for seq in $list ; do
 		# Run report for previous test!
-		if [ "$tc_status" == "fail" ]; then
-			bad+=("$seqnum")
-		fi
+		_stash_test_status "$seqnum" "$tc_status"
 		if $do_report && [[ ! $tc_status =~ ^(init|expunge)$ ]]; then
 			_make_testcase_report "$prev_seq" "$tc_status"
 		fi

The relevant section of check looks like this:

		# Run report for previous test!
		if $err ; then
			bad="$bad $seqnum"
			n_bad=`expr $n_bad + 1`
			tc_status="fail"
		fi

And "git blame" shows that this part of check hasn't changed since
2018, and I'm on the latest version upstream version of xfstests:

commit 568ac9fffeb6afec03e5d6c9936617232fd7fc6d (HEAD, tag: v2022.06.05, origin/master, origin/HEAD, kernel/master)
Author: Dave Chinner <dchinner@redhat.com>
Date:   Fri Jun 3 11:54:13 2022 +1000

    xfs/189: systemd monitoring of /etc/fstab sucks


Was your patch based xfstests with some out-of-tree patches?

> The caveats are:
> - rerun tests will be tracked as a single failure in @try and @bad
>   + xunit reports do not include any rerun details
> - .bad files generated on failure will be overwritten by test reruns
> 
> For xunit reports, I think it'll make sense to stash the aggregates in a
> separate <test>.agg-results file or something. Similarly for .bad file
> overwrites, I could add a .<rerun #> suffix for capturing all failure
> data.

For xunit results fie, was assuming that simply we would just have
multiple repeated testcase entries stored in the single results.xml
file.  For example:

<testcase classname="xfstests.global" name="generic/476" time="354">
		<failure message="Test  failed, reason unknown" type="TestFail" />
		<system-out>
		...
	</testcase>
<testcase classname="xfstests.global" name="generic/476" time="343">
	</testcase>
<testcase classname="xfstests.global" name="generic/476" time="353">
	</testcase>
	...

I don't know that we need a separate file for the rerun tests, since
it's not that hard to create a python script which parses the results
and calculates the pass/fail percentages for any test which is run
mutiple times.


As far as haivng the .bad and .full files, I agree that some kind of
.rerun-NN suffix would make a lot of sense.

Cheers,

						- Ted