Re: [RFC PATCH 0/2] add option to rerun failed tests

[David Disseldorp <ddiss@suse.de>]

Thanks for the feedback, Ted...

On Fri, 24 Jun 2022 00:42:03 -0400, Theodore Ts'o wrote:

> On Tue, Jun 21, 2022 at 06:01:51PM +0200, David Disseldorp wrote:
> > This RFC patchset adds support to loop on failed tests, as proposed by
> > Ted Ts'o in https://lwn.net/Articles/897061/:
> >   add a mode that will immediately rerun a failed test 25 or 100 times
> >   to establish a failure percentage.
> > 
> > There are a couple of things which I'd like to resolve before dropping
> > the RFC flag, but would appreciate early feedback on the approach here.  
> 
> This is really exciting!  I was hoping to try it out, but the first
> patch doesn't apply to origin/master on xfstests-dev.
> 
> For example this patch hunk:
> 
> @@ -729,9 +750,7 @@ function run_section()
>  	prev_seq=""
>  	for seq in $list ; do
>  		# Run report for previous test!
> -		if [ "$tc_status" == "fail" ]; then
> -			bad+=("$seqnum")
> -		fi
> +		_stash_test_status "$seqnum" "$tc_status"
>  		if $do_report && [[ ! $tc_status =~ ^(init|expunge)$ ]]; then
>  			_make_testcase_report "$prev_seq" "$tc_status"
>  		fi
> 
> The relevant section of check looks like this:
> 
> 		# Run report for previous test!
> 		if $err ; then
> 			bad="$bad $seqnum"
> 			n_bad=`expr $n_bad + 1`
> 			tc_status="fail"
> 		fi
> 
> And "git blame" shows that this part of check hasn't changed since
> 2018, and I'm on the latest version upstream version of xfstests:
> 
> commit 568ac9fffeb6afec03e5d6c9936617232fd7fc6d (HEAD, tag: v2022.06.05, origin/master, origin/HEAD, kernel/master)
> Author: Dave Chinner <dchinner@redhat.com>
> Date:   Fri Jun 3 11:54:13 2022 +1000
> 
>     xfs/189: systemd monitoring of /etc/fstab sucks
> 
> 
> Was your patch based xfstests with some out-of-tree patches?

Yes, I forgot to mention that, sorry. As Zorro indicated, these were
done atop the v2022.06.12 tag with the following series applied:
https://lore.kernel.org/fstests/20220620192934.21694-1-ddiss@suse.de/

> > The caveats are:
> > - rerun tests will be tracked as a single failure in @try and @bad
> >   + xunit reports do not include any rerun details
> > - .bad files generated on failure will be overwritten by test reruns
> > 
> > For xunit reports, I think it'll make sense to stash the aggregates in a
> > separate <test>.agg-results file or something. Similarly for .bad file
> > overwrites, I could add a .<rerun #> suffix for capturing all failure
> > data.  
> 
> For xunit results fie, was assuming that simply we would just have
> multiple repeated testcase entries stored in the single results.xml
> file.  For example:
> 
> <testcase classname="xfstests.global" name="generic/476" time="354">
> 		<failure message="Test  failed, reason unknown" type="TestFail" />
> 		<system-out>
> 		...
> 	</testcase>
> <testcase classname="xfstests.global" name="generic/476" time="343">
> 	</testcase>
> <testcase classname="xfstests.global" name="generic/476" time="353">
> 	</testcase>
> 	...

That seems sensible, I'll add this functionality.

> I don't know that we need a separate file for the rerun tests, since
> it's not that hard to create a python script which parses the results
> and calculates the pass/fail percentages for any test which is run
> mutiple times.

It's just a string, so doesn't need to be in a file. I'll add an extra
parameter to _make_testcase_report so that it can be stashed somewhere
like <testcase ... status="$aggregate_stats"> on the last rerun.

> As far as haivng the .bad and .full files, I agree that some kind of
> .rerun-NN suffix would make a lot of sense.

I had a bit of a play with this and it does get a bit ugly if we want to
prefix things like .dmesg as well. The xunit rerun entries will already
capture everything, but I suppose it's still needed for those not using
xunit reports.

Cheers, David