Re: [PATCH 5/6] check: run tests in exactly the order specified

["Darrick J. Wong" <djwong@kernel.org>]

On Thu, Feb 11, 2021 at 09:00:45AM -0500, Brian Foster wrote:
> On Tue, Feb 09, 2021 at 06:56:47PM -0800, Darrick J. Wong wrote:
> > From: Darrick J. Wong <djwong@kernel.org>
> > 
> > Introduce a new --exact-order switch to disable all sorting, filtering
> > of repeated lines, and shuffling of test order.  The goal of this is to
> > be able to run tests in a specific order, namely to try to reproduce
> > test failures that could be the result of a -r(andomize) run getting
> > lucky.
> > 
> > Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> > ---
> >  check |   36 ++++++++++++++++++++++++++++--------
> >  1 file changed, 28 insertions(+), 8 deletions(-)
> > 
> > 
> > diff --git a/check b/check
> > index 6f8db858..106ec8e1 100755
> > --- a/check
> > +++ b/check
> ...
> > @@ -249,17 +251,22 @@ _prepare_test_list()
> >  		trim_test_list $list
> >  	done
> >  
> > -	# sort the list of tests into numeric order
> > -	if $randomize; then
> > -		if type shuf >& /dev/null; then
> > -			sorter="shuf"
> > +	# sort the list of tests into numeric order unless we're running tests
> > +	# in the exact order specified
> > +	if ! $exact_order; then
> > +		if $randomize; then
> > +			if type shuf >& /dev/null; then
> > +				sorter="shuf"
> > +			else
> > +				sorter="awk -v seed=$RANDOM -f randomize.awk"
> > +			fi
> >  		else
> > -			sorter="awk -v seed=$RANDOM -f randomize.awk"
> > +			sorter="cat"
> >  		fi
> > +		list=`sort -n $tmp.list | uniq | $sorter`
> >  	else
> > -		sorter="cat"
> > +		list=`cat $tmp.list`
> 
> Do we want to still filter out duplicates (i.e. uniq) in exact order
> mode? LGTM either way:

I figure --exact-order means to run exactly what the user specified,
duplicates and all.

--D

> 
> Reviewed-by: Brian Foster <bfoster@redhat.com>
> 
> >  	fi
> > -	list=`sort -n $tmp.list | uniq | $sorter`
> >  	rm -f $tmp.list
> >  }
> >  
> > @@ -304,7 +311,20 @@ while [ $# -gt 0 ]; do
> >  	-udiff)	diff="$diff -u" ;;
> >  
> >  	-n)	showme=true ;;
> > -        -r)	randomize=true ;;
> > +	-r)
> > +		if $exact_order; then
> > +			echo "Cannot specify -r and --exact-order."
> > +			exit 1
> > +		fi
> > +		randomize=true
> > +		;;
> > +	--exact-order)
> > +		if $randomize; then
> > +			echo "Cannnot specify --exact-order and -r."
> > +			exit 1
> > +		fi
> > +		exact_order=true
> > +		;;
> >  	-i)	iterations=$2; shift ;;
> >  	-T)	timestamp=true ;;
> >  	-d)	DUMP_OUTPUT=true ;;
> > 
>