Re: [PATCH 5/6] check: run tests in exactly the order specified

[Brian Foster <bfoster@redhat.com>]

On Tue, Feb 09, 2021 at 06:56:47PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
> 
> Introduce a new --exact-order switch to disable all sorting, filtering
> of repeated lines, and shuffling of test order.  The goal of this is to
> be able to run tests in a specific order, namely to try to reproduce
> test failures that could be the result of a -r(andomize) run getting
> lucky.
> 
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> ---
>  check |   36 ++++++++++++++++++++++++++++--------
>  1 file changed, 28 insertions(+), 8 deletions(-)
> 
> 
> diff --git a/check b/check
> index 6f8db858..106ec8e1 100755
> --- a/check
> +++ b/check
...
> @@ -249,17 +251,22 @@ _prepare_test_list()
>  		trim_test_list $list
>  	done
>  
> -	# sort the list of tests into numeric order
> -	if $randomize; then
> -		if type shuf >& /dev/null; then
> -			sorter="shuf"
> +	# sort the list of tests into numeric order unless we're running tests
> +	# in the exact order specified
> +	if ! $exact_order; then
> +		if $randomize; then
> +			if type shuf >& /dev/null; then
> +				sorter="shuf"
> +			else
> +				sorter="awk -v seed=$RANDOM -f randomize.awk"
> +			fi
>  		else
> -			sorter="awk -v seed=$RANDOM -f randomize.awk"
> +			sorter="cat"
>  		fi
> +		list=`sort -n $tmp.list | uniq | $sorter`
>  	else
> -		sorter="cat"
> +		list=`cat $tmp.list`

Do we want to still filter out duplicates (i.e. uniq) in exact order
mode? LGTM either way:

Reviewed-by: Brian Foster <bfoster@redhat.com>

>  	fi
> -	list=`sort -n $tmp.list | uniq | $sorter`
>  	rm -f $tmp.list
>  }
>  
> @@ -304,7 +311,20 @@ while [ $# -gt 0 ]; do
>  	-udiff)	diff="$diff -u" ;;
>  
>  	-n)	showme=true ;;
> -        -r)	randomize=true ;;
> +	-r)
> +		if $exact_order; then
> +			echo "Cannot specify -r and --exact-order."
> +			exit 1
> +		fi
> +		randomize=true
> +		;;
> +	--exact-order)
> +		if $randomize; then
> +			echo "Cannnot specify --exact-order and -r."
> +			exit 1
> +		fi
> +		exact_order=true
> +		;;
>  	-i)	iterations=$2; shift ;;
>  	-T)	timestamp=true ;;
>  	-d)	DUMP_OUTPUT=true ;;
>