Re: [PATCH 5/6] check: run tests in exactly the order specified

Brian Foster
2021-02-11T14:05:13Z

On Tue, Feb 09, 2021 at 06:56:47PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
>
> Introduce a new --exact-order switch to disable all sorting, filtering
> of repeated lines, and shuffling of test order. The goal of this is to
> be able to run tests in a specific order, namely to try to reproduce
> test failures that could be the result of a -r(andomize) run getting
> lucky.
>
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> ---
> check | 36 ++++++++++++++++++++++++++++--------
> 1 file changed, 28 insertions(+), 8 deletions(-)
>
>
> diff --git a/check b/check
> index 6f8db858..106ec8e1 100755
> --- a/check
> +++ b/check
...
> @@ -249,17 +251,22 @@ _prepare_test_list()
> trim_test_list $list
> done
>
> - # sort the list of tests into numeric order
> - if $randomize; then
> - if type shuf >& /dev/null; then
> - sorter="shuf"
> + # sort the list of tests into numeric order unless we're running tests
> + # in the exact order specified
> + if ! $exact_order; then
> + if $randomize; then
> + if type shuf >& /dev/null; then
> + sorter="shuf"
> + else
> + sorter="awk -v seed=$RANDOM -f randomize.awk"
> + fi
> else
> - sorter="awk -v seed=$RANDOM -f randomize.awk"
> + sorter="cat"
> fi
> + list=`sort -n $tmp.list | uniq | $sorter`
> else
> - sorter="cat"
> + list=`cat $tmp.list`

Do we want to still filter out duplicates (i.e. uniq) in exact order
mode? LGTM either way:

Reviewed-by: Brian Foster <bfoster@redhat.com>

> fi
> - list=`sort -n $tmp.list | uniq | $sorter`
> rm -f $tmp.list
> }
>
> @@ -304,7 +311,20 @@ while [ $# -gt 0 ]; do
> -udiff) diff="$diff -u" ;;
>
> -n) showme=true ;;
> - -r) randomize=true ;;
> + -r)
> + if $exact_order; then
> + echo "Cannot specify -r and --exact-order."
> + exit 1
> + fi
> + randomize=true
> + ;;
> + --exact-order)
> + if $randomize; then
> + echo "Cannnot specify --exact-order and -r."
> + exit 1
> + fi
> + exact_order=true
> + ;;
> -i) iterations=$2; shift ;;
> -T) timestamp=true ;;
> -d) DUMP_OUTPUT=true ;;
>