[PATCH 0/5]: CLI and feature improvements for check-parallel

[PATCH 0/5]: CLI and feature improvements for check-parallel

[Dave Chinner]

Hi folks,

This patchset brings some new functionality to check-parallel to
make it easier to run across different machines and filesystems.
The patch seti adds these features:

- auto-adjusts concurrency for the machine it is running on.
- adds CLI support of selection of tests. These CLI parameters are
  identical to the check CLI parameters and the test lists are built
  using the same code.
- adds support for logwrites devices so that all the test that use
  dm-logwrites are now enabled
- adds support for specifying the initial filesystem type to test
  on the CLI. This makes it easy to select xfs, btrfs, ext4, etc as
  the target filesystem type that is to be tested.
- Only block device based filesystems can be used with
  check-parallel, and this is now enforced at FSTYP selection time.

For example, testing the rw group on ext4 is now a simple matter of
adding the "-f ext4" parameter to the command line like so:

$ time sudo ./check-parallel -D /mnt/xfs -f ext4 -g rw -x dump
Runner 12 Failures:  ext4/308
Runner 1 Failures:  generic/095
Runner 21 Failures:  generic/042
Runner 51 Failures:  generic/627
Runner 8 Failures:  generic/032
Runner 13 Failures:  generic/019
Runner 34 Failures:  generic/347
.....

Options like exclude files, groups and lists also work natively in
check-parallel now, same as they do in check...

-Dave.

[PATCH 1/5] check-parallel: adjust concurrency according to CPU count

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

COncurrency is currently hard coded at 64 worker threads. THis is
too many for small CPU count machines; the idea is to create a
sustained load of roughly one test per CPU as they are mostly single
threaded/single process tests. The number "64" was chosen because
I've been developing this functionality on a 64p VM.

Rather than hard coding the concurrency, probe the number of CPUs
available and create that many running contexts.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 check-parallel | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/check-parallel b/check-parallel
index c85437252..d34c73f66 100755
--- a/check-parallel
+++ b/check-parallel
@@ -13,7 +13,7 @@ export SRC_DIR="tests"
 basedir=$1
 shift
 check_args="$*"
-runners=64
+runners=$(getconf _NPROCESSORS_CONF)
 runner_list=()
 runtimes=()
 
-- 
2.45.2

[PATCH 2/5] check: factor out test list building code

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

factor out all the test list parsing and building code to
common/test_list so that it can be used by both check and
check-parallel.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 check            | 270 +++----------------------------------------
 common/report    |   2 +-
 common/test_list | 295 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 314 insertions(+), 253 deletions(-)
 create mode 100644 common/test_list

diff --git a/check b/check
index 607d2456e..4dc266dcf 100755
--- a/check
+++ b/check
@@ -15,19 +15,13 @@ notrun=()
 interrupt=true
 diff="diff -u"
 showme=false
-have_test_arg=false
-randomize=false
-exact_order=false
 export here=`pwd`
-xfile=""
-subdir_xfile=""
 brief_test_summary=false
 do_report=false
 DUMP_OUTPUT=false
 iterations=1
 istop=false
 loop_on_fail=0
-exclude_tests=()
 
 # This is a global variable used to pass test failure text to reporting gunk
 _err_msg=""
@@ -49,8 +43,9 @@ timestamp=${TIMESTAMP:=false}
 
 rm -f $tmp.list $tmp.tmp $tmp.grep $here/$iam.out $tmp.report.* $tmp.arglist
 
-SRC_GROUPS="generic"
-export SRC_DIR="tests"
+# We need to include the test list processing first as argument parsing
+# requires test list parsing and setup.
+. ./common/test_list
 
 usage()
 {
@@ -124,153 +119,12 @@ examples:
 	    exit 1
 }
 
-get_sub_group_list()
-{
-	local d=$1
-	local grp=$2
-
-	test -s "$SRC_DIR/$d/group.list" || return 1
-
-	local grpl=$(sed -n < $SRC_DIR/$d/group.list \
-		-e 's/#.*//' \
-		-e 's/$/ /' \
-		-e "s;^\($VALID_TEST_NAME\).* $grp .*;$SRC_DIR/$d/\1;p")
-	echo $grpl
-}
-
-get_group_list()
-{
-	local grp=$1
-	local grpl=""
-	local sub=$(dirname $grp)
-	local fsgroup="$FSTYP"
-
-	if [ -n "$sub" -a "$sub" != "." -a -d "$SRC_DIR/$sub" ]; then
-		# group is given as <subdir>/<group> (e.g. xfs/quick)
-		grp=$(basename $grp)
-		get_sub_group_list $sub $grp
-		return
-	fi
-
-	if [ "$FSTYP" = ext2 -o "$FSTYP" = ext3 ]; then
-	    fsgroup=ext4
-	fi
-	for d in $SRC_GROUPS $fsgroup; do
-		if ! test -d "$SRC_DIR/$d" ; then
-			continue
-		fi
-		grpl="$grpl $(get_sub_group_list $d $grp)"
-	done
-	echo $grpl
-}
-
-# Find all tests, excluding files that are test metadata such as group files.
-# It matches test names against $VALID_TEST_NAME defined in common/rc
-get_all_tests()
-{
-	touch $tmp.list
-	for d in $SRC_GROUPS $FSTYP; do
-		if ! test -d "$SRC_DIR/$d" ; then
-			continue
-		fi
-		ls $SRC_DIR/$d/* | \
-			grep -v "\..*" | \
-			grep "^$SRC_DIR/$d/$VALID_TEST_NAME"| \
-			grep -v "group\|Makefile" >> $tmp.list 2>/dev/null
-	done
-}
-
-# takes the list of tests to run in $tmp.list, and removes the tests passed to
-# the function from that list.
-trim_test_list()
-{
-	local test_list="$*"
-
-	rm -f $tmp.grep
-	local numsed=0
-	for t in $test_list
-	do
-	    if [ $numsed -gt 100 ]; then
-		grep -v -f $tmp.grep <$tmp.list >$tmp.tmp
-		mv $tmp.tmp $tmp.list
-		numsed=0
-		rm -f $tmp.grep
-	    fi
-	    echo "^$t\$" >>$tmp.grep
-	    numsed=`expr $numsed + 1`
-	done
-	grep -v -f $tmp.grep <$tmp.list >$tmp.tmp
-	mv $tmp.tmp $tmp.list
-	rm -f $tmp.grep
-}
-
 _timestamp()
 {
     local now=`date "+%T"`
     echo -n " [$now]"
 }
 
-_prepare_test_list()
-{
-	unset list
-	# Tests specified on the command line
-	if [ -s $tmp.arglist ]; then
-		cat $tmp.arglist > $tmp.list
-	else
-		touch $tmp.list
-	fi
-
-	# Specified groups to include
-	# Note that the CLI processing adds a leading space to the first group
-	# parameter, so we have to catch that here checking for "all"
-	if ! $have_test_arg && [ "$GROUP_LIST" == " all" ]; then
-		# no test numbers, do everything
-		get_all_tests
-	else
-		for group in $GROUP_LIST; do
-			list=$(get_group_list $group)
-			if [ -z "$list" ]; then
-				echo "Group \"$group\" is empty or not defined?"
-				exit 1
-			fi
-
-			for t in $list; do
-				grep -s "^$t\$" $tmp.list >/dev/null || \
-							echo "$t" >>$tmp.list
-			done
-		done
-	fi
-
-	# Specified groups to exclude
-	for xgroup in $XGROUP_LIST; do
-		list=$(get_group_list $xgroup)
-		if [ -z "$list" ]; then
-			echo "Group \"$xgroup\" is empty or not defined?"
-			continue
-		fi
-
-		trim_test_list $list
-	done
-
-	# sort the list of tests into numeric order unless we're running tests
-	# in the exact order specified
-	if ! $exact_order; then
-		if $randomize; then
-			if type shuf >& /dev/null; then
-				sorter="shuf"
-			else
-				sorter="awk -v seed=$RANDOM -f randomize.awk"
-			fi
-		else
-			sorter="cat"
-		fi
-		list=`sort -n $tmp.list | uniq | $sorter`
-	else
-		list=`cat $tmp.list`
-	fi
-	rm -f $tmp.list
-}
-
 # Process command arguments first.
 while [ $# -gt 0 ]; do
 	case "$1" in
@@ -285,48 +139,20 @@ while [ $# -gt 0 ]; do
 		export OVERLAY=true
 		;;
 
-	-g)	group=$2 ; shift ;
-		GROUP_LIST="$GROUP_LIST ${group//,/ }"
-		;;
+	-g)	_tl_setup_group $2 ; shift ;;
+	-e)	_tl_setup_exclude_tests $2 ; shift ;;
+	-E)	_tl_setup_exclude_file $2 ; shift ;;
+	-x)	_tl_setup_exclude_group $2; shift ;;
+	-X)	_tl_setup_exclude_subdir $2; shift ;;
+	-r)	_tl_setup_randomise ;;
+	--exact-order) _tl_setup_ordered ;;
 
-	-x)	xgroup=$2 ; shift ;
-		XGROUP_LIST="$XGROUP_LIST ${xgroup//,/ }"
-		;;
-
-	-X)	subdir_xfile=$2; shift ;
-		;;
-	-e)
-		xfile=$2; shift ;
-		readarray -t -O "${#exclude_tests[@]}" exclude_tests < \
-			<(echo "$xfile" | tr ', ' '\n\n')
-		;;
-
-	-E)	xfile=$2; shift ;
-		if [ -f $xfile ]; then
-			readarray -t -O ${#exclude_tests[@]} exclude_tests < \
-				<(sed "s/#.*$//" $xfile)
-		fi
-		;;
 	-s)	RUN_SECTION="$RUN_SECTION $2"; shift ;;
 	-S)	EXCLUDE_SECTION="$EXCLUDE_SECTION $2"; shift ;;
 	-l)	diff="diff" ;;
 	-udiff)	diff="$diff -u" ;;
 
 	-n)	showme=true ;;
-	-r)
-		if $exact_order; then
-			echo "Cannot specify -r and --exact-order."
-			exit 1
-		fi
-		randomize=true
-		;;
-	--exact-order)
-		if $randomize; then
-			echo "Cannnot specify --exact-order and -r."
-			exit 1
-		fi
-		exact_order=true
-		;;
 	-i)	iterations=$2; shift ;;
 	-I) 	iterations=$2; istop=true; shift ;;
 	-T)	timestamp=true ;;
@@ -344,13 +170,13 @@ while [ $# -gt 0 ]; do
 
 	-*)	usage ;;
 	*)	# not an argument, we've got tests now.
-		have_test_arg=true ;;
+		_tl_setup_cli $*
 	esac
 
 	# if we've found a test specification, the break out of the processing
 	# loop before we shift the arguments so that this is the first argument
 	# that we process in the test arg loop below.
-	if $have_test_arg; then
+	if $_tl_have_test_args; then
 		break;
 	fi
 
@@ -388,51 +214,6 @@ if [ -n "$FUZZ_REWRITE_DURATION" ]; then
 	fi
 fi
 
-if [ -n "$subdir_xfile" ]; then
-	for d in $SRC_GROUPS $FSTYP; do
-		[ -f $SRC_DIR/$d/$subdir_xfile ] || continue
-		for f in `sed "s/#.*$//" $SRC_DIR/$d/$subdir_xfile`; do
-			exclude_tests+=($d/$f)
-		done
-	done
-fi
-
-# Process tests from command line now.
-if $have_test_arg; then
-	while [ $# -gt 0 ]; do
-		case "$1" in
-		-*)	echo "Arguments before tests, please!"
-			status=1
-			exit $status
-			;;
-		*)	# Expand test pattern (e.g. xfs/???, *fs/001)
-			list=$(cd $SRC_DIR; echo $1)
-			for t in $list; do
-				t=${t#$SRC_DIR/}
-				test_dir=${t%%/*}
-				test_name=${t##*/}
-				group_file=$SRC_DIR/$test_dir/group.list
-
-				if grep -Eq "^$test_name" $group_file; then
-					# in group file ... OK
-					echo $SRC_DIR/$test_dir/$test_name \
-						>>$tmp.arglist
-				else
-					# oops
-					echo "$t - unknown test, ignored"
-				fi
-			done
-			;;
-		esac
-
-		shift
-	done
-elif [ -z "$GROUP_LIST" ]; then
-	# default group list is the auto group. If any other group or test is
-	# specified, we use that instead.
-	GROUP_LIST="auto"
-fi
-
 if [ `id -u` -ne 0 ]
 then
     echo "check: QA must be run as root"
@@ -593,21 +374,6 @@ _check_filesystems()
 	return $ret
 }
 
-_expunge_test()
-{
-	local TEST_ID="$1"
-
-	for f in "${exclude_tests[@]}"; do
-		# $f may contain traling spaces and comments
-		local id_regex="^${TEST_ID}\b"
-		if [[ "$f" =~ ${id_regex} ]]; then
-			echo "       [expunged]"
-			return 0
-		fi
-	done
-	return 1
-}
-
 # retain files which would be overwritten in subsequent reruns of the same test
 _stash_fail_loop_files() {
 	local seq_prefix="${REPORT_DIR}/${1}"
@@ -714,7 +480,7 @@ _run_seq() {
 }
 
 _detect_kmemleak
-_prepare_test_list
+_tl_prepare_test_list
 fstests_start_time="$(date +"%F %T")"
 
 if $OPTIONS_HAVE_SECTIONS; then
@@ -794,7 +560,7 @@ function run_section()
 		# TEST_DEV could be changed, source common/rc again with
 		# correct FSTYP to get FSTYP specific configs, e.g. common/xfs
 		. common/rc
-		_prepare_test_list
+		_tl_prepare_test_list
 	elif [ "$OLD_TEST_FS_MOUNT_OPTS" != "$TEST_FS_MOUNT_OPTS" ]; then
 		_test_unmount 2> /dev/null
 		if ! _test_mount
@@ -859,7 +625,7 @@ function run_section()
 
 	loop_status=()	# track rerun-on-failure state
 	local tc_status ix
-	local -a _list=( $list )
+	local -a _list=( $_tl_tests )
 	for ((ix = 0; ix < ${#_list[*]}; !${#loop_status[*]} && ix++)); do
 		seq="${_list[$ix]}"
 
@@ -880,7 +646,7 @@ function run_section()
 
 		# the filename for the test and the name output are different.
 		# we don't include the tests/ directory in the name output.
-		export seqnum=${seq#$SRC_DIR/}
+		export seqnum=$(_tl_strip_src_dir $seq)
 		group=${seqnum%%/*}
 		if $OPTIONS_HAVE_SECTIONS; then
 			REPORT_DIR="$RESULT_BASE/$section"
@@ -902,7 +668,7 @@ function run_section()
 		echo -n "$seqnum"
 
 		if $showme; then
-			if _expunge_test $seqnum; then
+			if _tl_expunge_test $seqnum; then
 				tc_status="expunge"
 			else
 				echo
@@ -928,7 +694,7 @@ function run_section()
 		rm -f $seqres.out.bad $seqres.hints
 
 		# check if we really should run it
-		if _expunge_test $seqnum; then
+		if _tl_expunge_test $seqnum; then
 			tc_status="expunge"
 			_stash_test_status "$seqnum" "$tc_status"
 			continue
diff --git a/common/report b/common/report
index 7128bbeba..5697d2540 100644
--- a/common/report
+++ b/common/report
@@ -196,7 +196,7 @@ _xunit_make_testcase_report()
 		echo -e "\t\t<skipped/>" >> $report
 		;;
 	"fail")
-		local out_src="${SRC_DIR}/${test_name}.out"
+		local out_src="${_tl_src_dir}/${test_name}.out"
 		local full_file="${REPORT_DIR}/${test_name}.full"
 		local dmesg_file="${REPORT_DIR}/${test_name}.dmesg"
 		local outbad_file="${REPORT_DIR}/${test_name}.out.bad"
diff --git a/common/test_list b/common/test_list
new file mode 100644
index 000000000..2432be6f7
--- /dev/null
+++ b/common/test_list
@@ -0,0 +1,295 @@
+##/bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (c) 2000-2002,2006 Silicon Graphics, Inc.  All Rights Reserved.
+# Copyright (c) 2024 Red Hat, Inc.  All Rights Reserved.
+#
+# Test list parsing and building functions
+#
+# Note: this file must stand alone and not be dependent on any other includes,
+# most especially common/rc and common/config. This is because we have to
+# include this file before option parsing, whilst the rc/config includes need to
+# be included -after- option parsing.
+#
+# Any function or variable that is public should have a "_tl_" prefix.
+
+export _tl_src_dir="tests"
+
+_SRC_GROUPS="generic"
+_GROUP_LIST=
+_XGROUP_LIST=
+_tl_exact_order=false
+_tl_randomise=false
+_tl_have_test_args=false
+_tl_file="$tmp.test_list"
+_tl_exclude_tests=()
+_tl_tests=
+
+_tl_strip_src_dir()
+{
+	local test="$1"
+
+	echo ${test#$_tl_src_dir/}
+}
+
+get_sub_group_list()
+{
+	local d=$1
+	local grp=$2
+
+	test -s "$_tl_src_dir/$d/group.list" || return 1
+
+	local grpl=$(sed -n < $_tl_src_dir/$d/group.list \
+		-e 's/#.*//' \
+		-e 's/$/ /' \
+		-e "s;^\($VALID_TEST_NAME\).* $grp .*;$_tl_src_dir/$d/\1;p")
+	echo $grpl
+}
+
+get_group_list()
+{
+	local grp=$1
+	local grpl=""
+	local sub=$(dirname $grp)
+	local fsgroup="$FSTYP"
+
+	if [ -n "$sub" -a "$sub" != "." -a -d "$_tl_src_dir/$sub" ]; then
+		# group is given as <subdir>/<group> (e.g. xfs/quick)
+		grp=$(basename $grp)
+		get_sub_group_list $sub $grp
+		return
+	fi
+
+	if [ "$FSTYP" = ext2 -o "$FSTYP" = ext3 ]; then
+	    fsgroup=ext4
+	fi
+	for d in $_SRC_GROUPS $fsgroup; do
+		if ! test -d "$_tl_src_dir/$d" ; then
+			continue
+		fi
+		grpl="$grpl $(get_sub_group_list $d $grp)"
+	done
+	echo $grpl
+}
+
+# Find all tests, excluding files that are test metadata such as group files.
+# It matches test names against $VALID_TEST_NAME defined in common/rc
+get_all_tests()
+{
+	touch $tmp.list
+	for d in $_SRC_GROUPS $FSTYP; do
+		if ! test -d "$_tl_src_dir/$d" ; then
+			continue
+		fi
+		ls $_tl_src_dir/$d/* | \
+			grep -v "\..*" | \
+			grep "^$_tl_src_dir/$d/$VALID_TEST_NAME"| \
+			grep -v "group\|Makefile" >> $tmp.list 2>/dev/null
+	done
+}
+
+# takes the list of tests to run in $tmp.list, and removes the tests passed to
+# the function from that list.
+trim_test_list()
+{
+	local test_list="$*"
+
+	rm -f $tmp.grep
+	local numsed=0
+	for t in $test_list
+	do
+	    if [ $numsed -gt 100 ]; then
+		grep -v -f $tmp.grep <$tmp.list >$tmp.tmp
+		mv $tmp.tmp $tmp.list
+		numsed=0
+		rm -f $tmp.grep
+	    fi
+	    echo "^$t\$" >>$tmp.grep
+	    numsed=`expr $numsed + 1`
+	done
+	grep -v -f $tmp.grep <$tmp.list >$tmp.tmp
+	mv $tmp.tmp $tmp.list
+	rm -f $tmp.grep
+}
+
+_tl_prepare_test_list()
+{
+	unset _tl_tests
+	# Tests specified on the command line
+	if [ -s $_tl_file ]; then
+		cat $_tl_file > $tmp.list
+	else
+		touch $tmp.list
+	fi
+
+	# Specified groups to include
+	# Note that the CLI processing adds a leading space to the first group
+	# parameter, so we have to catch that here checking for "all"
+	if ! $_tl_have_test_args && [ "$_GROUP_LIST" == " all" ]; then
+		# no test numbers, do everything
+		get_all_tests
+	else
+		for group in $_GROUP_LIST; do
+			list=$(get_group_list $group)
+			if [ -z "$list" ]; then
+				echo "Group \"$group\" is empty or not defined?"
+				exit 1
+			fi
+
+			for t in $list; do
+				grep -s "^$t\$" $tmp.list >/dev/null || \
+							echo "$t" >>$tmp.list
+			done
+		done
+	fi
+
+	# Specified groups to exclude
+	for xgroup in $_XGROUP_LIST; do
+		list=$(get_group_list $xgroup)
+		if [ -z "$list" ]; then
+			echo "Group \"$xgroup\" is empty or not defined?"
+			continue
+		fi
+
+		trim_test_list $list
+	done
+
+	# sort the list of tests into numeric order unless we're running tests
+	# in the exact order specified
+	if ! $_tl_exact_order; then
+		if $_tl_randomise; then
+			if type shuf >& /dev/null; then
+				sorter="shuf"
+			else
+				sorter="awk -v seed=$RANDOM -f randomize.awk"
+			fi
+		else
+			sorter="cat"
+		fi
+		_tl_tests=`sort -n $tmp.list | uniq | $sorter`
+	else
+		_tl_tests=`cat $tmp.list`
+	fi
+	rm -f $tmp.list
+}
+
+_tl_expunge_test()
+{
+	local TEST_ID="$1"
+
+	for f in "${_tl_exclude_tests[@]}"; do
+		# $f may contain traling spaces and comments
+		local id_regex="^${TEST_ID}\b"
+		if [[ "$f" =~ ${id_regex} ]]; then
+			echo "       [expunged]"
+			return 0
+		fi
+	done
+	return 1
+}
+
+_tl_setup_exclude_tests()
+{
+	local list="$1"
+
+	readarray -t -O "${#_tl_exclude_tests[@]}" _tl_exclude_tests < \
+		<(echo "$list" | tr ', ' '\n\n')
+}
+
+_tl_setup_exclude_file()
+{
+	local xfile="$1"
+
+	if [ -f $xfile ]; then
+		readarray -t -O ${#_tl_exclude_tests[@]} _tl_exclude_tests < \
+			<(sed "s/#.*$//" $xfile)
+	fi
+}
+
+_tl_setup_exclude_subdir()
+{
+	local xfile="$1"
+	local d
+	local f
+
+	[ -z "$xfile" ] && return
+
+	for d in $_SRC_GROUPS $FSTYP; do
+		[ -f $_tl_src_dir/$d/$xfile ] || continue
+		for f in `sed "s/#.*$//" $_tl_src_dir/$d/$xfile`; do
+			_tl_exclude_tests+=($d/$f)
+		done
+	done
+}
+
+_tl_setup_exclude_group()
+{
+	local xgroup="$1"
+
+	_XGROUP_LIST="$_XGROUP_LIST ${xgroup//,/ }"
+}
+
+_tl_setup_group()
+{
+	local group="$1"
+
+	_GROUP_LIST="$_GROUP_LIST ${group//,/ }"
+}
+
+_tl_setup_randomise()
+{
+	if $_tl_exact_order; then
+		echo "Cannot specify -r and --exact-order."
+		exit 1
+	fi
+	_tl_randomise=true
+}
+
+_tl_setup_ordered()
+{
+	if $_tl_randomise; then
+		echo "Cannnot specify --exact-order and -r."
+		exit 1
+	fi
+	_tl_exact_order=true
+}
+
+_tl_setup_cli()
+{
+	while [ $# -gt 0 ]; do
+		case "$1" in
+		-*)	echo "Arguments before tests, please!"
+			status=1
+			exit $status
+			;;
+		*)	# Expand test pattern (e.g. xfs/???, *fs/001)
+			local list=$(cd $_tl_src_dir; echo $1)
+			local t
+
+			for t in $list; do
+				t=${t#$_tl_src_dir/}
+				local test_dir=${t%%/*}
+				local test_name=${t##*/}
+				local group_file=$_tl_src_dir/$test_dir/group.list
+
+				if grep -Eq "^$test_name" $group_file; then
+					# in group file ... OK
+					echo $_tl_src_dir/$test_dir/$test_name \
+						>> $_tl_file
+					_tl_have_test_args=true
+				else
+					# oops
+					echo "$t - unknown test, ignored"
+				fi
+			done
+			;;
+		esac
+
+		shift
+	done
+
+	if ! $_tl_have_test_args && [ -z "$_GROUP_LIST" ]; then
+		# default group list is the auto group. If any other group or
+		# test is specified, we use that instead.
+		_GROUP_LIST="auto"
+	fi
+}
-- 
2.45.2

[PATCH 3/5] check-parallel: use common group list parsing code

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

Build the test list directly from command line prompts, rather
than hard coding the tests and using the check infrastructure to
filter that list.

We still pass exact test lists to check to execute the tests that
each runner needs to execute, but all other test list commands
are no longer passed to check.

As a result of this change, check-parallel no longer passes unknown
CLI parameters through to the internal check invocations. At this
point, the only non test-list related option is config file section
selection; more of the check options will be brought across as
needed in future patches.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 check-parallel   | 153 ++++++++++++++++++++++++++++++++++++++++-------
 common/test_list |   7 +++
 2 files changed, 139 insertions(+), 21 deletions(-)

diff --git a/check-parallel b/check-parallel
index d34c73f66..a341c9c2e 100755
--- a/check-parallel
+++ b/check-parallel
@@ -9,18 +9,115 @@
 # for them and runs the test in the background. When it completes, it tears down
 # the loop devices.
 
-export SRC_DIR="tests"
-basedir=$1
-shift
-check_args="$*"
+basedir=""
 runners=$(getconf _NPROCESSORS_CONF)
 runner_list=()
 runtimes=()
+show_test_list=
+run_section=""
 
+tmp=/tmp/check-parallel.$$
 
-# tests in auto group
-test_list=$(awk '/^[0-9].*auto/ { print "generic/" $1 }' tests/generic/group.list)
-test_list+=$(awk '/^[0-9].*auto/ { print "xfs/" $1 }' tests/xfs/group.list)
+export FSTYP=xfs
+
+# We need to include the test list processing first as argument parsing
+# requires test list parsing and setup.
+. ./common/test_names
+. ./common/test_list
+
+usage()
+{
+    echo "Usage: $0 [options] [testlist]"'
+
+check options
+    -D <dir>		Directory to run in
+    -n			Output test list, do not run tests
+    -r			randomize test order
+    --exact-order	run tests in the exact order specified
+    -s section		run only specified section from config file
+
+testlist options
+    -g group[,group...]	include tests from these groups
+    -x group[,group...]	exclude tests from these groups
+    -X exclude_file	exclude individual tests
+    -e testlist         exclude a specific list of tests
+    -E external_file	exclude individual tests
+    [testlist]		include tests matching names in testlist
+
+testlist argument is a list of tests in the form of <test dir>/<test name>.
+
+<test dir> is a directory under tests that contains a group file,
+with a list of the names of the tests in that directory.
+
+<test name> may be either a specific test file name (e.g. xfs/001) or
+a test file name match pattern (e.g. xfs/*).
+
+group argument is either a name of a tests group to collect from all
+the test dirs (e.g. quick) or a name of a tests group to collect from
+a specific tests dir in the form of <test dir>/<group name> (e.g. xfs/quick).
+If you want to run all the tests in the test suite, use "-g all" to specify all
+groups.
+
+exclude_file argument refers to a name of a file inside each test directory.
+for every test dir where this file is found, the listed test names are
+excluded from the list of tests to run from that test dir.
+
+external_file argument is a path to a single file containing a list of tests
+to exclude in the form of <test dir>/<test name>.
+
+examples:
+ check-parallel -D /mnt xfs/001
+ check-parallel -D /mnt -g quick
+ check-parallel -D /mnt -g xfs/quick
+ check-parallel -D /mnt -x stress xfs/*
+ check-parallel -D /mnt -X .exclude -g auto
+ check-parallel -D /mnt -E ~/.xfstests.exclude
+'
+	    exit 1
+}
+
+# Process command arguments first.
+while [ $# -gt 0 ]; do
+	case "$1" in
+	-\? | -h | --help) usage ;;
+
+	-D)	basedir=$2; shift ;;
+	-g)	_tl_setup_group $2 ; shift ;;
+	-e)	_tl_setup_exclude_tests $2 ; shift ;;
+	-E)	_tl_setup_exclude_file $2 ; shift ;;
+	-x)	_tl_setup_exclude_group $2; shift ;;
+	-X)	_tl_setup_exclude_subdir $2; shift ;;
+	-r)	_tl_setup_randomise ;;
+	--exact-order) _tl_setup_ordered ;;
+	-n)	show_test_list="yes" ;;
+
+	-s)	run_section="$run_section -s $2"; shift ;;
+
+	-*)	usage ;;
+	*)	# not an argument, we've got tests now.
+		_tl_setup_cli $*
+	esac
+
+	# if we've found a test specification, the break out of the processing
+	# loop before we shift the arguments so that this is the first argument
+	# that we process in the test arg loop below.
+	if $_tl_have_test_args; then
+		break;
+	fi
+
+	shift
+done
+
+if [ ! -d "$basedir" ]; then
+	echo "Invalid basedir specification"
+	usage
+fi
+if [ -d "$basedir/runner-0/" ]; then
+	prev_results=`ls -tr $basedir/runner-0/ | grep results | tail -1`
+fi
+
+_tl_prepare_test_list
+_tl_strip_test_list
 
 # grab all previously run tests and order them from highest runtime to lowest
 # We are going to try to run the longer tests first, hopefully so we can avoid
@@ -30,25 +127,23 @@ test_list+=$(awk '/^[0-9].*auto/ { print "xfs/" $1 }' tests/xfs/group.list)
 #
 # If we have tests in the test list that don't have runtimes recorded, then
 # append them to be run last.
-
-build_runner_list()
+time_order_test_list()
 {
 	local runtimes
 	local run_list=()
-	local prev_results=`ls -tr $basedir/runner-0/ | grep results | tail -1`
 
 	runtimes=$(cat $basedir/*/$prev_results/check.time | sort -k 2 -nr | cut -d " " -f 1)
 
 	# Iterate the timed list first. For every timed list entry that
 	# is found in the test_list, add it to the local runner list.
 	local -a _list=( $runtimes )
-	local -a _tlist=( $test_list )
+	local -a _tlist=( $_tl_tests )
 	local rx=0
 	local ix
 	local jx
 	#set -x
 	for ((ix = 0; ix < ${#_list[*]}; ix++)); do
-		echo $test_list | grep -q ${_list[$ix]}
+		echo $_tl_tests | grep -q ${_list[$ix]}
 		if [ $? == 0 ]; then
 			# add the test to the new run list and remove
 			# it from the remaining test list.
@@ -60,20 +155,21 @@ build_runner_list()
 
 	# The final test list is all the time ordered tests followed by
 	# all the tests we didn't find time records for.
-	test_list="${run_list[*]} ${_tlist[*]}"
+	_tl_tests="${run_list[*]} ${_tlist[*]}"
 }
 
-if [ -f $basedir/runner-0/results/check.time ]; then
-	build_runner_list
+if ! $_tl_randomise -a ! $_tl_exact_order; then
+	if [ -f $basedir/runner-0/$prev_results/check.time ]; then
+		time_order_test_list
+	fi
 fi
 
 # split the list amongst N runners
-
 split_runner_list()
 {
 	local ix
 	local rx
-	local -a _list=( $test_list )
+	local -a _list=( $_tl_tests )
 	for ((ix = 0; ix < ${#_list[*]}; ix++)); do
 		seq="${_list[$ix]}"
 		rx=$((ix % $runners))
@@ -137,7 +233,7 @@ runner_go()
 
 	# Run the tests in it's own mount namespace, as per the comment below
 	# that precedes making the basedir a private mount.
-	./src/nsexec -m ./check $check_args -x unreliable_in_parallel --exact-order ${runner_list[$id]} > $me/log 2>&1
+	./src/nsexec -m ./check $run_section -x unreliable_in_parallel --exact-order ${runner_list[$id]} > $me/log 2>&1
 
 	wait
 	sleep 1
@@ -165,6 +261,13 @@ cleanup()
 
 trap "cleanup; exit" HUP INT QUIT TERM
 
+split_runner_list
+if [ -n "$show_test_list" ]; then
+	echo Time ordered test list:
+	echo $_tl_tests
+	echo
+fi
+
 
 # Each parallel test runner needs to only see it's own mount points. If we
 # leave the basedir as shared, then all tests see all mounts and then we get
@@ -178,15 +281,23 @@ trap "cleanup; exit" HUP INT QUIT TERM
 # in it's own mount namespace so that they cannot see mounts that other tests
 # are performing.
 mount --make-private $basedir
-split_runner_list
+
 now=`date +%Y-%m-%d-%H:%M:%S`
 for ((i = 0; i < $runners; i++)); do
 
-	runner_go $i $now &
+	if [ -n "$show_test_list" ]; then
+		echo "Runner $i: ${runner_list[$i]}"
+	else
+		runner_go $i $now &
+	fi
 
 done;
 wait
 
+if [ -n "$show_test_list" ]; then
+	exit 0
+fi
+
 echo -n "Tests run: "
 grep Ran /mnt/xfs/*/log | sed -e 's,^.*:,,' -e 's, ,\n,g' | sort | uniq | wc -l
 
@@ -195,7 +306,7 @@ grep Failures: $basedir/*/log | uniq | sed -e "s/^.*Failures://" -e "s,\([0-9]\)
 echo
 
 echo Ten slowest tests - runtime in seconds:
-cat $basedir/*/results/check.time | sort -k 2 -nr | head -10
+cat $basedir/*/results-$now/check.time | sort -k 2 -nr | head -10
 
 echo
 echo Cleanup on Aisle 5?
diff --git a/common/test_list b/common/test_list
index 2432be6f7..2b3ae9fbf 100644
--- a/common/test_list
+++ b/common/test_list
@@ -24,6 +24,7 @@ _tl_file="$tmp.test_list"
 _tl_exclude_tests=()
 _tl_tests=
 
+# strip 'tests\' prefix from the provided test name
 _tl_strip_src_dir()
 {
 	local test="$1"
@@ -31,6 +32,12 @@ _tl_strip_src_dir()
 	echo ${test#$_tl_src_dir/}
 }
 
+# strip 'tests\' prefix from all the tests in the test list
+_tl_strip_test_list()
+{
+	_tl_tests=$(echo $_tl_tests | sed -e "s/$_tl_src_dir\///g")
+}
+
 get_sub_group_list()
 {
 	local d=$1
-- 
2.45.2

[PATCH 4/5] check-parallel: add logwrite device support

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

It is just another loop device per test runner....

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 check-parallel | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/check-parallel b/check-parallel
index a341c9c2e..498b53c5a 100755
--- a/check-parallel
+++ b/check-parallel
@@ -208,12 +208,14 @@ runner_go()
 	local me=$basedir/runner-$id
 	local _test=$me/test.img
 	local _scratch=$me/scratch.img
+	local _logwrites=$me/logwrites.img
 	local _results=$me/results-$2
 
 	mkdir -p $me
 
 	xfs_io -f -c 'truncate 2g' $_test
 	xfs_io -f -c 'truncate 8g' $_scratch
+	xfs_io -f -c 'truncate 1g' $_logwrites
 
 	mkfs.xfs -f $_test > /dev/null 2>&1
 
@@ -221,6 +223,7 @@ runner_go()
 	export TEST_DIR=$me/test
 	export SCRATCH_DEV=$(_create_loop_device $_scratch)
 	export SCRATCH_MNT=$me/scratch
+	export LOGWRITES_DEV=$(_create_loop_device $_logwrites)
 	export FSTYP=xfs
 	export RESULT_BASE=$_results
 
@@ -241,6 +244,7 @@ runner_go()
 	umount -R $SCRATCH_MNT 2> /dev/null
 	_destroy_loop_device $TEST_DEV
 	_destroy_loop_device $SCRATCH_DEV
+	_destroy_loop_device $LOGWRITES_DEV
 
 	grep -q Failures: $me/log
 	if [ $? -eq 0 ]; then
-- 
2.45.2

[PATCH 5/5] check-parallel: allow FSTYP selection from the CLI

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

Add a CLI option to specify the initial FSTYP to test. If this is
not specified the the default of "xfs" will be used. This option is
different to the way check has FSTYP specified as check-parallel
has no infrastructure to support non block device based filesystems
and hence we have to reject virtual or network based filesysetms
are this point in time.

This patch only implements default mkfs parameter support for the
test device. Config sections can be used to override this as check
will then format the test device when the section that defines
non-default test device mkfs options is selected.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 check-parallel | 44 +++++++++++++++++++++++++++++++++++++-------
 1 file changed, 37 insertions(+), 7 deletions(-)

diff --git a/check-parallel b/check-parallel
index 498b53c5a..38fa4f7f9 100755
--- a/check-parallel
+++ b/check-parallel
@@ -18,7 +18,7 @@ run_section=""
 
 tmp=/tmp/check-parallel.$$
 
-export FSTYP=xfs
+FSTYP=
 
 # We need to include the test list processing first as argument parsing
 # requires test list parsing and setup.
@@ -35,6 +35,7 @@ check options
     -r			randomize test order
     --exact-order	run tests in the exact order specified
     -s section		run only specified section from config file
+    -f <FSTYPE>		specify the filesystem type to test
 
 testlist options
     -g group[,group...]	include tests from these groups
@@ -66,16 +67,39 @@ external_file argument is a path to a single file containing a list of tests
 to exclude in the form of <test dir>/<test name>.
 
 examples:
- check-parallel -D /mnt xfs/001
- check-parallel -D /mnt -g quick
+ check-parallel -f xfs -D /mnt xfs/001
+ check-parallel -f ext4 -D /mnt -g quick
  check-parallel -D /mnt -g xfs/quick
  check-parallel -D /mnt -x stress xfs/*
- check-parallel -D /mnt -X .exclude -g auto
- check-parallel -D /mnt -E ~/.xfstests.exclude
+ check-parallel -f btrfs -D /mnt -X .exclude -g auto
+ check-parallel -f udf -D /mnt -E ~/.xfstests.exclude
 '
 	    exit 1
 }
 
+# Only support block device based filesystems with generic mkfs support
+# at the moment.
+is_supported_fstype()
+{
+	local fstype=$1
+
+	case $fstype in
+	xfs)		;;
+	ext2|ext3|ext4)	;;
+	udf)		;;
+	jfs)		;;
+	f2fs)		;;
+	btrfs)		;;
+	bcachefs)	;;
+	gfs2)		;;
+	ocfs2)		;;
+	*)
+		echo "unsupported FSTYPE: $fstype"
+		usage
+		;;
+	esac
+}
+
 # Process command arguments first.
 while [ $# -gt 0 ]; do
 	case "$1" in
@@ -91,6 +115,8 @@ while [ $# -gt 0 ]; do
 	--exact-order) _tl_setup_ordered ;;
 	-n)	show_test_list="yes" ;;
 
+	-f)	is_supported_fstype $2 ; FSTYP=$2; shift ;;
+
 	-s)	run_section="$run_section -s $2"; shift ;;
 
 	-*)	usage ;;
@@ -108,6 +134,8 @@ while [ $# -gt 0 ]; do
 	shift
 done
 
+export FSTYP=${FSTYP:=xfs}
+
 if [ ! -d "$basedir" ]; then
 	echo "Invalid basedir specification"
 	usage
@@ -217,8 +245,6 @@ runner_go()
 	xfs_io -f -c 'truncate 8g' $_scratch
 	xfs_io -f -c 'truncate 1g' $_logwrites
 
-	mkfs.xfs -f $_test > /dev/null 2>&1
-
 	export TEST_DEV=$(_create_loop_device $_test)
 	export TEST_DIR=$me/test
 	export SCRATCH_DEV=$(_create_loop_device $_scratch)
@@ -232,6 +258,10 @@ runner_go()
 	mkdir -p $RESULT_BASE
 	rm -f $RESULT_BASE/check.*
 
+	# Only supports default mkfs parameters right now
+	wipefs $TEST_DEV > /dev/null 2>&1
+	yes | mkfs -t $FSTYP $TEST_DEV > /dev/null 2>&1
+
 #	export DUMP_CORRUPT_FS=1
 
 	# Run the tests in it's own mount namespace, as per the comment below
-- 
2.45.2

Re: [PATCH 0/5]: CLI and feature improvements for check-parallel

[Darrick J. Wong]

On Wed, Jan 15, 2025 at 04:51:11PM +1100, Dave Chinner wrote:
> Hi folks,
> 
> This patchset brings some new functionality to check-parallel to
> make it easier to run across different machines and filesystems.
> The patch seti adds these features:
> 
> - auto-adjusts concurrency for the machine it is running on.
> - adds CLI support of selection of tests. These CLI parameters are
>   identical to the check CLI parameters and the test lists are built
>   using the same code.
> - adds support for logwrites devices so that all the test that use
>   dm-logwrites are now enabled
> - adds support for specifying the initial filesystem type to test
>   on the CLI. This makes it easy to select xfs, btrfs, ext4, etc as
>   the target filesystem type that is to be tested.
> - Only block device based filesystems can be used with
>   check-parallel, and this is now enforced at FSTYP selection time.

Can you please add a --help so that curious users don't have to go find
the calling conventions by reading the bash? :)

--D

> For example, testing the rw group on ext4 is now a simple matter of
> adding the "-f ext4" parameter to the command line like so:
> 
> $ time sudo ./check-parallel -D /mnt/xfs -f ext4 -g rw -x dump
> Runner 12 Failures:  ext4/308
> Runner 1 Failures:  generic/095
> Runner 21 Failures:  generic/042
> Runner 51 Failures:  generic/627
> Runner 8 Failures:  generic/032
> Runner 13 Failures:  generic/019
> Runner 34 Failures:  generic/347
> .....
> 
> Options like exclude files, groups and lists also work natively in
> check-parallel now, same as they do in check...
> 
> -Dave.
> 

Re: [PATCH 0/5]: CLI and feature improvements for check-parallel

[Darrick J. Wong]

On Tue, Jan 14, 2025 at 10:29:16PM -0800, Darrick J. Wong wrote:
> On Wed, Jan 15, 2025 at 04:51:11PM +1100, Dave Chinner wrote:
> > Hi folks,
> > 
> > This patchset brings some new functionality to check-parallel to
> > make it easier to run across different machines and filesystems.
> > The patch seti adds these features:
> > 
> > - auto-adjusts concurrency for the machine it is running on.
> > - adds CLI support of selection of tests. These CLI parameters are
> >   identical to the check CLI parameters and the test lists are built
> >   using the same code.
> > - adds support for logwrites devices so that all the test that use
> >   dm-logwrites are now enabled
> > - adds support for specifying the initial filesystem type to test
> >   on the CLI. This makes it easy to select xfs, btrfs, ext4, etc as
> >   the target filesystem type that is to be tested.
> > - Only block device based filesystems can be used with
> >   check-parallel, and this is now enforced at FSTYP selection time.
> 
> Can you please add a --help so that curious users don't have to go find
> the calling conventions by reading the bash? :)

Oh, there is one in patch 3, please disregard this message.

--D

> --D
> 
> > For example, testing the rw group on ext4 is now a simple matter of
> > adding the "-f ext4" parameter to the command line like so:
> > 
> > $ time sudo ./check-parallel -D /mnt/xfs -f ext4 -g rw -x dump
> > Runner 12 Failures:  ext4/308
> > Runner 1 Failures:  generic/095
> > Runner 21 Failures:  generic/042
> > Runner 51 Failures:  generic/627
> > Runner 8 Failures:  generic/032
> > Runner 13 Failures:  generic/019
> > Runner 34 Failures:  generic/347
> > .....
> > 
> > Options like exclude files, groups and lists also work natively in
> > check-parallel now, same as they do in check...
> > 
> > -Dave.
> > 
> 

Re: [PATCH 0/5]: CLI and feature improvements for check-parallel

[Dave Chinner]

On Tue, Jan 14, 2025 at 10:29:16PM -0800, Darrick J. Wong wrote:
> On Wed, Jan 15, 2025 at 04:51:11PM +1100, Dave Chinner wrote:
> > Hi folks,
> > 
> > This patchset brings some new functionality to check-parallel to
> > make it easier to run across different machines and filesystems.
> > The patch seti adds these features:
> > 
> > - auto-adjusts concurrency for the machine it is running on.
> > - adds CLI support of selection of tests. These CLI parameters are
> >   identical to the check CLI parameters and the test lists are built
> >   using the same code.
> > - adds support for logwrites devices so that all the test that use
> >   dm-logwrites are now enabled
> > - adds support for specifying the initial filesystem type to test
> >   on the CLI. This makes it easy to select xfs, btrfs, ext4, etc as
> >   the target filesystem type that is to be tested.
> > - Only block device based filesystems can be used with
> >   check-parallel, and this is now enforced at FSTYP selection time.
> 
> Can you please add a --help so that curious users don't have to go find
> the calling conventions by reading the bash? :)

Already implemented:

$ ./check-parallel --help
Usage: ./check-parallel [options] [testlist]

check options
    -D <dir>            Directory to run in
    -n                  Output test list, do not run tests
    -r                  randomize test order
    --exact-order       run tests in the exact order specified
    -s section          run only specified section from config file
    -f <FSTYPE>         specify the filesystem type to test

testlist options
    -g group[,group...] include tests from these groups
    -x group[,group...] exclude tests from these groups
    -X exclude_file     exclude individual tests
    -e testlist         exclude a specific list of tests
    -E external_file    exclude individual tests
    [testlist]          include tests matching names in testlist

testlist argument is a list of tests in the form of <test dir>/<test name>.

<test dir> is a directory under tests that contains a group file,
with a list of the names of the tests in that directory.

<test name> may be either a specific test file name (e.g. xfs/001) or
a test file name match pattern (e.g. xfs/*).

group argument is either a name of a tests group to collect from all
the test dirs (e.g. quick) or a name of a tests group to collect from
a specific tests dir in the form of <test dir>/<group name> (e.g. xfs/quick).
If you want to run all the tests in the test suite, use "-g all" to specify all
groups.

exclude_file argument refers to a name of a file inside each test directory.
for every test dir where this file is found, the listed test names are
excluded from the list of tests to run from that test dir.

external_file argument is a path to a single file containing a list of tests
to exclude in the form of <test dir>/<test name>.

examples:
 check-parallel -f xfs -D /mnt xfs/001
 check-parallel -f ext4 -D /mnt -g quick
 check-parallel -D /mnt -g xfs/quick
 check-parallel -D /mnt -x stress xfs/*
 check-parallel -f btrfs -D /mnt -X .exclude -g auto
 check-parallel -f udf -D /mnt -E ~/.xfstests.exclude
$

-Dave.

-- 
Dave Chinner
david@fromorbit.com