[RFC 0/5] Add support for central fsconfig and -q <n> unconditional loop

[RFC 0/5] Add support for central fsconfig and -q <n> unconditional loop

[Nirjhar Roy (IBM)]

This series introduces 2 new features, central fsconfig support and "-q <n>"
option to support unconditional looping. Along with this, it also brings
some improvements to output data both, in stdout and in result.log.
Central fsconfig feature allows us to define MKFS and MOUNT options for
different filesystems and use this central config to invoke any test.
We don't need to maintain different sections in local.config file for
different mkfs and mount options of any given FS.
e.g.
./check -c configs/xfs/4k,configs/ext4/1k -g quick

Central fsconfigs can be useful for below reasons:-
1. Testers and other FS developers can know what is being actively
tested and maintained by FS Maintainers.
2. It "reduces" the need to maintain different local.config files or
different sections within local.config file for subsystem maintainers
for testing different filesystem configurations.
-q <n> option can be useful for fast unconditional looping (compared to
-I <n>) to test a certain flakey test and get it's pass/fail metrics.

This is similar to -L <n>, although with -L the test only loops if it
fails the 1st time.
The individual patches can provide more details about other changes and
improvements.

This is something that was discussed during last LSFMM [1]
[1] https://lore.kernel.org/all/87h6h4sopf.fsf@doe.com/

Nirjhar Roy (IBM) (5):
  tests/selftest: Add a new pseudo flaky test.
  check: Add -q <n> option to support unconditional looping.
  check: Improve pass/fail metrics and section config output
  check,common/config: Add support for central fsconfig
  configs: Add a couple of xfs and ext4 configuration templates.

 README.config-sections |  12 +++
 check                  | 193 +++++++++++++++++++++++++++++++----------
 common/config          |  51 ++++++++++-
 common/report          |   2 +-
 configs/ext4/1k        |   3 +
 configs/ext4/4k        |   3 +
 configs/ext4/64k       |   3 +
 configs/xfs/1k         |   3 +
 configs/xfs/4k         |   3 +
 configs/xfs/64k        |   3 +
 configs/xfs/adv        |   3 +
 configs/xfs/quota      |   3 +
 tests/selftest/007     |  21 +++++
 tests/selftest/007.out |   2 +
 14 files changed, 256 insertions(+), 49 deletions(-)
 create mode 100644 configs/ext4/1k
 create mode 100644 configs/ext4/4k
 create mode 100644 configs/ext4/64k
 create mode 100644 configs/xfs/1k
 create mode 100644 configs/xfs/4k
 create mode 100644 configs/xfs/64k
 create mode 100644 configs/xfs/adv
 create mode 100644 configs/xfs/quota
 create mode 100755 tests/selftest/007
 create mode 100644 tests/selftest/007.out

--
2.34.1

[RFC 1/5] tests/selftest: Add a new pseudo flaky test.

[Nirjhar Roy (IBM)]

This test is to simulate the behavior of a flaky test. This will be required
when we will make some modifications to the pass/fail metric calculation of
the test infrastructure where we will need a test with non-zero pass
and non-zero failure rate.

Reviewed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Nirjhar Roy (IBM) <nirjhar.roy.lists@gmail.com>
---
 tests/selftest/007     | 21 +++++++++++++++++++++
 tests/selftest/007.out |  2 ++
 2 files changed, 23 insertions(+)
 create mode 100755 tests/selftest/007
 create mode 100644 tests/selftest/007.out

diff --git a/tests/selftest/007 b/tests/selftest/007
new file mode 100755
index 00000000..f100ec5f
--- /dev/null
+++ b/tests/selftest/007
@@ -0,0 +1,21 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 IBM Corporation.  All Rights Reserved.
+# Author: Nirjhar Roy (IBM) <nirjhar.roy.lists@gmail.com>
+#
+# FS QA Test 007
+#
+# This test is to simulate the behavior of a flakey test.
+#
+
+. ./common/preamble
+_begin_fstest selftest
+
+if (($RANDOM % 2)); then
+    echo "Silence is golden"
+else
+    echo "Silence is flakey"
+fi
+
+status=0
+exit
diff --git a/tests/selftest/007.out b/tests/selftest/007.out
new file mode 100644
index 00000000..fd3590e6
--- /dev/null
+++ b/tests/selftest/007.out
@@ -0,0 +1,2 @@
+QA output created by 007
+Silence is golden
-- 
2.34.1

Re: [RFC 1/5] tests/selftest: Add a new pseudo flaky test.

[Darrick J. Wong]

On Fri, Jan 10, 2025 at 09:10:25AM +0000, Nirjhar Roy (IBM) wrote:
> This test is to simulate the behavior of a flaky test. This will be required
> when we will make some modifications to the pass/fail metric calculation of
> the test infrastructure where we will need a test with non-zero pass
> and non-zero failure rate.
> 
> Reviewed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
> Signed-off-by: Nirjhar Roy (IBM) <nirjhar.roy.lists@gmail.com>
> ---
>  tests/selftest/007     | 21 +++++++++++++++++++++
>  tests/selftest/007.out |  2 ++
>  2 files changed, 23 insertions(+)
>  create mode 100755 tests/selftest/007
>  create mode 100644 tests/selftest/007.out
> 
> diff --git a/tests/selftest/007 b/tests/selftest/007
> new file mode 100755
> index 00000000..f100ec5f
> --- /dev/null
> +++ b/tests/selftest/007
> @@ -0,0 +1,21 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2024 IBM Corporation.  All Rights Reserved.
> +# Author: Nirjhar Roy (IBM) <nirjhar.roy.lists@gmail.com>
> +#
> +# FS QA Test 007
> +#
> +# This test is to simulate the behavior of a flakey test.
> +#
> +
> +. ./common/preamble
> +_begin_fstest selftest
> +
> +if (($RANDOM % 2)); then
> +    echo "Silence is golden"
> +else
> +    echo "Silence is flakey"

Oh is it now? ;)

Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>

--D

> +fi
> +
> +status=0
> +exit
> diff --git a/tests/selftest/007.out b/tests/selftest/007.out
> new file mode 100644
> index 00000000..fd3590e6
> --- /dev/null
> +++ b/tests/selftest/007.out
> @@ -0,0 +1,2 @@
> +QA output created by 007
> +Silence is golden
> -- 
> 2.34.1
> 
> 

[RFC 2/5] check: Add -q <n> option to support unconditional looping.

[Nirjhar Roy (IBM)]

This patch adds -q <n> option through which one can run a given test <n>
times unconditionally. It also prints pass/fail metrics at the end.

The advantage of this over -L <n> and -i/-I <n> is that:
    a. -L <n> will not re-run a flakey test if the test passes for the first time.
    b. -I/-i <n> sets up devices during each iteration and hence slower.
Note -q <n> will override -L <n>.

Also rename _stash_fail_loop_files() to _stash_loop_files()
because this function will now be used even when the test doesn't fail
(i.e. when ran with -q <n>).

Suggested-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Nirjhar Roy (IBM) <nirjhar.roy.lists@gmail.com>
Co-developed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
---
 check | 46 ++++++++++++++++++++++++++++++++++++----------
 1 file changed, 36 insertions(+), 10 deletions(-)

diff --git a/check b/check
index 607d2456..e8ac0ce9 100755
--- a/check
+++ b/check
@@ -11,6 +11,7 @@ needsum=true
 try=()
 sum_bad=0
 bad=()
+is_bad_test=false
 notrun=()
 interrupt=true
 diff="diff -u"
@@ -27,6 +28,7 @@ DUMP_OUTPUT=false
 iterations=1
 istop=false
 loop_on_fail=0
+loop_unconditional=0
 exclude_tests=()
 
 # This is a global variable used to pass test failure text to reporting gunk
@@ -83,6 +85,7 @@ check options
     -s section		run only specified section from config file
     -S section		exclude the specified section from the config file
     -L <n>		loop tests <n> times following a failure, measuring aggregate pass/fail metrics
+    -q <n>		loop tests <n> times irrespective of a pass or a failure, measuring aggregate pass/fail metrics
 
 testlist options
     -g group[,group...]	include tests from these groups
@@ -341,7 +344,9 @@ while [ $# -gt 0 ]; do
 	-L)	[[ $2 =~ ^[0-9]+$ ]] || usage
 		loop_on_fail=$2; shift
 		;;
-
+	-q)	[[ $2 =~ ^[0-9]+$ ]] || usage
+		loop_unconditional=$(($2 - 1)); shift
+		;;
 	-*)	usage ;;
 	*)	# not an argument, we've got tests now.
 		have_test_arg=true ;;
@@ -357,6 +362,11 @@ while [ $# -gt 0 ]; do
 	shift
 done
 
+# -q <n> overrides -L <m>
+if [ "$loop_unconditional" -gt 0 ]; then
+	loop_on_fail=0
+fi
+
 # we need common/rc, that also sources common/config. We need to source it
 # after processing args, overlay needs FSTYP set before sourcing common/config
 if ! . ./common/rc; then
@@ -609,7 +619,7 @@ _expunge_test()
 }
 
 # retain files which would be overwritten in subsequent reruns of the same test
-_stash_fail_loop_files() {
+_stash_loop_files() {
 	local seq_prefix="${REPORT_DIR}/${1}"
 	local cp_suffix="$2"
 
@@ -633,10 +643,18 @@ _stash_test_status() {
 	fi
 
 	if ((${#loop_status[*]} > 0)); then
-		# continuing or completing rerun-on-failure loop
-		_stash_fail_loop_files "$test_seq" ".rerun${#loop_status[*]}"
+		# continuing or completing rerun loop
+		_stash_loop_files "$test_seq" ".rerun${#loop_status[*]}"
 		loop_status+=("$test_status")
-		if ((${#loop_status[*]} > loop_on_fail)); then
+
+		# only stash @bad result for initial failure in loop
+		if [[ "$test_status" == "fail" ]] && ! $is_bad_test; then
+			bad+=("$test_seq")
+			is_bad_test=true
+		fi
+
+		if ((loop_on_fail && ${#loop_status[*]} > loop_on_fail)) || \
+		   ((loop_unconditional && ${#loop_status[*]} > loop_unconditional)); then
 			printf "%s aggregate results across %d runs: " \
 				"$test_seq" "${#loop_status[*]}"
 			awk "BEGIN {
@@ -650,23 +668,30 @@ _stash_test_status() {
 				}'
 			echo
 			loop_status=()
+			is_bad_test=false
 		fi
-		return	# only stash @bad result for initial failure in loop
+		return
 	fi
 
 	case "$test_status" in
 	fail)
-		if ((loop_on_fail > 0)); then
-			# initial failure, start rerun-on-failure loop
-			_stash_fail_loop_files "$test_seq" ".rerun0"
+		# re-run if either of the loop argument is set
+		if ((loop_on_fail > 0)) || ((loop_unconditional > 0)); then
+			_stash_loop_files "$test_seq" ".rerun0"
 			loop_status+=("$test_status")
 		fi
 		bad+=("$test_seq")
+		is_bad_test=true
 		;;
 	list|notrun)
 		notrun+=("$test_seq")
 		;;
 	pass|expunge)
+		# re-run if loop_unconditional argument is set
+		if ((loop_unconditional > 0)); then
+			_stash_loop_files "$test_seq" ".rerun0"
+			loop_status+=("$test_status")
+		fi
 		;;
 	*)
 		echo "Unexpected test $test_seq status: $test_status"
@@ -857,7 +882,8 @@ function run_section()
 	seqres="$check"
 	_check_test_fs
 
-	loop_status=()	# track rerun-on-failure state
+	is_bad_test=false
+	loop_status=()	# track loop rerun state
 	local tc_status ix
 	local -a _list=( $list )
 	for ((ix = 0; ix < ${#_list[*]}; !${#loop_status[*]} && ix++)); do
-- 
2.34.1

Re: [RFC 2/5] check: Add -q <n> option to support unconditional looping.

[Theodore Ts'o]

On Fri, Jan 10, 2025 at 09:10:26AM +0000, Nirjhar Roy (IBM) wrote:
> This patch adds -q <n> option through which one can run a given test <n>
> times unconditionally. It also prints pass/fail metrics at the end.
> 
> The advantage of this over -L <n> and -i/-I <n> is that:
>     a. -L <n> will not re-run a flakey test if the test passes for the first time.
>     b. -I/-i <n> sets up devices during each iteration and hence slower.
> Note -q <n> will override -L <n>.

This is great!  It's something that I've wanted for a while, since at
the moment I implement {gce,kvm}-xfstests -C 10 is to run check ten
times, and doing something which does the looping inside check instead
of outside will be much more efficient.

One other thing that has been on my todo list to update, but which
perhaps you might be willing to do while you are doing work in this
area (nudge, nudge :-), is an optional mode which interates but which
stops once a test fails.  This is essentially the reverse of -L, and
the reason why it's useful is when trying to bisect a flakey test,
which sometimes might only be failing 2-5% of the time, require
running a test 30-50 times.  But the moment the test fails, we don't
need to run the test any more, so this would speed up bisection tests
which today I do via:

   gce-xfstests ltm --repo linux.git --bisect-good v6.12 --bisect-bad \
	v6.13-rc1 -C 50 -c ext4/inline_data generic/273

Because of this, I wonder if we should have one option to specify the
number of interations, and then a different option which specifies the
iteration mode, which might be "unconditional", "until first failure",
"only if the test initially fails", etc, instead of separate options
for -q, -L, etc.

Thanks,

					- Ted

Re: [RFC 2/5] check: Add -q <n> option to support unconditional looping.

[Ritesh Harjani]

"Theodore Ts'o" <tytso@mit.edu> writes:

> On Fri, Jan 10, 2025 at 09:10:26AM +0000, Nirjhar Roy (IBM) wrote:
>> This patch adds -q <n> option through which one can run a given test <n>
>> times unconditionally. It also prints pass/fail metrics at the end.
>> 
>> The advantage of this over -L <n> and -i/-I <n> is that:
>>     a. -L <n> will not re-run a flakey test if the test passes for the first time.
>>     b. -I/-i <n> sets up devices during each iteration and hence slower.
>> Note -q <n> will override -L <n>.
>
> This is great!  It's something that I've wanted for a while, since at
> the moment I implement {gce,kvm}-xfstests -C 10 is to run check ten
> times, and doing something which does the looping inside check instead
> of outside will be much more efficient.

Yup "-q 10" could be used which can give pass/fail metrics of how many
times the test passed v/s failed by doing unconditional looping within
xfstest's check script itself. 

>
> One other thing that has been on my todo list to update, but which
> perhaps you might be willing to do while you are doing work in this
> area (nudge, nudge :-), is an optional mode which interates but which
> stops once a test fails.  This is essentially the reverse of -L, and
> the reason why it's useful is when trying to bisect a flakey test,
> which sometimes might only be failing 2-5% of the time, require
> running a test 30-50 times.  But the moment the test fails, we don't
> need to run the test any more, so this would speed up bisection tests
> which today I do via:
>
>    gce-xfstests ltm --repo linux.git --bisect-good v6.12 --bisect-bad \
> 	v6.13-rc1 -C 50 -c ext4/inline_data generic/273

There is -I which stops iterating on encountering a failure.
Does that work for this usecase?

>
> Because of this, I wonder if we should have one option to specify the
> number of interations, and then a different option which specifies the
> iteration mode, which might be "unconditional", "until first failure",
> "only if the test initially fails", etc, instead of separate options
> for -q, -L, etc.

I like the idea of iteration mode here. I will let others determine on
how easy it is to kill any option in xfstests today and replace it with
other. However here is a summary of different iteration and looping
options.

We have 3 options in xfstests today:
1. -i <n>              iterate the test list <n> times
2. -I <n>              iterate the test list <n> times, but stops iterating further in case of any test failure
3. -L <n>              loop tests <n> times following a failure, measuring aggregate pass/fail metrics

So we have -i/-I which are iterations and -q/-L which are loops. 
Looping happens when we can just loop over a particular test <n> times
and give pass/fail metrics.
Whereas in case of iterations it goes over from the beginning which will
also source different rc/config files and prepares the device etc. (hence
it's a bit slow too)

Using -q will be faster over using -i or -I similar to how -L is faster. However -L
will only re-run when there is a failure the 1st time.

-q v/s -l: Can we kill current -l option if it is not in use by anyone?
I would prefer -l since it looks a short form for looping. I don't see
we using -l anymore. But I will let others comment. 
    -l                  line mode diff


-ritesh

[RFC 3/5] check: Improve pass/fail metrics and section config output

[Nirjhar Roy (IBM)]

This patch improves the output by:
- Adding pass/fail aggregate results in the stdout for each section's
  summary. Also adds the location of the xunit xml files in the
  section's summary.
- Adds section config details into the result's global log file.

e.g.

$ ./check -q 2 -R xunit-quiet   selftest/001

SECTION       -- s1
=========================
selftest/001 aggregate results across 2 runs: pass=2 (100.0%)
Ran: selftest/001
Passed all 1 tests
Xunit report: /home/ubuntu/xfstests/results//s1/result.xml

SECTION       -- s2
=========================
selftest/001 aggregate results across 2 runs: pass=2 (100.0%)
Ran: selftest/001
Passed all 1 tests
Xunit report: /home/ubuntu/xfstests/results//s2/result.xml

Output in results/check.log file

Kernel version: 6.13.0-rc1-00182-gb8f52214c61a-dirty
Wed Jan  8 11:23:57 UTC 2025
SECTION       -- s1
=========================
FSTYP         -- ext4
PLATFORM      -- Linux/x86_64 citest-1 6.13.0-rc1-00182-gb8f52214c61a-dirty #5 SMP PREEMPT_DYNAMIC Wed Dec 18 14:03:34 IST 2024
MKFS_OPTIONS  -- -F /dev/loop1
MOUNT_OPTIONS -- -o acl,user_xattr /dev/loop1 /mnt1/scratch
selftest/001 aggregate results across 2 runs: pass=2 (100.0%)
Ran: selftest/001
Passed all 1 tests
Xunit report: /home/ubuntu/xfstests/results//s1/result.xml

Kernel version: 6.13.0-rc1-00182-gb8f52214c61a-dirty
Wed Jan  8 11:24:02 UTC 2025
SECTION       -- s2
=========================
FSTYP         -- ext4
PLATFORM      -- Linux/x86_64 citest-1 6.13.0-rc1-00182-gb8f52214c61a-dirty #5 SMP PREEMPT_DYNAMIC Wed Dec 18 14:03:34 IST 2024
MKFS_OPTIONS  -- -F /dev/loop1
MOUNT_OPTIONS -- -o acl,user_xattr /dev/loop1 /mnt1/scratch
selftest/001 aggregate results across 2 runs: pass=2 (100.0%)
Ran: selftest/001
Passed all 1 tests
Xunit report: /home/ubuntu/xfstests/results//s2/result.xml

Suggested-by: Ritesh Harjani <ritesh.list@gmail.com>
Co-developed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Nirjhar Roy (IBM) <nirjhar.roy.lists@gmail.com>
---
 check | 67 ++++++++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 46 insertions(+), 21 deletions(-)

diff --git a/check b/check
index e8ac0ce9..f7028836 100755
--- a/check
+++ b/check
@@ -468,6 +468,25 @@ if [ -n "$REPORT_GCOV" ]; then
 	_gcov_check_report_gcov
 fi
 
+_print_list()
+{
+	local -n list=$1
+	local item
+	for item in "${list[@]}"; do
+		echo "$item"
+	done
+}
+
+_display_test_configuration()
+{
+	echo "FSTYP         -- `_full_fstyp_details`"
+	echo "PLATFORM      -- `_full_platform_details`"
+	if [ ! -z "$SCRATCH_DEV" ]; then
+		echo "MKFS_OPTIONS  -- `_scratch_mkfs_options`"
+		echo "MOUNT_OPTIONS -- `_scratch_mount_options`"
+	fi
+}
+
 _wrapup()
 {
 	seq="check.$$"
@@ -505,11 +524,19 @@ _wrapup()
 
 		echo "SECTION       -- $section" >>$tmp.summary
 		echo "=========================" >>$tmp.summary
+
+		_global_log "SECTION       -- $section"
+		_global_log "=========================" >>$tmp.summary
+		_global_log "$(_display_test_configuration)"
 		if ((${#try[*]} > 0)); then
+			local test_aggr_stats=$(_print_list loop_test_stats_per_section)
 			if [ $brief_test_summary == "false" ]; then
+				[[ -n "$test_aggr_stats" ]] && echo "$test_aggr_stats" >> \
+					$tmp.summary
 				echo "Ran: ${try[*]}"
 				echo "Ran: ${try[*]}" >>$tmp.summary
 			fi
+			_global_log "$test_aggr_stats"
 			_global_log "Ran: ${try[*]}"
 		fi
 
@@ -539,12 +566,15 @@ _wrapup()
 			_global_log "Passed all ${#try[*]} tests"
 			echo "Passed all ${#try[*]} tests" >>$tmp.summary
 		fi
-		echo "" >>$tmp.summary
 		if $do_report; then
 			_make_section_report "$section" "${#try[*]}" \
 					     "${#bad[*]}" "${#notrun[*]}" \
 					     "$((sect_stop - sect_start))"
+			local out_fn="$REPORT_DIR/result.xml"
+			echo "Xunit report: $out_fn" >> $tmp.summary
+			_global_log "Xunit report: $out_fn"
 		fi
+		echo "" >>$tmp.summary
 
 		# Generate code coverage report
 		if [ -n "$REPORT_GCOV" ]; then
@@ -655,18 +685,19 @@ _stash_test_status() {
 
 		if ((loop_on_fail && ${#loop_status[*]} > loop_on_fail)) || \
 		   ((loop_unconditional && ${#loop_status[*]} > loop_unconditional)); then
-			printf "%s aggregate results across %d runs: " \
-				"$test_seq" "${#loop_status[*]}"
-			awk "BEGIN {
-				n=split(\"${loop_status[*]}\", arr);"'
-				for (i = 1; i <= n; i++)
-					stats[arr[i]]++;
-				for (x in stats)
-					printf("%s=%d (%.1f%%)",
-					       (i-- > n ? x : ", " x),
-					       stats[x], 100 * stats[x] / n);
-				}'
-			echo
+			local test_stats=$(printf "%s aggregate results across %d runs: " \
+				"$test_seq" "${#loop_status[*]}")
+			test_stats+=$(awk "BEGIN {
+					n=split(\"${loop_status[*]}\", arr);"'
+					for (i = 1; i <= n; i++)
+						stats[arr[i]]++;
+					for (x in stats)
+						printf("%s=%d (%.1f%%)",
+							   (i-- > n ? x : ", " x),
+							   stats[x], 100 * stats[x] / n);
+					}')
+			echo "$test_stats"
+			loop_test_stats_per_section+=("$test_stats")
 			loop_status=()
 			is_bad_test=false
 		fi
@@ -839,14 +870,7 @@ function run_section()
 	rm -f $check.full
 
 	[ -f $check.time ] || touch $check.time
-
-	# print out our test configuration
-	echo "FSTYP         -- `_full_fstyp_details`"
-	echo "PLATFORM      -- `_full_platform_details`"
-	if [ ! -z "$SCRATCH_DEV" ]; then
-	  echo "MKFS_OPTIONS  -- `_scratch_mkfs_options`"
-	  echo "MOUNT_OPTIONS -- `_scratch_mount_options`"
-	fi
+	_display_test_configuration
 	echo
 	test -n "$REPORT_GCOV" && _gcov_reset
 	needwrap=true
@@ -884,6 +908,7 @@ function run_section()
 
 	is_bad_test=false
 	loop_status=()	# track loop rerun state
+	loop_test_stats_per_section=() # store loop test statistics per section
 	local tc_status ix
 	local -a _list=( $list )
 	for ((ix = 0; ix < ${#_list[*]}; !${#loop_status[*]} && ix++)); do
-- 
2.34.1

[RFC 4/5] check,common/config: Add support for central fsconfig

[Nirjhar Roy (IBM)]

This adds support to pick and use any existing FS config from
configs/<fstype>/<config>. e.g.

configs/xfs/1k
configs/xfs/4k
configs/ext4/4k
configs/ext4/64k

This should help us maintain and test different fs test
configurations from a central place. We also hope that
this will be useful for both developers and testers to
look into what is being actively maintained and tested
by FS Maintainers.

When we will have fsconfigs set, then will be another subdirectory created
in results/<section>. For example let's look at the following:

The directory tree structure on running
sudo ./check -q 2 -R xunit-quiet -c xfs/4k,configs/xfs/1k selftest/001 selftest/007

$ tree results/
results/
├── check.full
├── check.log
├── check.time
├── xfs_1
│   └── xfs
│       ├── 1k
│       │   ├── check.log
│       │   ├── check.time
│       │   ├── result.xml
│       │   └── selftest
│       │       ├── 001.full
│       │       ├── 001.full.rerun0
│       │       ├── 001.full.rerun1
│       │       ├── 007.full
│       │       ├── 007.full.rerun0
│       │       ├── 007.full.rerun1
│       │       ├── 007.out.bad
│       │       ├── 007.out.bad.rerun0
│       │       ├── 007.out.bad.rerun1
│       │       └── dmesg_filter
│       └── 4k
│           ├── check.log
│           ├── check.time
│           ├── result.xml
│           └── selftest
│               ├── 001.full
│               ├── 001.full.rerun0
│               ├── 001.full.rerun1
│               ├── 007.full
│               ├── 007.full.rerun0
│               ├── 007.full.rerun1
│               ├── 007.out.bad
│               ├── 007.out.bad.rerun0
│               ├── 007.out.bad.rerun1
│               └── dmesg_filter
└── xfs_2
    └── xfs
        ├── 1k
        │   ├── check.log
        │   ├── check.time
        │   ├── result.xml
        │   └── selftest
        │       ├── 001.full
        │       ├── 001.full.rerun0
        │       ├── 001.full.rerun1
        │       ├── 007.full
        │       ├── 007.full.rerun0
        │       ├── 007.full.rerun1
        │       └── dmesg_filter
        └── 4k
            ├── check.log
            ├── check.time
            ├── result.xml
            └── selftest
                ├── 001.full
                ├── 001.full.rerun0
                ├── 001.full.rerun1
                ├── 007.full
                ├── 007.full.rerun0
                ├── 007.full.rerun1
                ├── 007.out.bad
                ├── 007.out.bad.rerun1
                └── dmesg_filter

12 directories, 51 files

Contents of one of the xml/xunit files (results/xfs_1/xfs/4k/result.xml):
(Please note that FS_CONFIG_OPTION is a new property that is added in this patch)

<?xml version="1.0" encoding="UTF-8"?>
<testsuite
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/tree/doc/xunit.xsd"

 name="xfstests"
 failures="1" skipped="0" tests="2" time="6"
 hostname="li-5d80d4cc-2782-11b2-a85c-bed59fe4c9e5"
  start_timestamp="2024-12-11T13:07:29+05:30"
        timestamp="2024-12-11T13:07:35+05:30"
 report_timestamp="2024-12-11T13:07:35+05:30"
>
	<properties>
		<property name="ARCH" value="x86_64"/>
                ...
		<property name="FSTYP" value="xfs"/>
		<property name="FS_CONFIG_OPTION" value="xfs/4k"/>
		<property name="HOST_OPTIONS" value="local.config"/>
                ...
	</properties>
	<testcase classname="xfstests.xfs_1" name="selftest/001" time="0">
	</testcase>
	<testcase classname="xfstests.xfs_1" name="selftest/001" time="1">
	</testcase>
	<testcase classname="xfstests.xfs_1" name="selftest/007" time="1">
		<failure message="- output mismatch (see /home/nirjhar/work/xfstests-dev/results//xfs_1/xfs/4k/selftest/007.out.bad)" type="TestFail" />
	</testcase>
	<testcase classname="xfstests.xfs_1" name="selftest/007" time="1">
		<failure message="- output mismatch (see /home/nirjhar/work/xfstests-dev/results//xfs_1/xfs/4k/selftest/007.out.bad)" type="TestFail" />
	</testcase>
</testsuite>

Suggested-by: Ritesh Harjani <ritesh.list@gmail.com>
Co-developed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Nirjhar Roy (IBM) <nirjhar.roy.lists@gmail.com>
---
 README.config-sections | 12 +++++++
 check                  | 82 +++++++++++++++++++++++++++++++++---------
 common/config          | 51 ++++++++++++++++++++++++--
 common/report          |  2 +-
 4 files changed, 128 insertions(+), 19 deletions(-)

diff --git a/README.config-sections b/README.config-sections
index a42d9d7b..0511b707 100644
--- a/README.config-sections
+++ b/README.config-sections
@@ -108,10 +108,22 @@ MKFS_OPTIONS="-q -F -b1024"
 [ext4_nojournal]
 MKFS_OPTIONS="-q -F -b4096 -O ^has_journal"
 
+# we could also do like: FS_CONFIG_OPTION=ext4/1k
+[ext4_1k_bs_fsconfig]
+FS_CONFIG_OPTION=configs/ext4/1k
+
+# we could also do like: FS_CONFIG_OPTION=ext4/nojournal
+[ext4_nojournal_fsconfig]
+FS_CONFIG_OPTION=configs/ext4/nojournal
+
 [xfs_filesystem]
 MKFS_OPTIONS="-f"
 FSTYP=xfs
 
+# we could also do like: FS_CONFIG_OPTION=xfs/4k
+[xfs_4k_bs_fsconfig]
+FS_CONFIG_OPTION=configs/xfs/4k
+
 [ext3_filesystem]
 FSTYP=ext3
 MOUNT_OPTIONS="-o noatime"
diff --git a/check b/check
index f7028836..90aa94ba 100755
--- a/check
+++ b/check
@@ -80,6 +80,7 @@ check options
     -I <n>		iterate the test list <n> times, but stops iterating further in case of any test failure
     -d			dump test output to stdout
     -b			brief test summary
+    -c			use fs configs from configs/<fstype>/<config> e.g. configs/xfs/4k
     -R fmt[,fmt]	generate report in formats specified. Supported formats: xunit, xunit-quiet
     --large-fs		optimise scratch device for large filesystems
     -s section		run only specified section from config file
@@ -116,6 +117,14 @@ excluded from the list of tests to run from that test dir.
 external_file argument is a path to a single file containing a list of tests
 to exclude in the form of <test dir>/<test name>.
 
+-c can be used in various formats as mentioned below. We can also set the
+FS_CONFIG_OPTION environment variable in the local.config file and get the
+same functionality. e.g.
+    1) FS_CONFIG_OPTION=xfs/4k
+    2) FS_CONFIG_OPTION=configs/xfs/4k
+Please note that cmdline will take precedence if FS_CONFIG_OPTION is
+passed in both local.config file and via cmdline.
+
 examples:
  check xfs/001
  check -g quick
@@ -123,6 +132,9 @@ examples:
  check -x stress xfs/*
  check -X .exclude -g auto
  check -E ~/.xfstests.exclude
+ check -c xfs/4k,ext4/1k
+ check -c xfs/4k -c configs/ext4/1k
+ check -c $(echo configs/xfs/* | tr ' ' ',') -q 5 tests/selftest/007
 '
 	    exit 1
 }
@@ -312,6 +324,8 @@ while [ $# -gt 0 ]; do
 		;;
 	-s)	RUN_SECTION="$RUN_SECTION $2"; shift ;;
 	-S)	EXCLUDE_SECTION="$EXCLUDE_SECTION $2"; shift ;;
+	-c)	CMD_FS_CONFIG_OPTION="$CMD_FS_CONFIG_OPTION$2,";
+		export OPTIONS_HAVE_FSCONFIGS=true; shift ;;
 	-l)	diff="diff" ;;
 	-udiff)	diff="$diff -u" ;;
 
@@ -458,7 +472,7 @@ _wipe_counters()
 
 _global_log() {
 	echo "$1" >> $check.log
-	if $OPTIONS_HAVE_SECTIONS; then
+	if options_have_sections_or_fs_configs; then
 		echo "$1" >> ${REPORT_DIR}/check.log
 	fi
 }
@@ -513,7 +527,7 @@ _wrapup()
 				}' \
 				| sort -n >$tmp.out
 			mv $tmp.out $check.time
-			if $OPTIONS_HAVE_SECTIONS; then
+			if options_have_sections_or_fs_configs; then
 				cp $check.time ${REPORT_DIR}/check.time
 			fi
 		fi
@@ -524,9 +538,14 @@ _wrapup()
 
 		echo "SECTION       -- $section" >>$tmp.summary
 		echo "=========================" >>$tmp.summary
-
 		_global_log "SECTION       -- $section"
 		_global_log "=========================" >>$tmp.summary
+
+		if [[ -n $FS_CONFIG_OPTION ]]; then
+			echo "FSCONFIG      -- $FS_CONFIG_OPTION" >> $tmp.summary
+			_global_log "FSCONFIG      -- $FS_CONFIG_OPTION"
+		fi
+
 		_global_log "$(_display_test_configuration)"
 		if ((${#try[*]} > 0)); then
 			local test_aggr_stats=$(_print_list loop_test_stats_per_section)
@@ -541,7 +560,7 @@ _wrapup()
 		fi
 
 		$interrupt && echo "Interrupted!" | tee -a $check.log
-		if $OPTIONS_HAVE_SECTIONS; then
+		if options_have_sections_or_fs_configs; then
 			$interrupt && echo "Interrupted!" | tee -a \
 				${REPORT_DIR}/check.log
 		fi
@@ -592,7 +611,7 @@ _wrapup()
 
 	sum_bad=`expr $sum_bad + ${#bad[*]}`
 	_wipe_counters
-	if ! $OPTIONS_HAVE_SECTIONS; then
+	if ! options_have_sections_or_fs_configs; then
 		rm -f $tmp.*
 	fi
 }
@@ -773,7 +792,7 @@ _detect_kmemleak
 _prepare_test_list
 fstests_start_time="$(date +"%F %T")"
 
-if $OPTIONS_HAVE_SECTIONS; then
+if options_have_sections_or_fs_configs; then
 	trap "_summary; exit \$status" 0 1 2 3 15
 else
 	trap "_wrapup; exit \$status" 0 1 2 3 15
@@ -782,6 +801,7 @@ fi
 function run_section()
 {
 	local section=$1 skip
+	local fsconfig=$2
 
 	OLD_FSTYP=$FSTYP
 	OLD_TEST_FS_MOUNT_OPTS=$TEST_FS_MOUNT_OPTS
@@ -814,7 +834,10 @@ function run_section()
 		fi
 	fi
 
-	get_next_config $section
+	# FS_CONFIG_OPTION (if provided) will only be set after get_next_config
+	# has run which will export the FS_CONFIG_OPTION, so that we can use it
+	# later
+	get_next_config $section $fsconfig
 	_canonicalize_devices
 
 	mkdir -p $RESULT_BASE
@@ -828,6 +851,10 @@ function run_section()
 		echo "SECTION       -- $section"
 	fi
 
+	if $OPTIONS_HAVE_FSCONFIGS; then
+		echo "FSCONFIG      -- $FS_CONFIG_OPTION"
+	fi
+
 	sect_start=`_wallclock`
 	if $RECREATE_TEST_DEV || [ "$OLD_FSTYP" != "$FSTYP" ]; then
 		echo "RECREATING    -- $FSTYP on $TEST_DEV"
@@ -849,7 +876,12 @@ function run_section()
 		# TEST_DEV has been recreated, previous FSTYP derived from
 		# TEST_DEV could be changed, source common/rc again with
 		# correct FSTYP to get FSTYP specific configs, e.g. common/xfs
+		# sourcing common/rc can re-set FS_CONFIG_OPTION to the value in
+		# local.config file(if present). So store and restore the value of
+		# FS_CONFIG_OPTION which is there currently at this point.
+		local fs_config_tmp="$FS_CONFIG_OPTION"
 		. common/rc
+		export FS_CONFIG_OPTION="$fs_config_tmp"
 		_prepare_test_list
 	elif [ "$OLD_TEST_FS_MOUNT_OPTS" != "$TEST_FS_MOUNT_OPTS" ]; then
 		_test_unmount 2> /dev/null
@@ -933,10 +965,14 @@ function run_section()
 		# we don't include the tests/ directory in the name output.
 		export seqnum=${seq#$SRC_DIR/}
 		group=${seqnum%%/*}
+
+		REPORT_DIR="$RESULT_BASE"
 		if $OPTIONS_HAVE_SECTIONS; then
-			REPORT_DIR="$RESULT_BASE/$section"
-		else
-			REPORT_DIR="$RESULT_BASE"
+			REPORT_DIR="$REPORT_DIR/$section"
+		fi
+		if $OPTIONS_HAVE_FSCONFIGS; then
+			local fsconfig_subdir="${FS_CONFIG_OPTION//configs/''}"
+			REPORT_DIR="$REPORT_DIR/$fsconfig_subdir"
 		fi
 		export RESULT_DIR="$REPORT_DIR/$group"
 		seqres="$REPORT_DIR/$seqnum"
@@ -1151,14 +1187,28 @@ function run_section()
 	_scratch_unmount 2> /dev/null
 }
 
+prepare_fs_config_list()
+{
+	IFS=',' read -r -a CMD_FS_CONFIG_OPTION_LIST <<< "$CMD_FS_CONFIG_OPTION"
+	CMD_FS_OPTION_LIST_LEN=${#CMD_FS_CONFIG_OPTION_LIST[@]}
+	if [ $CMD_FS_OPTION_LIST_LEN -eq 0 ]; then
+		# Create an empty fsconfig "" to pass to run_section in iters loop
+		CMD_FS_CONFIG_OPTION_LIST=("")
+	fi
+}
+
+prepare_fs_config_list
+
 for ((iters = 0; iters < $iterations; iters++)) do
 	for section in $HOST_OPTIONS_SECTIONS; do
-		run_section $section
-		if [ "$sum_bad" != 0 ] && [ "$istop" = true ]; then
-			interrupt=false
-			status=`expr $sum_bad != 0`
-			exit
-		fi
+		for fsconfig in "${CMD_FS_CONFIG_OPTION_LIST[@]}"; do
+			run_section $section $fsconfig
+			if [ "$sum_bad" != 0 ] && [ "$istop" = true ]; then
+				interrupt=false
+				status=`expr $sum_bad != 0`
+				exit
+			fi
+		done
 	done
 done
 
diff --git a/common/config b/common/config
index 5f86fc2c..901dc1e8 100644
--- a/common/config
+++ b/common/config
@@ -343,6 +343,11 @@ if [ -x /usr/sbin/selinuxenabled ] && /usr/sbin/selinuxenabled; then
 	export SELINUX_MOUNT_OPTIONS
 fi
 
+options_have_sections_or_fs_configs()
+{
+	$OPTIONS_HAVE_SECTIONS || $OPTIONS_HAVE_FSCONFIGS
+}
+
 _common_mount_opts()
 {
 	case $FSTYP in
@@ -783,8 +788,32 @@ parse_config_section() {
 		| sed -n -e "/^\[$SECTION\]/,/^\s*\[/{/^[^#].*\=.*/p;}"`
 }
 
+parse_fs_config_option() {
+	local fs_config_option
+	if ! $OPTIONS_HAVE_FSCONFIGS; then
+		return 0
+	fi
+
+	if [ -f "$1" ]; then
+		fs_config_option=$1
+	elif [ -f configs/"$1" ]; then
+		fs_config_option=configs/"$1"
+	else
+		echo "Invalid FS_CONFIG_OPTION $1"
+		exit 1
+	fi
+
+	eval `sed -e 's/[[:space:]]*\=[[:space:]]*/=/g' \
+		-e 's/#.*$//' \
+		-e 's/[[:space:]]*$//' \
+		-e 's/^[[:space:]]*//' \
+		-e "s/^\([^=]*\)=\"\?'\?\([^\"']*\)\"\?'\?$/export \1=\"\2\"/" \
+		< $fs_config_option`
+}
+
 get_next_config() {
-	if [ ! -z "$CONFIG_INCLUDED" ] && ! $OPTIONS_HAVE_SECTIONS; then
+	local cmd_fs_config_option=$2
+	if [[ $CONFIG_INCLUDED && ! options_have_sections_or_fs_configs ]]; then
 		return 0
 	fi
 
@@ -817,6 +846,23 @@ get_next_config() {
 	fi
 
 	parse_config_section $1
+
+	# FS_CONFIG_OPTION can be set either via cmdline -c xfs/4k (through
+	# CMD_FS_CONFIG_OPTION) or can be set in section config file
+	# FS_CONFIG_OPTION=xfs/4k.
+	# Please note that FS_CONFIG_OPTION in the local.config file can be
+	# overridden by the value passed through the command line.
+	if [[ -n "$cmd_fs_config_option" ]]; then
+		export FS_CONFIG_OPTION=$cmd_fs_config_option
+	fi
+
+	if [[ -n "$FS_CONFIG_OPTION" ]]; then
+		export OPTIONS_HAVE_FSCONFIGS=true
+		parse_fs_config_option $FS_CONFIG_OPTION
+	else
+		export OPTIONS_HAVE_FSCONFIGS=false
+	fi
+
 	if [ ! -z "$OLD_FSTYP" ] && [ $OLD_FSTYP != $FSTYP ]; then
 		[ -z "$MOUNT_OPTIONS" ] && _mount_opts
 		[ -z "$TEST_FS_MOUNT_OPTS" ] && _test_mount_opts
@@ -900,7 +946,8 @@ get_next_config() {
 }
 
 if [ -z "$CONFIG_INCLUDED" ]; then
-	get_next_config `echo $HOST_OPTIONS_SECTIONS | cut -f1 -d" "`
+	get_next_config `echo $HOST_OPTIONS_SECTIONS | cut -f1 -d" "` \
+		`echo $CMD_FS_CONFIG_OPTION | cut -f1 -d","`
 	export CONFIG_INCLUDED=true
 
 	# Autodetect fs type based on what's on $TEST_DEV unless it's been set
diff --git a/common/report b/common/report
index 7128bbeb..8723c510 100644
--- a/common/report
+++ b/common/report
@@ -12,7 +12,7 @@ REPORT_ENV_LIST=("SECTION" "FSTYP" "PLATFORM" "MKFS_OPTIONS" "MOUNT_OPTIONS" \
 # Variables that are captured in the report /if/ they are set.
 REPORT_ENV_LIST_OPT=("TAPE_DEV" "RMT_TAPE_DEV" "FSSTRESS_AVOID" "FSX_AVOID"
 		     "KCONFIG_PATH" "PERF_CONFIGNAME" "MIN_FSSIZE"
-		     "IDMAPPED_MOUNTS")
+		     "IDMAPPED_MOUNTS" "FS_CONFIG_OPTION")
 
 encode_xml()
 {
-- 
2.34.1

Re: [RFC 4/5] check,common/config: Add support for central fsconfig

[Theodore Ts'o]

On Fri, Jan 10, 2025 at 09:10:28AM +0000, Nirjhar Roy (IBM) wrote:
> This adds support to pick and use any existing FS config from
> configs/<fstype>/<config>. e.g.
> 
> configs/xfs/1k
> configs/xfs/4k
> configs/ext4/4k
> configs/ext4/64k
> 
> This should help us maintain and test different fs test
> configurations from a central place. We also hope that
> this will be useful for both developers and testers to
> look into what is being actively maintained and tested
> by FS Maintainers.

I haven't been using the current in-place configs in kvm-xfstests and
gce-xfstests because there are number of things that my setup can
support that xfstests native config doesn't support (and becuase my
system predates the FS config setup).  I don't mind just using my own
custom setup, but if we can keep feature parity, perhaps someday I can
switch over to xfstests's system.  This might also make it easier for
people to more easily test using the same setup as the FS maintainers,
regardless of which test running infrastructure they are using.

A) A way of specifying the minimum device size needed for the TEST and
      SCRATCH device.  Using a smaller file system size reduces test
      run time, and if you are paying for cloud test infrastructure,
      the size of the Google Persistent Disk or Amazon Elastic Block
      Storage has a direct impact on the cost per test, which in turn
      impacts how many tests we can afford to run.  But for certain
      test configurations, such as using a larger block size, or using
      bigalloc, a larger test device size might be needed in order for
      tests to be able to run correctly.

B) A way of specifying test exclusions, both at a global level, or on
	a per-fs, or on a per-configuration basis.  It should also be
	possible to specify the kernel version being tested, and so
	that certain test exclusions might only be done when testing
	LTS kernels (for example):

#if LINUX_VERSION_CODE < KERNEL_VERSION(6,6,30)
// This test failure is fixed by commit 631426ba1d45q ("mm/madvise:
// make MADV_POPULATE_(READ|WRITE) handle VM_FAULT_RETRY properly"),
// which first landed in v6.9, and was backported to 6.6.30 as commit
// 631426ba1d45.  Unfortunately, it's too involved to backport it and its
// dependencies to the 6.1 or earlier LTS kernels
generic/743
#endif

C) A way to run shell functions to do setups for testing things like
	overlayfs, or nfs (where I may be starting separate VM's for
	the NFS server, or needing to find the IP address for the NFS
	server running the appropriate kernel under test, which either
	been the same as the kernel under test, or which might be some
	standard server version, such as a RHEL or SLES kernel), as
	part of the setup and teardown of a particular test
	configuration.

D) I also have a really nice scheme for specifying a mkfs
	configuration for testing LTS kernels, since I use the same
	test appliance for testing upstream and LTS/product kernels,
	and the latest mkfs.xfs will produce a file system image that
	isn't supported by a 5.15 LTS kernel.  Product teams might
	also want to run tests using the mkfs configuration for that
	era kernel, even if a 6.1 kernel can support a file system
	created using the latest version of xfsprogs or e2fsprogs.

	Doing this is a bit non-trivial due to a misfeature of how
	mkfs.xfs works, but I have a workaround that some might find
	useful here:

   https://github.com/tytso/xfstests-bld/commit/f62433b74146e6ecacdeace306828c6c7510c4a6

	Note that this might also be useful for xfstests, where
	specific xfstests scripts have to handle cases where mkfs.xfs
	might unexpectedly fail due to an unfortunate combination
	between the test-specific _scratch_mkfs options, and the
	global MKFS_OPTIONS.  This never happens with ext4, due to how
	mkfs.ext4 handles conflicting command-line options, but it
	*is* a problem with mkfs.xfs.  If you think merging an 150
	line shell script library is easier than trying to get
	consensus from within the xfs community, here's a technical
	workaround to what might be charitably described as a
	disagreement between the xfs architects and the needs of the
	testing community.  :-)

If we're going to have critical mass, maybe this is something that's
worth discussing at the upcoming LSF/MM?  As I said, I'm happy having
this be an exclusive feature in gce-xfstests and kvm-xfstests, but
perhaps it would make sense to uplevel some of this feature into
xfstests so that more people can take advantage of it, and to make it
easier for us to share test configs across test teams and upstream
developers?

Cheers,

						- Ted

Re: [RFC 4/5] check,common/config: Add support for central fsconfig

[Anand Jain]

On 10/1/25 17:10, Nirjhar Roy (IBM) wrote:
> This adds support to pick and use any existing FS config from
> configs/<fstype>/<config>. e.g.
> 
> configs/xfs/1k
> configs/xfs/4k
> configs/ext4/4k
> configs/ext4/64k
> 
> This should help us maintain and test different fs test
> configurations from a central place. We also hope that
> this will be useful for both developers and testers to
> look into what is being actively maintained and tested
> by FS Maintainers.
> 
> When we will have fsconfigs set, then will be another subdirectory created
> in results/<section>. For example let's look at the following:
> 
> The directory tree structure on running
> sudo ./check -q 2 -R xunit-quiet -c xfs/4k,configs/xfs/1k selftest/001 selftest/007
> 


The -c option check makes sense to me. Is it possible to get this
feature implemented first while the -q option is still under discussion?

That said, I have a suggestion for the -c option—
  Global config variables should be overridden by file system-specific 
config files.

For example, if configs/localhost.config contains:

MKFS_OPTIONS="--sectorsize 64K"

but configs/<fstype>/some_config sets:

MKFS_OPTIONS=""

then the value from configs/<fstype>/some_config should take priority.

I ran some tests with btrfs, and I don’t see this behavior happening yet.

Thanks, Anand

Re: [RFC 4/5] check,common/config: Add support for central fsconfig

[Ritesh Harjani]

Anand Jain <anand.jain@oracle.com> writes:

> On 10/1/25 17:10, Nirjhar Roy (IBM) wrote:
>> This adds support to pick and use any existing FS config from
>> configs/<fstype>/<config>. e.g.
>> 
>> configs/xfs/1k
>> configs/xfs/4k
>> configs/ext4/4k
>> configs/ext4/64k
>> 
>> This should help us maintain and test different fs test
>> configurations from a central place. We also hope that
>> this will be useful for both developers and testers to
>> look into what is being actively maintained and tested
>> by FS Maintainers.
>> 
>> When we will have fsconfigs set, then will be another subdirectory created
>> in results/<section>. For example let's look at the following:
>> 
>> The directory tree structure on running
>> sudo ./check -q 2 -R xunit-quiet -c xfs/4k,configs/xfs/1k selftest/001 selftest/007
>> 
>
>
> The -c option check makes sense to me. Is it possible to get this
> feature implemented first while the -q option is still under discussion?

Hi Anand, 

Thanks for trying the patches. The design of -c option is still under
discussion [1]. But it will be helpful if you could help us understand
your reasons for finding -c option useful :) 

[1]: https://lore.kernel.org/linux-fsdevel/Z55RXUKB5O5l8QjM@li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com/

>
> That said, I have a suggestion for the -c option—
>   Global config variables should be overridden by file system-specific 
> config files.
>
> For example, if configs/localhost.config contains:
>
> MKFS_OPTIONS="--sectorsize 64K"
>
> but configs/<fstype>/some_config sets:
>
> MKFS_OPTIONS=""
>
> then the value from configs/<fstype>/some_config should take priority.
>
> I ran some tests with btrfs, and I don’t see this behavior happening yet.

I think that was intentional. I guess the reasoning was, we don't want to
break use cases for folks who still wanted to use local.config file
option.

However, in the new proposed design [2] we are thinking of having 1
large config per filesystem. e.g. configs/btrfs/config.btrfs which will
define all of the relevant sections e.g. btrfs_4k, btrfs_64k, ...  Then
on invokking "make", it will generate a single large fs config file i.e.
configs/.all-sections-configs which will club all filesystems section
configs together.

Now when someone invokes check script with different -s options, it will
first look into local.config file, if local.config not found, then it
will look into configs/.all-sections-configs to get the relevant section
defines.

This hopefully should address all the other concerns which were raised
on the current central fs config design.

[2]: https://lore.kernel.org/linux-fsdevel/87plj0hp7e.fsf@gmail.com/

-ritesh

>
> Thanks, Anand

Re: [RFC 4/5] check,common/config: Add support for central fsconfig

[Anand Jain]

On 2/3/25 01:30, Ritesh Harjani (IBM) wrote:
> Anand Jain <anand.jain@oracle.com> writes:
> 
>> On 10/1/25 17:10, Nirjhar Roy (IBM) wrote:
>>> This adds support to pick and use any existing FS config from
>>> configs/<fstype>/<config>. e.g.
>>>
>>> configs/xfs/1k
>>> configs/xfs/4k
>>> configs/ext4/4k
>>> configs/ext4/64k
>>>
>>> This should help us maintain and test different fs test
>>> configurations from a central place. We also hope that
>>> this will be useful for both developers and testers to
>>> look into what is being actively maintained and tested
>>> by FS Maintainers.
>>>
>>> When we will have fsconfigs set, then will be another subdirectory created
>>> in results/<section>. For example let's look at the following:
>>>
>>> The directory tree structure on running
>>> sudo ./check -q 2 -R xunit-quiet -c xfs/4k,configs/xfs/1k selftest/001 selftest/007
>>>
>>
>>
>> The -c option check makes sense to me. Is it possible to get this
>> feature implemented first while the -q option is still under discussion?
> 
> Hi Anand,
> 
> Thanks for trying the patches. The design of -c option is still under
> discussion [1]. But it will be helpful if you could help us understand
> your reasons for finding -c option useful :)
> 

Reason is exactly same as you mentioned and I copied it here:

---
1. Testers and other FS developers can know what is being actively
tested and maintained by FS Maintainers.
---

> [1]: https://lore.kernel.org/linux-fsdevel/Z55RXUKB5O5l8QjM@li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com/
> 
>>
>> That said, I have a suggestion for the -c option—
>>    Global config variables should be overridden by file system-specific
>> config files.
>>
>> For example, if configs/localhost.config contains:
>>
>> MKFS_OPTIONS="--sectorsize 64K"
>>
>> but configs/<fstype>/some_config sets:
>>
>> MKFS_OPTIONS=""
>>
>> then the value from configs/<fstype>/some_config should take priority.
>>
>> I ran some tests with btrfs, and I don’t see this behavior happening yet.
> 
> I think that was intentional. I guess the reasoning was, we don't want to
> break use cases for folks who still wanted to use local.config file
> option.
>

Ok.

> However, in the new proposed design [2] we are thinking of having 1
> large config per filesystem. e.g. configs/btrfs/config.btrfs which will
> define all of the relevant sections e.g. btrfs_4k, btrfs_64k, ...  Then
> on invokking "make", it will generate a single large fs config file i.e.
> configs/.all-sections-configs which will club all filesystems section
> configs together.
>
> Now when someone invokes check script with different -s options, it will
> first look into local.config file, if local.config not found, then it
> will look into configs/.all-sections-configs to get the relevant section
> defines.
> 
> This hopefully should address all the other concerns which were raised
> on the current central fs config design.
> 
> [2]: https://lore.kernel.org/linux-fsdevel/87plj0hp7e.fsf@gmail.com/
> 


I think it’s making things a bit more complicated than needed.

True. We’d probably be better off with one big config file.
No need for <fstype> directories under configs.
configs/<fstype>.config should work just fine.
I’m not fond of the need to run make; it seems excessive.

The way to run it is simple and works well:

  $ ./check -c configs/btrfs.config
  $ ./check -c configs/btrfs.config -s <section1> ...
  $ ./check -c configs/btrfs.config -S <exclude-section1> ...

These steps already work fine:

  $ cp configs/btrfs.config configs/$(hostname).config
  $ ./check -s <section> ..

(Patch containing configs/btrfs.config is already in the ML).

Also, a --dry-run option to check the config before running would
be super helpful.

Thanks, Anand

> -ritesh
> 
>>
>> Thanks, Anand

[RFC 5/5] configs: Add a couple of xfs and ext4 configuration templates.

[Nirjhar Roy (IBM)]

This commit adds a couple of simple configuration templates. Developers
should add more templates based on the tests they add.

Reviewed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Nirjhar Roy (IBM) <nirjhar.roy.lists@gmail.com>
---
 configs/ext4/1k   | 3 +++
 configs/ext4/4k   | 3 +++
 configs/ext4/64k  | 3 +++
 configs/xfs/1k    | 3 +++
 configs/xfs/4k    | 3 +++
 configs/xfs/64k   | 3 +++
 configs/xfs/adv   | 3 +++
 configs/xfs/quota | 3 +++
 8 files changed, 24 insertions(+)
 create mode 100644 configs/ext4/1k
 create mode 100644 configs/ext4/4k
 create mode 100644 configs/ext4/64k
 create mode 100644 configs/xfs/1k
 create mode 100644 configs/xfs/4k
 create mode 100644 configs/xfs/64k
 create mode 100644 configs/xfs/adv
 create mode 100644 configs/xfs/quota

diff --git a/configs/ext4/1k b/configs/ext4/1k
new file mode 100644
index 00000000..6f16fc44
--- /dev/null
+++ b/configs/ext4/1k
@@ -0,0 +1,3 @@
+FSTYP=ext4
+MKFS_OPTIONS="-F -b 1024"
+MOUNT_OPTIONS="-o block_validity "
diff --git a/configs/ext4/4k b/configs/ext4/4k
new file mode 100644
index 00000000..32f02ea9
--- /dev/null
+++ b/configs/ext4/4k
@@ -0,0 +1,3 @@
+FSTYP=ext4
+MKFS_OPTIONS="-F -b 4096"
+MOUNT_OPTIONS="-o block_validity "
diff --git a/configs/ext4/64k b/configs/ext4/64k
new file mode 100644
index 00000000..6c50b839
--- /dev/null
+++ b/configs/ext4/64k
@@ -0,0 +1,3 @@
+FSTYP=ext4
+MKFS_OPTIONS="-F -b 65536"
+MOUNT_OPTIONS="-o block_validity "
diff --git a/configs/xfs/1k b/configs/xfs/1k
new file mode 100644
index 00000000..ab74d3a7
--- /dev/null
+++ b/configs/xfs/1k
@@ -0,0 +1,3 @@
+FSTYP=xfs
+MKFS_OPTIONS="-b size=1024"
+MOUNT_OPTIONS=" "
diff --git a/configs/xfs/4k b/configs/xfs/4k
new file mode 100644
index 00000000..9a390404
--- /dev/null
+++ b/configs/xfs/4k
@@ -0,0 +1,3 @@
+FSTYP=xfs
+MKFS_OPTIONS="-b size=4096"
+MOUNT_OPTIONS=" "
diff --git a/configs/xfs/64k b/configs/xfs/64k
new file mode 100644
index 00000000..ddcd7977
--- /dev/null
+++ b/configs/xfs/64k
@@ -0,0 +1,3 @@
+FSTYP=xfs
+MKFS_OPTIONS="-b size=65536"
+MOUNT_OPTIONS=" "
diff --git a/configs/xfs/adv b/configs/xfs/adv
new file mode 100644
index 00000000..f1b2ec2e
--- /dev/null
+++ b/configs/xfs/adv
@@ -0,0 +1,3 @@
+FSTYP=xfs
+MKFS_OPTIONS="-m inobtcount=1,bigtime=1,rmapbt=1"
+MOUNT_OPTIONS=" "
diff --git a/configs/xfs/quota b/configs/xfs/quota
new file mode 100644
index 00000000..a7f0c7e7
--- /dev/null
+++ b/configs/xfs/quota
@@ -0,0 +1,3 @@
+FSTYP=xfs
+MKFS_OPTIONS="-f "
+MOUNT_OPTIONS="-o usrquota,grpquota,prjquota"
-- 
2.34.1