[PATCH 0/2] add smoketest group and try to provide template

[PATCH 0/2] add smoketest group and try to provide template

[Zorro Lang]

The patch [1/2] is a general patch, please review it as usual.

The patch [2/2] is RFC, by talking with Darrick, we agree to split this
part from original [1/2], as RFC patch, to get separated review points.
If any of you feel it's good, feel free to give it RVB, or feel free to
share your review points.

More details refer to each patch commit log.


History:
https://lore.kernel.org/fstests/20230727153046.dl4palugnyidxoe7@zlang-mailbox/T/#m32ce9c4c316438074b068bd38c5e2dc6b0d53f42
https://lore.kernel.org/fstests/20230727030529.r4ivp6dmtrht5zo2@zlang-mailbox/T/#t

Thanks,
Zorro

[PATCH 1/2] fstests: add smoketest group

[Zorro Lang]

Darrick suggests that fstests can provide a simple smoketest, by
running several generic filesystem smoke testing for five minutes
apiece (SOAK_DURATION="5m"). Since there are only five smoke tests,
this is effectively a 20min super-quick test.

With gcov enabled, running these tests yields about ~75% coverage for
iomap and ~60% for xfs; or ~50% for ext4 and ~75% for ext4; and ~45%
for btrfs.  Coverage was about ~65% for the pagecache.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Zorro Lang <zlang@kernel.org>
---
 doc/group-names.txt | 1 +
 tests/generic/475   | 2 +-
 tests/generic/476   | 2 +-
 tests/generic/521   | 2 +-
 tests/generic/522   | 2 +-
 tests/generic/642   | 2 +-
 6 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/doc/group-names.txt b/doc/group-names.txt
index 1c35a394..c3dcca37 100644
--- a/doc/group-names.txt
+++ b/doc/group-names.txt
@@ -118,6 +118,7 @@ selftest		tests with fixed results, used to validate testing setup
 send			btrfs send/receive
 shrinkfs		decreasing the size of a filesystem
 shutdown		FS_IOC_SHUTDOWN ioctl
+smoketest		Simple smoke tests
 snapshot		btrfs snapshots
 soak			long running soak tests whose runtime can be controlled
                         directly by setting the SOAK_DURATION variable
diff --git a/tests/generic/475 b/tests/generic/475
index 0cbf5131..ce7fe013 100755
--- a/tests/generic/475
+++ b/tests/generic/475
@@ -12,7 +12,7 @@
 # testing efforts.
 #
 . ./common/preamble
-_begin_fstest shutdown auto log metadata eio recoveryloop
+_begin_fstest shutdown auto log metadata eio recoveryloop smoketest
 
 # Override the default cleanup function.
 _cleanup()
diff --git a/tests/generic/476 b/tests/generic/476
index 8e93b734..b1ae4df4 100755
--- a/tests/generic/476
+++ b/tests/generic/476
@@ -8,7 +8,7 @@
 # bugs in the write path.
 #
 . ./common/preamble
-_begin_fstest auto rw long_rw stress soak
+_begin_fstest auto rw long_rw stress soak smoketest
 
 # Override the default cleanup function.
 _cleanup()
diff --git a/tests/generic/521 b/tests/generic/521
index 22dd31a8..0956e501 100755
--- a/tests/generic/521
+++ b/tests/generic/521
@@ -7,7 +7,7 @@
 # Long-soak directio fsx test
 #
 . ./common/preamble
-_begin_fstest soak long_rw
+_begin_fstest soak long_rw smoketest
 
 # Import common functions.
 . ./common/filter
diff --git a/tests/generic/522 b/tests/generic/522
index f0cbcb24..0e4e6009 100755
--- a/tests/generic/522
+++ b/tests/generic/522
@@ -7,7 +7,7 @@
 # Long-soak buffered fsx test
 #
 . ./common/preamble
-_begin_fstest soak long_rw
+_begin_fstest soak long_rw smoketest
 
 # Import common functions.
 . ./common/filter
diff --git a/tests/generic/642 b/tests/generic/642
index eba90903..e6a475a8 100755
--- a/tests/generic/642
+++ b/tests/generic/642
@@ -8,7 +8,7 @@
 # bugs in the xattr code.
 #
 . ./common/preamble
-_begin_fstest auto soak attr long_rw stress
+_begin_fstest auto soak attr long_rw stress smoketest
 
 _cleanup()
 {
-- 
2.40.1

[RFC PATCH 2/2] fstests: support test template

[Zorro Lang]

The fstests has too many big or small testing groups, and it keeps
growing. It's hard for many users to pick up test cases they need.
Likes the smoketest, soak test, random-load test, integrality test
and so on. So most of users might just run "quick" group, or "auto"
group, or "all" directly each time. Besides the group, there're
some global parameters (e.g. *_FACTOR, SOAK_DURATION, etc) too, so
there're many "portfolios" to use them.

So I think fstests can provide a test template, which is bigger than
group, base on group and global parameters, provide reference about
how to do some kinds of tests.

Some users who are familar with fstests have their own wrappers, they
do different kind of tests by fstests according their experience.
They have their different testing templates, some templates might be
helpful and recommended to others. So I'd like to let fstests provide
a template/ directory and a "-t" option to load template. By this
chance, hope more people can share their great test templates to
others. We can record these templates in fstests, then anyone can use
them directly or refer to them to write their wrapper.

Signed-off-by: Zorro Lang <zlang@kernel.org>
---
 check               |  8 ++++++++
 templates/smoketest | 16 ++++++++++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 templates/smoketest

diff --git a/check b/check
index 89e7e7bf..7100aae4 100755
--- a/check
+++ b/check
@@ -335,6 +335,14 @@ while [ $# -gt 0 ]; do
 		;;
 	-i)	iterations=$2; shift ;;
 	-I) 	iterations=$2; istop=true; shift ;;
+	-t)
+		source templates/$2
+		if [ $? -ne 0 ];then
+			echo "Cannot import the templates/$2"
+			exit 1
+		fi
+		shift
+		;;
 	-T)	timestamp=true ;;
 	-d)	DUMP_OUTPUT=true ;;
 	-b)	brief_test_summary=true;;
diff --git a/templates/smoketest b/templates/smoketest
new file mode 100644
index 00000000..40a0104b
--- /dev/null
+++ b/templates/smoketest
@@ -0,0 +1,16 @@
+##/bin/bash
+# For infrequent filesystem developers who simply want to run a quick test
+# of the most commonly used filesystem functionality, use this command:
+#
+#     ./check -t smoketest <other config options>
+#
+# This template helps fstests to run several tests to exercise the file I/O,
+# metadata, and crash recovery exercisers for four minutes apiece.  This
+# should complete in approximately 20 minutes.
+
+echo "**********************"
+echo "* A Quick Smoke Test *"
+echo "**********************"
+
+[ -z "$SOAK_DURATION" ] && SOAK_DURATION="4m"
+GROUP_LIST="smoketest"
-- 
2.40.1

Re: [PATCH 1/2] fstests: add smoketest group

[Darrick J. Wong]

On Fri, Jul 28, 2023 at 02:53:14AM +0800, Zorro Lang wrote:
> Darrick suggests that fstests can provide a simple smoketest, by
> running several generic filesystem smoke testing for five minutes
> apiece (SOAK_DURATION="5m"). Since there are only five smoke tests,
> this is effectively a 20min super-quick test.
> 
> With gcov enabled, running these tests yields about ~75% coverage for
> iomap and ~60% for xfs; or ~50% for ext4 and ~75% for ext4; and ~45%
> for btrfs.  Coverage was about ~65% for the pagecache.
> 
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> Signed-off-by: Zorro Lang <zlang@kernel.org>

Er... this excerpt looks good to me! :)

Reviewed-by: Darrick J. Wong <djwong@kernel.org>

(is that even allowed??)

--D

> ---
>  doc/group-names.txt | 1 +
>  tests/generic/475   | 2 +-
>  tests/generic/476   | 2 +-
>  tests/generic/521   | 2 +-
>  tests/generic/522   | 2 +-
>  tests/generic/642   | 2 +-
>  6 files changed, 6 insertions(+), 5 deletions(-)
> 
> diff --git a/doc/group-names.txt b/doc/group-names.txt
> index 1c35a394..c3dcca37 100644
> --- a/doc/group-names.txt
> +++ b/doc/group-names.txt
> @@ -118,6 +118,7 @@ selftest		tests with fixed results, used to validate testing setup
>  send			btrfs send/receive
>  shrinkfs		decreasing the size of a filesystem
>  shutdown		FS_IOC_SHUTDOWN ioctl
> +smoketest		Simple smoke tests
>  snapshot		btrfs snapshots
>  soak			long running soak tests whose runtime can be controlled
>                          directly by setting the SOAK_DURATION variable
> diff --git a/tests/generic/475 b/tests/generic/475
> index 0cbf5131..ce7fe013 100755
> --- a/tests/generic/475
> +++ b/tests/generic/475
> @@ -12,7 +12,7 @@
>  # testing efforts.
>  #
>  . ./common/preamble
> -_begin_fstest shutdown auto log metadata eio recoveryloop
> +_begin_fstest shutdown auto log metadata eio recoveryloop smoketest
>  
>  # Override the default cleanup function.
>  _cleanup()
> diff --git a/tests/generic/476 b/tests/generic/476
> index 8e93b734..b1ae4df4 100755
> --- a/tests/generic/476
> +++ b/tests/generic/476
> @@ -8,7 +8,7 @@
>  # bugs in the write path.
>  #
>  . ./common/preamble
> -_begin_fstest auto rw long_rw stress soak
> +_begin_fstest auto rw long_rw stress soak smoketest
>  
>  # Override the default cleanup function.
>  _cleanup()
> diff --git a/tests/generic/521 b/tests/generic/521
> index 22dd31a8..0956e501 100755
> --- a/tests/generic/521
> +++ b/tests/generic/521
> @@ -7,7 +7,7 @@
>  # Long-soak directio fsx test
>  #
>  . ./common/preamble
> -_begin_fstest soak long_rw
> +_begin_fstest soak long_rw smoketest
>  
>  # Import common functions.
>  . ./common/filter
> diff --git a/tests/generic/522 b/tests/generic/522
> index f0cbcb24..0e4e6009 100755
> --- a/tests/generic/522
> +++ b/tests/generic/522
> @@ -7,7 +7,7 @@
>  # Long-soak buffered fsx test
>  #
>  . ./common/preamble
> -_begin_fstest soak long_rw
> +_begin_fstest soak long_rw smoketest
>  
>  # Import common functions.
>  . ./common/filter
> diff --git a/tests/generic/642 b/tests/generic/642
> index eba90903..e6a475a8 100755
> --- a/tests/generic/642
> +++ b/tests/generic/642
> @@ -8,7 +8,7 @@
>  # bugs in the xattr code.
>  #
>  . ./common/preamble
> -_begin_fstest auto soak attr long_rw stress
> +_begin_fstest auto soak attr long_rw stress smoketest
>  
>  _cleanup()
>  {
> -- 
> 2.40.1
>