Re: [PATCH v3] selftest: add tests for debugging testing setup

[Leah Rumancik <leah.rumancik@gmail.com>]

On Wed, Feb 22, 2023 at 10:30:16PM +0800, Zorro Lang wrote:
> On Tue, Feb 21, 2023 at 02:36:49PM -0800, Leah Rumancik wrote:
> > diff --git a/README.selftest b/README.selftest
> > new file mode 100644
> > index 00000000..8f1d45af
> > --- /dev/null
> > +++ b/README.selftest
> > @@ -0,0 +1,22 @@
> > +Tests with consistent results are provided in the selftest folder.
> > +Since many people develop testing infrastructure around xfstests,
> > +these tests are helpful to confirm the testing setup is working as
> > +expected.
> > +
> > +The provided tests include:
> > +selftest/001 - pass
> > +selftest/002 - fail from output mismatch
> > +selftest/003 - fail via _fail
> > +selftest/004 - skip
> > +selftest/005 - crash
> > +selftest/006 - hang
> > +
> > +Two groups are used for these tests: selftest and dangerous_selftest.
> > +selftest/00[1-4] are in the selftest group and selftest/00[5-6] are
> > +in the dangerous_selftest group.
> > +
> > +The selftests will only be run if explicitly speficied. To run the
> > +selftests, you can specify individual tests, use "selftests/*", or
>                                                              ^
>                              The path name doesn't have the "s" :)
> 
> I'll change this when I merge it. Others looks good to me.

Good catch, thank you!

- Leah

> 
> Reviewed-by: Zorro Lang <zlang@redhat.com>
> 
> Thanks,
> Zorro
> 
> > +use the groups "-g selftest/selftest -g selftest/dangerous_selftest".
> > +Note, you cannot use the group names without including the folder name
> > +(ie. "-g selftest").
> > diff --git a/doc/group-names.txt b/doc/group-names.txt
> > index 771ce937..ace59e05 100644
> > --- a/doc/group-names.txt
> > +++ b/doc/group-names.txt
> > @@ -38,6 +38,7 @@ dangerous_fsstress_repair	race fsstress and xfs_scrub online repair
> >  dangerous_fsstress_scrub	race fsstress and xfs_scrub checking
> >  dangerous_repair	fuzzers to evaluate xfs_repair offline repair
> >  dangerous_scrub		fuzzers to evaluate xfs_scrub checking
> > +dangerous_selftest	selftests that crash/hang
> >  data			data loss checkers
> >  dax			direct access mode for persistent memory files
> >  db			xfs_db functional tests
> > @@ -111,6 +112,7 @@ samefs			overlayfs when all layers are on the same fs
> >  scrub			filesystem metadata scrubbers
> >  seed			btrfs seeded filesystems
> >  seek			llseek functionality
> > +selftest		tests with fixed results, used to validate testing setup
> >  send			btrfs send/receive
> >  shrinkfs		decreasing the size of a filesystem
> >  shutdown		FS_IOC_SHUTDOWN ioctl
> > diff --git a/tests/selftest/001 b/tests/selftest/001
> > new file mode 100755
> > index 00000000..f3480330
> > --- /dev/null
> > +++ b/tests/selftest/001
> > @@ -0,0 +1,13 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +#
> > +# FS QA Test No. 001
> > +#
> > +# A test that should always pass
> > +#
> > +. ./common/preamble
> > +_begin_fstest selftest
> > +
> > +echo "Silence is golden"
> > +status=0
> > +exit
> > diff --git a/tests/selftest/001.out b/tests/selftest/001.out
> > new file mode 100644
> > index 00000000..88678b8e
> > --- /dev/null
> > +++ b/tests/selftest/001.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 001
> > +Silence is golden
> > diff --git a/tests/selftest/002 b/tests/selftest/002
> > new file mode 100755
> > index 00000000..189ae2cb
> > --- /dev/null
> > +++ b/tests/selftest/002
> > @@ -0,0 +1,13 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +#
> > +# FS QA Test No. 002
> > +#
> > +# A test that should always fail due to output mismatch
> > +#
> > +. ./common/preamble
> > +_begin_fstest selftest
> > +
> > +echo "I am intentionally broken"
> > +status=0
> > +exit
> > diff --git a/tests/selftest/002.out b/tests/selftest/002.out
> > new file mode 100644
> > index 00000000..61705c7c
> > --- /dev/null
> > +++ b/tests/selftest/002.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 002
> > +Silence is golden
> > diff --git a/tests/selftest/003 b/tests/selftest/003
> > new file mode 100755
> > index 00000000..c7a2848c
> > --- /dev/null
> > +++ b/tests/selftest/003
> > @@ -0,0 +1,15 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +#
> > +# FS QA Test No. 003
> > +#
> > +# A test that _fail's
> > +#
> > +
> > +. ./common/preamble
> > +_begin_fstest selftest
> > +
> > +_fail "I have _fail'ed"
> > +
> > +status=0
> > +exit
> > diff --git a/tests/selftest/003.out b/tests/selftest/003.out
> > new file mode 100644
> > index 00000000..6895fc80
> > --- /dev/null
> > +++ b/tests/selftest/003.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 003
> > +Silence is golden
> > diff --git a/tests/selftest/004 b/tests/selftest/004
> > new file mode 100755
> > index 00000000..89a3884d
> > --- /dev/null
> > +++ b/tests/selftest/004
> > @@ -0,0 +1,15 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +#
> > +# FS QA Test No. 004
> > +#
> > +# A test that should always be skipped
> > +#
> > +. ./common/preamble
> > +_begin_fstest selftest
> > +
> > +_notrun "Always skip me"
> > +
> > +echo "I should be skipped"
> > +status=0
> > +exit
> > diff --git a/tests/selftest/004.out b/tests/selftest/004.out
> > new file mode 100644
> > index 00000000..af8614ae
> > --- /dev/null
> > +++ b/tests/selftest/004.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 004
> > +Silence is golden
> > diff --git a/tests/selftest/005 b/tests/selftest/005
> > new file mode 100755
> > index 00000000..738cf44d
> > --- /dev/null
> > +++ b/tests/selftest/005
> > @@ -0,0 +1,16 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +#
> > +# FS QA Test No. 005
> > +#
> > +# A test that crashes
> > +#
> > +. ./common/preamble
> > +_begin_fstest dangerous_selftest
> > +
> > +echo 1 > /proc/sys/kernel/sysrq
> > +echo c > /proc/sysrq-trigger
> > +
> > +echo "I should have crashed by now"
> > +status=0
> > +exit
> > diff --git a/tests/selftest/005.out b/tests/selftest/005.out
> > new file mode 100644
> > index 00000000..a5027f12
> > --- /dev/null
> > +++ b/tests/selftest/005.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 005
> > +Silence is golden
> > diff --git a/tests/selftest/006 b/tests/selftest/006
> > new file mode 100755
> > index 00000000..150de6e8
> > --- /dev/null
> > +++ b/tests/selftest/006
> > @@ -0,0 +1,19 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +#
> > +# FS QA Test No. 006
> > +#
> > +# A test that hangs
> > +#
> > +
> > +. ./common/preamble
> > +_begin_fstest dangerous_selftest
> > +
> > +while :
> > +do
> > +    sleep 1d
> > +done
> > +
> > +echo "I should still be sleeping"
> > +status=0
> > +exit
> > diff --git a/tests/selftest/006.out b/tests/selftest/006.out
> > new file mode 100644
> > index 00000000..675c1b7c
> > --- /dev/null
> > +++ b/tests/selftest/006.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 006
> > +Silence is golden
> > diff --git a/tests/selftest/Makefile b/tests/selftest/Makefile
> > new file mode 100644
> > index 00000000..3ddfca37
> > --- /dev/null
> > +++ b/tests/selftest/Makefile
> > @@ -0,0 +1,24 @@
> > +#
> > +# Copyright (c) 2023 Google, Inc. All Rights Reserved.
> > +#
> > +
> > +TOPDIR = ../..
> > +include $(TOPDIR)/include/builddefs
> > +include $(TOPDIR)/include/buildgrouplist
> > +
> > +SELFTEST_DIR = selftest
> > +TARGET_DIR = $(PKG_LIB_DIR)/$(TESTS_DIR)/$(SELFTEST_DIR)
> > +DIRT = group.list
> > +
> > +default: $(DIRT)
> > +
> > +include $(BUILDRULES)
> > +
> > +install:
> > +	$(INSTALL) -m 755 -d $(TARGET_DIR)
> > +	$(INSTALL) -m 755 $(TESTS) $(TARGET_DIR)
> > +	$(INSTALL) -m 644 group.list $(TARGET_DIR)
> > +	$(INSTALL) -m 644 $(OUTFILES) $(TARGET_DIR)
> > +
> > +# Nothing.
> > +install-dev install-lib:
> > -- 
> > 2.39.2.637.g21b0678d19-goog
> > 
>