[NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing

public inbox for fstests@vger.kernel.org
 help / color / mirror / Atom feed

* [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing
@ 2024-12-31 23:25 Darrick J. Wong
  2024-12-31 23:34 ` [PATCHSET 1/5] fstests: functional test for refcount reporting Darrick J. Wong
                   ` (5 more replies)
  0 siblings, 6 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:25 UTC (permalink / raw)
  To: Carlos Maiolino, Zorro Lang, Andrey Albershteyn,
	Christoph Hellwig
  Cc: xfs, greg.marsden, shirley.ma, konrad.wilk, fstests

Hi everyone,

Thank you all for helping get online repair, parent pointers, and
metadata directory trees, and realtime allocation groups merged this
year!  We got a lot done in 2024.

Having sent pull requests to Carlos for the last pieces of the realtime
modernization project, I have exactly two worthwhile projects left in my
development trees!  The stuff here isn't necessarily in mergeable state
yet, but I still believe everyone ought to know what I'm up to.

The first project implements (somewhat buggily; I never quite got back
to dealing with moving eof blocks) free space defragmentation so that we
can meaningfully shrink filesystems; garbage collect regions of the
filesystem; or prepare for large allocations.  There's not much new
kernel code other than exporting refcounts and gaining the ability to
map free space.

The second project initiates filesystem self healing routines whenever
problems start to crop up, which means that it can run fully
autonomously in the background.  The monitoring system uses some
pseudo-file and seqbuf tricks that I lifted from kmo last winter.

Both of these projects are largely userspace code.

Also I threw in some xfs_repair code to do dangerous fs upgrades.
Nobody should use these, ever.

Maintainers: please do not merge, this is a dog-and-pony show to attract
developer attention.

--D

PS: I'll be back after the holidays to look at the zoned/atomic/fsverity
patches.  And finally rebase fstests to 2024-12-08.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [PATCHSET 1/5] fstests: functional test for refcount reporting
  2024-12-31 23:25 [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing Darrick J. Wong
@ 2024-12-31 23:34 ` Darrick J. Wong
  2024-12-31 23:56   ` [PATCH 1/1] xfs: test output of new FSREFCOUNTS ioctl Darrick J. Wong
  2024-12-31 23:35 ` [PATCHSET 2/5] fstests: defragment free space Darrick J. Wong
                   ` (4 subsequent siblings)
  5 siblings, 1 reply; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:34 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

Hi all,

Add a short functional test for the new GETFSREFCOUNTS ioctl that allows
userspace to query reference count information for a given range of
physical blocks.

If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.

This has been running on the djcloud for months with no problems.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=report-refcounts

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=report-refcounts

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=report-refcounts
---
Commits in this patchset:
 * xfs: test output of new FSREFCOUNTS ioctl
---
 common/rc           |    4 +
 doc/group-names.txt |    1 
 tests/xfs/1921      |  164 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1921.out  |    4 +
 4 files changed, 171 insertions(+), 2 deletions(-)
 create mode 100755 tests/xfs/1921
 create mode 100644 tests/xfs/1921.out


^ permalink raw reply	[flat|nested] 24+ messages in thread

* [PATCH 1/1] xfs: test output of new FSREFCOUNTS ioctl
  2024-12-31 23:34 ` [PATCHSET 1/5] fstests: functional test for refcount reporting Darrick J. Wong
@ 2024-12-31 23:56   ` Darrick J. Wong
  0 siblings, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:56 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

From: Darrick J. Wong <djwong@kernel.org>

Make sure the cursors work properly and that refcounts are correct.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 common/rc           |    4 +
 doc/group-names.txt |    1 
 tests/xfs/1921      |  164 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1921.out  |    4 +
 4 files changed, 171 insertions(+), 2 deletions(-)
 create mode 100755 tests/xfs/1921
 create mode 100644 tests/xfs/1921.out


diff --git a/common/rc b/common/rc
index e04ca50e3140c0..c45a226849ce0f 100644
--- a/common/rc
+++ b/common/rc
@@ -2811,8 +2811,8 @@ _require_xfs_io_command()
 		echo $testio | grep -q "Operation not supported" && \
 			_notrun "O_TMPFILE is not supported"
 		;;
-	"fsmap")
-		testio=`$XFS_IO_PROG -f -c "fsmap" $testfile 2>&1`
+	"fsmap"|"fsrefcounts")
+		testio=`$XFS_IO_PROG -f -c "$command" $testfile 2>&1`
 		echo $testio | grep -q "Inappropriate ioctl" && \
 			_notrun "xfs_io $command support is missing"
 		;;
diff --git a/doc/group-names.txt b/doc/group-names.txt
index ed886caac058c3..b04d0180e8ec02 100644
--- a/doc/group-names.txt
+++ b/doc/group-names.txt
@@ -58,6 +58,7 @@ fsck			general fsck tests
 fsmap			FS_IOC_GETFSMAP ioctl
 fsproperties		Filesystem properties
 fsr			XFS free space reorganizer
+fsrefcounts		FS_IOC_GETFSREFCOUNTS ioctl
 fuzzers			filesystem fuzz tests
 growfs			increasing the size of a filesystem
 hardlink		hardlinks
diff --git a/tests/xfs/1921 b/tests/xfs/1921
new file mode 100755
index 00000000000000..2d0af845767ed2
--- /dev/null
+++ b/tests/xfs/1921
@@ -0,0 +1,164 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021-2025 Oracle.  All Rights Reserved.
+#
+# FS QA Test No. 1921
+#
+# Populate filesystem, check that fsrefcounts -n10000 matches fsrefcounts -n1,
+# then verify that the refcount information is consistent with the fsmap info.
+#
+. ./common/preamble
+_begin_fstest auto clone fsrefcounts fsmap
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.* $TEST_DIR/a $TEST_DIR/b
+}
+
+. ./common/filter
+
+_require_scratch
+_require_xfs_io_command "fsmap"
+_require_xfs_io_command "fsrefcounts"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+cpus=$(( $(src/feature -o) * 4))
+
+# Use fsstress to create a directory tree with some variability
+FSSTRESS_ARGS=$(_scale_fsstress_args -p 4 -d $SCRATCH_MNT -n 4000 $FSSTRESS_AVOID)
+$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full
+
+_scratch_cycle_mount	# flush all the background gc
+
+echo "Compare fsrefcounts" | tee -a $seqres.full
+$XFS_IO_PROG -c 'fsrefcounts -m -n 65536' $SCRATCH_MNT | grep -v 'EXT:' > $TEST_DIR/a
+$XFS_IO_PROG -c 'fsrefcounts -m -n 1' $SCRATCH_MNT | grep -v 'EXT:' > $TEST_DIR/b
+cat $TEST_DIR/a $TEST_DIR/b >> $seqres.full
+
+diff -uw $TEST_DIR/a $TEST_DIR/b
+
+echo "Compare fsrefcounts to fsmap" | tee -a $seqres.full
+$XFS_IO_PROG -c 'fsmap -m -n 65536' $SCRATCH_MNT | grep -v 'EXT:' > $TEST_DIR/b
+cat $TEST_DIR/b >> $seqres.full
+
+while IFS=',' read ext major minor pstart pend owners length crap; do
+	test "$ext" = "EXT" && continue
+
+	awk_args=(-'F' ',' '-v' "major=$major" '-v' "minor=$minor" \
+		  '-v' "pstart=$pstart" '-v' "pend=$pend" '-v' "owners=$owners")
+
+	if [ "$owners" -eq 1 ]; then
+		$AWK_PROG "${awk_args[@]}" \
+'
+BEGIN {
+	printf("Q:%s:%s:%s:%s:%s:\n", major, minor, pstart, pend, owners) > "/dev/stderr";
+	next_map = -1;
+}
+{
+	if ($2 != major || $3 != minor) {
+		next;
+	}
+	if ($5 <= pstart) {
+		next;
+	}
+
+	printf(" A:%s:%s:%s:%s\n", $2, $3, $4, $5) > "/dev/stderr";
+	if (next_map < 0) {
+		if ($4 > pstart) {
+			exit 1
+		}
+		next_map = $5 + 1;
+	} else {
+		if ($4 != next_map) {
+			exit 1
+		}
+		next_map = $5 + 1;
+	}
+	if (next_map >= pend) {
+		nextfile;
+	}
+}
+END {
+	exit 0;
+}
+' $TEST_DIR/b 2> $tmp.debug
+		res=$?
+	else
+		$AWK_PROG "${awk_args[@]}" \
+'
+function max(a, b) {
+	return a > b ? a : b;
+}
+function min(a, b) {
+	return a < b ? a : b;
+}
+BEGIN {
+	printf("Q:%s:%s:%s:%s:%s:\n", major, minor, pstart, pend, owners) > "/dev/stderr";
+	refcount_whole = 0;
+	aborted = 0;
+}
+{
+	if ($2 != major || $3 != minor) {
+		next;
+	}
+	if ($4 > pend) {
+		nextfile;
+	}
+	if ($5 < pstart) {
+		next;
+	}
+	if ($6 == "special_0:2") {
+		/* unknown owner means we cannot distinguish separate owners */
+		aborted = 1;
+		exit 0;
+	}
+
+	printf(" A:%s:%s:%s:%s -> %d\n", $2, $3, $4, $5, refcount_whole) > "/dev/stderr";
+	if ($4 <= pstart && $5 >= pend) {
+		/* Account for extents that span the whole range */
+		refcount_whole++;
+	} else {
+		/* Otherwise track refcounts per-block as we find them */
+		for (block = max($4, pstart); block <= min($5, pend); block++) {
+			refcounts[block]++;
+		}
+	}
+}
+END {
+	if (aborted) {
+		exit 0;
+	}
+	deficit = owners - refcount_whole;
+	printf(" W:%d:%d\n", owners, refcount_whole, deficit) > "/dev/stderr";
+	if (deficit == 0) {
+		exit 0;
+	}
+
+	refcount_slivers = deficit;
+	for (block in refcounts) {
+		printf(" X:%s:%d\n", block, refcounts[block]) > "/dev/stderr";
+		if (refcounts[block] != deficit) {
+			refcount_slivers = 0;
+		}
+	}
+
+	refcount_whole += refcount_slivers;
+	exit owners == refcount_whole ? 0 : 1;
+}
+' $TEST_DIR/b 2> $tmp.debug
+		res=$?
+	fi
+	if [ $res -ne 0 ]; then
+		echo "$major,$minor,$pstart,$pend,$owners not found in fsmap"
+		cat $tmp.debug >> $seqres.full
+		break
+	fi
+done < $TEST_DIR/a
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/1921.out b/tests/xfs/1921.out
new file mode 100644
index 00000000000000..f5ea660379bbdd
--- /dev/null
+++ b/tests/xfs/1921.out
@@ -0,0 +1,4 @@
+QA output created by 1921
+Format and mount
+Compare fsrefcounts
+Compare fsrefcounts to fsmap


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCHSET 2/5] fstests: defragment free space
  2024-12-31 23:25 [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing Darrick J. Wong
  2024-12-31 23:34 ` [PATCHSET 1/5] fstests: functional test for refcount reporting Darrick J. Wong
@ 2024-12-31 23:35 ` Darrick J. Wong
  2024-12-31 23:56   ` [PATCH 1/1] xfs: test clearing of " Darrick J. Wong
  2024-12-31 23:35 ` [PATCHSET 3/5] fstests: capture logs from mount failures Darrick J. Wong
                   ` (3 subsequent siblings)
  5 siblings, 1 reply; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:35 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

Hi all,

These patches contain experimental code to enable userspace to defragment
the free space in a filesystem.  Two purposes are imagined for this
functionality: clearing space at the end of a filesystem before
shrinking it, and clearing free space in anticipation of making a large
allocation.

The first patch adds a new fallocate mode that allows userspace to
allocate free space from the filesystem into a file.  The goal here is
to allow the filesystem shrink process to prevent allocation from a
certain part of the filesystem while a free space defragmenter evacuates
all the files from the doomed part of the filesystem.

The second patch amends the online repair system to allow the sysadmin
to forcibly rebuild metadata structures, even if they're not corrupt.
Without adding an ioctl to move metadata btree blocks, this is the only
way to dislodge metadata.

If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.

This has been running on the djcloud for months with no problems.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=defrag-freespace

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=defrag-freespace

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=defrag-freespace
---
Commits in this patchset:
 * xfs: test clearing of free space
---
 common/rc          |    5 ++++
 tests/xfs/1400     |   52 +++++++++++++++++++++++++++++++++++++++
 tests/xfs/1400.out |    2 +
 tests/xfs/1401     |   70 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1401.out |    2 +
 5 files changed, 131 insertions(+)
 create mode 100755 tests/xfs/1400
 create mode 100644 tests/xfs/1400.out
 create mode 100755 tests/xfs/1401
 create mode 100644 tests/xfs/1401.out

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [PATCH 1/1] xfs: test clearing of free space
  2024-12-31 23:35 ` [PATCHSET 2/5] fstests: defragment free space Darrick J. Wong
@ 2024-12-31 23:56   ` Darrick J. Wong
  0 siblings, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:56 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

From: Darrick J. Wong <djwong@kernel.org>

Simple regression test for the spaceman clearspace command, which tries
to free all the used space in some part of the filesystem.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 common/rc          |    5 ++++
 tests/xfs/1400     |   52 +++++++++++++++++++++++++++++++++++++++
 tests/xfs/1400.out |    2 +
 tests/xfs/1401     |   70 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1401.out |    2 +
 5 files changed, 131 insertions(+)
 create mode 100755 tests/xfs/1400
 create mode 100644 tests/xfs/1400.out
 create mode 100755 tests/xfs/1401
 create mode 100644 tests/xfs/1401.out


diff --git a/common/rc b/common/rc
index c45a226849ce0f..d7dfb55bbbd7e1 100644
--- a/common/rc
+++ b/common/rc
@@ -2786,6 +2786,11 @@ _require_xfs_io_command()
 			-c "fsync" -c "$command $blocksize $((2 * $blocksize))" \
 			$testfile 2>&1`
 		;;
+	"fmapfree")
+		local blocksize=$(_get_file_block_size $TEST_DIR)
+		testio=`$XFS_IO_PROG -F -f -c "$command $blocksize $((2 * $blocksize))" \
+			$testfile 2>&1`
+		;;
 	"fiemap")
 		# If 'ranged' is passed as argument then we check to see if fiemap supports
 		# ranged query params
diff --git a/tests/xfs/1400 b/tests/xfs/1400
new file mode 100755
index 00000000000000..ec3f7aec2a318a
--- /dev/null
+++ b/tests/xfs/1400
@@ -0,0 +1,52 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022-2025 Oracle.  All Rights Reserved.
+#
+# FS QA Test 1400
+#
+# Basic functionality testing for FALLOC_FL_MAP_FREE
+#
+. ./common/preamble
+_begin_fstest auto prealloc
+
+. ./common/filter
+
+_require_scratch
+_require_xfs_io_command "fmapfree"
+
+_scratch_mkfs | _filter_mkfs 2> $tmp.mkfs > /dev/null
+_scratch_mount >> $seqres.full
+. $tmp.mkfs
+
+testfile="$SCRATCH_MNT/$seq.txt"
+touch $testfile
+if $XFS_IO_PROG -c 'stat -v' $testfile | grep -q 'realtime'; then
+	# realtime
+	increment=$((dbsize * rtblocks / 100))
+	length=$((dbsize * rtblocks))
+else
+	# data
+	increment=$((dbsize * dblocks / 100))
+	length=$((dbsize * dblocks))
+fi
+
+free_bytes=$(stat -f -c '%f * %S' $testfile | bc)
+
+echo "free space: $free_bytes; increment: $increment; length: $length" >> $seqres.full
+
+# Map all the free space on that device, 10% at a time
+for ((start = 0; start < length; start += increment)); do
+	$XFS_IO_PROG -f -c "fmapfree $start $increment" $testfile
+done
+
+space_used=$(stat -c '%b * %B' $testfile | bc)
+
+echo "space captured: $space_used" >> $seqres.full
+$FILEFRAG_PROG -v $testfile >> $seqres.full
+
+# Did we get within 10% of the free space?
+_within_tolerance "mapfree space used" $space_used $free_bytes 10% -v
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/1400.out b/tests/xfs/1400.out
new file mode 100644
index 00000000000000..601404d7a46856
--- /dev/null
+++ b/tests/xfs/1400.out
@@ -0,0 +1,2 @@
+QA output created by 1400
+mapfree space used is in range
diff --git a/tests/xfs/1401 b/tests/xfs/1401
new file mode 100755
index 00000000000000..14675abd8ff985
--- /dev/null
+++ b/tests/xfs/1401
@@ -0,0 +1,70 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022-2025 Oracle.  All Rights Reserved.
+#
+# FS QA Test No. 1401
+#
+# Basic functionality testing for the free space defragmenter.
+#
+. ./common/preamble
+_begin_fstest auto defrag shrinkfs
+
+. ./common/filter
+
+_notrun "XXX test is not ready yet; you need to deal with eof blocks"
+_notrun "XXX clearfree cannot move unwritten extents; does fiexchange work for this?"
+_notrun "XXX csp_buffercopy never returns if we hit eof"
+
+_require_scratch
+_require_xfs_spaceman_command "clearfree"
+
+_scratch_mkfs | _filter_mkfs 2> $tmp.mkfs > /dev/null
+cat $tmp.mkfs >> $seqres.full
+. $tmp.mkfs
+_scratch_mount >> $seqres.full
+
+cpus=$(( $(src/feature -o) * 4))
+
+# Use fsstress to create a directory tree with some variability
+FSSTRESS_ARGS=$(_scale_fsstress_args -p 4 -d $SCRATCH_MNT -n 4000 $FSSTRESS_AVOID)
+$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full
+
+$XFS_IO_PROG -c 'stat -v' $SCRATCH_MNT >> $seqres.full
+
+if $XFS_IO_PROG -c 'stat -v' $SCRATCH_MNT | grep -q 'rt-inherit'; then
+	# realtime
+	increment=$((dbsize * rtblocks / agcount))
+	length=$((dbsize * rtblocks))
+	fsmap_devarg="-r"
+else
+	# data
+	increment=$((dbsize * agsize))
+	length=$((dbsize * dblocks))
+	fsmap_devarg="-d"
+fi
+
+echo "start: $start; increment: $increment; length: $length" >> $seqres.full
+$DF_PROG $SCRATCH_MNT >> $seqres.full
+
+TRACE_PROG="strace -s99 -e fallocate,ioctl,openat -o $tmp.strace"
+
+for ((start = 0; start < length; start += increment)); do
+	echo "---------------------------" >> $seqres.full
+	echo "start: $start end: $((start + increment))" >> $seqres.full
+	echo "---------------------------" >> $seqres.full
+
+	fsmap_args="-vvvv $fsmap_devarg $((start / 512)) $((increment / 512))"
+	clearfree_args="-v all $start $increment"
+
+	$XFS_IO_PROG -c "fsmap $fsmap_args" $SCRATCH_MNT > $tmp.before
+	$TRACE_PROG $XFS_SPACEMAN_PROG -c "clearfree $clearfree_args" $SCRATCH_MNT &>> $seqres.full || break
+	cat $tmp.strace >> $seqres.full
+	$XFS_IO_PROG -c "fsmap $fsmap_args" $SCRATCH_MNT > $tmp.after
+	cat $tmp.before >> $seqres.full
+	cat $tmp.after >> $seqres.full
+done
+
+# success, all done
+echo Silence is golden
+status=0
+exit
diff --git a/tests/xfs/1401.out b/tests/xfs/1401.out
new file mode 100644
index 00000000000000..504999381ea9a8
--- /dev/null
+++ b/tests/xfs/1401.out
@@ -0,0 +1,2 @@
+QA output created by 1401
+Silence is golden


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCHSET 3/5] fstests: capture logs from mount failures
  2024-12-31 23:25 [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing Darrick J. Wong
  2024-12-31 23:34 ` [PATCHSET 1/5] fstests: functional test for refcount reporting Darrick J. Wong
  2024-12-31 23:35 ` [PATCHSET 2/5] fstests: defragment free space Darrick J. Wong
@ 2024-12-31 23:35 ` Darrick J. Wong
  2024-12-31 23:56   ` [PATCH 1/2] treewide: convert all $MOUNT_PROG to _mount Darrick J. Wong
  2024-12-31 23:56   ` [PATCH 2/2] check: capture dmesg of mount failures if test fails Darrick J. Wong
  2024-12-31 23:35 ` [PATCHSET 4/5] fstests: live health monitoring of filesystems Darrick J. Wong
                   ` (2 subsequent siblings)
  5 siblings, 2 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:35 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

Hi all,

Whenever a mount fails, we should capture the kernel logs for the last
few seconds before the failure.  If the test fails, retain the log
contents for further analysis.

If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.

This has been running on the djcloud for months with no problems.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=capture-mount-failures
---
Commits in this patchset:
 * treewide: convert all $MOUNT_PROG to _mount
 * check: capture dmesg of mount failures if test fails
---
 check                  |   22 +++++++++++++++++++++-
 common/btrfs           |    4 ++--
 common/dmdelay         |    2 +-
 common/dmerror         |    2 +-
 common/dmlogwrites     |    2 +-
 common/overlay         |    6 +++---
 common/rc              |   26 +++++++++++++++++++++++++-
 common/report          |    8 ++++++++
 tests/btrfs/075        |    2 +-
 tests/btrfs/208        |    2 +-
 tests/ext4/032         |    2 +-
 tests/generic/067      |    6 +++---
 tests/generic/085      |    2 +-
 tests/generic/361      |    2 +-
 tests/generic/373      |    2 +-
 tests/generic/374      |    2 +-
 tests/generic/409      |    6 +++---
 tests/generic/410      |    8 ++++----
 tests/generic/411      |    8 ++++----
 tests/generic/589      |    8 ++++----
 tests/overlay/005      |    4 ++--
 tests/overlay/025      |    2 +-
 tests/overlay/035      |    2 +-
 tests/overlay/062      |    2 +-
 tests/overlay/083      |    6 +++---
 tests/overlay/086      |   12 ++++++------
 tests/selftest/008     |   20 ++++++++++++++++++++
 tests/selftest/008.out |    1 +
 tests/xfs/078          |    2 +-
 tests/xfs/149          |    4 ++--
 tests/xfs/289          |    4 ++--
 tests/xfs/544          |    2 +-
 32 files changed, 128 insertions(+), 55 deletions(-)
 create mode 100755 tests/selftest/008
 create mode 100644 tests/selftest/008.out


^ permalink raw reply	[flat|nested] 24+ messages in thread

* [PATCH 1/2] treewide: convert all $MOUNT_PROG to _mount
  2024-12-31 23:35 ` [PATCHSET 3/5] fstests: capture logs from mount failures Darrick J. Wong
@ 2024-12-31 23:56   ` Darrick J. Wong
  2024-12-31 23:56   ` [PATCH 2/2] check: capture dmesg of mount failures if test fails Darrick J. Wong
  1 sibling, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:56 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

From: Darrick J. Wong <djwong@kernel.org>

Going to add some new log scraping functionality when mount failures
occur, so we need everyone to use _mount instead of $MOUNT_PROG.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 common/btrfs       |    4 ++--
 common/dmdelay     |    2 +-
 common/dmerror     |    2 +-
 common/dmlogwrites |    2 +-
 common/overlay     |    6 +++---
 tests/btrfs/075    |    2 +-
 tests/btrfs/208    |    2 +-
 tests/ext4/032     |    2 +-
 tests/generic/067  |    6 +++---
 tests/generic/085  |    2 +-
 tests/generic/361  |    2 +-
 tests/generic/373  |    2 +-
 tests/generic/374  |    2 +-
 tests/generic/409  |    6 +++---
 tests/generic/410  |    8 ++++----
 tests/generic/411  |    8 ++++----
 tests/generic/589  |    8 ++++----
 tests/overlay/005  |    4 ++--
 tests/overlay/025  |    2 +-
 tests/overlay/035  |    2 +-
 tests/overlay/062  |    2 +-
 tests/overlay/083  |    6 +++---
 tests/overlay/086  |   12 ++++++------
 tests/xfs/078      |    2 +-
 tests/xfs/149      |    4 ++--
 tests/xfs/289      |    4 ++--
 tests/xfs/544      |    2 +-
 27 files changed, 53 insertions(+), 53 deletions(-)


diff --git a/common/btrfs b/common/btrfs
index 95a9c8e6c7f448..64f38cc240ab8b 100644
--- a/common/btrfs
+++ b/common/btrfs
@@ -351,7 +351,7 @@ _btrfs_stress_subvolume()
 	mkdir -p $subvol_mnt
 	while [ ! -e $stop_file ]; do
 		$BTRFS_UTIL_PROG subvolume create $btrfs_mnt/$subvol_name
-		$MOUNT_PROG -o subvol=$subvol_name $btrfs_dev $subvol_mnt
+		_mount -o subvol=$subvol_name $btrfs_dev $subvol_mnt
 		$UMOUNT_PROG $subvol_mnt
 		$BTRFS_UTIL_PROG subvolume delete $btrfs_mnt/$subvol_name
 	done
@@ -437,7 +437,7 @@ _btrfs_stress_remount_compress()
 	local btrfs_mnt=$1
 	while true; do
 		for algo in no zlib lzo; do
-			$MOUNT_PROG -o remount,compress=$algo $btrfs_mnt
+			_mount -o remount,compress=$algo $btrfs_mnt
 		done
 	done
 }
diff --git a/common/dmdelay b/common/dmdelay
index 66cac1a70c14c8..794ea37ba200ce 100644
--- a/common/dmdelay
+++ b/common/dmdelay
@@ -20,7 +20,7 @@ _init_delay()
 _mount_delay()
 {
 	_scratch_options mount
-	$MOUNT_PROG -t $FSTYP `_common_dev_mount_options` $SCRATCH_OPTIONS \
+	_mount -t $FSTYP `_common_dev_mount_options` $SCRATCH_OPTIONS \
 		$DELAY_DEV $SCRATCH_MNT
 }
 
diff --git a/common/dmerror b/common/dmerror
index 3494b6dd3b9479..2f006142a309fe 100644
--- a/common/dmerror
+++ b/common/dmerror
@@ -91,7 +91,7 @@ _dmerror_init()
 _dmerror_mount()
 {
 	_scratch_options mount
-	$MOUNT_PROG -t $FSTYP `_common_dev_mount_options $*` $SCRATCH_OPTIONS \
+	_mount -t $FSTYP `_common_dev_mount_options $*` $SCRATCH_OPTIONS \
 		$DMERROR_DEV $SCRATCH_MNT
 }
 
diff --git a/common/dmlogwrites b/common/dmlogwrites
index 7a8a9078cb8b65..c054acb875a384 100644
--- a/common/dmlogwrites
+++ b/common/dmlogwrites
@@ -139,7 +139,7 @@ _log_writes_mkfs()
 _log_writes_mount()
 {
 	_scratch_options mount
-	$MOUNT_PROG -t $FSTYP `_common_dev_mount_options $*` $SCRATCH_OPTIONS \
+	_mount -t $FSTYP `_common_dev_mount_options $*` $SCRATCH_OPTIONS \
 		$LOGWRITES_DMDEV $SCRATCH_MNT
 }
 
diff --git a/common/overlay b/common/overlay
index faa9339a6477f7..da1d8d2c3183f4 100644
--- a/common/overlay
+++ b/common/overlay
@@ -29,13 +29,13 @@ _overlay_mount_dirs()
 	[ -n "$upperdir" ] && [ "$upperdir" != "-" ] && \
 		diropts+=",upperdir=$upperdir,workdir=$workdir"
 
-	$MOUNT_PROG -t overlay $diropts `_common_dev_mount_options $*`
+	_mount -t overlay $diropts `_common_dev_mount_options $*`
 }
 
 # Mount with mnt/dev of scratch mount and custom mount options
 _overlay_scratch_mount_opts()
 {
-	$MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT $*
+	_mount -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT $*
 }
 
 # Mount with same options/mnt/dev of scratch mount, but optionally
@@ -127,7 +127,7 @@ _overlay_base_scratch_mount()
 _overlay_scratch_mount()
 {
 	if echo "$*" | grep -q remount; then
-		$MOUNT_PROG $SCRATCH_MNT $*
+		_mount $SCRATCH_MNT $*
 		return
 	fi
 
diff --git a/tests/btrfs/075 b/tests/btrfs/075
index 917993ca2da3a6..737c4ffdd57865 100755
--- a/tests/btrfs/075
+++ b/tests/btrfs/075
@@ -37,7 +37,7 @@ _scratch_mount
 subvol_mnt=$TEST_DIR/$seq.mnt
 mkdir -p $subvol_mnt
 $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/subvol >>$seqres.full 2>&1
-$MOUNT_PROG -o subvol=subvol $SELINUX_MOUNT_OPTIONS $SCRATCH_DEV $subvol_mnt
+_mount -o subvol=subvol $SELINUX_MOUNT_OPTIONS $SCRATCH_DEV $subvol_mnt
 status=$?
 
 exit
diff --git a/tests/btrfs/208 b/tests/btrfs/208
index 5ea732ae8f71a7..93a999541dab06 100755
--- a/tests/btrfs/208
+++ b/tests/btrfs/208
@@ -45,7 +45,7 @@ _scratch_unmount
 
 # Now we mount the subvol2, which makes subvol3 not accessible for this mount
 # point, but we should be able to delete it using it's subvolume id
-$MOUNT_PROG -o subvol=subvol2 $SCRATCH_DEV $SCRATCH_MNT
+_mount -o subvol=subvol2 $SCRATCH_DEV $SCRATCH_MNT
 _delete_and_list subvol3 "Last remaining subvolume:"
 _scratch_unmount
 
diff --git a/tests/ext4/032 b/tests/ext4/032
index 238ab178363c12..9a1b9312cc42cc 100755
--- a/tests/ext4/032
+++ b/tests/ext4/032
@@ -48,7 +48,7 @@ ext4_online_resize()
 		$seqres.full 2>&1 || _fail "mkfs failed"
 
 	echo "+++ mount image file" | tee -a $seqres.full
-	$MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${IMG_MNT} > \
+	_mount -t ${FSTYP} ${LOOP_DEVICE} ${IMG_MNT} > \
 		/dev/null 2>&1 || _fail "mount failed"
 
 	echo "+++ resize fs to $final_size" | tee -a $seqres.full
diff --git a/tests/generic/067 b/tests/generic/067
index b561b7bc5946a2..b6e984f5231753 100755
--- a/tests/generic/067
+++ b/tests/generic/067
@@ -34,7 +34,7 @@ mount_nonexistent_mnt()
 {
 	echo "# mount to nonexistent mount point" >>$seqres.full
 	rm -rf $TEST_DIR/nosuchdir
-	$MOUNT_PROG $SCRATCH_DEV $TEST_DIR/nosuchdir >>$seqres.full 2>&1
+	_mount $SCRATCH_DEV $TEST_DIR/nosuchdir >>$seqres.full 2>&1
 }
 
 # fs driver should be able to handle mounting a free loop device gracefully
@@ -43,7 +43,7 @@ mount_free_loopdev()
 {
 	echo "# mount a free loop device" >>$seqres.full
 	loopdev=`losetup -f`
-	$MOUNT_PROG -t $FSTYP $loopdev $SCRATCH_MNT >>$seqres.full 2>&1
+	_mount -t $FSTYP $loopdev $SCRATCH_MNT >>$seqres.full 2>&1
 }
 
 # mount with wrong fs type specified.
@@ -55,7 +55,7 @@ mount_wrong_fstype()
 		fs=xfs
 	fi
 	echo "# mount with wrong fs type" >>$seqres.full
-	$MOUNT_PROG -t $fs $SCRATCH_DEV $SCRATCH_MNT >>$seqres.full 2>&1
+	_mount -t $fs $SCRATCH_DEV $SCRATCH_MNT >>$seqres.full 2>&1
 }
 
 # umount a symlink to device, which is not mounted.
diff --git a/tests/generic/085 b/tests/generic/085
index cfe6112d6b444d..cbabd257cad8f0 100755
--- a/tests/generic/085
+++ b/tests/generic/085
@@ -69,7 +69,7 @@ for ((i=0; i<100; i++)); do
 done &
 pid=$!
 for ((i=0; i<100; i++)); do
-	$MOUNT_PROG $lvdev $SCRATCH_MNT >/dev/null 2>&1
+	_mount $lvdev $SCRATCH_MNT >/dev/null 2>&1
 	$UMOUNT_PROG $lvdev >/dev/null 2>&1
 done &
 pid="$pid $!"
diff --git a/tests/generic/361 b/tests/generic/361
index c56157391d3209..c2ebda3c1a01ad 100755
--- a/tests/generic/361
+++ b/tests/generic/361
@@ -52,7 +52,7 @@ fi
 $XFS_IO_PROG -fc "pwrite 0 520m" $fs_mnt/testfile >>$seqres.full 2>&1
 
 # remount should not hang
-$MOUNT_PROG -o remount,ro $fs_mnt >>$seqres.full 2>&1
+_mount -o remount,ro $fs_mnt >>$seqres.full 2>&1
 
 # success, all done
 echo "Silence is golden"
diff --git a/tests/generic/373 b/tests/generic/373
index 3bd46963a76686..0d5a50cbee40b8 100755
--- a/tests/generic/373
+++ b/tests/generic/373
@@ -42,7 +42,7 @@ blksz=65536
 sz=$((blksz * blocks))
 
 echo "Mount otherdir"
-$MOUNT_PROG --bind $SCRATCH_MNT $otherdir
+_mount --bind $SCRATCH_MNT $otherdir
 
 echo "Create file"
 _pwrite_byte 0x61 0 $sz $testdir/file >> $seqres.full
diff --git a/tests/generic/374 b/tests/generic/374
index acb23d17289784..977a2b268bbc98 100755
--- a/tests/generic/374
+++ b/tests/generic/374
@@ -41,7 +41,7 @@ blksz=65536
 sz=$((blocks * blksz))
 
 echo "Mount otherdir"
-$MOUNT_PROG --bind $SCRATCH_MNT $otherdir
+_mount --bind $SCRATCH_MNT $otherdir
 
 echo "Create file"
 _pwrite_byte 0x61 0 $sz $testdir/file >> $seqres.full
diff --git a/tests/generic/409 b/tests/generic/409
index b7edc2ac664461..79468e2b0ddb41 100755
--- a/tests/generic/409
+++ b/tests/generic/409
@@ -87,7 +87,7 @@ start_test()
 
 	_scratch_mkfs >$seqres.full 2>&1
 	_get_mount -t $FSTYP $SCRATCH_DEV $MNTHEAD
-	$MOUNT_PROG --make-"${type}" $MNTHEAD
+	_mount --make-"${type}" $MNTHEAD
 	mkdir $mpA $mpB $mpC $mpD
 }
 
@@ -107,9 +107,9 @@ bind_run()
 	echo "bind $source on $dest"
 	_get_mount -t $FSTYP $SCRATCH_DEV $mpA
 	mkdir -p $mpA/dir 2>/dev/null
-	$MOUNT_PROG --make-shared $mpA
+	_mount --make-shared $mpA
 	_get_mount --bind $mpA $mpB
-	$MOUNT_PROG --make-"$source" $mpB
+	_mount --make-"$source" $mpB
 	# maybe unbindable at here
 	_get_mount --bind $mpB $mpC 2>/dev/null
 	if [ $? -ne 0 ]; then
diff --git a/tests/generic/410 b/tests/generic/410
index 902f27144285e4..db8c97dbac7701 100755
--- a/tests/generic/410
+++ b/tests/generic/410
@@ -93,7 +93,7 @@ start_test()
 
 	_scratch_mkfs >>$seqres.full 2>&1
 	_get_mount -t $FSTYP $SCRATCH_DEV $MNTHEAD
-	$MOUNT_PROG --make-"${type}" $MNTHEAD
+	_mount --make-"${type}" $MNTHEAD
 	mkdir $mpA $mpB $mpC
 }
 
@@ -117,14 +117,14 @@ run()
 	echo "make-$cmd a $orgs mount"
 	_get_mount -t $FSTYP $SCRATCH_DEV $mpA
 	mkdir -p $mpA/dir 2>/dev/null
-	$MOUNT_PROG --make-shared $mpA
+	_mount --make-shared $mpA
 
 	# prepare the original status on mpB
 	_get_mount --bind $mpA $mpB
 	# shared&slave status need to do make-slave then make-shared
 	# two operations.
 	for t in $orgs; do
-		$MOUNT_PROG --make-"$t" $mpB
+		_mount --make-"$t" $mpB
 	done
 
 	# "before" for prepare and check original status
@@ -145,7 +145,7 @@ run()
 			_put_mount # umount C
 		fi
 		if [ "$i" = "before" ];then
-			$MOUNT_PROG --make-"${cmd}" $mpB
+			_mount --make-"${cmd}" $mpB
 		fi
 	done
 
diff --git a/tests/generic/411 b/tests/generic/411
index c35436c82e988e..09a813f5d3028e 100755
--- a/tests/generic/411
+++ b/tests/generic/411
@@ -76,7 +76,7 @@ start_test()
 
 	_scratch_mkfs >$seqres.full 2>&1
 	_get_mount -t $FSTYP $SCRATCH_DEV $MNTHEAD
-	$MOUNT_PROG --make-"${type}" $MNTHEAD
+	_mount --make-"${type}" $MNTHEAD
 	mkdir $mpA $mpB $mpC
 }
 
@@ -99,11 +99,11 @@ crash_test()
 
 	_get_mount -t $FSTYP $SCRATCH_DEV $mpA
 	mkdir $mpA/mnt1
-	$MOUNT_PROG --make-shared $mpA
+	_mount --make-shared $mpA
 	_get_mount --bind $mpA $mpB
 	_get_mount --bind $mpA $mpC
-	$MOUNT_PROG --make-slave $mpB
-	$MOUNT_PROG --make-slave $mpC
+	_mount --make-slave $mpB
+	_mount --make-slave $mpC
 	_get_mount -t $FSTYP $SCRATCH_DEV $mpA/mnt1
 	mkdir $mpA/mnt1/mnt2
 
diff --git a/tests/generic/589 b/tests/generic/589
index 0ce16556a05df9..6f69abd17ab01e 100755
--- a/tests/generic/589
+++ b/tests/generic/589
@@ -80,12 +80,12 @@ start_test()
 
 	_get_mount -t $FSTYP $SCRATCH_DEV $SRCHEAD
 	# make sure $SRCHEAD is private
-	$MOUNT_PROG --make-private $SRCHEAD
+	_mount --make-private $SRCHEAD
 
 	_get_mount -t $FSTYP $SCRATCH_DEV $DSTHEAD
 	# test start with a bind, then make-shared $DSTHEAD
 	_get_mount --bind $DSTHEAD $DSTHEAD
-	$MOUNT_PROG --make-"${type}" $DSTHEAD
+	_mount --make-"${type}" $DSTHEAD
 	mkdir $mpA $mpB $mpC $mpD
 }
 
@@ -105,10 +105,10 @@ move_run()
 	echo "move $source to $dest"
 	_get_mount -t $FSTYP $SCRATCH_DEV $mpA
 	mkdir -p $mpA/dir 2>/dev/null
-	$MOUNT_PROG --make-shared $mpA
+	_mount --make-shared $mpA
 	# need a peer for slave later
 	_get_mount --bind $mpA $mpB
-	$MOUNT_PROG --make-"$source" $mpB
+	_mount --make-"$source" $mpB
 	# maybe unbindable at here
 	_get_mount --move $mpB $mpC 2>/dev/null
 	if [ $? -ne 0 ]; then
diff --git a/tests/overlay/005 b/tests/overlay/005
index 4c11d5e1b6f701..01914ee17b9a30 100755
--- a/tests/overlay/005
+++ b/tests/overlay/005
@@ -50,8 +50,8 @@ $MKFS_XFS_PROG -f -n ftype=1 $upper_loop_dev >>$seqres.full 2>&1
 # mount underlying xfs
 mkdir -p ${OVL_BASE_SCRATCH_MNT}/lowermnt
 mkdir -p ${OVL_BASE_SCRATCH_MNT}/uppermnt
-$MOUNT_PROG $fs_loop_dev ${OVL_BASE_SCRATCH_MNT}/lowermnt
-$MOUNT_PROG $upper_loop_dev ${OVL_BASE_SCRATCH_MNT}/uppermnt
+_mount $fs_loop_dev ${OVL_BASE_SCRATCH_MNT}/lowermnt
+_mount $upper_loop_dev ${OVL_BASE_SCRATCH_MNT}/uppermnt
 
 # prepare dirs
 mkdir -p ${OVL_BASE_SCRATCH_MNT}/lowermnt/lower
diff --git a/tests/overlay/025 b/tests/overlay/025
index dc819a39348b69..6ba46191b557be 100755
--- a/tests/overlay/025
+++ b/tests/overlay/025
@@ -36,7 +36,7 @@ _require_extra_fs tmpfs
 # create a tmpfs in $TEST_DIR
 tmpfsdir=$TEST_DIR/tmpfs
 mkdir -p $tmpfsdir
-$MOUNT_PROG -t tmpfs tmpfs $tmpfsdir
+_mount -t tmpfs tmpfs $tmpfsdir
 
 mkdir -p $tmpfsdir/{lower,upper,work,mnt}
 mkdir -p -m 0 $tmpfsdir/upper/testd
diff --git a/tests/overlay/035 b/tests/overlay/035
index 0b3257c4cce09e..cede58790e1b9d 100755
--- a/tests/overlay/035
+++ b/tests/overlay/035
@@ -42,7 +42,7 @@ mkdir -p $lowerdir1 $lowerdir2 $upperdir $workdir
 # Verify that overlay is mounted read-only and that it cannot be remounted rw.
 _overlay_scratch_mount_opts -o"lowerdir=$lowerdir2:$lowerdir1"
 touch $SCRATCH_MNT/foo 2>&1 | _filter_scratch
-$MOUNT_PROG -o remount,rw $SCRATCH_MNT 2>&1 | _filter_ro_mount
+_mount -o remount,rw $SCRATCH_MNT 2>&1 | _filter_ro_mount
 $UMOUNT_PROG $SCRATCH_MNT
 
 # Make workdir immutable to prevent workdir re-create on mount
diff --git a/tests/overlay/062 b/tests/overlay/062
index e44628b7459bfb..9a1db7419c4ca2 100755
--- a/tests/overlay/062
+++ b/tests/overlay/062
@@ -60,7 +60,7 @@ lowertestdir=$lower2/testdir
 create_test_files $lowertestdir
 
 # bind mount to pin lower test dir dentry to dcache
-$MOUNT_PROG --bind $lowertestdir $lowertestdir
+_mount --bind $lowertestdir $lowertestdir
 
 # For non-upper overlay mount, nfs_export requires disabling redirect_dir.
 _overlay_scratch_mount_opts \
diff --git a/tests/overlay/083 b/tests/overlay/083
index d037d4c858e6a6..56e02f8cc77d73 100755
--- a/tests/overlay/083
+++ b/tests/overlay/083
@@ -40,14 +40,14 @@ mkdir -p "$lowerdir_spaces" "$lowerdir_colons" "$lowerdir_commas"
 
 # _overlay_mount_* helpers do not handle special chars well, so execute mount directly.
 # if escaped colons are not parsed correctly, mount will fail.
-$MOUNT_PROG -t overlay ovl_esc_test $SCRATCH_MNT \
+_mount -t overlay ovl_esc_test $SCRATCH_MNT \
 	-o"upperdir=$upperdir,workdir=$workdir" \
 	-o"lowerdir=$lowerdir_colons_esc:$lowerdir_spaces" \
 	2>&1 | tee -a $seqres.full
 
 # if spaces are not escaped when showing mount options,
 # mount command will not show the word 'spaces' after the spaces
-$MOUNT_PROG -t overlay | grep ovl_esc_test  | tee -a $seqres.full | grep -v spaces && \
+_mount -t overlay | grep ovl_esc_test  | tee -a $seqres.full | grep -v spaces && \
 	echo "ERROR: escaped spaces truncated from lowerdir mount option"
 
 # Re-create the upper/work dirs to mount them with a different lower
@@ -65,7 +65,7 @@ mkdir -p "$upperdir" "$workdir"
 # and this test will fail, but the failure would indicate a libmount issue, not
 # a kernel issue.  Therefore, force libmount to use mount(2) syscall, so we only
 # test the kernel fix.
-LIBMOUNT_FORCE_MOUNT2=always $MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_DEV $SCRATCH_MNT \
+LIBMOUNT_FORCE_MOUNT2=always _mount -t overlay $OVL_BASE_SCRATCH_DEV $SCRATCH_MNT \
 	-o"upperdir=$upperdir,workdir=$workdir,lowerdir=$lowerdir_commas_esc" 2>> $seqres.full || \
 	echo "ERROR: incorrect parsing of escaped comma in lowerdir mount option"
 
diff --git a/tests/overlay/086 b/tests/overlay/086
index 9c8a00588595f6..23c56d074ff34a 100755
--- a/tests/overlay/086
+++ b/tests/overlay/086
@@ -33,21 +33,21 @@ mkdir -p "$lowerdir_spaces" "$lowerdir_colons"
 # _overlay_mount_* helpers do not handle lowerdir+,datadir+, so execute mount directly.
 
 # check illegal combinations and order of lowerdir,lowerdir+,datadir+
-$MOUNT_PROG -t overlay none $SCRATCH_MNT \
+_mount -t overlay none $SCRATCH_MNT \
 	-o"lowerdir=$lowerdir,lowerdir+=$lowerdir_colons" \
 	2>> $seqres.full && \
 	echo "ERROR: invalid combination of lowerdir and lowerdir+ mount options"
 
 $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
 
-$MOUNT_PROG -t overlay none $SCRATCH_MNT \
+_mount -t overlay none $SCRATCH_MNT \
 	-o"lowerdir=$lowerdir,datadir+=$lowerdir_colons" \
 	-o redirect_dir=follow,metacopy=on 2>> $seqres.full && \
 	echo "ERROR: invalid combination of lowerdir and datadir+ mount options"
 
 $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
 
-$MOUNT_PROG -t overlay none $SCRATCH_MNT \
+_mount -t overlay none $SCRATCH_MNT \
 	-o"datadir+=$lowerdir,lowerdir+=$lowerdir_colons" \
 	-o redirect_dir=follow,metacopy=on 2>> $seqres.full && \
 	echo "ERROR: invalid order of lowerdir+ and datadir+ mount options"
@@ -55,7 +55,7 @@ $MOUNT_PROG -t overlay none $SCRATCH_MNT \
 $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
 
 # mount is expected to fail with escaped colons.
-$MOUNT_PROG -t overlay none $SCRATCH_MNT \
+_mount -t overlay none $SCRATCH_MNT \
 	-o"lowerdir+=$lowerdir_colons_esc" \
 	2>> $seqres.full && \
 	echo "ERROR: incorrect parsing of escaped colons in lowerdir+ mount option"
@@ -63,14 +63,14 @@ $MOUNT_PROG -t overlay none $SCRATCH_MNT \
 $UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
 
 # mount is expected to succeed without escaped colons.
-$MOUNT_PROG -t overlay ovl_esc_test $SCRATCH_MNT \
+_mount -t overlay ovl_esc_test $SCRATCH_MNT \
 	-o"lowerdir+=$lowerdir_colons,datadir+=$lowerdir_spaces" \
 	-o redirect_dir=follow,metacopy=on \
 	2>&1 | tee -a $seqres.full
 
 # if spaces are not escaped when showing mount options,
 # mount command will not show the word 'spaces' after the spaces
-$MOUNT_PROG -t overlay | grep ovl_esc_test | tee -a $seqres.full | \
+_mount -t overlay | grep ovl_esc_test | tee -a $seqres.full | \
 	grep -q 'datadir+'.*spaces || \
 	echo "ERROR: escaped spaces truncated from datadir+ mount option"
 
diff --git a/tests/xfs/078 b/tests/xfs/078
index 834c99a0020153..4224fd40bc9fea 100755
--- a/tests/xfs/078
+++ b/tests/xfs/078
@@ -75,7 +75,7 @@ _grow_loop()
 	$XFS_IO_PROG -c "pwrite $new_size $bsize" $LOOP_IMG | _filter_io
 	LOOP_DEV=`_create_loop_device $LOOP_IMG`
 	echo "*** mount loop filesystem"
-	$MOUNT_PROG -t xfs $LOOP_DEV $LOOP_MNT
+	_mount -t xfs $LOOP_DEV $LOOP_MNT
 
 	echo "*** grow loop filesystem"
 	$XFS_GROWFS_PROG $LOOP_MNT 2>&1 |  _filter_growfs 2>&1
diff --git a/tests/xfs/149 b/tests/xfs/149
index f1b2405e7bff11..bbaf86132dff37 100755
--- a/tests/xfs/149
+++ b/tests/xfs/149
@@ -64,7 +64,7 @@ $XFS_GROWFS_PROG $loop_symlink 2>&1 | sed -e s:$loop_symlink:LOOPSYMLINK:
 # These mounted operations should pass
 
 echo "=== mount ==="
-$MOUNT_PROG $loop_dev $mntdir || _fail "!!! failed to loopback mount"
+_mount $loop_dev $mntdir || _fail "!!! failed to loopback mount"
 
 echo "=== xfs_growfs - check device node ==="
 $XFS_GROWFS_PROG -D 8192 $loop_dev > /dev/null
@@ -76,7 +76,7 @@ echo "=== unmount ==="
 $UMOUNT_PROG $mntdir || _fail "!!! failed to unmount"
 
 echo "=== mount device symlink ==="
-$MOUNT_PROG $loop_symlink $mntdir || _fail "!!! failed to loopback mount"
+_mount $loop_symlink $mntdir || _fail "!!! failed to loopback mount"
 
 echo "=== xfs_growfs - check device symlink ==="
 $XFS_GROWFS_PROG -D 16384 $loop_symlink > /dev/null
diff --git a/tests/xfs/289 b/tests/xfs/289
index cf0f2883c4f373..089a3f8cc14a68 100755
--- a/tests/xfs/289
+++ b/tests/xfs/289
@@ -56,7 +56,7 @@ echo "=== xfs_growfs - plain file - should be rejected ==="
 $XFS_GROWFS_PROG $tmpfile 2>&1 | _filter_test_dir
 
 echo "=== mount ==="
-$MOUNT_PROG -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount"
+_mount -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount"
 
 echo "=== xfs_growfs - mounted - check absolute path ==="
 $XFS_GROWFS_PROG -D 8192 $tmpdir | _filter_test_dir > /dev/null
@@ -79,7 +79,7 @@ $XFS_GROWFS_PROG -D 28672 tmpsymlink.$$ > /dev/null
 
 echo "=== xfs_growfs - bind mount ==="
 mkdir $tmpbind
-$MOUNT_PROG -o bind $tmpdir $tmpbind
+_mount -o bind $tmpdir $tmpbind
 $XFS_GROWFS_PROG -D 32768 $tmpbind | _filter_test_dir > /dev/null
 
 echo "=== xfs_growfs - bind mount - relative path ==="
diff --git a/tests/xfs/544 b/tests/xfs/544
index bd694453d5409f..a3a23c1726ca1c 100755
--- a/tests/xfs/544
+++ b/tests/xfs/544
@@ -35,7 +35,7 @@ mkdir $TEST_DIR/dest.$seq
 # Test
 echo "*** dump with bind-mounted test ***" >> $seqres.full
 
-$MOUNT_PROG --bind $TEST_DIR/src.$seq $TEST_DIR/dest.$seq || _fail "Bind mount failed"
+_mount --bind $TEST_DIR/src.$seq $TEST_DIR/dest.$seq || _fail "Bind mount failed"
 
 $XFSDUMP_PROG -L session -M test -f $tmp.dump $TEST_DIR/dest.$seq \
 	>> $seqres.full 2>&1 && echo "dump with bind-mounted should be failed, but passed."


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 2/2] check: capture dmesg of mount failures if test fails
  2024-12-31 23:35 ` [PATCHSET 3/5] fstests: capture logs from mount failures Darrick J. Wong
  2024-12-31 23:56   ` [PATCH 1/2] treewide: convert all $MOUNT_PROG to _mount Darrick J. Wong
@ 2024-12-31 23:56   ` Darrick J. Wong
  2025-01-06 11:18     ` Nirjhar Roy
  1 sibling, 1 reply; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:56 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

From: Darrick J. Wong <djwong@kernel.org>

Capture the kernel output after a mount failure occurs.  If the test
itself fails, then keep the logging output for further diagnosis.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 check                  |   22 +++++++++++++++++++++-
 common/rc              |   26 +++++++++++++++++++++++++-
 common/report          |    8 ++++++++
 tests/selftest/008     |   20 ++++++++++++++++++++
 tests/selftest/008.out |    1 +
 5 files changed, 75 insertions(+), 2 deletions(-)
 create mode 100755 tests/selftest/008
 create mode 100644 tests/selftest/008.out


diff --git a/check b/check
index 9222cd7e4f8197..a46ea1a54d78bb 100755
--- a/check
+++ b/check
@@ -614,7 +614,7 @@ _stash_fail_loop_files() {
 	local seq_prefix="${REPORT_DIR}/${1}"
 	local cp_suffix="$2"
 
-	for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core" ".hints"; do
+	for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core" ".hints" ".mountfail"; do
 		rm -f "${seq_prefix}${i}${cp_suffix}"
 		if [ -f "${seq_prefix}${i}" ]; then
 			cp "${seq_prefix}${i}" "${seq_prefix}${i}${cp_suffix}"
@@ -994,6 +994,7 @@ function run_section()
 				      echo -n "	$seqnum -- "
 			cat $seqres.notrun
 			tc_status="notrun"
+			rm -f "$seqres.mountfail?"
 			_stash_test_status "$seqnum" "$tc_status"
 
 			# Unmount the scratch fs so that we can wipe the scratch
@@ -1053,6 +1054,7 @@ function run_section()
 		if [ ! -f $seq.out ]; then
 			_dump_err "no qualified output"
 			tc_status="fail"
+			rm -f "$seqres.mountfail?"
 			_stash_test_status "$seqnum" "$tc_status"
 			continue;
 		fi
@@ -1089,6 +1091,24 @@ function run_section()
 				rm -f $seqres.hints
 			fi
 		fi
+
+		if [ -f "$seqres.mountfail?" ]; then
+			if [ "$tc_status" = "fail" ]; then
+				# Let the user know if there were mount
+				# failures on a test that failed because that
+				# could be interesting.
+				mv "$seqres.mountfail?" "$seqres.mountfail"
+				_dump_err "check: possible mount failures (see $seqres.mountfail)"
+				test -f $seqres.mountfail && \
+					maybe_compress_logfile $seqres.mountfail $MAX_MOUNTFAIL_SIZE
+			else
+				# Don't retain mount failure logs for tests
+				# that pass or were skipped because some tests
+				# intentionally drive mount failures.
+				rm -f "$seqres.mountfail?"
+			fi
+		fi
+
 		_stash_test_status "$seqnum" "$tc_status"
 	done
 
diff --git a/common/rc b/common/rc
index d7dfb55bbbd7e1..0ede68eb912440 100644
--- a/common/rc
+++ b/common/rc
@@ -204,9 +204,33 @@ _get_hugepagesize()
 	awk '/Hugepagesize/ {print $2 * 1024}' /proc/meminfo
 }
 
+# Does dmesg have a --since flag?
+_dmesg_detect_since()
+{
+	if [ -z "$DMESG_HAS_SINCE" ]; then
+		test "$DMESG_HAS_SINCE" = "yes"
+		return
+	elif dmesg --help | grep -q -- --since; then
+		DMESG_HAS_SINCE=yes
+	else
+		DMESG_HAS_SINCE=no
+	fi
+}
+
 _mount()
 {
-    $MOUNT_PROG $*
+	$MOUNT_PROG $*
+	ret=$?
+	if [ "$ret" -ne 0 ]; then
+		echo "\"$MOUNT_PROG $*\" failed at $(date)" >> "$seqres.mountfail?"
+		if _dmesg_detect_since; then
+			dmesg --since '30s ago' >> "$seqres.mountfail?"
+		else
+			dmesg | tail -n 100 >> "$seqres.mountfail?"
+		fi
+	fi
+
+	return $ret
 }
 
 # Call _mount to do mount operation but also save mountpoint to
diff --git a/common/report b/common/report
index 0e91e481f9725a..b57697f76dafb2 100644
--- a/common/report
+++ b/common/report
@@ -199,6 +199,7 @@ _xunit_make_testcase_report()
 		local out_src="${SRC_DIR}/${test_name}.out"
 		local full_file="${REPORT_DIR}/${test_name}.full"
 		local dmesg_file="${REPORT_DIR}/${test_name}.dmesg"
+		local mountfail_file="${REPORT_DIR}/${test_name}.mountfail"
 		local outbad_file="${REPORT_DIR}/${test_name}.out.bad"
 		if [ -z "$_err_msg" ]; then
 			_err_msg="Test $test_name failed, reason unknown"
@@ -225,6 +226,13 @@ _xunit_make_testcase_report()
 			printf ']]>\n'	>>$report
 			echo -e "\t\t</system-err>" >> $report
 		fi
+		if [ -z "$quiet" -a -f "$mountfail_file" ]; then
+			echo -e "\t\t<mount-failure>" >> $report
+			printf	'<![CDATA[\n' >>$report
+			cat "$mountfail_file" | tr -dc '[:print:][:space:]' | encode_cdata >>$report
+			printf ']]>\n'	>>$report
+			echo -e "\t\t</mount-failure>" >> $report
+		fi
 		;;
 	*)
 		echo -e "\t\t<failure message=\"Unknown test_status=$test_status\" type=\"TestFail\"/>" >> $report
diff --git a/tests/selftest/008 b/tests/selftest/008
new file mode 100755
index 00000000000000..db80ffe6f77339
--- /dev/null
+++ b/tests/selftest/008
@@ -0,0 +1,20 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024-2025 Oracle.  All Rights Reserved.
+#
+# FS QA Test 008
+#
+# Test mount failure capture.
+#
+. ./common/preamble
+_begin_fstest selftest
+
+_require_command "$WIPEFS_PROG" wipefs
+_require_scratch
+
+$WIPEFS_PROG -a $SCRATCH_DEV
+_scratch_mount &>> $seqres.full
+
+# success, all done
+status=0
+exit
diff --git a/tests/selftest/008.out b/tests/selftest/008.out
new file mode 100644
index 00000000000000..aaff95f3f48372
--- /dev/null
+++ b/tests/selftest/008.out
@@ -0,0 +1 @@
+QA output created by 008


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* Re: [PATCH 2/2] check: capture dmesg of mount failures if test fails
  2024-12-31 23:56   ` [PATCH 2/2] check: capture dmesg of mount failures if test fails Darrick J. Wong
@ 2025-01-06 11:18     ` Nirjhar Roy
  2025-01-06 23:52       ` Darrick J. Wong
  0 siblings, 1 reply; 24+ messages in thread
From: Nirjhar Roy @ 2025-01-06 11:18 UTC (permalink / raw)
  To: Darrick J. Wong, zlang; +Cc: fstests, linux-xfs

On Tue, 2024-12-31 at 15:56 -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
> 
> Capture the kernel output after a mount failure occurs.  If the test
> itself fails, then keep the logging output for further diagnosis.
> 
> Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
> ---
>  check                  |   22 +++++++++++++++++++++-
>  common/rc              |   26 +++++++++++++++++++++++++-
>  common/report          |    8 ++++++++
>  tests/selftest/008     |   20 ++++++++++++++++++++
>  tests/selftest/008.out |    1 +
>  5 files changed, 75 insertions(+), 2 deletions(-)
>  create mode 100755 tests/selftest/008
>  create mode 100644 tests/selftest/008.out
> 
> 
> diff --git a/check b/check
> index 9222cd7e4f8197..a46ea1a54d78bb 100755
> --- a/check
> +++ b/check
> @@ -614,7 +614,7 @@ _stash_fail_loop_files() {
>  	local seq_prefix="${REPORT_DIR}/${1}"
>  	local cp_suffix="$2"
>  
> -	for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core"
> ".hints"; do
> +	for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core" ".hints"
> ".mountfail"; do
>  		rm -f "${seq_prefix}${i}${cp_suffix}"
>  		if [ -f "${seq_prefix}${i}" ]; then
>  			cp "${seq_prefix}${i}"
> "${seq_prefix}${i}${cp_suffix}"
> @@ -994,6 +994,7 @@ function run_section()
>  				      echo -n "	$seqnum -- "
>  			cat $seqres.notrun
>  			tc_status="notrun"
> +			rm -f "$seqres.mountfail?"
>  			_stash_test_status "$seqnum" "$tc_status"
>  
>  			# Unmount the scratch fs so that we can wipe
> the scratch
> @@ -1053,6 +1054,7 @@ function run_section()
>  		if [ ! -f $seq.out ]; then
>  			_dump_err "no qualified output"
>  			tc_status="fail"
> +			rm -f "$seqres.mountfail?"
>  			_stash_test_status "$seqnum" "$tc_status"
>  			continue;
>  		fi
> @@ -1089,6 +1091,24 @@ function run_section()
>  				rm -f $seqres.hints
>  			fi
>  		fi
> +
> +		if [ -f "$seqres.mountfail?" ]; then
> +			if [ "$tc_status" = "fail" ]; then
> +				# Let the user know if there were mount
> +				# failures on a test that failed
> because that
> +				# could be interesting.
> +				mv "$seqres.mountfail?"
> "$seqres.mountfail"
> +				_dump_err "check: possible mount
> failures (see $seqres.mountfail)"
> +				test -f $seqres.mountfail && \
> +					maybe_compress_logfile
> $seqres.mountfail $MAX_MOUNTFAIL_SIZE
> +			else
> +				# Don't retain mount failure logs for
> tests
> +				# that pass or were skipped because
> some tests
> +				# intentionally drive mount failures.
> +				rm -f "$seqres.mountfail?"
> +			fi
> +		fi
> +
>  		_stash_test_status "$seqnum" "$tc_status"
>  	done
>  
> diff --git a/common/rc b/common/rc
> index d7dfb55bbbd7e1..0ede68eb912440 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -204,9 +204,33 @@ _get_hugepagesize()
>  	awk '/Hugepagesize/ {print $2 * 1024}' /proc/meminfo
>  }
>  
> +# Does dmesg have a --since flag?
> +_dmesg_detect_since()
> +{
> +	if [ -z "$DMESG_HAS_SINCE" ]; then
> +		test "$DMESG_HAS_SINCE" = "yes"
> +		return
> +	elif dmesg --help | grep -q -- --since; then
> +		DMESG_HAS_SINCE=yes
> +	else
> +		DMESG_HAS_SINCE=no
> +	fi
> +}
> +
>  _mount()
>  {
> -    $MOUNT_PROG $*
> +	$MOUNT_PROG $*
> +	ret=$?
> +	if [ "$ret" -ne 0 ]; then
> +		echo "\"$MOUNT_PROG $*\" failed at $(date)" >>
> "$seqres.mountfail?"
> +		if _dmesg_detect_since; then
> +			dmesg --since '30s ago' >> "$seqres.mountfail?"
> +		else
> +			dmesg | tail -n 100 >> "$seqres.mountfail?"
Is it possible to grep for a mount failure message in dmesg and then
capture the last n lines? Do you think that will be more accurate?

Also, do you think it is useful to make this 100 configurable instead
of hardcoding? 
> +		fi
> +	fi
> +
> +	return $ret
>  }
>  
>  # Call _mount to do mount operation but also save mountpoint to
> diff --git a/common/report b/common/report
> index 0e91e481f9725a..b57697f76dafb2 100644
> --- a/common/report
> +++ b/common/report
> @@ -199,6 +199,7 @@ _xunit_make_testcase_report()
>  		local out_src="${SRC_DIR}/${test_name}.out"
>  		local full_file="${REPORT_DIR}/${test_name}.full"
>  		local dmesg_file="${REPORT_DIR}/${test_name}.dmesg"
> +		local
> mountfail_file="${REPORT_DIR}/${test_name}.mountfail"
>  		local outbad_file="${REPORT_DIR}/${test_name}.out.bad"
>  		if [ -z "$_err_msg" ]; then
>  			_err_msg="Test $test_name failed, reason
> unknown"
> @@ -225,6 +226,13 @@ _xunit_make_testcase_report()
>  			printf ']]>\n'	>>$report
>  			echo -e "\t\t</system-err>" >> $report
>  		fi
> +		if [ -z "$quiet" -a -f "$mountfail_file" ]; then
> +			echo -e "\t\t<mount-failure>" >> $report
> +			printf	'<![CDATA[\n' >>$report
> +			cat "$mountfail_file" | tr -dc
> '[:print:][:space:]' | encode_cdata >>$report
> +			printf ']]>\n'	>>$report
> +			echo -e "\t\t</mount-failure>" >> $report
> +		fi
>  		;;
>  	*)
>  		echo -e "\t\t<failure message=\"Unknown
> test_status=$test_status\" type=\"TestFail\"/>" >> $report
> diff --git a/tests/selftest/008 b/tests/selftest/008
> new file mode 100755
> index 00000000000000..db80ffe6f77339
> --- /dev/null
> +++ b/tests/selftest/008
> @@ -0,0 +1,20 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2024-2025 Oracle.  All Rights Reserved.
> +#
> +# FS QA Test 008
> +#
> +# Test mount failure capture.
> +#
> +. ./common/preamble
> +_begin_fstest selftest
> +
> +_require_command "$WIPEFS_PROG" wipefs
> +_require_scratch
> +
> +$WIPEFS_PROG -a $SCRATCH_DEV
> +_scratch_mount &>> $seqres.full
Minor: Do you think adding some filtered messages from the captured
dmesg logs in the output will be helpful?  
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/selftest/008.out b/tests/selftest/008.out
> new file mode 100644
> index 00000000000000..aaff95f3f48372
> --- /dev/null
> +++ b/tests/selftest/008.out
> @@ -0,0 +1 @@
> +QA output created by 008
> 


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 2/2] check: capture dmesg of mount failures if test fails
  2025-01-06 11:18     ` Nirjhar Roy
@ 2025-01-06 23:52       ` Darrick J. Wong
  2025-01-13  5:55         ` Nirjhar Roy
  0 siblings, 1 reply; 24+ messages in thread
From: Darrick J. Wong @ 2025-01-06 23:52 UTC (permalink / raw)
  To: Nirjhar Roy; +Cc: zlang, fstests, linux-xfs

On Mon, Jan 06, 2025 at 04:48:34PM +0530, Nirjhar Roy wrote:
> On Tue, 2024-12-31 at 15:56 -0800, Darrick J. Wong wrote:
> > From: Darrick J. Wong <djwong@kernel.org>
> > 
> > Capture the kernel output after a mount failure occurs.  If the test
> > itself fails, then keep the logging output for further diagnosis.
> > 
> > Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
> > ---
> >  check                  |   22 +++++++++++++++++++++-
> >  common/rc              |   26 +++++++++++++++++++++++++-
> >  common/report          |    8 ++++++++
> >  tests/selftest/008     |   20 ++++++++++++++++++++
> >  tests/selftest/008.out |    1 +
> >  5 files changed, 75 insertions(+), 2 deletions(-)
> >  create mode 100755 tests/selftest/008
> >  create mode 100644 tests/selftest/008.out
> > 
> > 
> > diff --git a/check b/check
> > index 9222cd7e4f8197..a46ea1a54d78bb 100755
> > --- a/check
> > +++ b/check
> > @@ -614,7 +614,7 @@ _stash_fail_loop_files() {
> >  	local seq_prefix="${REPORT_DIR}/${1}"
> >  	local cp_suffix="$2"
> >  
> > -	for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core"
> > ".hints"; do
> > +	for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core" ".hints"
> > ".mountfail"; do
> >  		rm -f "${seq_prefix}${i}${cp_suffix}"
> >  		if [ -f "${seq_prefix}${i}" ]; then
> >  			cp "${seq_prefix}${i}"
> > "${seq_prefix}${i}${cp_suffix}"
> > @@ -994,6 +994,7 @@ function run_section()
> >  				      echo -n "	$seqnum -- "
> >  			cat $seqres.notrun
> >  			tc_status="notrun"
> > +			rm -f "$seqres.mountfail?"
> >  			_stash_test_status "$seqnum" "$tc_status"
> >  
> >  			# Unmount the scratch fs so that we can wipe
> > the scratch
> > @@ -1053,6 +1054,7 @@ function run_section()
> >  		if [ ! -f $seq.out ]; then
> >  			_dump_err "no qualified output"
> >  			tc_status="fail"
> > +			rm -f "$seqres.mountfail?"
> >  			_stash_test_status "$seqnum" "$tc_status"
> >  			continue;
> >  		fi
> > @@ -1089,6 +1091,24 @@ function run_section()
> >  				rm -f $seqres.hints
> >  			fi
> >  		fi
> > +
> > +		if [ -f "$seqres.mountfail?" ]; then
> > +			if [ "$tc_status" = "fail" ]; then
> > +				# Let the user know if there were mount
> > +				# failures on a test that failed
> > because that
> > +				# could be interesting.
> > +				mv "$seqres.mountfail?"
> > "$seqres.mountfail"
> > +				_dump_err "check: possible mount
> > failures (see $seqres.mountfail)"
> > +				test -f $seqres.mountfail && \
> > +					maybe_compress_logfile
> > $seqres.mountfail $MAX_MOUNTFAIL_SIZE
> > +			else
> > +				# Don't retain mount failure logs for
> > tests
> > +				# that pass or were skipped because
> > some tests
> > +				# intentionally drive mount failures.
> > +				rm -f "$seqres.mountfail?"
> > +			fi
> > +		fi
> > +
> >  		_stash_test_status "$seqnum" "$tc_status"
> >  	done
> >  
> > diff --git a/common/rc b/common/rc
> > index d7dfb55bbbd7e1..0ede68eb912440 100644
> > --- a/common/rc
> > +++ b/common/rc
> > @@ -204,9 +204,33 @@ _get_hugepagesize()
> >  	awk '/Hugepagesize/ {print $2 * 1024}' /proc/meminfo
> >  }
> >  
> > +# Does dmesg have a --since flag?
> > +_dmesg_detect_since()
> > +{
> > +	if [ -z "$DMESG_HAS_SINCE" ]; then
> > +		test "$DMESG_HAS_SINCE" = "yes"
> > +		return
> > +	elif dmesg --help | grep -q -- --since; then
> > +		DMESG_HAS_SINCE=yes
> > +	else
> > +		DMESG_HAS_SINCE=no
> > +	fi
> > +}
> > +
> >  _mount()
> >  {
> > -    $MOUNT_PROG $*
> > +	$MOUNT_PROG $*
> > +	ret=$?
> > +	if [ "$ret" -ne 0 ]; then
> > +		echo "\"$MOUNT_PROG $*\" failed at $(date)" >>
> > "$seqres.mountfail?"
> > +		if _dmesg_detect_since; then
> > +			dmesg --since '30s ago' >> "$seqres.mountfail?"
> > +		else
> > +			dmesg | tail -n 100 >> "$seqres.mountfail?"
> Is it possible to grep for a mount failure message in dmesg and then
> capture the last n lines? Do you think that will be more accurate?

Alas no, because there's no standard mount failure log message for us to
latch onto.

> Also, do you think it is useful to make this 100 configurable instead
> of hardcoding? 

I suppose, but why do you need more than 100?

> > +		fi
> > +	fi
> > +
> > +	return $ret
> >  }
> >  
> >  # Call _mount to do mount operation but also save mountpoint to
> > diff --git a/common/report b/common/report
> > index 0e91e481f9725a..b57697f76dafb2 100644
> > --- a/common/report
> > +++ b/common/report
> > @@ -199,6 +199,7 @@ _xunit_make_testcase_report()
> >  		local out_src="${SRC_DIR}/${test_name}.out"
> >  		local full_file="${REPORT_DIR}/${test_name}.full"
> >  		local dmesg_file="${REPORT_DIR}/${test_name}.dmesg"
> > +		local
> > mountfail_file="${REPORT_DIR}/${test_name}.mountfail"
> >  		local outbad_file="${REPORT_DIR}/${test_name}.out.bad"
> >  		if [ -z "$_err_msg" ]; then
> >  			_err_msg="Test $test_name failed, reason
> > unknown"
> > @@ -225,6 +226,13 @@ _xunit_make_testcase_report()
> >  			printf ']]>\n'	>>$report
> >  			echo -e "\t\t</system-err>" >> $report
> >  		fi
> > +		if [ -z "$quiet" -a -f "$mountfail_file" ]; then
> > +			echo -e "\t\t<mount-failure>" >> $report
> > +			printf	'<![CDATA[\n' >>$report
> > +			cat "$mountfail_file" | tr -dc
> > '[:print:][:space:]' | encode_cdata >>$report
> > +			printf ']]>\n'	>>$report
> > +			echo -e "\t\t</mount-failure>" >> $report
> > +		fi
> >  		;;
> >  	*)
> >  		echo -e "\t\t<failure message=\"Unknown
> > test_status=$test_status\" type=\"TestFail\"/>" >> $report
> > diff --git a/tests/selftest/008 b/tests/selftest/008
> > new file mode 100755
> > index 00000000000000..db80ffe6f77339
> > --- /dev/null
> > +++ b/tests/selftest/008
> > @@ -0,0 +1,20 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +# Copyright (c) 2024-2025 Oracle.  All Rights Reserved.
> > +#
> > +# FS QA Test 008
> > +#
> > +# Test mount failure capture.
> > +#
> > +. ./common/preamble
> > +_begin_fstest selftest
> > +
> > +_require_command "$WIPEFS_PROG" wipefs
> > +_require_scratch
> > +
> > +$WIPEFS_PROG -a $SCRATCH_DEV
> > +_scratch_mount &>> $seqres.full
> Minor: Do you think adding some filtered messages from the captured
> dmesg logs in the output will be helpful?  

No, this test exists to make sure that the dmesg log is captured in
$RESULT_DIR.  We don't care about the mount(8) output.

--D

> > +
> > +# success, all done
> > +status=0
> > +exit
> > diff --git a/tests/selftest/008.out b/tests/selftest/008.out
> > new file mode 100644
> > index 00000000000000..aaff95f3f48372
> > --- /dev/null
> > +++ b/tests/selftest/008.out
> > @@ -0,0 +1 @@
> > +QA output created by 008
> > 
> 
> 

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH 2/2] check: capture dmesg of mount failures if test fails
  2025-01-06 23:52       ` Darrick J. Wong
@ 2025-01-13  5:55         ` Nirjhar Roy
  0 siblings, 0 replies; 24+ messages in thread
From: Nirjhar Roy @ 2025-01-13  5:55 UTC (permalink / raw)
  To: Darrick J. Wong; +Cc: zlang, fstests, linux-xfs

On Mon, 2025-01-06 at 15:52 -0800, Darrick J. Wong wrote:
> On Mon, Jan 06, 2025 at 04:48:34PM +0530, Nirjhar Roy wrote:
> > On Tue, 2024-12-31 at 15:56 -0800, Darrick J. Wong wrote:
> > > From: Darrick J. Wong <djwong@kernel.org>
> > > 
> > > Capture the kernel output after a mount failure occurs.  If the
> > > test
> > > itself fails, then keep the logging output for further diagnosis.
> > > 
> > > Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
> > > ---
> > >  check                  |   22 +++++++++++++++++++++-
> > >  common/rc              |   26 +++++++++++++++++++++++++-
> > >  common/report          |    8 ++++++++
> > >  tests/selftest/008     |   20 ++++++++++++++++++++
> > >  tests/selftest/008.out |    1 +
> > >  5 files changed, 75 insertions(+), 2 deletions(-)
> > >  create mode 100755 tests/selftest/008
> > >  create mode 100644 tests/selftest/008.out
> > > 
> > > 
> > > diff --git a/check b/check
> > > index 9222cd7e4f8197..a46ea1a54d78bb 100755
> > > --- a/check
> > > +++ b/check
> > > @@ -614,7 +614,7 @@ _stash_fail_loop_files() {
> > >  	local seq_prefix="${REPORT_DIR}/${1}"
> > >  	local cp_suffix="$2"
> > >  
> > > -	for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core"
> > > ".hints"; do
> > > +	for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core" ".hints"
> > > ".mountfail"; do
> > >  		rm -f "${seq_prefix}${i}${cp_suffix}"
> > >  		if [ -f "${seq_prefix}${i}" ]; then
> > >  			cp "${seq_prefix}${i}"
> > > "${seq_prefix}${i}${cp_suffix}"
> > > @@ -994,6 +994,7 @@ function run_section()
> > >  				      echo -n "	$seqnum -- "
> > >  			cat $seqres.notrun
> > >  			tc_status="notrun"
> > > +			rm -f "$seqres.mountfail?"
> > >  			_stash_test_status "$seqnum" "$tc_status"
> > >  
> > >  			# Unmount the scratch fs so that we can wipe
> > > the scratch
> > > @@ -1053,6 +1054,7 @@ function run_section()
> > >  		if [ ! -f $seq.out ]; then
> > >  			_dump_err "no qualified output"
> > >  			tc_status="fail"
> > > +			rm -f "$seqres.mountfail?"
> > >  			_stash_test_status "$seqnum" "$tc_status"
> > >  			continue;
> > >  		fi
> > > @@ -1089,6 +1091,24 @@ function run_section()
> > >  				rm -f $seqres.hints
> > >  			fi
> > >  		fi
> > > +
> > > +		if [ -f "$seqres.mountfail?" ]; then
> > > +			if [ "$tc_status" = "fail" ]; then
> > > +				# Let the user know if there were mount
> > > +				# failures on a test that failed
> > > because that
> > > +				# could be interesting.
> > > +				mv "$seqres.mountfail?"
> > > "$seqres.mountfail"
> > > +				_dump_err "check: possible mount
> > > failures (see $seqres.mountfail)"
> > > +				test -f $seqres.mountfail && \
> > > +					maybe_compress_logfile
> > > $seqres.mountfail $MAX_MOUNTFAIL_SIZE
> > > +			else
> > > +				# Don't retain mount failure logs for
> > > tests
> > > +				# that pass or were skipped because
> > > some tests
> > > +				# intentionally drive mount failures.
> > > +				rm -f "$seqres.mountfail?"
> > > +			fi
> > > +		fi
> > > +
> > >  		_stash_test_status "$seqnum" "$tc_status"
> > >  	done
> > >  
> > > diff --git a/common/rc b/common/rc
> > > index d7dfb55bbbd7e1..0ede68eb912440 100644
> > > --- a/common/rc
> > > +++ b/common/rc
> > > @@ -204,9 +204,33 @@ _get_hugepagesize()
> > >  	awk '/Hugepagesize/ {print $2 * 1024}' /proc/meminfo
> > >  }
> > >  
> > > +# Does dmesg have a --since flag?
> > > +_dmesg_detect_since()
> > > +{
> > > +	if [ -z "$DMESG_HAS_SINCE" ]; then
> > > +		test "$DMESG_HAS_SINCE" = "yes"
> > > +		return
> > > +	elif dmesg --help | grep -q -- --since; then
> > > +		DMESG_HAS_SINCE=yes
> > > +	else
> > > +		DMESG_HAS_SINCE=no
> > > +	fi
> > > +}
> > > +
> > >  _mount()
> > >  {
> > > -    $MOUNT_PROG $*
> > > +	$MOUNT_PROG $*
> > > +	ret=$?
> > > +	if [ "$ret" -ne 0 ]; then
> > > +		echo "\"$MOUNT_PROG $*\" failed at $(date)" >>
> > > "$seqres.mountfail?"
> > > +		if _dmesg_detect_since; then
> > > +			dmesg --since '30s ago' >> "$seqres.mountfail?"
> > > +		else
> > > +			dmesg | tail -n 100 >> "$seqres.mountfail?"
> > Is it possible to grep for a mount failure message in dmesg and
> > then
> > capture the last n lines? Do you think that will be more accurate?
> 
> Alas no, because there's no standard mount failure log message for us
> to
> latch onto.
Okay makes sense. 
> 
> > Also, do you think it is useful to make this 100 configurable
> > instead
> > of hardcoding? 
> 
> I suppose, but why do you need more than 100?
So my thought behind this is that in case, the dmesg gets cluttered
with noisy logs from other processes. No hard preferences though. 
> 
> > > +		fi
> > > +	fi
> > > +
> > > +	return $ret
> > >  }
> > >  
> > >  # Call _mount to do mount operation but also save mountpoint to
> > > diff --git a/common/report b/common/report
> > > index 0e91e481f9725a..b57697f76dafb2 100644
> > > --- a/common/report
> > > +++ b/common/report
> > > @@ -199,6 +199,7 @@ _xunit_make_testcase_report()
> > >  		local out_src="${SRC_DIR}/${test_name}.out"
> > >  		local full_file="${REPORT_DIR}/${test_name}.full"
> > >  		local dmesg_file="${REPORT_DIR}/${test_name}.dmesg"
> > > +		local
> > > mountfail_file="${REPORT_DIR}/${test_name}.mountfail"
> > >  		local outbad_file="${REPORT_DIR}/${test_name}.out.bad"
> > >  		if [ -z "$_err_msg" ]; then
> > >  			_err_msg="Test $test_name failed, reason
> > > unknown"
> > > @@ -225,6 +226,13 @@ _xunit_make_testcase_report()
> > >  			printf ']]>\n'	>>$report
> > >  			echo -e "\t\t</system-err>" >> $report
> > >  		fi
> > > +		if [ -z "$quiet" -a -f "$mountfail_file" ]; then
> > > +			echo -e "\t\t<mount-failure>" >> $report
> > > +			printf	'<![CDATA[\n' >>$report
> > > +			cat "$mountfail_file" | tr -dc
> > > '[:print:][:space:]' | encode_cdata >>$report
> > > +			printf ']]>\n'	>>$report
> > > +			echo -e "\t\t</mount-failure>" >> $report
> > > +		fi
> > >  		;;
> > >  	*)
> > >  		echo -e "\t\t<failure message=\"Unknown
> > > test_status=$test_status\" type=\"TestFail\"/>" >> $report
> > > diff --git a/tests/selftest/008 b/tests/selftest/008
> > > new file mode 100755
> > > index 00000000000000..db80ffe6f77339
> > > --- /dev/null
> > > +++ b/tests/selftest/008
> > > @@ -0,0 +1,20 @@
> > > +#! /bin/bash
> > > +# SPDX-License-Identifier: GPL-2.0
> > > +# Copyright (c) 2024-2025 Oracle.  All Rights Reserved.
> > > +#
> > > +# FS QA Test 008
> > > +#
> > > +# Test mount failure capture.
> > > +#
> > > +. ./common/preamble
> > > +_begin_fstest selftest
> > > +
> > > +_require_command "$WIPEFS_PROG" wipefs
> > > +_require_scratch
> > > +
> > > +$WIPEFS_PROG -a $SCRATCH_DEV
> > > +_scratch_mount &>> $seqres.full
> > Minor: Do you think adding some filtered messages from the captured
> > dmesg logs in the output will be helpful?  
> 
> No, this test exists to make sure that the dmesg log is captured in
> $RESULT_DIR.  We don't care about the mount(8) output.
> 
> --D
Okay, got it.
--NR 
> 
> > > +
> > > +# success, all done
> > > +status=0
> > > +exit
> > > diff --git a/tests/selftest/008.out b/tests/selftest/008.out
> > > new file mode 100644
> > > index 00000000000000..aaff95f3f48372
> > > --- /dev/null
> > > +++ b/tests/selftest/008.out
> > > @@ -0,0 +1 @@
> > > +QA output created by 008
> > > 


^ permalink raw reply	[flat|nested] 24+ messages in thread

* [PATCHSET 4/5] fstests: live health monitoring of filesystems
  2024-12-31 23:25 [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing Darrick J. Wong
                   ` (2 preceding siblings ...)
  2024-12-31 23:35 ` [PATCHSET 3/5] fstests: capture logs from mount failures Darrick J. Wong
@ 2024-12-31 23:35 ` Darrick J. Wong
  2024-12-31 23:57   ` [PATCH 1/6] misc: convert all $UMOUNT_PROG to a _umount helper Darrick J. Wong
                     ` (5 more replies)
  2024-12-31 23:35 ` [PATCHSET 5/5] fstests: add difficult V5 features to filesystems Darrick J. Wong
  2025-01-02  1:37 ` [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing Stephen Zhang
  5 siblings, 6 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:35 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

Hi all,

This patchset builds off of Kent Overstreet's thread_with_file code to
deliver live information about filesystem health events to userspace.
This is done by creating a twf file and hooking internal operations so
that the event information can be queued to the twf without stalling the
kernel if the twf client program is nonresponsive.  This is a private
ioctl, so events are expressed using simple json objects so that we can
enrich the output later on without having to rev a ton of C structs.

In userspace, we create a new daemon program that will read the json
event objects and initiate repairs automatically.  This daemon is
managed entirely by systemd and will not block unmounting of the
filesystem unless repairs are ongoing.  It is autostarted via some
horrible udev rules.

If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.

This has been running on the djcloud for months with no problems.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=health-monitoring

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=health-monitoring

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=health-monitoring
---
Commits in this patchset:
 * misc: convert all $UMOUNT_PROG to a _umount helper
 * misc: convert all umount(1) invocations to _umount
 * xfs: test health monitoring code
 * xfs: test for metadata corruption error reporting via healthmon
 * xfs: test io error reporting via healthmon
 * xfs: test new xfs_scrubbed daemon
---
 common/btrfs        |    2 +
 common/config       |    6 +++
 common/dmdelay      |    4 +-
 common/dmdust       |    4 +-
 common/dmerror      |    4 +-
 common/dmflakey     |    4 +-
 common/dmhugedisk   |    2 +
 common/dmlogwrites  |    4 +-
 common/dmthin       |    4 +-
 common/overlay      |   10 +++---
 common/populate     |    8 ++---
 common/quota        |    2 +
 common/rc           |   47 ++++++++++++++++++---------
 common/systemd      |    9 +++++
 common/xfs          |   18 ++++++++++
 doc/group-names.txt |    1 +
 tests/btrfs/012     |    2 +
 tests/btrfs/020     |    2 +
 tests/btrfs/029     |    2 +
 tests/btrfs/031     |    2 +
 tests/btrfs/060     |    2 +
 tests/btrfs/065     |    2 +
 tests/btrfs/066     |    2 +
 tests/btrfs/067     |    2 +
 tests/btrfs/068     |    2 +
 tests/btrfs/075     |    2 +
 tests/btrfs/089     |    2 +
 tests/btrfs/124     |    2 +
 tests/btrfs/125     |    2 +
 tests/btrfs/185     |    4 +-
 tests/btrfs/197     |    4 +-
 tests/btrfs/199     |    2 +
 tests/btrfs/219     |   12 +++----
 tests/btrfs/254     |    2 +
 tests/btrfs/291     |    2 +
 tests/btrfs/298     |    4 +-
 tests/ext4/006      |    4 +-
 tests/ext4/007      |    4 +-
 tests/ext4/008      |    4 +-
 tests/ext4/009      |    8 ++---
 tests/ext4/010      |    6 ++-
 tests/ext4/011      |    2 +
 tests/ext4/012      |    2 +
 tests/ext4/013      |    6 ++-
 tests/ext4/014      |    6 ++-
 tests/ext4/015      |    6 ++-
 tests/ext4/016      |    6 ++-
 tests/ext4/017      |    6 ++-
 tests/ext4/018      |    6 ++-
 tests/ext4/019      |    6 ++-
 tests/ext4/032      |    4 +-
 tests/ext4/033      |    2 +
 tests/ext4/052      |    4 +-
 tests/ext4/053      |   32 +++++++++---------
 tests/ext4/056      |    2 +
 tests/generic/042   |    4 +-
 tests/generic/067   |    6 ++-
 tests/generic/081   |    2 +
 tests/generic/085   |    4 +-
 tests/generic/108   |    2 +
 tests/generic/171   |    2 +
 tests/generic/172   |    2 +
 tests/generic/173   |    2 +
 tests/generic/174   |    2 +
 tests/generic/306   |    2 +
 tests/generic/330   |    2 +
 tests/generic/332   |    2 +
 tests/generic/361   |    2 +
 tests/generic/373   |    2 +
 tests/generic/374   |    2 +
 tests/generic/395   |    2 +
 tests/generic/459   |    2 +
 tests/generic/563   |    4 +-
 tests/generic/604   |    2 +
 tests/generic/631   |    2 +
 tests/generic/648   |    6 ++-
 tests/generic/698   |    4 +-
 tests/generic/699   |    8 ++---
 tests/generic/704   |    2 +
 tests/generic/717   |    2 +
 tests/generic/730   |    2 +
 tests/generic/731   |    2 +
 tests/generic/732   |    4 +-
 tests/generic/746   |    8 ++---
 tests/overlay/003   |    2 +
 tests/overlay/004   |    2 +
 tests/overlay/005   |    6 ++-
 tests/overlay/014   |    4 +-
 tests/overlay/022   |    2 +
 tests/overlay/025   |    4 +-
 tests/overlay/029   |    6 ++-
 tests/overlay/031   |    8 ++---
 tests/overlay/035   |    2 +
 tests/overlay/036   |    8 ++---
 tests/overlay/037   |    6 ++-
 tests/overlay/040   |    2 +
 tests/overlay/041   |    2 +
 tests/overlay/042   |    2 +
 tests/overlay/043   |    2 +
 tests/overlay/044   |    2 +
 tests/overlay/048   |    4 +-
 tests/overlay/049   |    2 +
 tests/overlay/050   |    2 +
 tests/overlay/051   |    4 +-
 tests/overlay/052   |    2 +
 tests/overlay/053   |    4 +-
 tests/overlay/054   |    2 +
 tests/overlay/055   |    4 +-
 tests/overlay/056   |    2 +
 tests/overlay/057   |    4 +-
 tests/overlay/059   |    2 +
 tests/overlay/060   |    2 +
 tests/overlay/062   |    2 +
 tests/overlay/063   |    2 +
 tests/overlay/065   |   22 ++++++-------
 tests/overlay/067   |    2 +
 tests/overlay/068   |    4 +-
 tests/overlay/069   |    6 ++-
 tests/overlay/070   |    6 ++-
 tests/overlay/071   |    6 ++-
 tests/overlay/076   |    2 +
 tests/overlay/077   |    2 +
 tests/overlay/078   |    2 +
 tests/overlay/079   |    2 +
 tests/overlay/080   |    2 +
 tests/overlay/081   |   14 ++++----
 tests/overlay/083   |    2 +
 tests/overlay/084   |   10 +++---
 tests/overlay/085   |    2 +
 tests/overlay/086   |    8 ++---
 tests/xfs/014       |    4 +-
 tests/xfs/049       |    8 ++---
 tests/xfs/073       |    8 ++---
 tests/xfs/074       |    4 +-
 tests/xfs/078       |    4 +-
 tests/xfs/083       |    6 ++-
 tests/xfs/085       |    4 +-
 tests/xfs/086       |    8 ++---
 tests/xfs/087       |    6 ++-
 tests/xfs/088       |    8 ++---
 tests/xfs/089       |    8 ++---
 tests/xfs/091       |    8 ++---
 tests/xfs/093       |    6 ++-
 tests/xfs/097       |    6 ++-
 tests/xfs/098       |    4 +-
 tests/xfs/099       |    6 ++-
 tests/xfs/100       |    6 ++-
 tests/xfs/101       |    6 ++-
 tests/xfs/102       |    6 ++-
 tests/xfs/105       |    6 ++-
 tests/xfs/112       |    8 ++---
 tests/xfs/113       |    6 ++-
 tests/xfs/117       |    6 ++-
 tests/xfs/120       |    6 ++-
 tests/xfs/123       |    6 ++-
 tests/xfs/124       |    6 ++-
 tests/xfs/125       |    6 ++-
 tests/xfs/126       |    6 ++-
 tests/xfs/130       |    2 +
 tests/xfs/148       |    6 ++-
 tests/xfs/149       |    4 +-
 tests/xfs/152       |    2 +
 tests/xfs/169       |    6 ++-
 tests/xfs/186       |    4 +-
 tests/xfs/1878      |   80 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1878.out  |   10 ++++++
 tests/xfs/1879      |   89 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1879.out  |   12 +++++++
 tests/xfs/1882      |   64 +++++++++++++++++++++++++++++++++++++
 tests/xfs/1882.out  |    2 +
 tests/xfs/1883      |   75 +++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1883.out  |    2 +
 tests/xfs/1884      |   87 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1884.out  |    2 +
 tests/xfs/1885      |   53 ++++++++++++++++++++++++++++++
 tests/xfs/1885.out  |    5 +++
 tests/xfs/206       |    2 +
 tests/xfs/216       |    2 +
 tests/xfs/217       |    2 +
 tests/xfs/235       |    6 ++-
 tests/xfs/236       |    6 ++-
 tests/xfs/239       |    2 +
 tests/xfs/241       |    2 +
 tests/xfs/250       |    4 +-
 tests/xfs/265       |    6 ++-
 tests/xfs/289       |    4 +-
 tests/xfs/310       |    4 +-
 tests/xfs/507       |    2 +
 tests/xfs/513       |    4 +-
 tests/xfs/544       |    2 +
 tests/xfs/716       |    4 +-
 tests/xfs/806       |    4 +-
 192 files changed, 921 insertions(+), 391 deletions(-)
 create mode 100755 tests/xfs/1878
 create mode 100644 tests/xfs/1878.out
 create mode 100755 tests/xfs/1879
 create mode 100644 tests/xfs/1879.out
 create mode 100755 tests/xfs/1882
 create mode 100644 tests/xfs/1882.out
 create mode 100755 tests/xfs/1883
 create mode 100644 tests/xfs/1883.out
 create mode 100755 tests/xfs/1884
 create mode 100644 tests/xfs/1884.out
 create mode 100755 tests/xfs/1885
 create mode 100644 tests/xfs/1885.out


^ permalink raw reply	[flat|nested] 24+ messages in thread

* [PATCH 1/6] misc: convert all $UMOUNT_PROG to a _umount helper
  2024-12-31 23:35 ` [PATCHSET 4/5] fstests: live health monitoring of filesystems Darrick J. Wong
@ 2024-12-31 23:57   ` Darrick J. Wong
  2024-12-31 23:57   ` [PATCH 2/6] misc: convert all umount(1) invocations to _umount Darrick J. Wong
                     ` (4 subsequent siblings)
  5 siblings, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:57 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

From: Darrick J. Wong <djwong@kernel.org>

We're going to start collecting ephemeral(ish) filesystem stats in the
next patch, so switch all the $UMOUNT_PROG to a helper.

sed -e 's/$UMOUNT_PROG/_umount/g' -i $(git ls-files common tests check)

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 common/btrfs       |    2 +-
 common/dmdelay     |    4 ++--
 common/dmdust      |    4 ++--
 common/dmerror     |    2 +-
 common/dmflakey    |    4 ++--
 common/dmhugedisk  |    2 +-
 common/dmlogwrites |    4 ++--
 common/dmthin      |    4 ++--
 common/overlay     |   10 +++++-----
 common/rc          |   33 +++++++++++++++++++--------------
 tests/btrfs/020    |    2 +-
 tests/btrfs/029    |    2 +-
 tests/btrfs/031    |    2 +-
 tests/btrfs/060    |    2 +-
 tests/btrfs/065    |    2 +-
 tests/btrfs/066    |    2 +-
 tests/btrfs/067    |    2 +-
 tests/btrfs/068    |    2 +-
 tests/btrfs/075    |    2 +-
 tests/btrfs/089    |    2 +-
 tests/btrfs/124    |    2 +-
 tests/btrfs/125    |    2 +-
 tests/btrfs/185    |    4 ++--
 tests/btrfs/197    |    4 ++--
 tests/btrfs/219    |   12 ++++++------
 tests/btrfs/254    |    2 +-
 tests/ext4/032     |    4 ++--
 tests/ext4/052     |    4 ++--
 tests/ext4/053     |   32 ++++++++++++++++----------------
 tests/ext4/056     |    2 +-
 tests/generic/042  |    4 ++--
 tests/generic/067  |    6 +++---
 tests/generic/081  |    2 +-
 tests/generic/085  |    4 ++--
 tests/generic/108  |    2 +-
 tests/generic/361  |    2 +-
 tests/generic/373  |    2 +-
 tests/generic/374  |    2 +-
 tests/generic/459  |    2 +-
 tests/generic/604  |    2 ++
 tests/generic/648  |    6 +++---
 tests/generic/698  |    4 ++--
 tests/generic/699  |    8 ++++----
 tests/generic/704  |    2 +-
 tests/generic/730  |    2 +-
 tests/generic/731  |    2 +-
 tests/generic/732  |    4 ++--
 tests/generic/746  |    8 ++++----
 tests/overlay/003  |    2 +-
 tests/overlay/004  |    2 +-
 tests/overlay/005  |    6 +++---
 tests/overlay/014  |    4 ++--
 tests/overlay/022  |    2 +-
 tests/overlay/025  |    4 ++--
 tests/overlay/029  |    6 +++---
 tests/overlay/031  |    8 ++++----
 tests/overlay/035  |    2 +-
 tests/overlay/036  |    8 ++++----
 tests/overlay/037  |    6 +++---
 tests/overlay/040  |    2 +-
 tests/overlay/041  |    2 +-
 tests/overlay/042  |    2 +-
 tests/overlay/043  |    2 +-
 tests/overlay/044  |    2 +-
 tests/overlay/048  |    4 ++--
 tests/overlay/049  |    2 +-
 tests/overlay/050  |    2 +-
 tests/overlay/051  |    4 ++--
 tests/overlay/052  |    2 +-
 tests/overlay/053  |    4 ++--
 tests/overlay/054  |    2 +-
 tests/overlay/055  |    4 ++--
 tests/overlay/056  |    2 +-
 tests/overlay/057  |    4 ++--
 tests/overlay/059  |    2 +-
 tests/overlay/060  |    2 +-
 tests/overlay/062  |    2 +-
 tests/overlay/063  |    2 +-
 tests/overlay/065  |   22 +++++++++++-----------
 tests/overlay/067  |    2 +-
 tests/overlay/068  |    4 ++--
 tests/overlay/069  |    6 +++---
 tests/overlay/070  |    6 +++---
 tests/overlay/071  |    6 +++---
 tests/overlay/076  |    2 +-
 tests/overlay/077  |    2 +-
 tests/overlay/078  |    2 +-
 tests/overlay/079  |    2 +-
 tests/overlay/080  |    2 +-
 tests/overlay/081  |   14 +++++++-------
 tests/overlay/083  |    2 +-
 tests/overlay/084  |   10 +++++-----
 tests/overlay/085  |    2 +-
 tests/overlay/086  |    8 ++++----
 tests/xfs/078      |    4 ++--
 tests/xfs/148      |    6 +++---
 tests/xfs/149      |    4 ++--
 tests/xfs/186      |    4 ++--
 tests/xfs/289      |    4 ++--
 tests/xfs/507      |    2 +-
 tests/xfs/513      |    4 ++--
 tests/xfs/544      |    2 +-
 tests/xfs/806      |    4 ++--
 103 files changed, 226 insertions(+), 219 deletions(-)


diff --git a/common/btrfs b/common/btrfs
index 64f38cc240ab8b..b82c8f5a934cfd 100644
--- a/common/btrfs
+++ b/common/btrfs
@@ -352,7 +352,7 @@ _btrfs_stress_subvolume()
 	while [ ! -e $stop_file ]; do
 		$BTRFS_UTIL_PROG subvolume create $btrfs_mnt/$subvol_name
 		_mount -o subvol=$subvol_name $btrfs_dev $subvol_mnt
-		$UMOUNT_PROG $subvol_mnt
+		_umount $subvol_mnt
 		$BTRFS_UTIL_PROG subvolume delete $btrfs_mnt/$subvol_name
 	done
 }
diff --git a/common/dmdelay b/common/dmdelay
index 794ea37ba200ce..691e22538a622b 100644
--- a/common/dmdelay
+++ b/common/dmdelay
@@ -26,7 +26,7 @@ _mount_delay()
 
 _unmount_delay()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 _cleanup_delay()
@@ -34,7 +34,7 @@ _cleanup_delay()
 	# If dmsetup load fails then we need to make sure to do resume here
 	# otherwise the umount will hang
 	$DMSETUP_PROG resume delay-test > /dev/null 2>&1
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_umount $SCRATCH_MNT > /dev/null 2>&1
 	_dmsetup_remove delay-test
 }
 
diff --git a/common/dmdust b/common/dmdust
index 56fcc0e0fffa1e..13461c2dd3a006 100644
--- a/common/dmdust
+++ b/common/dmdust
@@ -22,7 +22,7 @@ _mount_dust()
 
 _unmount_dust()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 _cleanup_dust()
@@ -30,6 +30,6 @@ _cleanup_dust()
 	# If dmsetup load fails then we need to make sure to do resume here
 	# otherwise the umount will hang
 	$DMSETUP_PROG resume dust-test > /dev/null 2>&1
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_umount $SCRATCH_MNT > /dev/null 2>&1
 	_dmsetup_remove dust-test
 }
diff --git a/common/dmerror b/common/dmerror
index 2f006142a309fe..1e6a35230f3ccb 100644
--- a/common/dmerror
+++ b/common/dmerror
@@ -106,7 +106,7 @@ _dmerror_cleanup()
 	test -n "$NON_ERROR_RTDEV" && $DMSETUP_PROG resume error-rttest &>/dev/null
 	$DMSETUP_PROG resume error-test > /dev/null 2>&1
 
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_umount $SCRATCH_MNT > /dev/null 2>&1
 
 	test -n "$NON_ERROR_LOGDEV" && _dmsetup_remove error-logtest
 	test -n "$NON_ERROR_RTDEV" && _dmsetup_remove error-rttest
diff --git a/common/dmflakey b/common/dmflakey
index 52da3b100fbe45..64723f983b27ec 100644
--- a/common/dmflakey
+++ b/common/dmflakey
@@ -67,7 +67,7 @@ _mount_flakey()
 
 _unmount_flakey()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 _cleanup_flakey()
@@ -78,7 +78,7 @@ _cleanup_flakey()
 	test -n "$NON_FLAKEY_RTDEV" && $DMSETUP_PROG resume flakey-rttest &> /dev/null
 	$DMSETUP_PROG resume flakey-test > /dev/null 2>&1
 
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_umount $SCRATCH_MNT > /dev/null 2>&1
 
 	_dmsetup_remove flakey-test
 	test -n "$NON_FLAKEY_LOGDEV" && _dmsetup_remove flakey-logtest
diff --git a/common/dmhugedisk b/common/dmhugedisk
index 502f0243772d52..a02bff4351d9be 100644
--- a/common/dmhugedisk
+++ b/common/dmhugedisk
@@ -39,7 +39,7 @@ _dmhugedisk_init()
 
 _dmhugedisk_cleanup()
 {
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_umount $SCRATCH_MNT > /dev/null 2>&1
 	_dmsetup_remove huge-test
 	_dmsetup_remove huge-test-zero
 }
diff --git a/common/dmlogwrites b/common/dmlogwrites
index c054acb875a384..a1a5c415338276 100644
--- a/common/dmlogwrites
+++ b/common/dmlogwrites
@@ -145,7 +145,7 @@ _log_writes_mount()
 
 _log_writes_unmount()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 # _log_writes_replay_log <mark>
@@ -177,7 +177,7 @@ _log_writes_remove()
 
 _log_writes_cleanup()
 {
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_umount $SCRATCH_MNT > /dev/null 2>&1
 	_log_writes_remove
 }
 
diff --git a/common/dmthin b/common/dmthin
index 7107d50804896e..38d561c8eb25d6 100644
--- a/common/dmthin
+++ b/common/dmthin
@@ -23,7 +23,7 @@ DMTHIN_VOL_DEV="/dev/mapper/$DMTHIN_VOL_NAME"
 
 _dmthin_cleanup()
 {
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_umount $SCRATCH_MNT > /dev/null 2>&1
 	_dmsetup_remove $DMTHIN_VOL_NAME
 	_dmsetup_remove $DMTHIN_POOL_NAME
 	_dmsetup_remove $DMTHIN_META_NAME
@@ -32,7 +32,7 @@ _dmthin_cleanup()
 
 _dmthin_check_fs()
 {
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_umount $SCRATCH_MNT > /dev/null 2>&1
 	_check_scratch_fs $DMTHIN_VOL_DEV
 }
 
diff --git a/common/overlay b/common/overlay
index da1d8d2c3183f4..2877f31e22ebd9 100644
--- a/common/overlay
+++ b/common/overlay
@@ -142,18 +142,18 @@ _overlay_base_unmount()
 
 	[ -n "$dev" -a -n "$mnt" ] || return 0
 
-	$UMOUNT_PROG $mnt
+	_umount $mnt
 }
 
 _overlay_test_unmount()
 {
-	$UMOUNT_PROG $TEST_DIR
+	_umount $TEST_DIR
 	_overlay_base_unmount "$OVL_BASE_TEST_DEV" "$OVL_BASE_TEST_DIR"
 }
 
 _overlay_scratch_unmount()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 	_overlay_base_unmount "$OVL_BASE_SCRATCH_DEV" "$OVL_BASE_SCRATCH_MNT"
 }
 
@@ -342,7 +342,7 @@ _overlay_check_scratch_dirs()
 
 	# Need to umount overlay for scratch dir check
 	local ovl_mounted=`_is_dir_mountpoint $SCRATCH_MNT`
-	[ -z "$ovl_mounted" ] || $UMOUNT_PROG $SCRATCH_MNT
+	[ -z "$ovl_mounted" ] || _umount $SCRATCH_MNT
 
 	# Check dirs with extra overlay options
 	_overlay_check_dirs $lowerdir $upperdir $workdir $*
@@ -387,7 +387,7 @@ _overlay_check_fs()
 	else
 		# Check and umount overlay for dir check
 		ovl_mounted=`_is_dir_mountpoint $ovl_mnt`
-		[ -z "$ovl_mounted" ] || $UMOUNT_PROG $ovl_mnt
+		[ -z "$ovl_mounted" ] || _umount $ovl_mnt
 	fi
 
 	_overlay_check_dirs $base_mnt/$OVL_LOWER $base_mnt/$OVL_UPPER \
diff --git a/common/rc b/common/rc
index 0ede68eb912440..d3ee76e01db892 100644
--- a/common/rc
+++ b/common/rc
@@ -233,6 +233,11 @@ _mount()
 	return $ret
 }
 
+_umount()
+{
+	$UMOUNT_PROG $*
+}
+
 # Call _mount to do mount operation but also save mountpoint to
 # MOUNTED_POINT_STACK. Note that the mount point must be the last parameter
 _get_mount()
@@ -266,7 +271,7 @@ _put_mount()
 	local last_mnt=`echo $MOUNTED_POINT_STACK | awk '{print $1}'`
 
 	if [ -n "$last_mnt" ]; then
-		$UMOUNT_PROG $last_mnt
+		_umount $last_mnt
 	fi
 	MOUNTED_POINT_STACK=`echo $MOUNTED_POINT_STACK | cut -d\  -f2-`
 }
@@ -275,7 +280,7 @@ _put_mount()
 _clear_mount_stack()
 {
 	if [ -n "$MOUNTED_POINT_STACK" ]; then
-		$UMOUNT_PROG $MOUNTED_POINT_STACK
+		_umount $MOUNTED_POINT_STACK
 	fi
 	MOUNTED_POINT_STACK=""
 }
@@ -420,20 +425,20 @@ _scratch_unmount()
 		_overlay_scratch_unmount
 		;;
 	btrfs)
-		$UMOUNT_PROG $SCRATCH_MNT
+		_umount $SCRATCH_MNT
 		;;
 	tmpfs)
 		$UMOUNT_PROG $SCRATCH_MNT
 		;;
 	*)
-		$UMOUNT_PROG $SCRATCH_DEV
+		_umount $SCRATCH_DEV
 		;;
 	esac
 }
 
 _scratch_umount_idmapped()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 _scratch_remount()
@@ -457,7 +462,7 @@ _scratch_cycle_mount()
         ;;
     overlay)
         if [ "$OVL_BASE_FSTYP" = tmpfs ]; then
-            $UMOUNT_PROG $SCRATCH_MNT
+            _umount $SCRATCH_MNT
             unmounted=true
         fi
         ;;
@@ -505,9 +510,9 @@ _move_mount()
 
 	# Replace $mnt with $tmp. Use a temporary bind-mount because
 	# mount --move will fail with certain mount propagation layouts.
-	$UMOUNT_PROG $mnt || _fail "Failed to unmount $mnt"
+	_umount $mnt || _fail "Failed to unmount $mnt"
 	_mount --bind $tmp $mnt || _fail "Failed to bind-mount $tmp to $mnt"
-	$UMOUNT_PROG $tmp || _fail "Failed to unmount $tmp"
+	_umount $tmp || _fail "Failed to unmount $tmp"
 	rmdir $tmp
 }
 
@@ -573,7 +578,7 @@ _test_unmount()
 	if [ "$FSTYP" == "overlay" ]; then
 		_overlay_test_unmount
 	else
-		$UMOUNT_PROG $TEST_DEV
+		_umount $TEST_DEV
 	fi
 }
 
@@ -587,7 +592,7 @@ _test_cycle_mount()
         ;;
     overlay)
         if [ "$OVL_BASE_FSTYP" = tmpfs ]; then
-            $UMOUNT_PROG $TEST_DIR
+            _umount $TEST_DIR
             unmounted=true
         fi
         ;;
@@ -1375,7 +1380,7 @@ _repair_scratch_fs()
 		# Fall through to repair base fs
 		dev=$OVL_BASE_SCRATCH_DEV
 		fstyp=$OVL_BASE_FSTYP
-		$UMOUNT_PROG $OVL_BASE_SCRATCH_MNT
+		_umount $OVL_BASE_SCRATCH_MNT
 	fi
 	# Let's hope fsck -y suffices...
 	fsck -t $fstyp -y $dev 2>&1
@@ -2189,7 +2194,7 @@ _require_logdev()
         _notrun "This test requires USE_EXTERNAL to be enabled"
 
     # ensure its not mounted
-    $UMOUNT_PROG $SCRATCH_LOGDEV 2>/dev/null
+    _umount $SCRATCH_LOGDEV 2>/dev/null
 }
 
 # This test requires that an external log device is not in use
@@ -3281,7 +3286,7 @@ _umount_or_remount_ro()
     local mountpoint=`_is_dev_mounted $device`
 
     if [ $USE_REMOUNT -eq 0 ]; then
-        $UMOUNT_PROG $device
+        _umount $device
     else
         _remount $device ro
     fi
@@ -3799,7 +3804,7 @@ _require_scratch_dev_pool()
 			_notrun "$i is part of TEST_DEV, this test requires unique disks"
 		fi
 		if _mount | grep -q $i; then
-			if ! $UMOUNT_PROG $i; then
+			if ! _umount $i; then
 		            echo "failed to unmount $i - aborting"
 		            exit 1
 		        fi
diff --git a/tests/btrfs/020 b/tests/btrfs/020
index 7e5c6fd7b25229..f6fadab1f00bdb 100755
--- a/tests/btrfs/020
+++ b/tests/btrfs/020
@@ -17,7 +17,7 @@ _cleanup()
 {
 	cd /
 	rm -f $tmp.*
-	$UMOUNT_PROG $loop_mnt
+	_umount $loop_mnt
 	_destroy_loop_device $loop_dev1
 	losetup -d $loop_dev2 >/dev/null 2>&1
 	_destroy_loop_device $loop_dev3
diff --git a/tests/btrfs/029 b/tests/btrfs/029
index c37ad63fb613db..9799b275250e5a 100755
--- a/tests/btrfs/029
+++ b/tests/btrfs/029
@@ -74,7 +74,7 @@ cp --reflink=always $orig_file $copy_file >> $seqres.full 2>&1 || echo "cp refli
 md5sum $orig_file | _filter_testdir_and_scratch
 md5sum $copy_file | _filter_testdir_and_scratch
 
-$UMOUNT_PROG $reflink_test_dir
+_umount $reflink_test_dir
 
 # success, all done
 status=0
diff --git a/tests/btrfs/031 b/tests/btrfs/031
index 8ac73d3a86e70b..92c1d26f865ba9 100755
--- a/tests/btrfs/031
+++ b/tests/btrfs/031
@@ -99,7 +99,7 @@ mv $testdir2/file* $subvol2/
 echo "Verify the file contents:"
 _checksum_files
 
-$UMOUNT_PROG $cross_mount_test_dir
+_umount $cross_mount_test_dir
 
 # success, all done
 status=0
diff --git a/tests/btrfs/060 b/tests/btrfs/060
index 75c10bd23c36f5..0bf88f86ca822b 100755
--- a/tests/btrfs/060
+++ b/tests/btrfs/060
@@ -82,7 +82,7 @@ run_test()
 	fi
 
 	# in case the subvolume is still mounted
-	$UMOUNT_PROG $subvol_mnt >/dev/null 2>&1
+	_umount $subvol_mnt >/dev/null 2>&1
 	_scratch_unmount
 	# we called _require_scratch_nocheck instead of _require_scratch
 	# do check after test for each profile config
diff --git a/tests/btrfs/065 b/tests/btrfs/065
index b87c66d6e3d45e..9cd38fefe46875 100755
--- a/tests/btrfs/065
+++ b/tests/btrfs/065
@@ -90,7 +90,7 @@ run_test()
 	fi
 
 	# in case the subvolume is still mounted
-	$UMOUNT_PROG $subvol_mnt >/dev/null 2>&1
+	_umount $subvol_mnt >/dev/null 2>&1
 	_scratch_unmount
 	# we called _require_scratch_nocheck instead of _require_scratch
 	# do check after test for each profile config
diff --git a/tests/btrfs/066 b/tests/btrfs/066
index cc7cd9b7273d1c..b3db57049714ad 100755
--- a/tests/btrfs/066
+++ b/tests/btrfs/066
@@ -82,7 +82,7 @@ run_test()
 	fi
 
 	# in case the subvolume is still mounted
-	$UMOUNT_PROG $subvol_mnt >/dev/null 2>&1
+	_umount $subvol_mnt >/dev/null 2>&1
 	_scratch_unmount
 	# we called _require_scratch_nocheck instead of _require_scratch
 	# do check after test for each profile config
diff --git a/tests/btrfs/067 b/tests/btrfs/067
index 0b473050027a0a..ede9abbc689fe0 100755
--- a/tests/btrfs/067
+++ b/tests/btrfs/067
@@ -83,7 +83,7 @@ run_test()
 	fi
 
 	# in case the subvolume is still mounted
-	$UMOUNT_PROG $subvol_mnt >/dev/null 2>&1
+	_umount $subvol_mnt >/dev/null 2>&1
 	_scratch_unmount
 	# we called _require_scratch_nocheck instead of _require_scratch
 	# do check after test for each profile config
diff --git a/tests/btrfs/068 b/tests/btrfs/068
index 83e932e8417c0d..82dac5fd90ba85 100755
--- a/tests/btrfs/068
+++ b/tests/btrfs/068
@@ -83,7 +83,7 @@ run_test()
 	fi
 
 	# in case the subvolume is still mounted
-	$UMOUNT_PROG $subvol_mnt >/dev/null 2>&1
+	_umount $subvol_mnt >/dev/null 2>&1
 	_scratch_unmount
 	# we called _require_scratch_nocheck instead of _require_scratch
 	# do check after test for each profile config
diff --git a/tests/btrfs/075 b/tests/btrfs/075
index 737c4ffdd57865..8e78bd3d4b2336 100755
--- a/tests/btrfs/075
+++ b/tests/btrfs/075
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	rm -f $tmp.*
-	$UMOUNT_PROG $subvol_mnt >/dev/null 2>&1
+	_umount $subvol_mnt >/dev/null 2>&1
 }
 
 . ./common/filter
diff --git a/tests/btrfs/089 b/tests/btrfs/089
index 8f8e37b6fde87b..ade38a6d189eaa 100755
--- a/tests/btrfs/089
+++ b/tests/btrfs/089
@@ -35,7 +35,7 @@ mount --bind "$SCRATCH_MNT/testvol/testdir" "$SCRATCH_MNT/testvol/mnt"
 $BTRFS_UTIL_PROG subvolume delete "$SCRATCH_MNT/testvol" >>$seqres.full 2>&1
 
 # Unmount the bind mount, which should still be alive.
-$UMOUNT_PROG "$SCRATCH_MNT/testvol/mnt"
+_umount "$SCRATCH_MNT/testvol/mnt"
 
 echo "Silence is golden"
 status=0
diff --git a/tests/btrfs/124 b/tests/btrfs/124
index af079c2864de8e..19f8bbfc6b922e 100755
--- a/tests/btrfs/124
+++ b/tests/btrfs/124
@@ -132,7 +132,7 @@ if [ "$checkpoint1" != "$checkpoint3" ]; then
 	echo "Inital sum does not match with data on dev2 written by balance"
 fi
 
-$UMOUNT_PROG $dev2
+_umount $dev2
 _scratch_dev_pool_put
 _btrfs_rescan_devices
 _test_mount
diff --git a/tests/btrfs/125 b/tests/btrfs/125
index c8c0dd422f72b6..7acef2d38cda46 100755
--- a/tests/btrfs/125
+++ b/tests/btrfs/125
@@ -144,7 +144,7 @@ if [ "$checkpoint1" != "$checkpoint3" ]; then
 	echo "Inital sum does not match with data on dev2 written by balance"
 fi
 
-$UMOUNT_PROG $dev2
+_umount $dev2
 _scratch_dev_pool_put
 _btrfs_rescan_devices
 _test_mount
diff --git a/tests/btrfs/185 b/tests/btrfs/185
index 8d0643450f5d7d..c3b52fc2dbff66 100755
--- a/tests/btrfs/185
+++ b/tests/btrfs/185
@@ -15,7 +15,7 @@ mnt=$TEST_DIR/$seq.mnt
 # Override the default cleanup function.
 _cleanup()
 {
-	$UMOUNT_PROG $mnt > /dev/null 2>&1
+	_umount $mnt > /dev/null 2>&1
 	rm -rf $mnt > /dev/null 2>&1
 	cd /
 	rm -f $tmp.*
@@ -62,7 +62,7 @@ $BTRFS_UTIL_PROG device scan $device_1 >> $seqres.full 2>&1
 	_fail "if it fails here, then it means subvolume mount at boot may fail "\
 	      "in some configs."
 
-$UMOUNT_PROG $mnt > /dev/null 2>&1
+_umount $mnt > /dev/null 2>&1
 _scratch_dev_pool_put
 
 # success, all done
diff --git a/tests/btrfs/197 b/tests/btrfs/197
index 9f1d879a4e267a..913dbb2d3a50ef 100755
--- a/tests/btrfs/197
+++ b/tests/btrfs/197
@@ -15,7 +15,7 @@ _begin_fstest auto quick volume
 # Override the default cleanup function.
 _cleanup()
 {
-	$UMOUNT_PROG $TEST_DIR/$seq.mnt >/dev/null 2>&1
+	_umount $TEST_DIR/$seq.mnt >/dev/null 2>&1
 	rm -rf $TEST_DIR/$seq.mnt
 	cd /
 	rm -f $tmp.*
@@ -67,7 +67,7 @@ workout()
 	grep -q "${SCRATCH_DEV_NAME[1]}" $tmp.output && _fail "found stale device"
 
 	$BTRFS_UTIL_PROG device remove "${SCRATCH_DEV_NAME[1]}" "$TEST_DIR/$seq.mnt"
-	$UMOUNT_PROG $TEST_DIR/$seq.mnt
+	_umount $TEST_DIR/$seq.mnt
 	_scratch_unmount
 	_spare_dev_put
 	_scratch_dev_pool_put
diff --git a/tests/btrfs/219 b/tests/btrfs/219
index 052f61a399ae66..efe5096746652a 100755
--- a/tests/btrfs/219
+++ b/tests/btrfs/219
@@ -21,8 +21,8 @@ _cleanup()
 	rm -f $tmp.*
 
 	# The variables are set before the test case can fail.
-	$UMOUNT_PROG ${loop_mnt1} &> /dev/null
-	$UMOUNT_PROG ${loop_mnt2} &> /dev/null
+	_umount ${loop_mnt1} &> /dev/null
+	_umount ${loop_mnt2} &> /dev/null
 	rm -rf $loop_mnt1
 	rm -rf $loop_mnt2
 
@@ -66,7 +66,7 @@ loop_dev2=`_create_loop_device $fs_img2`
 # Normal single device case, should pass just fine
 _mount $loop_dev1 $loop_mnt1 > /dev/null  2>&1 || \
 	_fail "Couldn't do initial mount"
-$UMOUNT_PROG $loop_mnt1
+_umount $loop_mnt1
 
 _btrfs_forget_or_module_reload
 
@@ -75,15 +75,15 @@ _btrfs_forget_or_module_reload
 # measure.
 _mount $loop_dev1 $loop_mnt1 > /dev/null 2>&1 || \
 	_fail "Failed to mount the second time"
-$UMOUNT_PROG $loop_mnt1
+_umount $loop_mnt1
 
 _mount $loop_dev2 $loop_mnt2 > /dev/null 2>&1 || \
 	_fail "We couldn't mount the old generation"
-$UMOUNT_PROG $loop_mnt2
+_umount $loop_mnt2
 
 _mount $loop_dev1 $loop_mnt1 > /dev/null 2>&1 || \
 	_fail "Failed to mount the second time"
-$UMOUNT_PROG $loop_mnt1
+_umount $loop_mnt1
 
 # Now try mount them at the same time, if kernel does not support
 # temp-fsid feature then mount will fail.
diff --git a/tests/btrfs/254 b/tests/btrfs/254
index d9c9eea9c7bf23..eda32be1c2b1d1 100755
--- a/tests/btrfs/254
+++ b/tests/btrfs/254
@@ -96,7 +96,7 @@ test_add_device()
 	$BTRFS_UTIL_PROG filesystem show -m $SCRATCH_MNT | \
 					_filter_btrfs_filesystem_show
 
-	$UMOUNT_PROG $seq_mnt
+	_umount $seq_mnt
 	_scratch_unmount
 	cleanup_dmdev
 }
diff --git a/tests/ext4/032 b/tests/ext4/032
index 9a1b9312cc42cc..6e98f4f4ebb8de 100755
--- a/tests/ext4/032
+++ b/tests/ext4/032
@@ -63,7 +63,7 @@ ext4_online_resize()
 	fi
 	cat $tmp.resize2fs >> $seqres.full
 	echo "+++ umount fs" | tee -a $seqres.full
-	$UMOUNT_PROG ${IMG_MNT}
+	_umount ${IMG_MNT}
 
 	echo "+++ check fs" | tee -a $seqres.full
 	_check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
@@ -77,7 +77,7 @@ _cleanup()
 	cd /
 	[ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
 	rm -f $tmp.*
-	$UMOUNT_PROG ${IMG_MNT} > /dev/null 2>&1
+	_umount ${IMG_MNT} > /dev/null 2>&1
 	rm -f ${IMG_FILE} > /dev/null 2>&1
 }
 
diff --git a/tests/ext4/052 b/tests/ext4/052
index edcdc02515f725..ce3f90eb7e6d02 100755
--- a/tests/ext4/052
+++ b/tests/ext4/052
@@ -18,7 +18,7 @@ _cleanup()
 	cd /
 	rm -r -f $tmp.*
 	if [ ! -z "$loop_mnt" ]; then
-		$UMOUNT_PROG $loop_mnt
+		_umount $loop_mnt
 		rm -rf $loop_mnt
 	fi
 	[ ! -z "$fs_img" ] && rm -rf $fs_img
@@ -63,7 +63,7 @@ then
     status=1
 fi
 
-$UMOUNT_PROG $loop_mnt || _fail "umount failed"
+_umount $loop_mnt || _fail "umount failed"
 loop_mnt=
 
 $E2FSCK_PROG -fn $fs_img >> $seqres.full 2>&1 || _fail "file system corrupted"
diff --git a/tests/ext4/053 b/tests/ext4/053
index 4f20d217d5fd7a..0beb2201260162 100755
--- a/tests/ext4/053
+++ b/tests/ext4/053
@@ -20,7 +20,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _cleanup()
 {
 	cd /
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_umount $SCRATCH_MNT > /dev/null 2>&1
 	if [ -n "$LOOP_LOGDEV" ];then
 		_destroy_loop_device $LOOP_LOGDEV 2>/dev/null
 	fi
@@ -237,7 +237,7 @@ not_mnt() {
 	if simple_mount -o $1 $SCRATCH_DEV $SCRATCH_MNT; then
 		print_log "(mount unexpectedly succeeded)"
 		fail
-		$UMOUNT_PROG $SCRATCH_MNT
+		_umount $SCRATCH_MNT
 		return
 	fi
 	ok
@@ -248,7 +248,7 @@ not_mnt() {
 		return
 	fi
 	not_remount $1
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 mnt_only() {
@@ -270,7 +270,7 @@ mnt() {
 	fi
 
 	mnt_only $*
-	$UMOUNT_PROG $SCRATCH_MNT 2> /dev/null
+	_umount $SCRATCH_MNT 2> /dev/null
 
 	[ "$t2fs" -eq 0 ] && return
 
@@ -289,7 +289,7 @@ mnt() {
 				    -e 's/data=writeback/journal_data_writeback/')
 	$TUNE2FS_PROG -o $op_set $SCRATCH_DEV > /dev/null 2>&1
 	mnt_only "defaults" $check
-	$UMOUNT_PROG $SCRATCH_MNT 2> /dev/null
+	_umount $SCRATCH_MNT 2> /dev/null
 	if [ "$op_set" = ^* ]; then
 		op_set=${op_set#^}
 	else
@@ -309,12 +309,12 @@ remount() {
 	do_mnt remount,$2 $3
 	if [ $? -ne 0 ]; then
 		fail
-		$UMOUNT_PROG $SCRATCH_MNT 2> /dev/null
+		_umount $SCRATCH_MNT 2> /dev/null
 		return
 	else
 		ok
 	fi
-	$UMOUNT_PROG $SCRATCH_MNT 2> /dev/null
+	_umount $SCRATCH_MNT 2> /dev/null
 
 	# Now just specify mnt
 	print_log "mounting $fstype \"$1\" "
@@ -328,7 +328,7 @@ remount() {
 		ok
 	fi
 
-	$UMOUNT_PROG $SCRATCH_MNT 2> /dev/null
+	_umount $SCRATCH_MNT 2> /dev/null
 }
 
 # Test that the filesystem cannot be remounted with option(s) $1 (meaning that
@@ -364,7 +364,7 @@ mnt_then_not_remount() {
 		return
 	fi
 	not_remount $2
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 
@@ -400,8 +400,8 @@ LOGDEV_DEVNUM=`echo "${majmin%:*}*2^8 + ${majmin#*:}" | bc`
 fstype=
 for fstype in ext2 ext3 ext4; do
 
-	$UMOUNT_PROG $SCRATCH_MNT 2> /dev/null
-	$UMOUNT_PROG $SCRATCH_DEV 2> /dev/null
+	_umount $SCRATCH_MNT 2> /dev/null
+	_umount $SCRATCH_DEV 2> /dev/null
 
 	do_mkfs $SCRATCH_DEV ${SIZE}k
 
@@ -418,7 +418,7 @@ for fstype in ext2 ext3 ext4; do
 		continue
 	fi
 
-	$UMOUNT_PROG $SCRATCH_MNT 2> /dev/null
+	_umount $SCRATCH_MNT 2> /dev/null
 
 	not_mnt failme
 	mnt
@@ -552,7 +552,7 @@ for fstype in ext2 ext3 ext4; do
 	# dax mount options
 	simple_mount -o dax=always $SCRATCH_DEV $SCRATCH_MNT > /dev/null 2>&1
 	if [ $? -eq 0 ]; then
-		$UMOUNT_PROG $SCRATCH_MNT 2> /dev/null
+		_umount $SCRATCH_MNT 2> /dev/null
 		mnt dax
 		mnt dax=always
 		mnt dax=never
@@ -633,7 +633,7 @@ for fstype in ext2 ext3 ext4; do
 	not_remount jqfmt=vfsv1
 	not_remount noquota
 	mnt_only remount,usrquota,grpquota ^usrquota,^grpquota
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_umount $SCRATCH_MNT > /dev/null 2>&1
 
 	# test clearing/changing quota when enabled
 	do_mkfs -E quotatype=^prjquota $SCRATCH_DEV ${SIZE}k
@@ -654,7 +654,7 @@ for fstype in ext2 ext3 ext4; do
 	mnt_only remount,usrquota,grpquota usrquota,grpquota
 	quotaoff -f $SCRATCH_MNT >> $seqres.full 2>&1
 	mnt_only remount,noquota ^usrquota,^grpquota,quota
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_umount $SCRATCH_MNT > /dev/null 2>&1
 
 	# Quota feature
 	echo "== Testing quota feature " >> $seqres.full
@@ -696,7 +696,7 @@ for fstype in ext2 ext3 ext4; do
 
 done #for fstype in ext2 ext3 ext4; do
 
-$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+_umount $SCRATCH_MNT > /dev/null 2>&1
 echo "$ERR errors encountered" >> $seqres.full
 
 status=$ERR
diff --git a/tests/ext4/056 b/tests/ext4/056
index 8a290b11d69772..f9cb690fdfc80b 100755
--- a/tests/ext4/056
+++ b/tests/ext4/056
@@ -70,7 +70,7 @@ do_resize()
 	# delay
 	sleep 0.2
 	_scratch_unmount >> $seqres.full 2>&1 \
-		|| _fail "$UMOUNT_PROG failed. Exiting"
+		|| _fail "_umount failed. Exiting"
 }
 
 run_test()
diff --git a/tests/generic/042 b/tests/generic/042
index fd0ef705a18c3e..bea23ce29ac327 100755
--- a/tests/generic/042
+++ b/tests/generic/042
@@ -44,7 +44,7 @@ _crashtest()
 		_filter_xfs_io
 	$here/src/godown -f $mnt
 
-	$UMOUNT_PROG $mnt
+	_umount $mnt
 	_mount $img $mnt
 
 	# We should /never/ see 0xCD in the file, because we wrote that pattern
@@ -54,7 +54,7 @@ _crashtest()
 		_hexdump $file
 	fi
 
-	$UMOUNT_PROG $mnt
+	_umount $mnt
 }
 
 # Modify as appropriate.
diff --git a/tests/generic/067 b/tests/generic/067
index b6e984f5231753..19ee28d2cd945e 100755
--- a/tests/generic/067
+++ b/tests/generic/067
@@ -66,7 +66,7 @@ umount_symlink_device()
 	rm -f $symlink
 	echo "# umount symlink to device, which is not mounted" >>$seqres.full
 	ln -s $SCRATCH_DEV $symlink
-	$UMOUNT_PROG $symlink >>$seqres.full 2>&1
+	_umount $symlink >>$seqres.full 2>&1
 }
 
 # umount a path name that is 256 bytes long, this should fail gracefully,
@@ -78,7 +78,7 @@ umount_toolong_name()
 	_scratch_mount 2>&1 | tee -a $seqres.full
 
 	echo "# umount a too-long name" >>$seqres.full
-	$UMOUNT_PROG $longname >>$seqres.full 2>&1
+	_umount $longname >>$seqres.full 2>&1
 	_scratch_unmount 2>&1 | tee -a $seqres.full
 }
 
@@ -93,7 +93,7 @@ lazy_umount_symlink()
 	rm -f $symlink
 	ln -s $SCRATCH_MNT/testdir $symlink
 
-	$UMOUNT_PROG -l $symlink >>$seqres.full 2>&1
+	_umount -l $symlink >>$seqres.full 2>&1
 	# _scratch_unmount should not be blocked
 	_scratch_unmount 2>&1 | tee -a $seqres.full
 }
diff --git a/tests/generic/081 b/tests/generic/081
index 468c87ac9a9f0a..57dc07a36395f8 100755
--- a/tests/generic/081
+++ b/tests/generic/081
@@ -32,7 +32,7 @@ _cleanup()
 	# other tests to fail.
 	while test -e /dev/mapper/$vgname-$snapname || \
 	      test -e /dev/mapper/$vgname-$lvname; do
-		$UMOUNT_PROG $mnt >> $seqres.full 2>&1
+		_umount $mnt >> $seqres.full 2>&1
 		$LVM_PROG lvremove -f $vgname/$snapname >>$seqres.full 2>&1
 		$LVM_PROG lvremove -f $vgname/$lvname >>$seqres.full 2>&1
 		$LVM_PROG vgremove -f $vgname >>$seqres.full 2>&1
diff --git a/tests/generic/085 b/tests/generic/085
index cbabd257cad8f0..8c33386b7c383e 100755
--- a/tests/generic/085
+++ b/tests/generic/085
@@ -27,7 +27,7 @@ cleanup_dmdev()
 	$DMSETUP_PROG resume $lvdev >/dev/null 2>&1
 	[ -n "$pid" ] && kill -9 $pid 2>/dev/null
 	wait $pid
-	$UMOUNT_PROG $lvdev >/dev/null 2>&1
+	_umount $lvdev >/dev/null 2>&1
 	_dmsetup_remove $node
 }
 
@@ -70,7 +70,7 @@ done &
 pid=$!
 for ((i=0; i<100; i++)); do
 	_mount $lvdev $SCRATCH_MNT >/dev/null 2>&1
-	$UMOUNT_PROG $lvdev >/dev/null 2>&1
+	_umount $lvdev >/dev/null 2>&1
 done &
 pid="$pid $!"
 
diff --git a/tests/generic/108 b/tests/generic/108
index da13715f27ac21..e1df7ee1886cde 100755
--- a/tests/generic/108
+++ b/tests/generic/108
@@ -18,7 +18,7 @@ _cleanup()
 {
 	cd /
 	echo running > /sys/block/`_short_dev $SCSI_DEBUG_DEV`/device/state
-	$UMOUNT_PROG $SCRATCH_MNT >>$seqres.full 2>&1
+	_umount $SCRATCH_MNT >>$seqres.full 2>&1
 	$LVM_PROG vgremove -f $vgname >>$seqres.full 2>&1
 	$LVM_PROG pvremove -f $SCRATCH_DEV $SCSI_DEBUG_DEV >>$seqres.full 2>&1
 	$UDEV_SETTLE_PROG
diff --git a/tests/generic/361 b/tests/generic/361
index c2ebda3c1a01ad..456271b8d80308 100755
--- a/tests/generic/361
+++ b/tests/generic/361
@@ -16,7 +16,7 @@ _begin_fstest auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-	$UMOUNT_PROG $fs_mnt
+	_umount $fs_mnt
 	_destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
diff --git a/tests/generic/373 b/tests/generic/373
index 0d5a50cbee40b8..6ede189ead70bd 100755
--- a/tests/generic/373
+++ b/tests/generic/373
@@ -60,7 +60,7 @@ md5sum $testdir/file | _filter_scratch
 md5sum $othertestdir/otherfile | filter_otherdir
 
 echo "Unmount otherdir"
-$UMOUNT_PROG $otherdir
+_umount $otherdir
 rm -rf $otherdir
 
 # success, all done
diff --git a/tests/generic/374 b/tests/generic/374
index 977a2b268bbc98..bbdd8e66b4897b 100755
--- a/tests/generic/374
+++ b/tests/generic/374
@@ -59,7 +59,7 @@ echo "Check output"
 md5sum $testdir/file $othertestdir/otherfile | filter_md5
 
 echo "Unmount otherdir"
-$UMOUNT_PROG $otherdir
+_umount $otherdir
 rm -rf $otherdir
 
 # success, all done
diff --git a/tests/generic/459 b/tests/generic/459
index 32ee899f929819..e8799f75bf8e05 100755
--- a/tests/generic/459
+++ b/tests/generic/459
@@ -28,7 +28,7 @@ _cleanup()
 	xfs_freeze -u $SCRATCH_MNT 2>/dev/null
 	cd /
 	rm -f $tmp.*
-	$UMOUNT_PROG $SCRATCH_MNT >>$seqres.full 2>&1
+	_umount $SCRATCH_MNT >>$seqres.full 2>&1
 	$LVM_PROG vgremove -ff $vgname >>$seqres.full 2>&1
 	$LVM_PROG pvremove -ff $SCRATCH_DEV >>$seqres.full 2>&1
 	$UDEV_SETTLE_PROG
diff --git a/tests/generic/604 b/tests/generic/604
index c2e03c2eabb871..124eea853ecf70 100755
--- a/tests/generic/604
+++ b/tests/generic/604
@@ -26,6 +26,8 @@ done
 # mount the base fs.  Delay the mount attempt by a small amount in the hope
 # that the mount() call will try to lock s_umount /after/ umount has already
 # taken it.
+# This is the /one/ place in fstests where we need to call the umount binary
+# directly.
 $UMOUNT_PROG $SCRATCH_MNT &
 sleep 0.01s ; _scratch_mount
 wait
diff --git a/tests/generic/648 b/tests/generic/648
index 29d1b470bded4a..3e995a02983931 100755
--- a/tests/generic/648
+++ b/tests/generic/648
@@ -20,7 +20,7 @@ _cleanup()
 	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
 	wait
 	if [ -n "$loopmnt" ]; then
-		$UMOUNT_PROG $loopmnt 2>/dev/null
+		_umount $loopmnt 2>/dev/null
 		rm -r -f $loopmnt
 	fi
 	rm -f $tmp.*
@@ -111,7 +111,7 @@ while _soak_loop_running $((25 * TIME_FACTOR)); do
 
 	# Mount again to replay log after loading working table, so we have a
 	# consistent fs after test.
-	$UMOUNT_PROG $loopmnt
+	_umount $loopmnt
 	is_unmounted=1
 	# We must unmount dmerror at here, or whole later testing will crash.
 	# So try to umount enough times, before we have no choice.
@@ -137,7 +137,7 @@ done
 # Make sure the fs image file is ok
 if [ -f "$loopimg" ]; then
 	if _mount $loopimg $loopmnt -o loop; then
-		$UMOUNT_PROG $loopmnt &> /dev/null
+		_umount $loopmnt &> /dev/null
 	else
 		_metadump_dev $DMERROR_DEV $seqres.scratch.final.md
 		echo "final scratch mount failed"
diff --git a/tests/generic/698 b/tests/generic/698
index 28928b2fb32532..f432837a216f82 100755
--- a/tests/generic/698
+++ b/tests/generic/698
@@ -17,8 +17,8 @@ _begin_fstest auto quick perms attr idmapped mount
 _cleanup()
 {
 	cd /
-	$UMOUNT_PROG $SCRATCH_MNT/target-mnt 2>/dev/null
-	$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+	_umount $SCRATCH_MNT/target-mnt 2>/dev/null
+	_umount $SCRATCH_MNT 2>/dev/null
 	rm -r -f $tmp.*
 }
 
diff --git a/tests/generic/699 b/tests/generic/699
index 677307538a484b..5cff1cbaa67c4e 100755
--- a/tests/generic/699
+++ b/tests/generic/699
@@ -15,9 +15,9 @@ _begin_fstest auto quick perms attr idmapped mount
 _cleanup()
 {
 	cd /
-	$UMOUNT_PROG $SCRATCH_MNT/target-mnt
-	$UMOUNT_PROG $SCRATCH_MNT/ovl-merge 2>/dev/null
-	$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+	_umount $SCRATCH_MNT/target-mnt
+	_umount $SCRATCH_MNT/ovl-merge 2>/dev/null
+	_umount $SCRATCH_MNT 2>/dev/null
 	rm -r -f $tmp.*
 }
 
@@ -113,7 +113,7 @@ setup_overlayfs_idmapped_lower_metacopy_on()
 
 reset_overlayfs()
 {
-	$UMOUNT_PROG $SCRATCH_MNT/ovl-merge 2>/dev/null
+	_umount $SCRATCH_MNT/ovl-merge 2>/dev/null
 	rm -rf $upper $work $merge
 }
 
diff --git a/tests/generic/704 b/tests/generic/704
index f39d47066ccc4a..31d52a97b37f9d 100755
--- a/tests/generic/704
+++ b/tests/generic/704
@@ -14,7 +14,7 @@ _cleanup()
 {
 	cd /
 	rm -r -f $tmp.*
-	[ -d "$SCSI_DEBUG_MNT" ] && $UMOUNT_PROG $SCSI_DEBUG_MNT 2>/dev/null
+	[ -d "$SCSI_DEBUG_MNT" ] && _umount $SCSI_DEBUG_MNT 2>/dev/null
 	_put_scsi_debug_dev
 }
 
diff --git a/tests/generic/730 b/tests/generic/730
index 062314ea01e7b5..650c604d5fbefd 100755
--- a/tests/generic/730
+++ b/tests/generic/730
@@ -12,7 +12,7 @@ _begin_fstest auto quick
 _cleanup()
 {
 	cd /
-	$UMOUNT_PROG $SCSI_DEBUG_MNT >>$seqres.full 2>&1
+	_umount $SCSI_DEBUG_MNT >>$seqres.full 2>&1
 	_put_scsi_debug_dev
 	rm -f $tmp.*
 }
diff --git a/tests/generic/731 b/tests/generic/731
index cd39e8b09e3906..2621f6e237741d 100755
--- a/tests/generic/731
+++ b/tests/generic/731
@@ -13,7 +13,7 @@ _begin_fstest auto quick
 _cleanup()
 {
 	cd /
-	$UMOUNT_PROG $SCSI_DEBUG_MNT >>$seqres.full 2>&1
+	_umount $SCSI_DEBUG_MNT >>$seqres.full 2>&1
 	_put_scsi_debug_dev
 	rm -f $tmp.*
 }
diff --git a/tests/generic/732 b/tests/generic/732
index d08028c2333d1b..63406ddc163f2c 100755
--- a/tests/generic/732
+++ b/tests/generic/732
@@ -15,8 +15,8 @@ _begin_fstest auto quick rename
 # Override the default cleanup function.
 _cleanup()
 {
-	$UMOUNT_PROG $testdir1 2>/dev/null
-	$UMOUNT_PROG $testdir2 2>/dev/null
+	_umount $testdir1 2>/dev/null
+	_umount $testdir2 2>/dev/null
 	cd /
 	rm -r -f $tmp.*
 }
diff --git a/tests/generic/746 b/tests/generic/746
index 651affe07b40bc..2b40c964371175 100755
--- a/tests/generic/746
+++ b/tests/generic/746
@@ -38,7 +38,7 @@ esac
 # Override the default cleanup function.
 _cleanup()
 {
-	$UMOUNT_PROG $loop_dev &> /dev/null
+	_umount $loop_dev &> /dev/null
 	_destroy_loop_device $loop_dev
 	if [ $status -eq 0 ]; then
 		rm -rf $tmp
@@ -53,7 +53,7 @@ get_holes()
 	# in-core state that will perturb the free space map on umount.  Stick
 	# to established convention which requires the filesystem to be
 	# unmounted while we probe the underlying file.
-	$UMOUNT_PROG $loop_mnt
+	_umount $loop_mnt
 
 	# FIEMAP only works on regular files, so call it on the backing file
 	# and not the loop device like everything else
@@ -66,7 +66,7 @@ get_free_sectors()
 {
 	case $FSTYP in
 	ext4)
-	$UMOUNT_PROG $loop_mnt
+	_umount $loop_mnt
 	$DUMPE2FS_PROG $loop_dev  2>&1 | grep " Free blocks" | cut -d ":" -f2- | \
 		tr ',' '\n' | $SED_PROG 's/^ //' | \
 		$AWK_PROG -v spb=$sectors_per_block 'BEGIN{FS="-"};
@@ -80,7 +80,7 @@ get_free_sectors()
 	xfs)
 	agsize=`$XFS_INFO_PROG $loop_mnt | $SED_PROG -n 's/.*agsize=\(.*\) blks.*/\1/p'`
 	# Convert free space (agno, block, length) to (start sector, end sector)
-	$UMOUNT_PROG $loop_mnt
+	_umount $loop_mnt
 	$XFS_DB_PROG -r -c "freesp -d" $loop_dev | $SED_PROG '/^.*from/,$d'| \
 		 $AWK_PROG -v spb=$sectors_per_block -v agsize=$agsize \
 		'{ print spb * ($1 * agsize + $2), spb * ($1 * agsize + $2 + $3) - 1 }'
diff --git a/tests/overlay/003 b/tests/overlay/003
index 41ad99e794d8ee..0a2cb928ea5c58 100755
--- a/tests/overlay/003
+++ b/tests/overlay/003
@@ -56,7 +56,7 @@ rm -rf ${SCRATCH_MNT}/*
 ls ${SCRATCH_MNT}/
 
 # unmount overlayfs but not base fs
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 echo "Silence is golden"
 # success, all done
diff --git a/tests/overlay/004 b/tests/overlay/004
index bea4bb543f3611..4591d4e8487ce2 100755
--- a/tests/overlay/004
+++ b/tests/overlay/004
@@ -53,7 +53,7 @@ _user_do "chmod u-X ${SCRATCH_MNT}/attr_file2 > /dev/null 2>&1"
 stat -c %a ${SCRATCH_MNT}/attr_file2
 
 # unmount overlayfs but not base fs
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # check mode bits of the file that has been copied up, and
 # the file that should not have been copied up.
diff --git a/tests/overlay/005 b/tests/overlay/005
index 01914ee17b9a30..6b382ddb50d873 100755
--- a/tests/overlay/005
+++ b/tests/overlay/005
@@ -75,14 +75,14 @@ $XFS_IO_PROG -f -c "o" ${SCRATCH_MNT}/test_file \
 	>>$seqres.full 2>&1
 
 # unmount overlayfs
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # check overlayfs
 _overlay_check_scratch_dirs $lowerd $upperd $workd
 
 # unmount undelying xfs, this tiggers panic if memleak happens
-$UMOUNT_PROG ${OVL_BASE_SCRATCH_MNT}/uppermnt
-$UMOUNT_PROG ${OVL_BASE_SCRATCH_MNT}/lowermnt
+_umount ${OVL_BASE_SCRATCH_MNT}/uppermnt
+_umount ${OVL_BASE_SCRATCH_MNT}/lowermnt
 
 # success, all done
 echo "Silence is golden"
diff --git a/tests/overlay/014 b/tests/overlay/014
index f07fc685572b92..08850d489e4b49 100755
--- a/tests/overlay/014
+++ b/tests/overlay/014
@@ -46,7 +46,7 @@ _overlay_scratch_mount_dirs $lowerdir1 $lowerdir2 $workdir2
 rm -rf $SCRATCH_MNT/testdir
 mkdir -p $SCRATCH_MNT/testdir/visibledir
 # unmount overlayfs but not base fs
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # check overlayfs
 _overlay_check_scratch_dirs $lowerdir1 $lowerdir2 $workdir2
@@ -59,7 +59,7 @@ touch $SCRATCH_MNT/testdir/visiblefile
 
 # umount and mount overlay again, buggy kernel treats the copied-up dir as
 # opaque, visibledir is not seen in merged dir.
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 _overlay_scratch_mount_dirs "$lowerdir2:$lowerdir1" $upperdir $workdir
 ls $SCRATCH_MNT/testdir
 
diff --git a/tests/overlay/022 b/tests/overlay/022
index d33bd29781a356..40b0dd64f6fc6c 100755
--- a/tests/overlay/022
+++ b/tests/overlay/022
@@ -17,7 +17,7 @@ _begin_fstest auto quick mount nested
 _cleanup()
 {
 	cd /
-	$UMOUNT_PROG $tmp/mnt > /dev/null 2>&1
+	_umount $tmp/mnt > /dev/null 2>&1
 	rm -rf $tmp
 	rm -f $tmp.*
 }
diff --git a/tests/overlay/025 b/tests/overlay/025
index 6ba46191b557be..0abc8bf80b1716 100755
--- a/tests/overlay/025
+++ b/tests/overlay/025
@@ -19,8 +19,8 @@ _begin_fstest auto quick attr
 _cleanup()
 {
 	cd /
-	$UMOUNT_PROG $tmpfsdir/mnt
-	$UMOUNT_PROG $tmpfsdir
+	_umount $tmpfsdir/mnt
+	_umount $tmpfsdir
 	rm -rf $tmpfsdir
 	rm -f $tmp.*
 }
diff --git a/tests/overlay/029 b/tests/overlay/029
index 4bade9a0e129a4..007973dc075923 100755
--- a/tests/overlay/029
+++ b/tests/overlay/029
@@ -22,7 +22,7 @@ _begin_fstest auto quick nested
 _cleanup()
 {
 	cd /
-	$UMOUNT_PROG $tmp/mnt
+	_umount $tmp/mnt
 	rm -rf $tmp
 	rm -f $tmp.*
 }
@@ -56,7 +56,7 @@ _overlay_mount_dirs $SCRATCH_MNT/up $tmp/{upper,work} \
   overlay $tmp/mnt
 # accessing file in the second mount
 cat $tmp/mnt/foo
-$UMOUNT_PROG $tmp/mnt
+_umount $tmp/mnt
 
 # re-create upper/work to avoid ovl_verify_origin() mount failure
 # when index is enabled
@@ -66,7 +66,7 @@ mkdir -p $tmp/{upper,work}
 _overlay_mount_dirs $SCRATCH_MNT/low $tmp/{upper,work} \
   overlay $tmp/mnt
 cat $tmp/mnt/bar
-$UMOUNT_PROG $tmp/mnt
+_umount $tmp/mnt
 
 rm -rf $tmp/{upper,work}
 mkdir -p $tmp/{upper,work}
diff --git a/tests/overlay/031 b/tests/overlay/031
index dd9dfcdb970ac7..31d22d1cadae41 100755
--- a/tests/overlay/031
+++ b/tests/overlay/031
@@ -28,7 +28,7 @@ create_whiteout()
 
 	rm -f $SCRATCH_MNT/testdir/$file
 
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 # Import common functions.
@@ -68,7 +68,7 @@ rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
 
 # umount overlay again, create a new file with the same name and
 # mount overlay again.
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 touch $lowerdir1/testdir
 
 _overlay_scratch_mount_dirs $lowerdir1 $upperdir $workdir
@@ -77,7 +77,7 @@ _overlay_scratch_mount_dirs $lowerdir1 $upperdir $workdir
 # it will not clean up the dir and lead to residue.
 rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
 
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # let lower dir have invalid whiteouts, repeat ls and rmdir test again.
 rm -rf $lowerdir1/testdir
@@ -92,7 +92,7 @@ _overlay_scratch_mount_dirs "$lowerdir1:$lowerdir2" $upperdir $workdir
 ls $SCRATCH_MNT/testdir
 rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
 
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # let lower dir and upper dir both have invalid whiteouts, repeat ls and rmdir again.
 rm -rf $lowerdir1/testdir
diff --git a/tests/overlay/035 b/tests/overlay/035
index cede58790e1b9d..c6ce1318fbbb37 100755
--- a/tests/overlay/035
+++ b/tests/overlay/035
@@ -43,7 +43,7 @@ mkdir -p $lowerdir1 $lowerdir2 $upperdir $workdir
 _overlay_scratch_mount_opts -o"lowerdir=$lowerdir2:$lowerdir1"
 touch $SCRATCH_MNT/foo 2>&1 | _filter_scratch
 _mount -o remount,rw $SCRATCH_MNT 2>&1 | _filter_ro_mount
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # Make workdir immutable to prevent workdir re-create on mount
 $CHATTR_PROG +i $workdir
diff --git a/tests/overlay/036 b/tests/overlay/036
index 19a181bbdd9361..f902617d4ab0a2 100755
--- a/tests/overlay/036
+++ b/tests/overlay/036
@@ -34,8 +34,8 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	# unmount the two extra mounts in case they did not fail
-	$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
-	$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+	_umount $SCRATCH_MNT 2>/dev/null
+	_umount $SCRATCH_MNT 2>/dev/null
 }
 
 # Import common functions.
@@ -66,13 +66,13 @@ _overlay_mount_dirs $lowerdir $upperdir $workdir \
 # with index=off - expect success
 _overlay_mount_dirs $lowerdir $upperdir $workdir2 \
 		    overlay0 $SCRATCH_MNT -oindex=off && \
-		    $UMOUNT_PROG $SCRATCH_MNT
+		    _umount $SCRATCH_MNT
 
 # Try to mount another overlay with the same workdir
 # with index=off - expect success
 _overlay_mount_dirs $lowerdir2 $upperdir2 $workdir \
 		    overlay1 $SCRATCH_MNT -oindex=off && \
-		    $UMOUNT_PROG $SCRATCH_MNT
+		    _umount $SCRATCH_MNT
 
 # Try to mount another overlay with the same upperdir
 # with index=on - expect EBUSY
diff --git a/tests/overlay/037 b/tests/overlay/037
index 834e176380ebea..c278e7cab1fe05 100755
--- a/tests/overlay/037
+++ b/tests/overlay/037
@@ -39,17 +39,17 @@ mkdir -p $lowerdir $lowerdir2 $upperdir $upperdir2 $workdir
 # Mount overlay with lowerdir, upperdir, workdir and index=on
 # to store the file handles of lowerdir and upperdir in overlay.origin xattr
 _overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -oindex=on
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # Try to mount an overlay with the same upperdir and different lowerdir - expect ESTALE
 _overlay_scratch_mount_dirs $lowerdir2 $upperdir $workdir -oindex=on \
 	2>&1 | _filter_error_mount
-$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+_umount $SCRATCH_MNT 2>/dev/null
 
 # Try to mount an overlay with the same workdir and different upperdir - expect ESTALE
 _overlay_scratch_mount_dirs $lowerdir $upperdir2 $workdir -oindex=on \
 	2>&1 | _filter_error_mount
-$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+_umount $SCRATCH_MNT 2>/dev/null
 
 # Mount overlay with original lowerdir, upperdir, workdir and index=on - expect success
 _overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -oindex=on
diff --git a/tests/overlay/040 b/tests/overlay/040
index 11c7bf129a3626..47f50eb0638da0 100755
--- a/tests/overlay/040
+++ b/tests/overlay/040
@@ -48,7 +48,7 @@ _scratch_mount
 # modify lower origin file.
 $CHATTR_PROG +i $SCRATCH_MNT/foo > /dev/null 2>&1
 
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # touching origin file in lower, should succeed
 touch $lowerdir/foo
diff --git a/tests/overlay/041 b/tests/overlay/041
index 36491b8fa0edf6..52ca351b66d86c 100755
--- a/tests/overlay/041
+++ b/tests/overlay/041
@@ -142,7 +142,7 @@ subdir_d=$($here/src/t_dir_type $pure_lower_dir $pure_lower_subdir_st_ino)
 [[ $subdir_d == "subdir d" ]] || \
 	echo "Merged dir: Invalid d_ino reported for subdir"
 
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # check overlayfs
 _overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o xino=on
diff --git a/tests/overlay/042 b/tests/overlay/042
index aaa10da33e0249..ddd4173abee8ce 100755
--- a/tests/overlay/042
+++ b/tests/overlay/042
@@ -45,7 +45,7 @@ _scratch_mount -o index=off
 # Copy up lower and create upper hardlink with no index
 ln $SCRATCH_MNT/0 $SCRATCH_MNT/1
 
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # Add lower hardlinks while overlay is offline
 ln $lowerdir/0 $lowerdir/2
diff --git a/tests/overlay/043 b/tests/overlay/043
index 7325c653ab5cab..15cb9bf4bafaca 100755
--- a/tests/overlay/043
+++ b/tests/overlay/043
@@ -126,7 +126,7 @@ echo 3 > /proc/sys/vm/drop_caches
 check_inode_numbers $testdir $tmp.after_copyup $tmp.after_move
 
 # Verify that the inode numbers survive a mount cycle
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 _overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o redirect_dir=on,xino=on
 
 # Compare inode numbers before/after mount cycle
diff --git a/tests/overlay/044 b/tests/overlay/044
index 4d04d883efd695..5f09cc31c32a1e 100755
--- a/tests/overlay/044
+++ b/tests/overlay/044
@@ -99,7 +99,7 @@ cat $FILES
 check_ino_nlink $SCRATCH_MNT $tmp.before $tmp.after_one
 
 # Verify that the hardlinks survive a mount cycle
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 _overlay_check_scratch_dirs $lowerdir $upperdir $workdir -o index=on,xino=on
 _overlay_scratch_mount_dirs $lowerdir $upperdir $workdir -o index=on,xino=on
 
diff --git a/tests/overlay/048 b/tests/overlay/048
index 897e797e2ff549..4bd9753666bf6c 100755
--- a/tests/overlay/048
+++ b/tests/overlay/048
@@ -32,7 +32,7 @@ report_nlink()
 		_ls_l $SCRATCH_MNT/$f | awk '{ print $2, $9 }' | _filter_scratch
 	done
 
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 # Create lower hardlinks
@@ -101,7 +101,7 @@ touch $SCRATCH_MNT/1
 touch $SCRATCH_MNT/2
 
 # Perform the rest of the changes offline
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 test_hardlinks_offline
 
diff --git a/tests/overlay/049 b/tests/overlay/049
index 3ee500c5dd13b8..b091330ea26e2c 100755
--- a/tests/overlay/049
+++ b/tests/overlay/049
@@ -32,7 +32,7 @@ create_redirect()
 	touch $SCRATCH_MNT/origin/file
 	mv $SCRATCH_MNT/origin $SCRATCH_MNT/$redirect
 
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 # Import common functions.
diff --git a/tests/overlay/050 b/tests/overlay/050
index ec936e2a758f81..7c8ed1a4e96e8c 100755
--- a/tests/overlay/050
+++ b/tests/overlay/050
@@ -76,7 +76,7 @@ mount_dirs()
 # Unmount the overlay without unmounting base fs
 unmount_dirs()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 # Check non-stale file handles of lower/upper files and verify
diff --git a/tests/overlay/051 b/tests/overlay/051
index 9404dbbab90f15..2dadb5a3027180 100755
--- a/tests/overlay/051
+++ b/tests/overlay/051
@@ -28,7 +28,7 @@ _cleanup()
 	# Cleanup overlay scratch mount that is holding base test mount
 	# to prevent _check_test_fs and _test_umount from failing before
 	# _check_scratch_fs _scratch_umount
-	$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+	_umount $SCRATCH_MNT 2>/dev/null
 }
 
 # Import common functions.
@@ -103,7 +103,7 @@ mount_dirs()
 # underlying dirs
 unmount_dirs()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 
 	_overlay_check_scratch_dirs $middle:$lower $upper $work \
 				-o "index=on,nfs_export=on"
diff --git a/tests/overlay/052 b/tests/overlay/052
index 37402067dbe65e..e3366ea44147cb 100755
--- a/tests/overlay/052
+++ b/tests/overlay/052
@@ -73,7 +73,7 @@ mount_dirs()
 # Unmount the overlay without unmounting base fs
 unmount_dirs()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 # Check non-stale file handles of lower/upper moved files
diff --git a/tests/overlay/053 b/tests/overlay/053
index f7891aceda7246..87f748cefd3338 100755
--- a/tests/overlay/053
+++ b/tests/overlay/053
@@ -30,7 +30,7 @@ _cleanup()
 	# Cleanup overlay scratch mount that is holding base test mount
 	# to prevent _check_test_fs and _test_umount from failing before
 	# _check_scratch_fs _scratch_umount
-	$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+	_umount $SCRATCH_MNT 2>/dev/null
 }
 
 # Import common functions.
@@ -99,7 +99,7 @@ mount_dirs()
 # underlying dirs
 unmount_dirs()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 
 	_overlay_check_scratch_dirs $middle:$lower $upper $work \
 				-o "index=on,nfs_export=on,redirect_dir=on"
diff --git a/tests/overlay/054 b/tests/overlay/054
index 8d7f026a2d9b00..566d266a1ad788 100755
--- a/tests/overlay/054
+++ b/tests/overlay/054
@@ -87,7 +87,7 @@ mount_dirs()
 # Unmount the overlay without unmounting base fs
 unmount_dirs()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 # Check encode/decode/read file handles of dir with non-indexed ancestor
diff --git a/tests/overlay/055 b/tests/overlay/055
index 87a348c94489b8..a5b169956f4c09 100755
--- a/tests/overlay/055
+++ b/tests/overlay/055
@@ -37,7 +37,7 @@ _cleanup()
 	# Cleanup overlay scratch mount that is holding base test mount
 	# to prevent _check_test_fs and _test_umount from failing before
 	# _check_scratch_fs _scratch_umount
-	$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+	_umount $SCRATCH_MNT 2>/dev/null
 }
 
 # Import common functions.
@@ -109,7 +109,7 @@ mount_dirs()
 # underlying dirs
 unmount_dirs()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 
 	_overlay_check_scratch_dirs $middle:$lower $upper $work \
 				-o "index=on,nfs_export=on,redirect_dir=on"
diff --git a/tests/overlay/056 b/tests/overlay/056
index 158f34d05c22e9..01c319d7263f3c 100755
--- a/tests/overlay/056
+++ b/tests/overlay/056
@@ -73,7 +73,7 @@ mkdir $lowerdir/testdir2/subdir
 _overlay_scratch_mount_dirs $lowerdir $upperdir $workdir
 touch $SCRATCH_MNT/testdir1/foo
 touch $SCRATCH_MNT/testdir2/subdir
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 remove_impure $upperdir/testdir1
 remove_impure $upperdir/testdir2
 
diff --git a/tests/overlay/057 b/tests/overlay/057
index da7ffda30277d9..b631d431a37b47 100755
--- a/tests/overlay/057
+++ b/tests/overlay/057
@@ -48,7 +48,7 @@ _overlay_scratch_mount_dirs $lowerdir $lowerdir2 $workdir2 -o redirect_dir=on
 # Create opaque parent with absolute redirect child in middle layer
 mkdir $SCRATCH_MNT/pure
 mv $SCRATCH_MNT/origin $SCRATCH_MNT/pure/redirect
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 _overlay_scratch_mount_dirs $lowerdir2:$lowerdir $upperdir $workdir -o redirect_dir=on
 mv $SCRATCH_MNT/pure/redirect $SCRATCH_MNT/redirect
 # List content of renamed merge dir before mount cycle
@@ -56,7 +56,7 @@ ls $SCRATCH_MNT/redirect/
 
 # Verify that redirects are followed by listing content of renamed merge dir
 # after mount cycle
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 _overlay_scratch_mount_dirs $lowerdir2:$lowerdir $upperdir $workdir -o redirect_dir=on
 ls $SCRATCH_MNT/redirect/
 
diff --git a/tests/overlay/059 b/tests/overlay/059
index c48d2a82c76ec4..84b5c80eb984de 100755
--- a/tests/overlay/059
+++ b/tests/overlay/059
@@ -33,7 +33,7 @@ create_origin_ref()
 	_scratch_mount -o redirect_dir=on
 	mv $SCRATCH_MNT/origin $SCRATCH_MNT/$ref
 
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 # Import common functions.
diff --git a/tests/overlay/060 b/tests/overlay/060
index bb61fcfa644342..3d0ea353feaa9a 100755
--- a/tests/overlay/060
+++ b/tests/overlay/060
@@ -130,7 +130,7 @@ mount_ro_overlay()
 
 umount_overlay()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 # Assumes it is called with overlay mounted.
diff --git a/tests/overlay/062 b/tests/overlay/062
index 9a1db7419c4ca2..97a1bd8c12f20e 100755
--- a/tests/overlay/062
+++ b/tests/overlay/062
@@ -18,7 +18,7 @@ _cleanup()
 {
 	cd /
 	rm -f $tmp.*
-	$UMOUNT_PROG $lowertestdir
+	_umount $lowertestdir
 }
 
 # Import common functions.
diff --git a/tests/overlay/063 b/tests/overlay/063
index d9f30606a92d44..a50e63665202f0 100755
--- a/tests/overlay/063
+++ b/tests/overlay/063
@@ -40,7 +40,7 @@ rm ${upperdir}/file
 mkdir ${SCRATCH_MNT}/file > /dev/null 2>&1
 
 # unmount overlayfs
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 echo "Silence is golden"
 # success, all done
diff --git a/tests/overlay/065 b/tests/overlay/065
index fb6d6dd1bfcc0e..26f1c4bde4da90 100755
--- a/tests/overlay/065
+++ b/tests/overlay/065
@@ -30,7 +30,7 @@ _cleanup()
 {
 	cd /
 	rm -f $tmp.*
-	$UMOUNT_PROG $mnt2 2>/dev/null
+	_umount $mnt2 2>/dev/null
 }
 
 # Import common functions.
@@ -63,7 +63,7 @@ mkdir -p $lowerdir/lower $upperdir $workdir
 echo Conflicting upperdir/lowerdir
 _overlay_scratch_mount_dirs $upperdir $upperdir $workdir \
 	2>&1 | _filter_error_mount
-$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+_umount $SCRATCH_MNT 2>/dev/null
 
 # Use new upper/work dirs for each test to avoid ESTALE errors
 # on mismatch lowerdir/upperdir (see test overlay/037)
@@ -75,7 +75,7 @@ mkdir $upperdir $workdir
 echo Conflicting workdir/lowerdir
 _overlay_scratch_mount_dirs $workdir $upperdir $workdir \
 	-oindex=off 2>&1 | _filter_error_mount
-$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+_umount $SCRATCH_MNT 2>/dev/null
 
 rm -rf $upperdir $workdir
 mkdir -p $upperdir/lower $workdir
@@ -85,7 +85,7 @@ mkdir -p $upperdir/lower $workdir
 echo Overlapping upperdir/lowerdir
 _overlay_scratch_mount_dirs $upperdir/lower $upperdir $workdir \
 	2>&1 | _filter_error_mount
-$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+_umount $SCRATCH_MNT 2>/dev/null
 
 rm -rf $upperdir $workdir
 mkdir $upperdir $workdir
@@ -94,7 +94,7 @@ mkdir $upperdir $workdir
 echo Conflicting lower layers
 _overlay_scratch_mount_dirs $lowerdir:$lowerdir $upperdir $workdir \
 	2>&1 | _filter_error_mount
-$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+_umount $SCRATCH_MNT 2>/dev/null
 
 rm -rf $upperdir $workdir
 mkdir $upperdir $workdir
@@ -103,7 +103,7 @@ mkdir $upperdir $workdir
 echo Overlapping lower layers below
 _overlay_scratch_mount_dirs $lowerdir:$lowerdir/lower $upperdir $workdir \
 	2>&1 | _filter_error_mount
-$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+_umount $SCRATCH_MNT 2>/dev/null
 
 rm -rf $upperdir $workdir
 mkdir $upperdir $workdir
@@ -112,7 +112,7 @@ mkdir $upperdir $workdir
 echo Overlapping lower layers above
 _overlay_scratch_mount_dirs $lowerdir/lower:$lowerdir $upperdir $workdir \
 	2>&1 | _filter_error_mount
-$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+_umount $SCRATCH_MNT 2>/dev/null
 
 rm -rf $upperdir $workdir
 mkdir -p $upperdir/upper $workdir $mnt2
@@ -129,14 +129,14 @@ mkdir -p $upperdir2 $workdir2 $mnt2
 echo "Overlapping with upperdir of another instance (index=on)"
 _overlay_scratch_mount_dirs $upperdir/upper $upperdir2 $workdir2 \
 	-oindex=on 2>&1 | _filter_busy_mount
-$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+_umount $SCRATCH_MNT 2>/dev/null
 
 rm -rf $upperdir2 $workdir2
 mkdir -p $upperdir2 $workdir2
 
 echo "Overlapping with upperdir of another instance (index=off)"
 _overlay_scratch_mount_dirs $upperdir/upper $upperdir2 $workdir2 \
-	-oindex=off && $UMOUNT_PROG $SCRATCH_MNT
+	-oindex=off && _umount $SCRATCH_MNT
 
 rm -rf $upperdir2 $workdir2
 mkdir -p $upperdir2 $workdir2
@@ -146,14 +146,14 @@ mkdir -p $upperdir2 $workdir2
 echo "Overlapping with workdir of another instance (index=on)"
 _overlay_scratch_mount_dirs $workdir/work $upperdir2 $workdir2 \
 	-oindex=on 2>&1 | _filter_busy_mount
-$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+_umount $SCRATCH_MNT 2>/dev/null
 
 rm -rf $upperdir2 $workdir2
 mkdir -p $upperdir2 $workdir2
 
 echo "Overlapping with workdir of another instance (index=off)"
 _overlay_scratch_mount_dirs $workdir/work $upperdir2 $workdir2 \
-	-oindex=off && $UMOUNT_PROG $SCRATCH_MNT
+	-oindex=off && _umount $SCRATCH_MNT
 
 # Move upper layer root into lower layer after mount
 echo Overlapping upperdir/lowerdir after mount
diff --git a/tests/overlay/067 b/tests/overlay/067
index bb09a6042b275d..12a1781c149644 100755
--- a/tests/overlay/067
+++ b/tests/overlay/067
@@ -70,7 +70,7 @@ stat $testfile >>$seqres.full
 diff -q $realfile $testfile >>$seqres.full &&
 	echo "diff with middle layer file doesn't know right from wrong! (cold cache)"
 
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 # check overlayfs
 _overlay_check_scratch_dirs $middle:$lower $upper $work -o xino=off
 
diff --git a/tests/overlay/068 b/tests/overlay/068
index 0d33cf12de8550..480ba67e33ea74 100755
--- a/tests/overlay/068
+++ b/tests/overlay/068
@@ -28,7 +28,7 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	# Unmount the nested overlay mount
-	$UMOUNT_PROG $mnt2 2>/dev/null
+	_umount $mnt2 2>/dev/null
 }
 
 # Import common functions.
@@ -100,7 +100,7 @@ mount_dirs()
 unmount_dirs()
 {
 	# unmount & check nested overlay
-	$UMOUNT_PROG $mnt2
+	_umount $mnt2
 	_overlay_check_dirs $SCRATCH_MNT $upper2 $work2 \
 		-o "index=on,nfs_export=on,redirect_dir=on"
 
diff --git a/tests/overlay/069 b/tests/overlay/069
index 373ab1ee3dc115..67969eebbfcaa3 100755
--- a/tests/overlay/069
+++ b/tests/overlay/069
@@ -28,7 +28,7 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	# Unmount the nested overlay mount
-	$UMOUNT_PROG $mnt2 2>/dev/null
+	_umount $mnt2 2>/dev/null
 }
 
 # Import common functions.
@@ -108,12 +108,12 @@ mount_dirs()
 unmount_dirs()
 {
 	# unmount & check nested overlay
-	$UMOUNT_PROG $mnt2
+	_umount $mnt2
 	_overlay_check_dirs $SCRATCH_MNT $upper2 $work2 \
 		-o "index=on,nfs_export=on,redirect_dir=on"
 
 	# unmount & check underlying overlay
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 	_overlay_check_dirs $lower $upper $work \
 		-o "index=on,nfs_export=on,redirect_dir=on"
 }
diff --git a/tests/overlay/070 b/tests/overlay/070
index 36991229f28fe7..104b5f492088d6 100755
--- a/tests/overlay/070
+++ b/tests/overlay/070
@@ -26,7 +26,7 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	# Unmount the nested overlay mount
-	$UMOUNT_PROG $mnt2 2>/dev/null
+	_umount $mnt2 2>/dev/null
 	[ -z "$loopdev" ] || _destroy_loop_device $loopdev
 }
 
@@ -93,12 +93,12 @@ mount_dirs()
 unmount_dirs()
 {
 	# unmount & check nested overlay
-	$UMOUNT_PROG $mnt2
+	_umount $mnt2
 	_overlay_check_dirs $SCRATCH_MNT $upper2 $work2 \
 		-o "redirect_dir=on,index=on,xino=on"
 
 	# unmount & check underlying overlay
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 	_overlay_check_scratch_dirs $lower $upper $work \
 		-o "index=on,nfs_export=on"
 }
diff --git a/tests/overlay/071 b/tests/overlay/071
index 2a6313142d09d2..c58347f6cdb1c6 100755
--- a/tests/overlay/071
+++ b/tests/overlay/071
@@ -29,7 +29,7 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	# Unmount the nested overlay mount
-	$UMOUNT_PROG $mnt2 2>/dev/null
+	_umount $mnt2 2>/dev/null
 	[ -z "$loopdev" ] || _destroy_loop_device $loopdev
 }
 
@@ -103,12 +103,12 @@ mount_dirs()
 unmount_dirs()
 {
 	# unmount & check nested overlay
-	$UMOUNT_PROG $mnt2
+	_umount $mnt2
 	_overlay_check_dirs $SCRATCH_MNT $upper2 $work2 \
 		-o "redirect_dir=on,index=on,xino=on"
 
 	# unmount & check underlying overlay
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 	_overlay_check_dirs $lower $upper $work \
 		-o "index=on,nfs_export=on"
 }
diff --git a/tests/overlay/076 b/tests/overlay/076
index fb94dff685b6cc..28bf2d305b94d7 100755
--- a/tests/overlay/076
+++ b/tests/overlay/076
@@ -47,7 +47,7 @@ _scratch_mount
 # on kernel v5.10..v5.10.14.  Anything but hang is considered a test success.
 $CHATTR_PROG +i $SCRATCH_MNT/foo > /dev/null 2>&1
 
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # success, all done
 echo "Silence is golden"
diff --git a/tests/overlay/077 b/tests/overlay/077
index 00de0825aea6dc..cff24800469362 100755
--- a/tests/overlay/077
+++ b/tests/overlay/077
@@ -65,7 +65,7 @@ mv $SCRATCH_MNT/f100 $SCRATCH_MNT/former/
 
 # Remove the lower directory and mount overlay again to create
 # a "former merge dir"
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 rm -rf $lowerdir/former
 _scratch_mount
 
diff --git a/tests/overlay/078 b/tests/overlay/078
index d6df11f6852f45..bcc5aff1b7dc89 100755
--- a/tests/overlay/078
+++ b/tests/overlay/078
@@ -61,7 +61,7 @@ do_check()
 
 	echo "Test chattr +$1 $2" >> $seqres.full
 
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 
 	# Add attribute to lower file
 	$CHATTR_PROG +$attr $lowertestfile
diff --git a/tests/overlay/079 b/tests/overlay/079
index cfcafceea56e66..f8926e091ca137 100755
--- a/tests/overlay/079
+++ b/tests/overlay/079
@@ -156,7 +156,7 @@ mount_ro_overlay()
 
 umount_overlay()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 test_no_access()
diff --git a/tests/overlay/080 b/tests/overlay/080
index ce5c2375fb3154..94fe33ae7db4d2 100755
--- a/tests/overlay/080
+++ b/tests/overlay/080
@@ -264,7 +264,7 @@ mount_overlay()
 
 umount_overlay()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 
diff --git a/tests/overlay/081 b/tests/overlay/081
index 2270a04750da1f..454eea2cd96576 100755
--- a/tests/overlay/081
+++ b/tests/overlay/081
@@ -46,7 +46,7 @@ ovl_fsid=$(stat -f -c '%i' $test_dir)
 	echo "Overlayfs (uuid=null) and upper fs fsid differ"
 
 # Keep base fs mounted in case it has a volatile fsid (e.g. tmpfs)
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # Test legacy behavior is preserved by default for existing "impure" overlayfs
 _scratch_mount
@@ -55,7 +55,7 @@ ovl_fsid=$(stat -f -c '%i' $test_dir)
 [[ "$ovl_fsid" == "$upper_fsid" ]] || \
 	echo "Overlayfs (after uuid=null) and upper fs fsid differ"
 
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # Test unique fsid on explicit opt-in for existing "impure" overlayfs
 _scratch_mount -o uuid=on
@@ -65,7 +65,7 @@ ovl_unique_fsid=$ovl_fsid
 [[ "$ovl_fsid" != "$upper_fsid" ]] || \
 	echo "Overlayfs (uuid=on) and upper fs fsid are the same"
 
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # Test unique fsid is persistent by default after it was created
 _scratch_mount
@@ -74,7 +74,7 @@ ovl_fsid=$(stat -f -c '%i' $test_dir)
 [[ "$ovl_fsid" == "$ovl_unique_fsid" ]] || \
 	echo "Overlayfs (after uuid=on) unique fsid is not persistent"
 
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # Test ignore existing persistent fsid on explicit opt-out
 _scratch_mount -o uuid=null
@@ -83,7 +83,7 @@ ovl_fsid=$(stat -f -c '%i' $test_dir)
 [[ "$ovl_fsid" == "$upper_fsid" ]] || \
 	echo "Overlayfs (uuid=null) and upper fs fsid differ"
 
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # Test fallback to uuid=null with non-upper ovelray
 _overlay_scratch_mount_dirs "$upperdir:$lowerdir" "-" "-" -o ro,uuid=on
@@ -110,7 +110,7 @@ ovl_unique_fsid=$ovl_fsid
 [[ "$ovl_fsid" != "$upper_fsid" ]] || \
 	echo "Overlayfs (new) and upper fs fsid are the same"
 
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 # Test unique fsid is persistent by default after it was created
 _scratch_mount -o uuid=on
@@ -119,7 +119,7 @@ ovl_fsid=$(stat -f -c '%i' $test_dir)
 [[ "$ovl_fsid" == "$ovl_unique_fsid" ]] || \
 	echo "Overlayfs (uuid=on) unique fsid is not persistent"
 
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 echo "Silence is golden"
 status=0
diff --git a/tests/overlay/083 b/tests/overlay/083
index 56e02f8cc77d73..aaa3fdb9ad139a 100755
--- a/tests/overlay/083
+++ b/tests/overlay/083
@@ -52,7 +52,7 @@ _mount -t overlay | grep ovl_esc_test  | tee -a $seqres.full | grep -v spaces &&
 
 # Re-create the upper/work dirs to mount them with a different lower
 # This is required in case index feature is enabled
-$UMOUNT_PROG $SCRATCH_MNT
+_umount $SCRATCH_MNT
 rm -rf "$upperdir" "$workdir"
 mkdir -p "$upperdir" "$workdir"
 
diff --git a/tests/overlay/084 b/tests/overlay/084
index 28e9a76dc734c0..67321bc7618389 100755
--- a/tests/overlay/084
+++ b/tests/overlay/084
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	# Unmount nested mounts if things fail
-	$UMOUNT_PROG $OVL_BASE_SCRATCH_MNT/nested  2>/dev/null
+	_umount $OVL_BASE_SCRATCH_MNT/nested  2>/dev/null
 	rm -rf $tmp
 }
 
@@ -44,7 +44,7 @@ nesteddir=$OVL_BASE_SCRATCH_MNT/nested
 
 umount_overlay()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 test_escape()
@@ -88,12 +88,12 @@ test_escape()
 	echo "nested xattr mount with trusted.overlay"
 	_overlay_mount_dirs $SCRATCH_MNT/layer2:$SCRATCH_MNT/layer1 - - overlayfs $nesteddir
 	stat $nesteddir/dir/file  2>&1 | _filter_scratch
-	$UMOUNT_PROG $nesteddir
+	_umount $nesteddir
 
 	echo "nested xattr mount with user.overlay"
 	_overlay_mount_dirs $SCRATCH_MNT/layer2:$SCRATCH_MNT/layer1 - - -o userxattr overlayfs $nesteddir
 	stat $nesteddir/dir/file  2>&1 | _filter_scratch
-	$UMOUNT_PROG $nesteddir
+	_umount $nesteddir
 
 	# Also ensure propagate the escaped xattr when we copy-up layer2/dir
 	echo "copy-up of escaped xattrs"
@@ -164,7 +164,7 @@ test_escaped_xwhiteout()
 
 	do_test_xwhiteout $prefix $nesteddir
 
-	$UMOUNT_PROG $nesteddir
+	_umount $nesteddir
 }
 
 test_escaped_xwhiteout trusted
diff --git a/tests/overlay/085 b/tests/overlay/085
index 046d01d161d829..8396ceb7c72b90 100755
--- a/tests/overlay/085
+++ b/tests/overlay/085
@@ -157,7 +157,7 @@ mount_ro_overlay()
 
 umount_overlay()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 test_no_access()
diff --git a/tests/overlay/086 b/tests/overlay/086
index 23c56d074ff34a..45e5b45a279853 100755
--- a/tests/overlay/086
+++ b/tests/overlay/086
@@ -38,21 +38,21 @@ _mount -t overlay none $SCRATCH_MNT \
 	2>> $seqres.full && \
 	echo "ERROR: invalid combination of lowerdir and lowerdir+ mount options"
 
-$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+_umount $SCRATCH_MNT 2>/dev/null
 
 _mount -t overlay none $SCRATCH_MNT \
 	-o"lowerdir=$lowerdir,datadir+=$lowerdir_colons" \
 	-o redirect_dir=follow,metacopy=on 2>> $seqres.full && \
 	echo "ERROR: invalid combination of lowerdir and datadir+ mount options"
 
-$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+_umount $SCRATCH_MNT 2>/dev/null
 
 _mount -t overlay none $SCRATCH_MNT \
 	-o"datadir+=$lowerdir,lowerdir+=$lowerdir_colons" \
 	-o redirect_dir=follow,metacopy=on 2>> $seqres.full && \
 	echo "ERROR: invalid order of lowerdir+ and datadir+ mount options"
 
-$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+_umount $SCRATCH_MNT 2>/dev/null
 
 # mount is expected to fail with escaped colons.
 _mount -t overlay none $SCRATCH_MNT \
@@ -60,7 +60,7 @@ _mount -t overlay none $SCRATCH_MNT \
 	2>> $seqres.full && \
 	echo "ERROR: incorrect parsing of escaped colons in lowerdir+ mount option"
 
-$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+_umount $SCRATCH_MNT 2>/dev/null
 
 # mount is expected to succeed without escaped colons.
 _mount -t overlay ovl_esc_test $SCRATCH_MNT \
diff --git a/tests/xfs/078 b/tests/xfs/078
index 4224fd40bc9fea..799d8881220582 100755
--- a/tests/xfs/078
+++ b/tests/xfs/078
@@ -16,7 +16,7 @@ _cleanup()
 {
 	cd /
 	rm -f $tmp.*
-	$UMOUNT_PROG $LOOP_MNT 2>/dev/null
+	_umount $LOOP_MNT 2>/dev/null
 	[ -n "$LOOP_DEV" ] && _destroy_loop_device $LOOP_DEV 2>/dev/null
 	# try to keep the image file if test fails
 	[ $status -eq 0 ] && rm -f $LOOP_IMG
@@ -81,7 +81,7 @@ _grow_loop()
 	$XFS_GROWFS_PROG $LOOP_MNT 2>&1 |  _filter_growfs 2>&1
 
 	echo "*** unmount"
-	$UMOUNT_PROG -d $LOOP_MNT && LOOP_DEV=
+	_umount -d $LOOP_MNT && LOOP_DEV=
 
 	# Large grows takes forever to check..
 	if [ "$check" -gt "0" ]
diff --git a/tests/xfs/148 b/tests/xfs/148
index 9e6798f999b356..7c9badd3c1b3a0 100755
--- a/tests/xfs/148
+++ b/tests/xfs/148
@@ -14,7 +14,7 @@ _begin_fstest auto quick fuzzers
 _cleanup()
 {
 	cd /
-	$UMOUNT_PROG $mntpt > /dev/null 2>&1
+	_umount $mntpt > /dev/null 2>&1
 	_destroy_loop_device $loopdev > /dev/null 2>&1
 	rm -r -f $tmp.*
 }
@@ -90,7 +90,7 @@ cat $tmp.log >> $seqres.full
 cat $tmp.log | _filter_test_dir
 
 # Corrupt the entries
-$UMOUNT_PROG $mntpt
+_umount $mntpt
 _destroy_loop_device $loopdev
 cp $imgfile $imgfile.old
 sed -b \
@@ -121,7 +121,7 @@ fi
 echo "does repair complain?" >> $seqres.full
 
 # Does repair complain about this?
-$UMOUNT_PROG $mntpt
+_umount $mntpt
 $XFS_REPAIR_PROG -n $loopdev >> $seqres.full 2>&1
 res=$?
 test $res -eq 1 || \
diff --git a/tests/xfs/149 b/tests/xfs/149
index bbaf86132dff37..ceb80b646f5784 100755
--- a/tests/xfs/149
+++ b/tests/xfs/149
@@ -22,7 +22,7 @@ loop_symlink=$TEST_DIR/loop_symlink.$$
 # Override the default cleanup function.
 _cleanup()
 {
-    $UMOUNT_PROG $mntdir
+    _umount $mntdir
     [ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
     rmdir $mntdir
     rm -f $loop_symlink
@@ -73,7 +73,7 @@ echo "=== xfs_growfs - check device symlink ==="
 $XFS_GROWFS_PROG -D 12288 $loop_symlink > /dev/null
 
 echo "=== unmount ==="
-$UMOUNT_PROG $mntdir || _fail "!!! failed to unmount"
+_umount $mntdir || _fail "!!! failed to unmount"
 
 echo "=== mount device symlink ==="
 _mount $loop_symlink $mntdir || _fail "!!! failed to loopback mount"
diff --git a/tests/xfs/186 b/tests/xfs/186
index 88f02585e7f667..2bd4fe10ab8930 100755
--- a/tests/xfs/186
+++ b/tests/xfs/186
@@ -87,7 +87,7 @@ _do_eas()
 		_create_eas $2 $3
 	fi
 	echo ""
-	cd /; $UMOUNT_PROG $SCRATCH_MNT
+	cd /; _umount $SCRATCH_MNT
 	_print_inode
 }
 
@@ -99,7 +99,7 @@ _do_dirents()
 	echo ""
 	_scratch_mount
 	_create_dirents $1 $2
-	cd /; $UMOUNT_PROG $SCRATCH_MNT
+	cd /; _umount $SCRATCH_MNT
 	_print_inode
 }
 
diff --git a/tests/xfs/289 b/tests/xfs/289
index 089a3f8cc14a68..aab5f96293b3a5 100755
--- a/tests/xfs/289
+++ b/tests/xfs/289
@@ -13,8 +13,8 @@ _begin_fstest growfs auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    $UMOUNT_PROG $tmpdir
-    $UMOUNT_PROG $tmpbind
+    _umount $tmpdir
+    _umount $tmpbind
     rmdir $tmpdir
     rm -f $tmpsymlink
     rmdir $tmpbind
diff --git a/tests/xfs/507 b/tests/xfs/507
index 75c183c07a9fce..60542112fbd5a1 100755
--- a/tests/xfs/507
+++ b/tests/xfs/507
@@ -22,7 +22,7 @@ _register_cleanup "_cleanup" BUS
 _cleanup()
 {
 	cd /
-	test -n "$loop_mount" && $UMOUNT_PROG $loop_mount > /dev/null 2>&1
+	test -n "$loop_mount" && _umount $loop_mount > /dev/null 2>&1
 	test -n "$loop_dev" && _destroy_loop_device $loop_dev
 	rm -rf $tmp.*
 }
diff --git a/tests/xfs/513 b/tests/xfs/513
index 5585a9c8e76703..cb8d0aca841530 100755
--- a/tests/xfs/513
+++ b/tests/xfs/513
@@ -14,7 +14,7 @@ _cleanup()
 {
 	cd /
 	rm -f $tmp.*
-	$UMOUNT_PROG $LOOP_MNT 2>/dev/null
+	_umount $LOOP_MNT 2>/dev/null
 	if [ -n "$LOOP_DEV" ];then
 		_destroy_loop_device $LOOP_DEV 2>/dev/null
 	fi
@@ -89,7 +89,7 @@ get_mount_info()
 
 force_unmount()
 {
-	$UMOUNT_PROG $LOOP_MNT >/dev/null 2>&1
+	_umount $LOOP_MNT >/dev/null 2>&1
 }
 
 # _do_test <mount options> <should be mounted?> [<key string> <key should be found?>]
diff --git a/tests/xfs/544 b/tests/xfs/544
index a3a23c1726ca1c..f1b5cc74983a62 100755
--- a/tests/xfs/544
+++ b/tests/xfs/544
@@ -15,7 +15,7 @@ _cleanup()
 	_cleanup_dump
 	cd /
 	rm -r -f $tmp.*
-	$UMOUNT_PROG $TEST_DIR/dest.$seq 2> /dev/null
+	_umount $TEST_DIR/dest.$seq 2> /dev/null
 	rmdir $TEST_DIR/src.$seq 2> /dev/null
 	rmdir $TEST_DIR/dest.$seq 2> /dev/null
 }
diff --git a/tests/xfs/806 b/tests/xfs/806
index 09c55332cc8800..9334d1780c6855 100755
--- a/tests/xfs/806
+++ b/tests/xfs/806
@@ -23,7 +23,7 @@ _cleanup()
 {
 	cd /
 	rm -r -f $tmp.*
-	umount $dummymnt &>/dev/null
+	_umount $dummymnt &>/dev/null
 	rmdir $dummymnt &>/dev/null
 	rm -f $dummyfile
 }
@@ -46,7 +46,7 @@ testme() {
 	XFS_SCRUB_PHASE=7 $XFS_SCRUB_PROG -d -o autofsck $dummymnt 2>&1 | \
 		grep autofsck | _filter_test_dir | \
 		sed -e 's/\(directive.\).*$/\1/g'
-	umount $dummymnt
+	_umount $dummymnt
 }
 
 # We don't test the absence of an autofsck directive because xfs_scrub behaves


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 2/6] misc: convert all umount(1) invocations to _umount
  2024-12-31 23:35 ` [PATCHSET 4/5] fstests: live health monitoring of filesystems Darrick J. Wong
  2024-12-31 23:57   ` [PATCH 1/6] misc: convert all $UMOUNT_PROG to a _umount helper Darrick J. Wong
@ 2024-12-31 23:57   ` Darrick J. Wong
  2024-12-31 23:57   ` [PATCH 3/6] xfs: test health monitoring code Darrick J. Wong
                     ` (3 subsequent siblings)
  5 siblings, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:57 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

From: Darrick J. Wong <djwong@kernel.org>

Find all the places where we call umount(1) directly and convert all of
those to _umount calls as well.

sed \
 -e 's/\([[:space:]]\)umount\([[:space:]]*"\$\)/\1_umount\2/g' \
 -e 's/\([[:space:]]\)umount\([[:space:]]*\$\)/\1_umount\2/g' \
 -e 's/^umount\([[:space:]]*"\$\)/_umount\1/g' \
 -e 's/^umount\([[:space:]]*\$\)/_umount\1/g' \
 -i $(git ls-files tests common check)

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 common/dmerror    |    2 +-
 common/populate   |    8 ++++----
 common/quota      |    2 +-
 common/rc         |    4 ++--
 common/xfs        |    2 +-
 tests/btrfs/012   |    2 +-
 tests/btrfs/199   |    2 +-
 tests/btrfs/291   |    2 +-
 tests/btrfs/298   |    4 ++--
 tests/ext4/006    |    4 ++--
 tests/ext4/007    |    4 ++--
 tests/ext4/008    |    4 ++--
 tests/ext4/009    |    8 ++++----
 tests/ext4/010    |    6 +++---
 tests/ext4/011    |    2 +-
 tests/ext4/012    |    2 +-
 tests/ext4/013    |    6 +++---
 tests/ext4/014    |    6 +++---
 tests/ext4/015    |    6 +++---
 tests/ext4/016    |    6 +++---
 tests/ext4/017    |    6 +++---
 tests/ext4/018    |    6 +++---
 tests/ext4/019    |    6 +++---
 tests/ext4/033    |    2 +-
 tests/generic/171 |    2 +-
 tests/generic/172 |    2 +-
 tests/generic/173 |    2 +-
 tests/generic/174 |    2 +-
 tests/generic/306 |    2 +-
 tests/generic/330 |    2 +-
 tests/generic/332 |    2 +-
 tests/generic/395 |    2 +-
 tests/generic/563 |    4 ++--
 tests/generic/631 |    2 +-
 tests/generic/717 |    2 +-
 tests/xfs/014     |    4 ++--
 tests/xfs/049     |    8 ++++----
 tests/xfs/073     |    8 ++++----
 tests/xfs/074     |    4 ++--
 tests/xfs/083     |    6 +++---
 tests/xfs/085     |    4 ++--
 tests/xfs/086     |    8 ++++----
 tests/xfs/087     |    6 +++---
 tests/xfs/088     |    8 ++++----
 tests/xfs/089     |    8 ++++----
 tests/xfs/091     |    8 ++++----
 tests/xfs/093     |    6 +++---
 tests/xfs/097     |    6 +++---
 tests/xfs/098     |    4 ++--
 tests/xfs/099     |    6 +++---
 tests/xfs/100     |    6 +++---
 tests/xfs/101     |    6 +++---
 tests/xfs/102     |    6 +++---
 tests/xfs/105     |    6 +++---
 tests/xfs/112     |    8 ++++----
 tests/xfs/113     |    6 +++---
 tests/xfs/117     |    6 +++---
 tests/xfs/120     |    6 +++---
 tests/xfs/123     |    6 +++---
 tests/xfs/124     |    6 +++---
 tests/xfs/125     |    6 +++---
 tests/xfs/126     |    6 +++---
 tests/xfs/130     |    2 +-
 tests/xfs/152     |    2 +-
 tests/xfs/169     |    6 +++---
 tests/xfs/206     |    2 +-
 tests/xfs/216     |    2 +-
 tests/xfs/217     |    2 +-
 tests/xfs/235     |    6 +++---
 tests/xfs/236     |    6 +++---
 tests/xfs/239     |    2 +-
 tests/xfs/241     |    2 +-
 tests/xfs/250     |    4 ++--
 tests/xfs/265     |    6 +++---
 tests/xfs/310     |    4 ++--
 tests/xfs/716     |    4 ++--
 76 files changed, 172 insertions(+), 172 deletions(-)


diff --git a/common/dmerror b/common/dmerror
index 1e6a35230f3ccb..2b6f001b8427f6 100644
--- a/common/dmerror
+++ b/common/dmerror
@@ -97,7 +97,7 @@ _dmerror_mount()
 
 _dmerror_unmount()
 {
-	umount $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 }
 
 _dmerror_cleanup()
diff --git a/common/populate b/common/populate
index 96e6a0f0572f12..e6bcdf346ac4ff 100644
--- a/common/populate
+++ b/common/populate
@@ -540,7 +540,7 @@ _scratch_xfs_populate() {
 	__populate_fragment_file "${SCRATCH_MNT}/REFCOUNTBT"
 	__populate_fragment_file "${SCRATCH_MNT}/RTREFCOUNTBT"
 
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 }
 
 # Populate an ext4 on the scratch device with (we hope) all known
@@ -642,7 +642,7 @@ _scratch_ext4_populate() {
 	# Make sure we get all the fragmentation we asked for
 	__populate_fragment_file "${SCRATCH_MNT}/S_IFREG.FMT_ETREE"
 
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 }
 
 # Find the inode number of a file
@@ -831,7 +831,7 @@ _scratch_xfs_populate_check() {
 	dblksz="$(_xfs_get_dir_blocksize "$SCRATCH_MNT")"
 	leaf_lblk="$((32 * 1073741824 / blksz))"
 	node_lblk="$((64 * 1073741824 / blksz))"
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 
 	__populate_check_xfs_dformat "${extents_file}" "extents"
 	__populate_check_xfs_dformat "${btree_file}" "btree"
@@ -948,7 +948,7 @@ _scratch_ext4_populate_check() {
 	extents_slink="$(__populate_find_inode "${SCRATCH_MNT}/S_IFLNK.FMT_EXTENTS")"
 	local_attr="$(__populate_find_inode "${SCRATCH_MNT}/ATTR.FMT_LOCAL")"
 	block_attr="$(__populate_find_inode "${SCRATCH_MNT}/ATTR.FMT_BLOCK")"
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 
 	__populate_check_ext4_dformat "${extents_file}" "extents"
 	__populate_check_ext4_dformat "${etree_file}" "etree"
diff --git a/common/quota b/common/quota
index 344c942045e5f2..7399819bb2579b 100644
--- a/common/quota
+++ b/common/quota
@@ -92,7 +92,7 @@ _require_xfs_quota_acct_enabled()
 	if [ -z "$umount" ] && [ "$dev" = "$SCRATCH_DEV" ]; then
 		umount="-u"
 	fi
-	test "$umount" = "-u" && umount "$dev" &>/dev/null
+	test "$umount" = "-u" && _umount "$dev" &>/dev/null
 
 	case "$dev" in
 	"$TEST_DEV")	fsname="test";;
diff --git a/common/rc b/common/rc
index d3ee76e01db892..0d5c785cecc017 100644
--- a/common/rc
+++ b/common/rc
@@ -1348,7 +1348,7 @@ _repair_scratch_fs()
 			_scratch_xfs_repair -L 2>&1
 			echo "log zap returns $?"
 		else
-			umount "$SCRATCH_MNT"
+			_umount "$SCRATCH_MNT"
 		fi
 		_scratch_xfs_repair "$@" 2>&1
 		res=$?
@@ -1413,7 +1413,7 @@ _repair_test_fs()
 				_test_xfs_repair -L >>$tmp.repair 2>&1
 				echo "log zap returns $?" >> $tmp.repair
 			else
-				umount "$TEST_DEV"
+				_umount "$TEST_DEV"
 			fi
 			_test_xfs_repair "$@" >>$tmp.repair 2>&1
 			res=$?
diff --git a/common/xfs b/common/xfs
index 86654a9379cf89..b9e897e0e8839a 100644
--- a/common/xfs
+++ b/common/xfs
@@ -466,7 +466,7 @@ _require_xfs_has_feature()
 
 	_xfs_has_feature "$1" "$2" && return 0
 
-	test "$umount" = "-u" && umount "$fs" &>/dev/null
+	test "$umount" = "-u" && _umount "$fs" &>/dev/null
 
 	test -n "$message" && _notrun "$message"
 
diff --git a/tests/btrfs/012 b/tests/btrfs/012
index 5811b3b339cb3e..7bb075dc2d0e93 100755
--- a/tests/btrfs/012
+++ b/tests/btrfs/012
@@ -70,7 +70,7 @@ mount -o loop $SCRATCH_MNT/ext2_saved/image $SCRATCH_MNT/mnt || \
 
 echo "Checking saved ext2 image against the original one:"
 $FSSUM_PROG -r $tmp.original $SCRATCH_MNT/mnt/$BASENAME
-umount $SCRATCH_MNT/mnt
+_umount $SCRATCH_MNT/mnt
 
 echo "Generating new data on the converted btrfs" >> $seqres.full
 mkdir -p $SCRATCH_MNT/new 
diff --git a/tests/btrfs/199 b/tests/btrfs/199
index f161e55057ff27..bdad1cb934c91f 100755
--- a/tests/btrfs/199
+++ b/tests/btrfs/199
@@ -19,7 +19,7 @@ _begin_fstest auto quick trim fiemap
 _cleanup()
 {
 	cd /
-	umount $loop_mnt &> /dev/null
+	_umount $loop_mnt &> /dev/null
 	_destroy_loop_device $loop_dev &> /dev/null
 	rm -rf $tmp.*
 }
diff --git a/tests/btrfs/291 b/tests/btrfs/291
index c31de3a96ef1f5..f69b65114ed696 100755
--- a/tests/btrfs/291
+++ b/tests/btrfs/291
@@ -134,7 +134,7 @@ do
 	_mount $snap_dev $SCRATCH_MNT || _fail "mount failed at entry $cur"
 	fsverity measure $SCRATCH_MNT/fsv >>$seqres.full 2>&1
 	measured=$?
-	umount $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 	[ $state -eq 1 ] && [ $measured -eq 0 ] && state=2
 	[ $state -eq 2 ] && ([ $measured -eq 0 ] || _fail "verity done, but measurement failed at entry $cur")
 	post_mount=$(count_merkle_items $snap_dev)
diff --git a/tests/btrfs/298 b/tests/btrfs/298
index d4aee55e785a94..c5b65772d428b1 100755
--- a/tests/btrfs/298
+++ b/tests/btrfs/298
@@ -31,11 +31,11 @@ $BTRFS_UTIL_PROG device scan --forget
 echo "#Scan seed device and check using mount" >> $seqres.full
 $BTRFS_UTIL_PROG device scan $SCRATCH_DEV >> $seqres.full
 _mount $SPARE_DEV $SCRATCH_MNT
-umount $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 echo "#check again, ensures seed device still in kernel" >> $seqres.full
 _mount $SPARE_DEV $SCRATCH_MNT
-umount $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 echo "#Now scan of non-seed device makes kernel forget" >> $seqres.full
 $BTRFS_TUNE_PROG -f -S 0 $SCRATCH_DEV >> $seqres.full 2>&1
diff --git a/tests/ext4/006 b/tests/ext4/006
index d7862073114872..579eab55b32d26 100755
--- a/tests/ext4/006
+++ b/tests/ext4/006
@@ -97,7 +97,7 @@ echo "++ modify scratch" >> $seqres.full
 _scratch_fuzz_modify >> $seqres.full 2>&1
 
 echo "++ unmount" >> $seqres.full
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 # repair in a loop...
 for p in $(seq 1 "${FSCK_PASSES}"); do
@@ -122,7 +122,7 @@ echo "++ modify scratch" >> $ROUND2_LOG
 _scratch_fuzz_modify >> $ROUND2_LOG 2>&1
 
 echo "++ unmount" >> $ROUND2_LOG
-umount "${SCRATCH_MNT}" >> $ROUND2_LOG 2>&1
+_umount "${SCRATCH_MNT}" >> $ROUND2_LOG 2>&1
 
 cat "$ROUND2_LOG" >> $seqres.full
 
diff --git a/tests/ext4/007 b/tests/ext4/007
index deedbd9e8fb3d8..24cc2290f79a29 100755
--- a/tests/ext4/007
+++ b/tests/ext4/007
@@ -54,7 +54,7 @@ done
 for x in `seq 2 64`; do
 	touch "${TESTFILE}.${x}"
 done
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
@@ -89,7 +89,7 @@ for x in `seq 1 64`; do
 	test $? -ne 0 && broken=1
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
diff --git a/tests/ext4/008 b/tests/ext4/008
index b4b20ac10d6d2a..a586bf681dfd34 100755
--- a/tests/ext4/008
+++ b/tests/ext4/008
@@ -50,7 +50,7 @@ done
 for x in `seq 2 64`; do
 	echo moo >> "${TESTFILE}.${x}"
 done
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
@@ -70,7 +70,7 @@ e2fsck -fy "${SCRATCH_DEV}" >> $seqres.full 2>&1
 echo "+ mount image (2)"
 _scratch_mount
 
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
diff --git a/tests/ext4/009 b/tests/ext4/009
index 06a42fd77ffa0c..f6fe1e5f0d8d2a 100755
--- a/tests/ext4/009
+++ b/tests/ext4/009
@@ -45,13 +45,13 @@ done
 blksz="$(stat -f -c '%s' "${SCRATCH_MNT}")"
 freeblks="$(stat -f -c '%a' "${SCRATCH_MNT}")"
 $XFS_IO_PROG -f -c "falloc 0 $((blksz * freeblks))" "${SCRATCH_MNT}/bigfile2" >> $seqres.full
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ make some files"
 _scratch_mount
 rm -rf "${SCRATCH_MNT}/bigfile2"
 touch "${SCRATCH_MNT}/bigfile"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
@@ -70,7 +70,7 @@ $XFS_IO_PROG -f -c "falloc 0 $((blksz * freeblks))" "${SCRATCH_MNT}/bigfile" >>
 after="$(stat -c '%b' "${SCRATCH_MNT}/bigfile")"
 echo "$((after * b_bytes))" lt "$((blksz * freeblks / 4))" >> $seqres.full
 test "$((after * b_bytes))" -lt "$((blksz * freeblks / 4))" || _fail "falloc should fail"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ repair fs"
 e2fsck -fy "${SCRATCH_DEV}" >> $seqres.full 2>&1
@@ -80,7 +80,7 @@ _scratch_mount
 
 echo "+ modify files (2)"
 $XFS_IO_PROG -f -c "falloc 0 $((blksz * freeblks))" "${SCRATCH_MNT}/bigfile" >> $seqres.full
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
diff --git a/tests/ext4/010 b/tests/ext4/010
index 1139c79e80d538..27ce20f822256f 100755
--- a/tests/ext4/010
+++ b/tests/ext4/010
@@ -46,7 +46,7 @@ echo "+ make some files"
 for i in `seq 1 $((nr_groups * 8))`; do
 	mkdir -p "${SCRATCH_MNT}/d_${i}"
 done
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
@@ -61,7 +61,7 @@ _scratch_mount
 
 echo "+ modify files"
 touch "${SCRATCH_MNT}/file0" > /dev/null 2>&1 && _fail "touch should fail"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ repair fs"
 e2fsck -fy "${SCRATCH_DEV}" >> $seqres.full 2>&1
@@ -71,7 +71,7 @@ _scratch_mount
 
 echo "+ modify files (2)"
 touch "${SCRATCH_MNT}/file1"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
diff --git a/tests/ext4/011 b/tests/ext4/011
index cae4fb6b84768b..cb085c95596de1 100755
--- a/tests/ext4/011
+++ b/tests/ext4/011
@@ -39,7 +39,7 @@ blksz="$(stat -f -c '%s' "${SCRATCH_MNT}")"
 
 echo "+ make some files"
 echo moo > "${SCRATCH_MNT}/file0"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
diff --git a/tests/ext4/012 b/tests/ext4/012
index f7f2b0fb455762..e7adc617c4db17 100755
--- a/tests/ext4/012
+++ b/tests/ext4/012
@@ -39,7 +39,7 @@ blksz="$(stat -f -c '%s' "${SCRATCH_MNT}")"
 
 echo "+ make some files"
 echo moo > "${SCRATCH_MNT}/file0"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
diff --git a/tests/ext4/013 b/tests/ext4/013
index 7d2a9154a66936..4363e3d104b716 100755
--- a/tests/ext4/013
+++ b/tests/ext4/013
@@ -50,7 +50,7 @@ for x in `seq 2 64`; do
 	touch "${TESTFILE}.${x}"
 done
 inode="$(stat -c '%i' "${TESTFILE}.1")"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
@@ -72,7 +72,7 @@ for x in `seq 1 64`; do
 	test $? -ne 0 && broken=1
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ repair fs"
 e2fsck -fy "${SCRATCH_DEV}" >> $seqres.full 2>&1
@@ -93,7 +93,7 @@ for x in `seq 1 64`; do
 	test $? -ne 0 && broken=1
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
diff --git a/tests/ext4/014 b/tests/ext4/014
index ffed795ad4e93c..c874a62335d1f3 100755
--- a/tests/ext4/014
+++ b/tests/ext4/014
@@ -49,7 +49,7 @@ done
 for x in `seq 2 64`; do
 	touch "${TESTFILE}.${x}"
 done
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
@@ -70,7 +70,7 @@ for x in `seq 1 64`; do
 	test $? -ne 0 && broken=1
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ repair fs"
 e2fsck -fy "${SCRATCH_DEV}" >> $seqres.full 2>&1 && _fail "e2fsck should not succeed"
@@ -91,7 +91,7 @@ for x in `seq 1 64`; do
 	test $? -ne 0 && broken=1
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
diff --git a/tests/ext4/015 b/tests/ext4/015
index 81feda5c9423fb..32b3884de32035 100755
--- a/tests/ext4/015
+++ b/tests/ext4/015
@@ -45,7 +45,7 @@ $XFS_IO_PROG -f -c "falloc 0 $((blksz * freeblks))" "${SCRATCH_MNT}/bigfile" >>
 seq 1 2 ${freeblks} | while read lblk; do
 	$XFS_IO_PROG -f -c "fpunch $((lblk * blksz)) ${blksz}" "${SCRATCH_MNT}/bigfile" >> $seqres.full
 done
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
@@ -60,7 +60,7 @@ _scratch_mount
 
 echo "+ modify files"
 echo moo >> "${SCRATCH_MNT}/bigfile" 2> /dev/null && _fail "extent tree should be corrupt"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ repair fs"
 e2fsck -fy "${SCRATCH_DEV}" >> $seqres.full 2>&1
@@ -70,7 +70,7 @@ _scratch_mount
 
 echo "+ modify files (2)"
 $XFS_IO_PROG -f -c "pwrite ${blksz} ${blksz}" "${SCRATCH_MNT}/bigfile" >> $seqres.full
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
diff --git a/tests/ext4/016 b/tests/ext4/016
index b7db4cfda649ef..f0f1709b6c208a 100755
--- a/tests/ext4/016
+++ b/tests/ext4/016
@@ -40,7 +40,7 @@ echo "+ make some files"
 for x in `seq 1 15`; do
 	mkdir -p "${SCRATCH_MNT}/test/d_${x}"
 done
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
@@ -53,7 +53,7 @@ _scratch_mount
 
 echo "+ modify dirs"
 mkdir -p "${SCRATCH_MNT}/test/newdir" 2> /dev/null && _fail "directory should be corrupt"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ repair fs"
 e2fsck -fy "${SCRATCH_DEV}" >> $seqres.full 2>&1
@@ -63,7 +63,7 @@ _scratch_mount
 
 echo "+ modify dirs (2)"
 mkdir -p "${SCRATCH_MNT}/test/newdir" || _fail "directory should be corrupt"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
diff --git a/tests/ext4/017 b/tests/ext4/017
index fc867442c3da3a..7fa563106d676c 100755
--- a/tests/ext4/017
+++ b/tests/ext4/017
@@ -43,7 +43,7 @@ for x in `seq 1 $((blksz * 4 / 256))`; do
 	fname="$(printf "%.255s\n" "$(perl -e "print \"${x}_\" x 500;")")"
 	touch "${SCRATCH_MNT}/test/${fname}"
 done
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
@@ -56,7 +56,7 @@ _scratch_mount
 
 echo "+ modify dirs"
 mkdir -p "${SCRATCH_MNT}/test/newdir" 2> /dev/null && _fail "htree should be corrupt"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ repair fs"
 e2fsck -fy "${SCRATCH_DEV}" >> $seqres.full 2>&1
@@ -66,7 +66,7 @@ _scratch_mount
 
 echo "+ modify dirs (2)"
 mkdir -p "${SCRATCH_MNT}/test/newdir" || _fail "htree should not be corrupt"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
diff --git a/tests/ext4/018 b/tests/ext4/018
index f7377f059fb826..2e24fe2e82918d 100755
--- a/tests/ext4/018
+++ b/tests/ext4/018
@@ -40,7 +40,7 @@ blksz="$(stat -f -c '%s' "${SCRATCH_MNT}")"
 echo "+ make some files"
 $XFS_IO_PROG -f -c "pwrite -S 0x62 0 ${blksz}" "${SCRATCH_MNT}/attrfile" >> $seqres.full
 setfattr -n user.key -v "$(perl -e 'print "v" x 300;')" "${SCRATCH_MNT}/attrfile"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
@@ -54,7 +54,7 @@ _scratch_mount
 
 echo "+ modify attrs"
 setfattr -n user.newkey -v "$(perl -e 'print "v" x 300;')" "${SCRATCH_MNT}/attrfile" 2> /dev/null && _fail "xattr should be corrupt"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ repair fs"
 e2fsck -fy "${SCRATCH_DEV}" >> $seqres.full 2>&1
@@ -64,7 +64,7 @@ _scratch_mount
 
 echo "+ modify attrs (2)"
 setfattr -n user.newkey -v "$(perl -e 'print "v" x 300;')" "${SCRATCH_MNT}/attrfile" || _fail "xattr should not be corrupt"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
diff --git a/tests/ext4/019 b/tests/ext4/019
index 987972a80a3704..7df7ccbed5e50d 100755
--- a/tests/ext4/019
+++ b/tests/ext4/019
@@ -43,7 +43,7 @@ echo "file contents: moo" > "${SCRATCH_MNT}/x"
 str="$(perl -e "print './' x $(( (blksz / 2) - 16));")x"
 (cd $SCRATCH_MNT; ln -s "${str}" "long_symlink")
 cat "${SCRATCH_MNT}/long_symlink"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
@@ -54,7 +54,7 @@ debugfs -w -R 'zap -f /long_symlink -p 0x62 0' "${SCRATCH_DEV}" 2> /dev/null
 echo "+ mount image"
 _scratch_mount 2> /dev/null
 cat "${SCRATCH_MNT}/long_symlink" 2>/dev/null && _fail "symlink should be broken"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ repair fs"
 e2fsck -fy "${SCRATCH_DEV}" >> $seqres.full 2>&1
@@ -62,7 +62,7 @@ e2fsck -fy "${SCRATCH_DEV}" >> $seqres.full 2>&1
 echo "+ mount image (2)"
 _scratch_mount
 cat "${SCRATCH_MNT}/long_symlink" 2>/dev/null && _fail "symlink should be broken"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || _fail "fsck should not fail"
diff --git a/tests/ext4/033 b/tests/ext4/033
index 53f7106e2c6ba4..19cd1fb6f20d4c 100755
--- a/tests/ext4/033
+++ b/tests/ext4/033
@@ -14,7 +14,7 @@ _begin_fstest auto ioctl resize
 # Override the default cleanup function.
 _cleanup()
 {
-	umount $SCRATCH_MNT >/dev/null 2>&1
+	_umount $SCRATCH_MNT >/dev/null 2>&1
 	_dmhugedisk_cleanup
 	cd /
 	rm -f $tmp.*
diff --git a/tests/generic/171 b/tests/generic/171
index dd56aa792afbd5..f51f58e9495f8e 100755
--- a/tests/generic/171
+++ b/tests/generic/171
@@ -36,7 +36,7 @@ mkdir $testdir
 echo "Reformat with appropriate size"
 blksz="$(_get_block_size $testdir)"
 nr_blks=10240
-umount $SCRATCH_MNT
+_umount $SCRATCH_MNT
 sz_bytes=$((nr_blks * 8 * blksz))
 if [ $sz_bytes -lt $((32 * 1048576)) ]; then
 	sz_bytes=$((32 * 1048576))
diff --git a/tests/generic/172 b/tests/generic/172
index c23a1228455464..8d32f0288b1556 100755
--- a/tests/generic/172
+++ b/tests/generic/172
@@ -35,7 +35,7 @@ mkdir $testdir
 
 echo "Reformat with appropriate size"
 blksz="$(_get_block_size $testdir)"
-umount $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 file_size=$((768 * 1024 * 1024))
 fs_size=$((1024 * 1024 * 1024))
diff --git a/tests/generic/173 b/tests/generic/173
index 8df3c6df21b29c..2f1ea96ef6238e 100755
--- a/tests/generic/173
+++ b/tests/generic/173
@@ -36,7 +36,7 @@ mkdir $testdir
 echo "Reformat with appropriate size"
 blksz="$(_get_block_size $testdir)"
 nr_blks=10240
-umount $SCRATCH_MNT
+_umount $SCRATCH_MNT
 sz_bytes=$((nr_blks * 8 * blksz))
 if [ $sz_bytes -lt $((32 * 1048576)) ]; then
 	sz_bytes=$((32 * 1048576))
diff --git a/tests/generic/174 b/tests/generic/174
index b9c292071445fe..d93546eeb35581 100755
--- a/tests/generic/174
+++ b/tests/generic/174
@@ -37,7 +37,7 @@ mkdir $testdir
 echo "Reformat with appropriate size"
 blksz="$(_get_block_size $testdir)"
 nr_blks=10240
-umount $SCRATCH_MNT
+_umount $SCRATCH_MNT
 sz_bytes=$((nr_blks * 8 * blksz))
 if [ $sz_bytes -lt $((32 * 1048576)) ]; then
 	sz_bytes=$((32 * 1048576))
diff --git a/tests/generic/306 b/tests/generic/306
index a6ea654b67d179..e6502cb881e21e 100755
--- a/tests/generic/306
+++ b/tests/generic/306
@@ -12,7 +12,7 @@ _begin_fstest auto quick rw
 # Override the default cleanup function.
 _cleanup()
 {
-    umount $BINDFILE
+    _umount $BINDFILE
     cd /
     rm -f $tmp.*
 }
diff --git a/tests/generic/330 b/tests/generic/330
index 4fa81f9913ee7e..ab9af84611d725 100755
--- a/tests/generic/330
+++ b/tests/generic/330
@@ -61,7 +61,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "Check for damage"
-umount $SCRATCH_MNT
+_umount $SCRATCH_MNT
 _repair_scratch_fs >> $seqres.full
 
 # success, all done
diff --git a/tests/generic/332 b/tests/generic/332
index 4a61e4a02a7cdc..b15546d66a41e0 100755
--- a/tests/generic/332
+++ b/tests/generic/332
@@ -61,7 +61,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "Check for damage"
-umount $SCRATCH_MNT
+_umount $SCRATCH_MNT
 _repair_scratch_fs >> $seqres.full
 
 # success, all done
diff --git a/tests/generic/395 b/tests/generic/395
index 45787fff06be1d..d0600d0282c6a4 100755
--- a/tests/generic/395
+++ b/tests/generic/395
@@ -75,7 +75,7 @@ mount --bind $SCRATCH_MNT $SCRATCH_MNT/ro_bind_mnt
 mount -o remount,ro,bind $SCRATCH_MNT/ro_bind_mnt
 _set_encpolicy $SCRATCH_MNT/ro_bind_mnt/ro_dir |& _filter_scratch
 _get_encpolicy $SCRATCH_MNT/ro_bind_mnt/ro_dir |& _filter_scratch
-umount $SCRATCH_MNT/ro_bind_mnt
+_umount $SCRATCH_MNT/ro_bind_mnt
 
 # success, all done
 status=0
diff --git a/tests/generic/563 b/tests/generic/563
index ade66f93fbf30b..166774653a66d6 100755
--- a/tests/generic/563
+++ b/tests/generic/563
@@ -21,7 +21,7 @@ _cleanup()
 
 	echo $$ > $cgdir/cgroup.procs
 	rmdir $cgdir/$seq-cg* > /dev/null 2>&1
-	umount $SCRATCH_MNT > /dev/null 2>&1
+	_umount $SCRATCH_MNT > /dev/null 2>&1
 	_destroy_loop_device $LOOP_DEV > /dev/null 2>&1
 }
 
@@ -80,7 +80,7 @@ reset()
 	rmdir $cgdir/$seq-cg* > /dev/null 2>&1
 	$XFS_IO_PROG -fc "pwrite 0 $iosize" $SCRATCH_MNT/file \
 		>> $seqres.full 2>&1
-	umount $SCRATCH_MNT || _fail "umount failed"
+	_umount $SCRATCH_MNT || _fail "umount failed"
 	_mount $LOOP_DEV $SCRATCH_MNT || _fail "mount failed"
 	stat $SCRATCH_MNT/file > /dev/null
 }
diff --git a/tests/generic/631 b/tests/generic/631
index c7c95e5608b760..c9f8299c948f83 100755
--- a/tests/generic/631
+++ b/tests/generic/631
@@ -84,7 +84,7 @@ worker() {
 		touch $mergedir/etc/access.conf
 		mv $mergedir/etc/access.conf $mergedir/etc/access.conf.bak
 		touch $mergedir/etc/access.conf
-		umount $mergedir
+		_umount $mergedir
 	done
 	rm -f $SCRATCH_MNT/workers/$tag
 }
diff --git a/tests/generic/717 b/tests/generic/717
index 4378e964ab8597..7ff356e255b3d1 100755
--- a/tests/generic/717
+++ b/tests/generic/717
@@ -85,7 +85,7 @@ mkdir -p $SCRATCH_MNT/xyz
 mount --bind $dir $SCRATCH_MNT/xyz --bind
 _pwrite_byte 0x60 0 $((blksz * (nrblks + 2))) $dir/c >> $seqres.full
 $XFS_IO_PROG -c "exchangerange $SCRATCH_MNT/xyz/c" $dir/a
-umount $SCRATCH_MNT/xyz
+_umount $SCRATCH_MNT/xyz
 
 echo Swapping a file with itself
 $XFS_IO_PROG -c "exchangerange $dir/a" $dir/a
diff --git a/tests/xfs/014 b/tests/xfs/014
index 098f64186e1134..efae4efa5138f5 100755
--- a/tests/xfs/014
+++ b/tests/xfs/014
@@ -22,7 +22,7 @@ _begin_fstest auto enospc quick quota prealloc
 _cleanup()
 {
 	cd /
-	umount $LOOP_MNT 2>/dev/null
+	_umount $LOOP_MNT 2>/dev/null
 	_scratch_unmount 2>/dev/null
 	rm -f $tmp.*
 }
@@ -174,7 +174,7 @@ mount -t xfs -o loop,uquota,gquota $LOOP_FILE $LOOP_MNT || \
 _test_enospc $LOOP_MNT
 _test_edquot $LOOP_MNT
 
-umount $LOOP_MNT
+_umount $LOOP_MNT
 
 echo $orig_sp_time > /proc/sys/fs/xfs/speculative_prealloc_lifetime
 
diff --git a/tests/xfs/049 b/tests/xfs/049
index 668ac374576a69..89ee1dbdff4f10 100755
--- a/tests/xfs/049
+++ b/tests/xfs/049
@@ -13,8 +13,8 @@ _begin_fstest rw auto quick
 _cleanup()
 {
     cd /
-    umount $SCRATCH_MNT/test2 > /dev/null 2>&1
-    umount $SCRATCH_MNT/test > /dev/null 2>&1
+    _umount $SCRATCH_MNT/test2 > /dev/null 2>&1
+    _umount $SCRATCH_MNT/test > /dev/null 2>&1
     rm -f $tmp.*
 
     if [ -w $seqres.full ]
@@ -96,11 +96,11 @@ rm -rf $SCRATCH_MNT/test/* >> $seqres.full 2>&1 \
     || _fail "!!! clean failed"
 
 _log "umount ext2 on xfs"
-umount $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
+_umount $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! umount ext2 failed"
 
 _log "umount xfs"
-umount $SCRATCH_MNT/test >> $seqres.full 2>&1 \
+_umount $SCRATCH_MNT/test >> $seqres.full 2>&1 \
     || _fail "!!! umount xfs failed"
 
 echo "--- mounts at end (before cleanup)" >> $seqres.full
diff --git a/tests/xfs/073 b/tests/xfs/073
index 28f1fad08b8c96..7d99179b7bc974 100755
--- a/tests/xfs/073
+++ b/tests/xfs/073
@@ -21,9 +21,9 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount 2>/dev/null
-	umount $imgs.loop 2>/dev/null
+	_umount $imgs.loop 2>/dev/null
 	[ -d $imgs.loop ] && rmdir $imgs.loop
-	umount $imgs.source_dir 2>/dev/null
+	_umount $imgs.source_dir 2>/dev/null
 	[ -d $imgs.source_dir ] && rm -rf $imgs.source_dir
 	rm -f $imgs.* $tmp.* /var/tmp/xfs_copy.log.*
 }
@@ -98,8 +98,8 @@ _verify_copy()
 	diff -u $tmp.geometry1 $tmp.geometry2
 
 	echo unmounting and removing new image
-	umount $source_dir
-	umount $target_dir > /dev/null 2>&1
+	_umount $source_dir
+	_umount $target_dir > /dev/null 2>&1
 	rm -f $target
 }
 
diff --git a/tests/xfs/074 b/tests/xfs/074
index 278f0ade694d22..282642a8674557 100755
--- a/tests/xfs/074
+++ b/tests/xfs/074
@@ -59,7 +59,7 @@ $XFS_IO_PROG -ft \
 	-c "falloc 0 $(($BLOCK_SIZE * 2097152))" \
 	$LOOP_MNT/foo >> $seqres.full
 
-umount $LOOP_MNT
+_umount $LOOP_MNT
 _check_xfs_filesystem $LOOP_DEV none none
 
 _mkfs_dev -f $LOOP_DEV
@@ -72,7 +72,7 @@ $XFS_IO_PROG -ft \
 	-c "falloc 1023m 2g" \
 	$LOOP_MNT/foo >> $seqres.full
 
-umount $LOOP_MNT
+_umount $LOOP_MNT
 _check_xfs_filesystem $LOOP_DEV none none
 
 # success, all done
diff --git a/tests/xfs/083 b/tests/xfs/083
index 9291c8c0382489..875937e6ffe3b3 100755
--- a/tests/xfs/083
+++ b/tests/xfs/083
@@ -57,7 +57,7 @@ scratch_repair() {
 			_scratch_xfs_repair -L >> "${FSCK_LOG}" 2>&1
 			echo "+++ returns $?" >> "${FSCK_LOG}"
 		else
-			umount "${SCRATCH_MNT}" >> "${FSCK_LOG}" 2>&1
+			_umount "${SCRATCH_MNT}" >> "${FSCK_LOG}" 2>&1
 		fi
 	elif [ "${fsck_pass}" -eq "${FSCK_PASSES}" ]; then
 		echo "++ fsck did not fix in ${FSCK_PASSES} passes." >> "${FSCK_LOG}"
@@ -109,7 +109,7 @@ echo "+++ modify scratch" >> $seqres.full
 _scratch_fuzz_modify >> $seqres.full 2>&1
 
 echo "++ umount" >> $seqres.full
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 # repair in a loop...
 for p in $(seq 1 "${FSCK_PASSES}"); do
@@ -134,7 +134,7 @@ echo "+++ modify scratch" >> $ROUND2_LOG
 _scratch_fuzz_modify >> $ROUND2_LOG 2>&1
 
 echo "++ umount" >> $ROUND2_LOG
-umount "${SCRATCH_MNT}" >> $ROUND2_LOG 2>&1
+_umount "${SCRATCH_MNT}" >> $ROUND2_LOG 2>&1
 
 cat "$ROUND2_LOG" >> $seqres.full
 
diff --git a/tests/xfs/085 b/tests/xfs/085
index d33dd199e6f9c1..9faf16fde5cdab 100755
--- a/tests/xfs/085
+++ b/tests/xfs/085
@@ -54,7 +54,7 @@ for x in `seq 2 64`; do
 done
 inode="$(stat -c '%i' "${TESTFILE}.1")"
 agcount="$(_xfs_mount_agcount $SCRATCH_MNT)"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -82,7 +82,7 @@ for x in `seq 1 64`; do
 	test $? -ne 0 && broken=1
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/086 b/tests/xfs/086
index 44985f3913254d..03327cdeaf3f08 100755
--- a/tests/xfs/086
+++ b/tests/xfs/086
@@ -56,7 +56,7 @@ done
 inode="$(stat -c '%i' "${TESTFILE}.1")"
 agcount="$(_xfs_mount_agcount $SCRATCH_MNT)"
 test "${agcount}" -gt 1 || _notrun "Single-AG XFS not supported"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -73,7 +73,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 	for x in `seq 1 64`; do
 		$XFS_IO_PROG -f -c "pwrite -S 0x62 0 ${blksz}" "${TESTFILE}.${x}" >> $seqres.full
 	done
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -97,7 +97,7 @@ echo "+ modify files (2)"
 for x in `seq 1 64`; do
 	$XFS_IO_PROG -f -c "pwrite -S 0x62 ${blksz} ${blksz}" "${TESTFILE}.${x}" >> $seqres.full
 done
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ repair fs"
 _repair_scratch_fs >> $seqres.full 2>&1
@@ -114,7 +114,7 @@ for x in `seq 1 64`; do
 	test -s "${TESTFILE}.${x}" || broken=1
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/087 b/tests/xfs/087
index 3cca105685fc6a..aeef30657b9491 100755
--- a/tests/xfs/087
+++ b/tests/xfs/087
@@ -55,7 +55,7 @@ for x in `seq 2 64`; do
 done
 inode="$(stat -c '%i' "${TESTFILE}.1")"
 agcount="$(_xfs_mount_agcount $SCRATCH_MNT)"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -72,7 +72,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 	for x in `seq 65 70`; do
 		touch "${TESTFILE}.${x}" 2> /dev/null && broken=0
 	done
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 echo "broken: ${broken}"
 
@@ -91,7 +91,7 @@ for x in `seq 65 70`; do
 	touch "${TESTFILE}.${x}" || broken=1
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/088 b/tests/xfs/088
index b54a1ab7d00342..de100136014ba7 100755
--- a/tests/xfs/088
+++ b/tests/xfs/088
@@ -56,7 +56,7 @@ for x in `seq 2 64`; do
 done
 inode="$(stat -c '%i' "${TESTFILE}.1")"
 agcount="$(_xfs_mount_agcount $SCRATCH_MNT)"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -73,7 +73,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 	for x in `seq 1 64`; do
 		$XFS_IO_PROG -f -c "pwrite -S 0x62 0 ${blksz}" "${TESTFILE}.${x}" >> $seqres.full 2>> $seqres.full
 	done
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -97,7 +97,7 @@ echo "+ modify files (2)"
 for x in `seq 1 64`; do
 	$XFS_IO_PROG -f -c "pwrite -S 0x62 ${blksz} ${blksz}" "${TESTFILE}.${x}" >> $seqres.full
 done
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ repair fs"
 _repair_scratch_fs >> $seqres.full 2>&1
@@ -114,7 +114,7 @@ for x in `seq 1 64`; do
 	test -s "${TESTFILE}.${x}" || broken=1
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/089 b/tests/xfs/089
index ff3ae719326eca..f5640a46177578 100755
--- a/tests/xfs/089
+++ b/tests/xfs/089
@@ -56,7 +56,7 @@ for x in `seq 2 64`; do
 done
 inode="$(stat -c '%i' "${TESTFILE}.1")"
 agcount="$(_xfs_mount_agcount $SCRATCH_MNT)"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -73,7 +73,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 	for x in `seq 1 64`; do
 		$XFS_IO_PROG -f -c "pwrite -S 0x62 0 ${blksz}" "${TESTFILE}.${x}" >> $seqres.full 2>> $seqres.full
 	done
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -98,7 +98,7 @@ echo "+ modify files (2)"
 for x in `seq 1 64`; do
 	$XFS_IO_PROG -f -c "pwrite -S 0x62 ${blksz} ${blksz}" "${TESTFILE}.${x}" >> $seqres.full
 done
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ repair fs"
 _repair_scratch_fs >> $seqres.full 2>&1
@@ -115,7 +115,7 @@ for x in `seq 1 64`; do
 	test -s "${TESTFILE}.${x}" || broken=1
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/091 b/tests/xfs/091
index 3f606f8845797d..c7857cdf1b690b 100755
--- a/tests/xfs/091
+++ b/tests/xfs/091
@@ -56,7 +56,7 @@ for x in `seq 2 64`; do
 done
 inode="$(stat -c '%i' "${TESTFILE}.1")"
 agcount="$(_xfs_mount_agcount $SCRATCH_MNT)"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -73,7 +73,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 	for x in `seq 1 64`; do
 		$XFS_IO_PROG -f -c "pwrite -S 0x62 0 ${blksz}" "${TESTFILE}.${x}" >> $seqres.full 2>> $seqres.full
 	done
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -98,7 +98,7 @@ echo "+ modify files (2)"
 for x in `seq 1 64`; do
 	$XFS_IO_PROG -f -c "pwrite -S 0x62 ${blksz} ${blksz}" "${TESTFILE}.${x}" >> $seqres.full
 done
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ repair fs"
 _repair_scratch_fs >> $seqres.full 2>&1
@@ -115,7 +115,7 @@ for x in `seq 1 64`; do
 	test -s "${TESTFILE}.${x}" || broken=1
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/093 b/tests/xfs/093
index c4e8006063e121..cfb2a8c80c1770 100755
--- a/tests/xfs/093
+++ b/tests/xfs/093
@@ -55,7 +55,7 @@ for x in `seq 2 64`; do
 done
 inode="$(stat -c '%i' "${TESTFILE}.1")"
 agcount="$(_xfs_mount_agcount $SCRATCH_MNT)"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -72,7 +72,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 	for x in `seq 65 70`; do
 		touch "${TESTFILE}.${x}" 2> /dev/null && broken=0
 	done
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 echo "broken: ${broken}"
 
@@ -94,7 +94,7 @@ for x in `seq 65 70`; do
 	touch "${TESTFILE}.${x}" || broken=1
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/097 b/tests/xfs/097
index 384c76080ddcf4..0fcf65a2a8f65a 100755
--- a/tests/xfs/097
+++ b/tests/xfs/097
@@ -58,7 +58,7 @@ for x in `seq 2 64`; do
 done
 inode="$(stat -c '%i' "${TESTFILE}.1")"
 agcount="$(_xfs_mount_agcount $SCRATCH_MNT)"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -74,7 +74,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 	for x in `seq 65 70`; do
 		touch "${TESTFILE}.${x}" 2> /dev/null && broken=0
 	done
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 echo "broken: ${broken}"
 
@@ -93,7 +93,7 @@ for x in `seq 65 70`; do
 	touch "${TESTFILE}.${x}" || broken=1
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/098 b/tests/xfs/098
index a47cda67e14e29..48eb3fa2b3a753 100755
--- a/tests/xfs/098
+++ b/tests/xfs/098
@@ -56,7 +56,7 @@ for x in `seq 2 64`; do
 	touch "${TESTFILE}.${x}"
 done
 inode="$(stat -c '%i' "${TESTFILE}.1")"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -98,7 +98,7 @@ for x in `seq 1 64`; do
 	test $? -ne 0 && broken=1
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/099 b/tests/xfs/099
index f5321fe3d20b1c..17e1e8df7bf751 100755
--- a/tests/xfs/099
+++ b/tests/xfs/099
@@ -44,7 +44,7 @@ node_lblk="$((64 * 1073741824 / blksz))"
 echo "+ make some files"
 __populate_create_dir "${SCRATCH_MNT}/blockdir" "${nr}"
 inode="$(stat -c '%i' "${SCRATCH_MNT}/blockdir")"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -60,7 +60,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 
 	rm -rf "${SCRATCH_MNT}/blockdir/00000000" 2> /dev/null && _fail "modified corrupt directory"
 	mkdir "${SCRATCH_MNT}/blockdir/xxxxxxxx" 2> /dev/null && _fail "add to corrupt directory"
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -77,7 +77,7 @@ echo "+ modify dir (2)"
 mkdir -p "${SCRATCH_MNT}/blockdir"
 rm -rf "${SCRATCH_MNT}/blockdir/00000000" || _fail "couldn't modify repaired directory"
 mkdir "${SCRATCH_MNT}/blockdir/xxxxxxxx" || _fail "add to repaired directory"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/100 b/tests/xfs/100
index 6f465a79c926d2..dd50d984800335 100755
--- a/tests/xfs/100
+++ b/tests/xfs/100
@@ -44,7 +44,7 @@ node_lblk="$((64 * 1073741824 / blksz))"
 echo "+ make some files"
 __populate_create_dir "${SCRATCH_MNT}/blockdir" "${nr}"
 inode="$(stat -c '%i' "${SCRATCH_MNT}/blockdir")"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -65,7 +65,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 
 	rm -rf "${SCRATCH_MNT}/blockdir/00000000" 2> /dev/null && _fail "modified corrupt directory"
 	mkdir "${SCRATCH_MNT}/blockdir/xxxxxxxx" 2> /dev/null && _fail "add to corrupt directory"
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -82,7 +82,7 @@ echo "+ modify dir (2)"
 mkdir -p "${SCRATCH_MNT}/blockdir"
 rm -rf "${SCRATCH_MNT}/blockdir/00000000" || _fail "couldn't modify repaired directory"
 mkdir "${SCRATCH_MNT}/blockdir/xxxxxxxx" || _fail "add to repaired directory"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/101 b/tests/xfs/101
index a926acb0bc6735..2abcd711b18703 100755
--- a/tests/xfs/101
+++ b/tests/xfs/101
@@ -44,7 +44,7 @@ node_lblk="$((64 * 1073741824 / blksz))"
 echo "+ make some files"
 __populate_create_dir "${SCRATCH_MNT}/blockdir" "${nr}"
 inode="$(stat -c '%i' "${SCRATCH_MNT}/blockdir")"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -60,7 +60,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 
 	rm -rf "${SCRATCH_MNT}/blockdir/00000000" 2> /dev/null && _fail "modified corrupt directory"
 	mkdir "${SCRATCH_MNT}/blockdir/xxxxxxxx" 2> /dev/null && _fail "add to corrupt directory"
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -77,7 +77,7 @@ echo "+ modify dir (2)"
 mkdir -p "${SCRATCH_MNT}/blockdir"
 rm -rf "${SCRATCH_MNT}/blockdir/00000000" || _fail "couldn't modify repaired directory"
 mkdir "${SCRATCH_MNT}/blockdir/xxxxxxxx" || _fail "add to repaired directory"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/102 b/tests/xfs/102
index c3ddec5e432dc5..5a7c036ce55751 100755
--- a/tests/xfs/102
+++ b/tests/xfs/102
@@ -44,7 +44,7 @@ node_lblk="$((64 * 1073741824 / blksz))"
 echo "+ make some files"
 __populate_create_dir "${SCRATCH_MNT}/blockdir" "${nr}" true
 inode="$(stat -c '%i' "${SCRATCH_MNT}/blockdir")"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -65,7 +65,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 
 	rm -rf "${SCRATCH_MNT}/blockdir/00000000" 2> /dev/null && _fail "modified corrupt directory"
 	mkdir "${SCRATCH_MNT}/blockdir/xxxxxxxx" 2> /dev/null && _fail "add to corrupt directory"
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -82,7 +82,7 @@ echo "+ modify dir (2)"
 mkdir -p "${SCRATCH_MNT}/blockdir"
 rm -rf "${SCRATCH_MNT}/blockdir/00000000" || _fail "couldn't modify repaired directory"
 mkdir "${SCRATCH_MNT}/blockdir/xxxxxxxx" || _fail "add to repaired directory"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/105 b/tests/xfs/105
index 132aa07f8300ef..30d4dc47ec1fed 100755
--- a/tests/xfs/105
+++ b/tests/xfs/105
@@ -44,7 +44,7 @@ node_lblk="$((64 * 1073741824 / blksz))"
 echo "+ make some files"
 __populate_create_dir "${SCRATCH_MNT}/blockdir" "${nr}" true
 inode="$(stat -c '%i' "${SCRATCH_MNT}/blockdir")"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -65,7 +65,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 
 	rm -rf "${SCRATCH_MNT}/blockdir/00000000" 2> /dev/null && _fail "modified corrupt directory"
 	mkdir "${SCRATCH_MNT}/blockdir/xxxxxxxx" 2> /dev/null && _fail "add to corrupt directory"
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -82,7 +82,7 @@ echo "+ modify dir (2)"
 mkdir -p "${SCRATCH_MNT}/blockdir"
 rm -rf "${SCRATCH_MNT}/blockdir/00000000" || _fail "couldn't modify repaired directory"
 mkdir "${SCRATCH_MNT}/blockdir/xxxxxxxx" || _fail "add to repaired directory"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/112 b/tests/xfs/112
index f0e717cf26d8c9..267432a863a92d 100755
--- a/tests/xfs/112
+++ b/tests/xfs/112
@@ -44,7 +44,7 @@ node_lblk="$((64 * 1073741824 / blksz))"
 echo "+ make some files"
 __populate_create_dir "${SCRATCH_MNT}/blockdir" "${nr}" true
 inode="$(stat -c '%i' "${SCRATCH_MNT}/blockdir")"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -65,14 +65,14 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 
 	rm -rf "${SCRATCH_MNT}/blockdir/00000000" 2> /dev/null && _fail "modified corrupt directory"
 	mkdir "${SCRATCH_MNT}/blockdir/xxxxxxxx" 2> /dev/null && _fail "add to corrupt directory"
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
 _repair_scratch_fs >> $seqres.full 2>&1
 if [ $? -eq 2 ]; then
 	_scratch_mount
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 	_repair_scratch_fs >> $seqres.full 2>&1
 fi
 
@@ -86,7 +86,7 @@ echo "+ modify dir (2)"
 mkdir -p "${SCRATCH_MNT}/blockdir"
 rm -rf "${SCRATCH_MNT}/blockdir/00000000" || _fail "couldn't modify repaired directory"
 mkdir "${SCRATCH_MNT}/blockdir/xxxxxxxx" || _fail "add to repaired directory"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/113 b/tests/xfs/113
index 22ac8c3fd51b80..2f19346aa74b3d 100755
--- a/tests/xfs/113
+++ b/tests/xfs/113
@@ -44,7 +44,7 @@ node_lblk="$((64 * 1073741824 / blksz))"
 echo "+ make some files"
 __populate_create_dir "${SCRATCH_MNT}/blockdir" "${nr}" true
 inode="$(stat -c '%i' "${SCRATCH_MNT}/blockdir")"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -86,7 +86,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 
 	rm -rf "${SCRATCH_MNT}/blockdir/00000000" 2> /dev/null && _fail "modified corrupt directory"
 	mkdir "${SCRATCH_MNT}/blockdir/xxxxxxxx" 2> /dev/null && _fail "add to corrupt directory"
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -103,7 +103,7 @@ echo "+ modify dir (2)"
 mkdir -p "${SCRATCH_MNT}/blockdir"
 rm -rf "${SCRATCH_MNT}/blockdir/00000000" || _fail "couldn't modify repaired directory"
 mkdir "${SCRATCH_MNT}/blockdir/xxxxxxxx" || _fail "add to repaired directory"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/117 b/tests/xfs/117
index 0ca8f1b96ddfd9..ae73ddbebfd53b 100755
--- a/tests/xfs/117
+++ b/tests/xfs/117
@@ -65,7 +65,7 @@ for ((i = 0; i < 64; i++)); do
 done
 echo "First victim inode is: " >> $seqres.full
 stat -c '%i' "$fname" >> $seqres.full
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -85,7 +85,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 		touch "$fname" &>> $seqres.full
 		test $? -eq 0 && broken=0
 	done
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 echo "broken: ${broken}"
 
@@ -110,7 +110,7 @@ for x in `seq 1 64`; do
 	echo "${x}: broken=${broken}" >> $seqres.full
 done
 echo "broken: ${broken}"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/120 b/tests/xfs/120
index f1f047f53a351b..9d0cc12a3e8b8d 100755
--- a/tests/xfs/120
+++ b/tests/xfs/120
@@ -45,7 +45,7 @@ for i in $(seq 1 2 ${nr}); do
 	$XFS_IO_PROG -f -c "fpunch $((i * blksz)) ${blksz}" "${SCRATCH_MNT}/bigfile" >> $seqres.full
 done
 inode="$(stat -c '%i' "${SCRATCH_MNT}/bigfile")"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -60,7 +60,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 	$XFS_IO_PROG -f -c "pwrite -S 0x62 ${blksz} ${blksz}" -c 'fsync' "${SCRATCH_MNT}/bigfile" >> $seqres.full 2> /dev/null
 	after="$(stat -c '%b' "${SCRATCH_MNT}/bigfile")"
 	test "${before}" -eq "${after}" || _fail "pwrite should fail on corrupt bmbt"
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -78,7 +78,7 @@ before="$(stat -c '%b' "${SCRATCH_MNT}/bigfile")"
 $XFS_IO_PROG -f -c "pwrite -S 0x62 ${blksz} ${blksz}" -c 'fsync' "${SCRATCH_MNT}/bigfile" >> $seqres.full 2> /dev/null
 after="$(stat -c '%b' "${SCRATCH_MNT}/bigfile")"
 test "${before}" -ne "${after}" || _fail "pwrite failed after fixing corrupt bmbt"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/123 b/tests/xfs/123
index 6b56551374cd8f..5bd3c86372058e 100755
--- a/tests/xfs/123
+++ b/tests/xfs/123
@@ -44,7 +44,7 @@ str="$(perl -e "print './' x $reps;")x"
 (cd $SCRATCH_MNT; ln -s "${str}" "long_symlink")
 cat "${SCRATCH_MNT}/long_symlink"
 inode="$(stat -c '%i' "${SCRATCH_MNT}/long_symlink")"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -55,7 +55,7 @@ _scratch_xfs_db -x -c "inode ${inode}" -c "dblock 0" -c "stack" -c "blocktrash -
 echo "+ mount image"
 if _try_scratch_mount >> $seqres.full 2>&1; then
 	cat "${SCRATCH_MNT}/long_symlink" 2>/dev/null && _fail "symlink should be broken"
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -64,7 +64,7 @@ _repair_scratch_fs >> $seqres.full 2>&1
 echo "+ mount image (2)"
 _scratch_mount
 cat "${SCRATCH_MNT}/long_symlink" 2>/dev/null && _fail "symlink should be broken"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/124 b/tests/xfs/124
index fe870dc96cc783..7890434b397262 100755
--- a/tests/xfs/124
+++ b/tests/xfs/124
@@ -46,7 +46,7 @@ seq 0 "${nr}" | while read d; do
 	setfattr -n "user.x$(printf "%.08d" "$d")" -v "0000000000000000" "${SCRATCH_MNT}/attrfile"
 done
 inode="$(stat -c '%i' "${SCRATCH_MNT}/attrfile")"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -64,7 +64,7 @@ echo "+ mount image && modify xattr"
 if _try_scratch_mount >> $seqres.full 2>&1; then
 
 	setfattr -x "user.x00000000" "${SCRATCH_MNT}/attrfile" 2> /dev/null && _fail "modified corrupt xattr"
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -80,7 +80,7 @@ $CHATTR_PROG -R -f -i "${SCRATCH_MNT}/"
 echo "+ modify xattr (2)"
 getfattr "${SCRATCH_MNT}/attrfile" -n "user.x00000000" > /dev/null 2>&1 && (setfattr -x "user.x00000000" "${SCRATCH_MNT}/attrfile" || _fail "remove corrupt xattr")
 setfattr -n "user.x00000000" -v 'x0x0x0x0' "${SCRATCH_MNT}/attrfile" || _fail "add corrupt xattr"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/125 b/tests/xfs/125
index 89e93650556e40..c3770c185b4063 100755
--- a/tests/xfs/125
+++ b/tests/xfs/125
@@ -47,7 +47,7 @@ seq 1 2 "${nr}" | while read d; do
 	setfattr -x "user.x$(printf "%.08d" "$d")" "${SCRATCH_MNT}/attrfile"
 done
 inode="$(stat -c '%i' "${SCRATCH_MNT}/attrfile")"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -64,7 +64,7 @@ echo "+ mount image && modify xattr"
 if _try_scratch_mount >> $seqres.full 2>&1; then
 
 	setfattr -x "user.x00000000" "${SCRATCH_MNT}/attrfile" 2> /dev/null && _fail "modified corrupt xattr"
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -80,7 +80,7 @@ $CHATTR_PROG -R -f -i "${SCRATCH_MNT}/"
 echo "+ modify xattr (2)"
 setfattr -n "user.x00000000" -v "1111111111111111" "${SCRATCH_MNT}/attrfile" || _fail "modified corrupt xattr"
 setfattr -x "user.x00000000" "${SCRATCH_MNT}/attrfile" || _fail "delete corrupt xattr"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/126 b/tests/xfs/126
index 5614ea398c0142..14eb2a6157e141 100755
--- a/tests/xfs/126
+++ b/tests/xfs/126
@@ -47,7 +47,7 @@ seq 1 2 "${nr}" | while read d; do
 	setfattr -x "user.x$(printf "%.08d" "$d")" "${SCRATCH_MNT}/attrfile"
 done
 inode="$(stat -c '%i' "${SCRATCH_MNT}/attrfile")"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
@@ -69,7 +69,7 @@ echo "+ mount image && modify xattr"
 if _try_scratch_mount >> $seqres.full 2>&1; then
 
 	setfattr -x "user.x00000000" "${SCRATCH_MNT}/attrfile" 2> /dev/null && _fail "modified corrupt xattr"
-	umount "${SCRATCH_MNT}"
+	_umount "${SCRATCH_MNT}"
 fi
 
 echo "+ repair fs"
@@ -84,7 +84,7 @@ $CHATTR_PROG -R -f -i "${SCRATCH_MNT}/"
 
 echo "+ modify xattr (2)"
 getfattr "${SCRATCH_MNT}/attrfile" -n "user.x00000000" 2> /dev/null && (setfattr -x "user.x00000000" "${SCRATCH_MNT}/attrfile" || _fail "modified corrupt xattr")
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || _fail "xfs_repair should not fail"
diff --git a/tests/xfs/130 b/tests/xfs/130
index 3e6dd861c47851..b1792a98e57db6 100755
--- a/tests/xfs/130
+++ b/tests/xfs/130
@@ -78,7 +78,7 @@ $CHATTR_PROG -R -f -i "${SCRATCH_MNT}/"
 echo "+ reflink more (2)"
 _cp_reflink "${SCRATCH_MNT}/file1" "${SCRATCH_MNT}/file5" || \
 	_fail "modified refcount tree"
-umount "${SCRATCH_MNT}"
+_umount "${SCRATCH_MNT}"
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> "$seqres.full" 2>&1 || \
diff --git a/tests/xfs/152 b/tests/xfs/152
index 7ba00c4bfac9ff..66577cfb4617fc 100755
--- a/tests/xfs/152
+++ b/tests/xfs/152
@@ -15,7 +15,7 @@ _begin_fstest auto quick quota idmapped
 
 wipe_mounts()
 {
-	umount "${SCRATCH_MNT}/idmapped" >/dev/null 2>&1
+	_umount "${SCRATCH_MNT}/idmapped" >/dev/null 2>&1
 	_scratch_unmount >/dev/null 2>&1
 }
 
diff --git a/tests/xfs/169 b/tests/xfs/169
index 6400fd9e6bdc8b..16c5385cf4815a 100755
--- a/tests/xfs/169
+++ b/tests/xfs/169
@@ -15,7 +15,7 @@ _begin_fstest auto clone
 _cleanup()
 {
     cd /
-    umount $SCRATCH_MNT > /dev/null 2>&1
+    _umount $SCRATCH_MNT > /dev/null 2>&1
     rm -rf $tmp.*
 }
 
@@ -43,7 +43,7 @@ for i in 1 2 x; do
 		_reflink_range  $testdir/file1 $((nr * blksz)) \
 				$testdir/file2 $((nr * blksz)) $blksz >> $seqres.full
 	done
-	umount $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 	_check_scratch_fs
 	_scratch_mount
 
@@ -51,7 +51,7 @@ for i in 1 2 x; do
 
 	echo "$i: Delete both files"
 	rm -rf $testdir/file1 $testdir/file2
-	umount $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 	_check_scratch_fs
 	_scratch_mount
 done
diff --git a/tests/xfs/206 b/tests/xfs/206
index bfd2dee939ddd7..16a734c3751194 100755
--- a/tests/xfs/206
+++ b/tests/xfs/206
@@ -18,7 +18,7 @@ _begin_fstest growfs auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    umount $tmpdir
+    _umount $tmpdir
     rmdir $tmpdir
     rm -f $tmp
     rm -f $tmpfile
diff --git a/tests/xfs/216 b/tests/xfs/216
index 680239b4ef788d..149c8fdfec887d 100755
--- a/tests/xfs/216
+++ b/tests/xfs/216
@@ -52,7 +52,7 @@ _do_mkfs()
 			-d name=$LOOP_DEV,size=${i}g $loop_mkfs_opts |grep log
 		mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
-		umount $LOOP_MNT > /dev/null 2>&1
+		_umount $LOOP_MNT > /dev/null 2>&1
 	done
 }
 # make large holey file
diff --git a/tests/xfs/217 b/tests/xfs/217
index 41caaf738267d4..30a186d7294940 100755
--- a/tests/xfs/217
+++ b/tests/xfs/217
@@ -31,7 +31,7 @@ _do_mkfs()
 			-d name=$LOOP_DEV,size=${i}g |grep log
 		mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
-		umount $LOOP_MNT > /dev/null 2>&1
+		_umount $LOOP_MNT > /dev/null 2>&1
 
 		# punch out the previous blocks so that we keep the amount of
 		# disk space the test requires down to a minimum.
diff --git a/tests/xfs/235 b/tests/xfs/235
index 5b201d93076952..0184ff71f2878c 100755
--- a/tests/xfs/235
+++ b/tests/xfs/235
@@ -31,7 +31,7 @@ _pwrite_byte 0x62 0 $((blksz * 64)) ${SCRATCH_MNT}/file0 >> $seqres.full
 _pwrite_byte 0x61 0 $((blksz * 64)) ${SCRATCH_MNT}/file1 >> $seqres.full
 cp -p ${SCRATCH_MNT}/file0 ${SCRATCH_MNT}/file2
 cp -p ${SCRATCH_MNT}/file1 ${SCRATCH_MNT}/file3
-umount ${SCRATCH_MNT}
+_umount ${SCRATCH_MNT}
 
 echo "+ check fs"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || \
@@ -49,7 +49,7 @@ if _try_scratch_mount >> $seqres.full 2>&1; then
 
 	$XFS_IO_PROG -f -c "pwrite -S 0x63 0 $((blksz * 64))" -c "fsync" ${SCRATCH_MNT}/file4 >> $seqres.full 2>&1
 	test -s ${SCRATCH_MNT}/file4 && _fail "should not be able to copy with busted rmap btree"
-	umount ${SCRATCH_MNT}
+	_umount ${SCRATCH_MNT}
 fi
 
 echo "+ repair fs"
@@ -66,7 +66,7 @@ $CHATTR_PROG -R -f -i ${SCRATCH_MNT}/
 echo "+ copy more (2)"
 cp -p ${SCRATCH_MNT}/file1 ${SCRATCH_MNT}/file5 || \
 	_fail "modified rmap tree"
-umount ${SCRATCH_MNT}
+_umount ${SCRATCH_MNT}
 
 echo "+ check fs (2)"
 _scratch_xfs_repair -n >> $seqres.full 2>&1 || \
diff --git a/tests/xfs/236 b/tests/xfs/236
index a374a300d1905a..277a9a402e2e05 100755
--- a/tests/xfs/236
+++ b/tests/xfs/236
@@ -15,7 +15,7 @@ _begin_fstest auto rmap punch
 _cleanup()
 {
     cd /
-    umount $SCRATCH_MNT > /dev/null 2>&1
+    _umount $SCRATCH_MNT > /dev/null 2>&1
     rm -rf $tmp.*
 }
 
@@ -44,7 +44,7 @@ for i in 1 2 x; do
 	seq 1 2 $((nr_blks - 1)) | while read nr; do
 		$XFS_IO_PROG -c "fpunch $((nr * blksz)) $blksz" $testdir/file2 >> $seqres.full
 	done
-	umount $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 	_check_scratch_fs
 	_scratch_mount
 
@@ -52,7 +52,7 @@ for i in 1 2 x; do
 
 	echo "$i: Delete both files"
 	rm -rf $testdir/file1 $testdir/file2
-	umount $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 	_check_scratch_fs
 	_scratch_mount
 done
diff --git a/tests/xfs/239 b/tests/xfs/239
index bfe722c0add020..7dc9be7d2edfe0 100755
--- a/tests/xfs/239
+++ b/tests/xfs/239
@@ -66,7 +66,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "Check for damage"
-umount $SCRATCH_MNT
+_umount $SCRATCH_MNT
 _repair_scratch_fs >> $seqres.full
 
 # success, all done
diff --git a/tests/xfs/241 b/tests/xfs/241
index 1532493979ffa7..a779e321417520 100755
--- a/tests/xfs/241
+++ b/tests/xfs/241
@@ -66,7 +66,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "Check for damage"
-umount $SCRATCH_MNT
+_umount $SCRATCH_MNT
 _repair_scratch_fs >> $seqres.full
 
 # success, all done
diff --git a/tests/xfs/250 b/tests/xfs/250
index f8846be6e197aa..82ab08d65192e7 100755
--- a/tests/xfs/250
+++ b/tests/xfs/250
@@ -13,7 +13,7 @@ _begin_fstest auto quick rw prealloc metadata
 _cleanup()
 {
 	cd /
-	umount $LOOP_MNT 2>/dev/null
+	_umount $LOOP_MNT 2>/dev/null
 	rm -f $LOOP_DEV
 	rmdir $LOOP_MNT
 }
@@ -60,7 +60,7 @@ _test_loop()
 	$XFS_IO_PROG -f -c "resvsp 0 $fsize" $LOOP_MNT/foo | _filter_io
 
 	echo "*** unmount loop filesystem"
-	umount $LOOP_MNT > /dev/null 2>&1
+	_umount $LOOP_MNT > /dev/null 2>&1
 
 	echo "*** check loop filesystem"
 	 _check_xfs_filesystem $LOOP_DEV none none
diff --git a/tests/xfs/265 b/tests/xfs/265
index 21de4c054a573f..2ba7342d066bb6 100755
--- a/tests/xfs/265
+++ b/tests/xfs/265
@@ -16,7 +16,7 @@ _begin_fstest auto clone
 _cleanup()
 {
     cd /
-    umount $SCRATCH_MNT > /dev/null 2>&1
+    _umount $SCRATCH_MNT > /dev/null 2>&1
     rm -rf $tmp.*
 }
 
@@ -51,7 +51,7 @@ for i in 1 2 x; do
 		truncate -s $((blksz * (nr_blks - nr))) $testdir/file1.$nr >> $seqres.full
 	done
 
-	umount $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 	_check_scratch_fs
 	_scratch_mount
 
@@ -60,7 +60,7 @@ for i in 1 2 x; do
 	echo "$i: Delete both files"
 	rm -rf $testdir
 	mkdir -p $testdir
-	umount $SCRATCH_MNT
+	_umount $SCRATCH_MNT
 	_check_scratch_fs
 	_scratch_mount
 done
diff --git a/tests/xfs/310 b/tests/xfs/310
index 34d17be97f36dd..f2a7ca50f67199 100755
--- a/tests/xfs/310
+++ b/tests/xfs/310
@@ -13,7 +13,7 @@ _begin_fstest auto clone rmap prealloc
 _cleanup()
 {
 	cd /
-	umount $SCRATCH_MNT > /dev/null 2>&1
+	_umount $SCRATCH_MNT > /dev/null 2>&1
 	_dmhugedisk_cleanup
 	rm -rf $tmp.*
 }
@@ -53,7 +53,7 @@ $XFS_IO_PROG -f -c "falloc 0 $((nr_blks * blksz))" $testdir/file1 >> $seqres.ful
 echo "Check extent count"
 xfs_bmap -l -p -v $testdir/file1 | grep '^[[:space:]]*2:' -q && xfs_bmap -l -p -v $testdir/file1
 inum=$(stat -c '%i' $testdir/file1)
-umount $SCRATCH_MNT
+_umount $SCRATCH_MNT
 
 echo "Check bmap count"
 nr_bmaps=$(xfs_db -c "inode $inum" -c "bmap" $DMHUGEDISK_DEV | grep 'data offset' | wc -l)
diff --git a/tests/xfs/716 b/tests/xfs/716
index cd4fffef298d31..55c66d1cf8bb19 100755
--- a/tests/xfs/716
+++ b/tests/xfs/716
@@ -49,7 +49,7 @@ ino=$(stat -c '%i' $file)
 
 # Figure out how many extents we need to have to create a data fork that's in
 # btree format.
-umount $SCRATCH_MNT
+_umount $SCRATCH_MNT
 di_forkoff=$(_scratch_xfs_db -c "inode $ino" -c "p core.forkoff" | \
 	awk '{print $3}')
 _scratch_xfs_db -c "inode $ino" -c "p" >> $seqres.full
@@ -61,7 +61,7 @@ $XFS_IO_PROG -c "falloc 0 $(( (min_ext_for_btree + 1) * 2 * blksz))" $file
 $here/src/punch-alternating $file
 
 # Make sure the data fork is in btree format.
-umount $SCRATCH_MNT
+_umount $SCRATCH_MNT
 _scratch_xfs_db -c "inode $ino" -c "p core.format" | grep -q "btree" || \
 	echo "data fork not in btree format?"
 echo "about to start test" >> $seqres.full


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 3/6] xfs: test health monitoring code
  2024-12-31 23:35 ` [PATCHSET 4/5] fstests: live health monitoring of filesystems Darrick J. Wong
  2024-12-31 23:57   ` [PATCH 1/6] misc: convert all $UMOUNT_PROG to a _umount helper Darrick J. Wong
  2024-12-31 23:57   ` [PATCH 2/6] misc: convert all umount(1) invocations to _umount Darrick J. Wong
@ 2024-12-31 23:57   ` Darrick J. Wong
  2024-12-31 23:57   ` [PATCH 4/6] xfs: test for metadata corruption error reporting via healthmon Darrick J. Wong
                     ` (2 subsequent siblings)
  5 siblings, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:57 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

From: Darrick J. Wong <djwong@kernel.org>

Add some functionality tests for the new health monitoring code.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 doc/group-names.txt |    1 +
 tests/xfs/1885      |   53 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1885.out  |    5 +++++
 3 files changed, 59 insertions(+)
 create mode 100755 tests/xfs/1885
 create mode 100644 tests/xfs/1885.out


diff --git a/doc/group-names.txt b/doc/group-names.txt
index b04d0180e8ec02..8fbb260d8c7bb5 100644
--- a/doc/group-names.txt
+++ b/doc/group-names.txt
@@ -117,6 +117,7 @@ samefs			overlayfs when all layers are on the same fs
 scrub			filesystem metadata scrubbers
 seed			btrfs seeded filesystems
 seek			llseek functionality
+selfhealing		self healing filesystem code
 selftest		tests with fixed results, used to validate testing setup
 send			btrfs send/receive
 shrinkfs		decreasing the size of a filesystem
diff --git a/tests/xfs/1885 b/tests/xfs/1885
new file mode 100755
index 00000000000000..1b87af3a9178fc
--- /dev/null
+++ b/tests/xfs/1885
@@ -0,0 +1,53 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024-2025 Oracle.  All Rights Reserved.
+#
+# FS QA Test 1885
+#
+# Make sure that healthmon handles module refcount correctly.
+#
+. ./common/preamble
+_begin_fstest auto selfhealing
+
+. ./common/filter
+. ./common/module
+
+refcount_file="/sys/module/xfs/refcnt"
+test -e "$refcount_file" || _notrun "cannot find xfs module refcount"
+
+_require_test
+_require_xfs_io_command healthmon
+
+# Capture mod refcount without the test fs mounted
+_test_unmount
+init_refcount="$(cat "$refcount_file")"
+
+# Capture mod refcount with the test fs mounted
+_test_mount
+nomon_mount_refcount="$(cat "$refcount_file")"
+
+# Capture mod refcount with test fs mounted and the healthmon fd open.
+# Pause the xfs_io process so that it doesn't actually respond to events.
+$XFS_IO_PROG -c 'healthmon -c -v' $TEST_DIR &
+sleep 0.5
+kill -STOP %1
+mon_mount_refcount="$(cat "$refcount_file")"
+
+# Capture mod refcount with only the healthmon fd open.
+_test_unmount
+mon_nomount_refcount="$(cat "$refcount_file")"
+
+# Capture mod refcount after continuing healthmon (which should exit due to the
+# unmount) and killing it.
+kill -CONT %1
+kill %1
+wait
+nomon_nomount_refcount="$(cat "$refcount_file")"
+
+_within_tolerance "mount refcount" "$nomon_mount_refcount" "$((init_refcount + 1))" 0 -v
+_within_tolerance "mount + healthmon refcount" "$mon_mount_refcount" "$((init_refcount + 2))" 0 -v
+_within_tolerance "healthmon refcount" "$mon_nomount_refcount" "$((init_refcount + 1))" 0 -v
+_within_tolerance "end refcount" "$nomon_nomount_refcount" "$init_refcount" 0 -v
+
+status=0
+exit
diff --git a/tests/xfs/1885.out b/tests/xfs/1885.out
new file mode 100644
index 00000000000000..f152cef0525609
--- /dev/null
+++ b/tests/xfs/1885.out
@@ -0,0 +1,5 @@
+QA output created by 1885
+mount refcount is in range
+mount + healthmon refcount is in range
+healthmon refcount is in range
+end refcount is in range


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 4/6] xfs: test for metadata corruption error reporting via healthmon
  2024-12-31 23:35 ` [PATCHSET 4/5] fstests: live health monitoring of filesystems Darrick J. Wong
                     ` (2 preceding siblings ...)
  2024-12-31 23:57   ` [PATCH 3/6] xfs: test health monitoring code Darrick J. Wong
@ 2024-12-31 23:57   ` Darrick J. Wong
  2024-12-31 23:58   ` [PATCH 5/6] xfs: test io " Darrick J. Wong
  2024-12-31 23:58   ` [PATCH 6/6] xfs: test new xfs_scrubbed daemon Darrick J. Wong
  5 siblings, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:57 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

From: Darrick J. Wong <djwong@kernel.org>

Check if we can detect runtime metadata corruptions via the health
monitor.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 common/rc          |   10 ++++++
 tests/xfs/1879     |   89 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1879.out |   12 +++++++
 3 files changed, 111 insertions(+)
 create mode 100755 tests/xfs/1879
 create mode 100644 tests/xfs/1879.out


diff --git a/common/rc b/common/rc
index 0d5c785cecc017..dd6857461e14dd 100644
--- a/common/rc
+++ b/common/rc
@@ -2850,6 +2850,16 @@ _require_xfs_io_command()
 		echo $testio | grep -q "Inappropriate ioctl" && \
 			_notrun "xfs_io $command support is missing"
 		;;
+	"healthmon")
+		testio=`$XFS_IO_PROG -c "$command -p $param" $TEST_DIR 2>&1`
+		echo $testio | grep -q "bad argument count" && \
+			_notrun "xfs_io $command $param support is missing"
+		echo $testio | grep -q "Inappropriate ioctl" && \
+			_notrun "xfs_io $command $param ioctl support is missing"
+		echo $testio | grep -q "Operation not supported" && \
+			_notrun "xfs_io $command $param kernel support is missing"
+		param_checked="$param"
+		;;
 	"label")
 		testio=`$XFS_IO_PROG -c "label" $TEST_DIR 2>&1`
 		;;
diff --git a/tests/xfs/1879 b/tests/xfs/1879
new file mode 100755
index 00000000000000..aab7bf9fa1f6e4
--- /dev/null
+++ b/tests/xfs/1879
@@ -0,0 +1,89 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024-2025 Oracle.  All Rights Reserved.
+#
+# FS QA Test No. 1879
+#
+# Corrupt some metadata and try to access it with the health monitoring program
+# running.  Check that healthmon observes a metadata error.
+#
+. ./common/preamble
+_begin_fstest auto quick eio
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.* $testdir
+}
+
+. ./common/filter
+
+_require_scratch_nocheck
+_require_xfs_io_command healthmon
+
+# Disable the scratch rt device to avoid test failures relating to the rt
+# bitmap consuming all the free space in our small data device.
+unset SCRATCH_RTDEV
+
+echo "Format and mount"
+_scratch_mkfs -d agcount=1 | _filter_mkfs 2> $tmp.mkfs >> $seqres.full
+. $tmp.mkfs
+_scratch_mount
+mkdir $SCRATCH_MNT/a/
+# Enough entries to get to a single block directory
+for ((i = 0; i < ( (isize + 255) / 256); i++)); do
+	path="$(printf "%s/a/%0255d" "$SCRATCH_MNT" "$i")"
+	touch "$path"
+done
+inum="$(stat -c %i "$SCRATCH_MNT/a")"
+_scratch_unmount
+
+# Fuzz the directory block so that the touch below will be guaranteed to trip
+# a runtime sickness report in exactly the manner we desire.
+_scratch_xfs_db -x -c "inode $inum" -c "dblock 0" -c 'fuzz bhdr.hdr.owner add' -c print &>> $seqres.full
+
+# Try to allocate space to trigger a metadata corruption event
+echo "Runtime corruption detection"
+_scratch_mount
+$XFS_IO_PROG -c 'healthmon -c -v' $SCRATCH_MNT > $tmp.healthmon &
+sleep 1	# wait for python program to start up
+touch $SCRATCH_MNT/a/farts &>> $seqres.full
+_scratch_unmount
+
+wait	# for healthmon to finish
+
+# Did we get errors?
+filter_healthmon()
+{
+	cat $tmp.healthmon >> $seqres.full
+	grep -A2 -E '(sick|corrupt)' $tmp.healthmon | grep -v -- '--' | sort | uniq
+}
+filter_healthmon
+
+# Run scrub to trigger a health event from there too.
+echo "Scrub corruption detection"
+_scratch_mount
+if _supports_xfs_scrub $SCRATCH_MNT $SCRATCH_DEV; then
+	$XFS_IO_PROG -c 'healthmon -c -v' $SCRATCH_MNT > $tmp.healthmon &
+	sleep 1	# wait for python program to start up
+	$XFS_SCRUB_PROG -n $SCRATCH_MNT &>> $seqres.full
+	_scratch_unmount
+
+	wait	# for healthmon to finish
+
+	# Did we get errors?
+	filter_healthmon
+else
+	# mock the output since we don't support scrub
+	_scratch_unmount
+	cat << ENDL
+  "domain":     "inode",
+  "structures":  ["directory"],
+  "structures":  ["parent"],
+  "type":       "corrupt",
+  "type":       "sick",
+ENDL
+fi
+
+status=0
+exit
diff --git a/tests/xfs/1879.out b/tests/xfs/1879.out
new file mode 100644
index 00000000000000..f02eefbf58ad6c
--- /dev/null
+++ b/tests/xfs/1879.out
@@ -0,0 +1,12 @@
+QA output created by 1879
+Format and mount
+Runtime corruption detection
+  "domain":     "inode",
+  "structures":  ["directory"],
+  "type":       "sick",
+Scrub corruption detection
+  "domain":     "inode",
+  "structures":  ["directory"],
+  "structures":  ["parent"],
+  "type":       "corrupt",
+  "type":       "sick",


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 5/6] xfs: test io error reporting via healthmon
  2024-12-31 23:35 ` [PATCHSET 4/5] fstests: live health monitoring of filesystems Darrick J. Wong
                     ` (3 preceding siblings ...)
  2024-12-31 23:57   ` [PATCH 4/6] xfs: test for metadata corruption error reporting via healthmon Darrick J. Wong
@ 2024-12-31 23:58   ` Darrick J. Wong
  2024-12-31 23:58   ` [PATCH 6/6] xfs: test new xfs_scrubbed daemon Darrick J. Wong
  5 siblings, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:58 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

From: Darrick J. Wong <djwong@kernel.org>

Create a new test to make sure the kernel can report IO errors via
health monitoring.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 tests/xfs/1878     |   80 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1878.out |   10 +++++++
 2 files changed, 90 insertions(+)
 create mode 100755 tests/xfs/1878
 create mode 100644 tests/xfs/1878.out


diff --git a/tests/xfs/1878 b/tests/xfs/1878
new file mode 100755
index 00000000000000..882d0dcca03cb1
--- /dev/null
+++ b/tests/xfs/1878
@@ -0,0 +1,80 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024-2025 Oracle.  All Rights Reserved.
+#
+# FS QA Test No. 1878
+#
+# Attempt to read and write a file in buffered and directio mode with the
+# health monitoring program running.  Check that healthmon observes all four
+# types of IO errors.
+#
+. ./common/preamble
+_begin_fstest auto quick eio
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.* $testdir
+	_dmerror_cleanup
+}
+
+. ./common/filter
+. ./common/dmerror
+
+_require_scratch_nocheck
+_require_xfs_io_command healthmon
+_require_dm_target error
+
+# Disable the scratch rt device to avoid test failures relating to the rt
+# bitmap consuming all the free space in our small data device.
+unset SCRATCH_RTDEV
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_dmerror_init no_log
+_dmerror_mount
+
+_require_fs_space $SCRATCH_MNT 65536
+
+# Create a file with written regions far enough apart that the pagecache can't
+# possibly be caching the regions with a single folio.
+testfile=$SCRATCH_MNT/fsync-err-test
+$XFS_IO_PROG -f \
+	-c 'pwrite -b 1m 0 1m' \
+	-c 'pwrite -b 1m 10g 1m' \
+	-c 'pwrite -b 1m 20g 1m' \
+	-c fsync $testfile >> $seqres.full
+
+# First we check if directio errors get reported
+$XFS_IO_PROG -c 'healthmon -c -v' $SCRATCH_MNT >> $tmp.healthmon &
+sleep 1	# wait for python program to start up
+_dmerror_load_error_table
+$XFS_IO_PROG -d -c 'pwrite -b 256k 12k 16k' $testfile >> $seqres.full
+$XFS_IO_PROG -d -c 'pread -b 256k 10g 16k' $testfile >> $seqres.full
+_dmerror_load_working_table
+
+_dmerror_unmount
+wait	# for healthmon to finish
+_dmerror_mount
+
+# Next we check if buffered io errors get reported.  We have to write something
+# before loading the error table to ensure the dquots get loaded.
+$XFS_IO_PROG -c 'pwrite -b 256k 20g 1k' -c fsync $testfile >> $seqres.full
+$XFS_IO_PROG -c 'healthmon -c -v' $SCRATCH_MNT >> $tmp.healthmon &
+sleep 1	# wait for python program to start up
+_dmerror_load_error_table
+$XFS_IO_PROG -c 'pread -b 256k 12k 16k' $testfile >> $seqres.full
+$XFS_IO_PROG -c 'pwrite -b 256k 20g 16k' -c fsync $testfile >> $seqres.full
+_dmerror_load_working_table
+
+_dmerror_unmount
+wait	# for healthmon to finish
+
+# Did we get errors?
+cat $tmp.healthmon >> $seqres.full
+grep -E '(diowrite|dioread|readahead|writeback)' $tmp.healthmon | sort | uniq
+
+_dmerror_cleanup
+
+status=0
+exit
diff --git a/tests/xfs/1878.out b/tests/xfs/1878.out
new file mode 100644
index 00000000000000..a8070c3c1afd23
--- /dev/null
+++ b/tests/xfs/1878.out
@@ -0,0 +1,10 @@
+QA output created by 1878
+Format and mount
+pwrite: Input/output error
+pread: Input/output error
+pread: Input/output error
+fsync: Input/output error
+  "type":       "dioread",
+  "type":       "diowrite",
+  "type":       "readahead",
+  "type":       "writeback",


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 6/6] xfs: test new xfs_scrubbed daemon
  2024-12-31 23:35 ` [PATCHSET 4/5] fstests: live health monitoring of filesystems Darrick J. Wong
                     ` (4 preceding siblings ...)
  2024-12-31 23:58   ` [PATCH 5/6] xfs: test io " Darrick J. Wong
@ 2024-12-31 23:58   ` Darrick J. Wong
  5 siblings, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:58 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

From: Darrick J. Wong <djwong@kernel.org>

Make sure the daemon in charge of self healing xfs actually does what it
says it does.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 common/config      |    6 ++++
 common/systemd     |    9 +++++
 common/xfs         |   16 ++++++++++
 tests/xfs/1882     |   64 ++++++++++++++++++++++++++++++++++++++
 tests/xfs/1882.out |    2 +
 tests/xfs/1883     |   75 +++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1883.out |    2 +
 tests/xfs/1884     |   87 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1884.out |    2 +
 9 files changed, 263 insertions(+)
 create mode 100755 tests/xfs/1882
 create mode 100644 tests/xfs/1882.out
 create mode 100755 tests/xfs/1883
 create mode 100644 tests/xfs/1883.out
 create mode 100755 tests/xfs/1884
 create mode 100644 tests/xfs/1884.out


diff --git a/common/config b/common/config
index fcff0660b05a97..2b3f946f3d308d 100644
--- a/common/config
+++ b/common/config
@@ -166,6 +166,12 @@ export XFS_ADMIN_PROG="$(type -P xfs_admin)"
 export XFS_GROWFS_PROG=$(type -P xfs_growfs)
 export XFS_SPACEMAN_PROG="$(type -P xfs_spaceman)"
 export XFS_SCRUB_PROG="$(type -P xfs_scrub)"
+XFS_SCRUBBED_PROG="$(type -P xfs_scrubbed)"
+# Normally the scrubbed daemon is installed in libexec
+if [ -n "$XFS_SCRUBBED_PROG" ] && [ -e /usr/libexec/xfs_scrubbed ]; then
+	XFS_SCRUBBED_PROG=/usr/libexec/xfs_scrubbed
+fi
+export XFS_SCRUBBED_PROG
 export XFS_PARALLEL_REPAIR_PROG="$(type -P xfs_prepair)"
 export XFS_PARALLEL_REPAIR64_PROG="$(type -P xfs_prepair64)"
 export __XFSDUMP_PROG="$(type -P xfsdump)"
diff --git a/common/systemd b/common/systemd
index b2e24f267b2d93..8366d4cba39d85 100644
--- a/common/systemd
+++ b/common/systemd
@@ -71,3 +71,12 @@ _systemd_unit_status() {
 	_systemd_installed || return 1
 	systemctl status "$1"
 }
+
+# Start a running systemd unit
+_systemd_unit_start() {
+	systemctl start "$1"
+}
+# Stop a running systemd unit
+_systemd_unit_stop() {
+	systemctl stop "$1"
+}
diff --git a/common/xfs b/common/xfs
index b9e897e0e8839a..b4f69403e7396e 100644
--- a/common/xfs
+++ b/common/xfs
@@ -2224,3 +2224,19 @@ _scratch_find_rt_metadir_entry() {
 
 	return 1
 }
+
+# Run the xfs_scrubbed self healing daemon
+_scratch_xfs_scrubbed() {
+	local scrubbed_args=()
+	local daemon_dir
+	daemon_dir=$(dirname "$XFS_SCRUBBED_PROG")
+
+	# If we're being run from a development branch, we might need to find
+	# the schema file on our own.
+	local maybe_schema="$daemon_dir/../libxfs/xfs_healthmon.schema.json"
+	if [ -f "$maybe_schema" ]; then
+		scrubbed_args+=(--event-schema "$maybe_schema")
+	fi
+
+	$XFS_SCRUBBED_PROG "${scrubbed_args[@]}" "$@" $SCRATCH_MNT
+}
diff --git a/tests/xfs/1882 b/tests/xfs/1882
new file mode 100755
index 00000000000000..b6a8bd545dbcf5
--- /dev/null
+++ b/tests/xfs/1882
@@ -0,0 +1,64 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024-2025 Oracle.  All Rights Reserved.
+#
+# FS QA Test 1882
+#
+# Make sure that xfs_scrubbed correctly handles all the reports that it gets
+# from the kernel.  We simulate this by using the --everything mode so we get
+# all the events, not just the sickness reports.
+#
+. ./common/preamble
+_begin_fstest auto selfhealing
+
+. ./common/filter
+. ./common/fuzzy
+. ./common/systemd
+. ./common/populate
+
+_require_scrub
+_require_xfs_io_command "scrub"		# online check support
+_require_command "$XFS_SCRUBBED_PROG" "xfs_scrubbed"
+_require_scratch
+
+# Does this fs support health monitoring?
+_scratch_mkfs >> $seqres.full
+_scratch_mount
+
+_scratch_xfs_scrubbed --check || \
+	_notrun "health monitoring not supported on this kernel"
+_scratch_xfs_scrubbed --require-validation --check && \
+	_notrun "skipping this test in favor of the one that does json validation"
+_scratch_unmount
+
+# Create a sample fs with all the goodies
+_scratch_populate_cached nofill &>> $seqres.full
+_scratch_mount
+
+# If the system xfsprogs has self healing enabled, we need to shut down the
+# daemon before we try to capture things.
+if _systemd_is_running; then
+	scratch_path=$(systemd-escape --path "$SCRATCH_MNT")
+	_systemd_unit_stop "xfs_scrubbed@${scratch_path}" &>> $seqres.full
+fi
+
+# Start the health monitor, have it log everything
+_scratch_xfs_scrubbed --everything --log > $tmp.scrubbed &
+scrubbed_pid=$!
+sleep 1
+
+# Run scrub to make some noise
+_scratch_scrub -b -n >> $seqres.full
+
+# Unmount fs to kill scrubbed, then wait for it to finish
+while ! _scratch_unmount &>/dev/null; do
+	sleep 0.5
+done
+kill $scrubbed_pid
+wait
+
+cat $tmp.scrubbed >> $seqres.full
+
+echo Silence is golden
+status=0
+exit
diff --git a/tests/xfs/1882.out b/tests/xfs/1882.out
new file mode 100644
index 00000000000000..9b31ccb735cabd
--- /dev/null
+++ b/tests/xfs/1882.out
@@ -0,0 +1,2 @@
+QA output created by 1882
+Silence is golden
diff --git a/tests/xfs/1883 b/tests/xfs/1883
new file mode 100755
index 00000000000000..9bba989386b37e
--- /dev/null
+++ b/tests/xfs/1883
@@ -0,0 +1,75 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024-2025 Oracle.  All Rights Reserved.
+#
+# FS QA Test 1883
+#
+# Make sure that xfs_scrubbed correctly validates the json events that it gets
+# from the kernel.  We simulate this by using the --everything mode so we get
+# all the events, not just the sickness reports.
+#
+. ./common/preamble
+_begin_fstest auto selfhealing
+
+. ./common/filter
+. ./common/fuzzy
+. ./common/systemd
+. ./common/populate
+
+_require_scrub
+_require_xfs_io_command "scrub"		# online check support
+_require_command "$XFS_SCRUBBED_PROG" "xfs_scrubbed"
+_require_scratch
+
+# Does this fs support health monitoring?
+_scratch_mkfs >> $seqres.full
+_scratch_mount
+
+_scratch_xfs_scrubbed --require-validation --check || \
+	_notrun "health monitoring with validation not supported on this kernel"
+_scratch_unmount
+
+# Create a sample fs with all the goodies
+_scratch_populate_cached nofill &>> $seqres.full
+_scratch_mount
+
+# If the system xfsprogs has self healing enabled, we need to shut down the
+# daemon before we try to capture things.
+if _systemd_is_running; then
+	scratch_path=$(systemd-escape --path "$SCRATCH_MNT")
+	_systemd_unit_stop "xfs_scrubbed@${scratch_path}" &>> $seqres.full
+fi
+
+# Start the health monitor, have it validate everything
+_scratch_xfs_scrubbed --require-validation --everything --debug-fast --log &> $tmp.scrubbed &
+scrubbed_pid=$!
+sleep 1
+
+# Run scrub to make some noise
+_scratch_scrub -b -n >> $seqres.full
+
+# Wait for up to 60 seconds for the log file to stop growing
+old_logsz=
+new_logsz=$(stat -c '%s' $tmp.scrubbed)
+for ((i = 0; i < 60; i++)); do
+	test "$old_logsz" = "$new_logsz" && break
+	old_logsz="$new_logsz"
+	sleep 1
+	new_logsz=$(stat -c '%s' $tmp.scrubbed)
+done
+
+# Unmount fs to kill scrubbed, then wait for it to finish
+while ! _scratch_unmount &>/dev/null; do
+	sleep 0.5
+done
+kill $scrubbed_pid
+wait
+
+# Look for schema validation errors
+grep -q 'not valid under any of the given schemas' $tmp.scrubbed && \
+	echo "Should not have found schema validation errors"
+cat $tmp.scrubbed >> $seqres.full
+
+echo Silence is golden
+status=0
+exit
diff --git a/tests/xfs/1883.out b/tests/xfs/1883.out
new file mode 100644
index 00000000000000..bc9c390c778b6e
--- /dev/null
+++ b/tests/xfs/1883.out
@@ -0,0 +1,2 @@
+QA output created by 1883
+Silence is golden
diff --git a/tests/xfs/1884 b/tests/xfs/1884
new file mode 100755
index 00000000000000..fc6e0a48372fda
--- /dev/null
+++ b/tests/xfs/1884
@@ -0,0 +1,87 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024-2025 Oracle.  All Rights Reserved.
+#
+# FS QA Test 1884
+#
+# Ensure that autonomous self healing works fixes the filesystem correctly.
+#
+. ./common/preamble
+_begin_fstest auto selfhealing
+
+. ./common/filter
+. ./common/fuzzy
+. ./common/systemd
+
+_require_scrub
+_require_xfs_io_command "repair"	# online repair support
+_require_xfs_db_command "blocktrash"
+_require_command "$XFS_SCRUBBED_PROG" "xfs_scrubbed"
+_require_scratch
+
+_scratch_mkfs >> $seqres.full
+_scratch_mount
+
+_xfs_has_feature $SCRATCH_MNT parent || \
+	_notrun "parent pointers required to test directory auto-repair"
+_scratch_xfs_scrubbed --repair --check || \
+	_notrun "health monitoring with repair not supported on this kernel"
+
+# Create a largeish directory
+dblksz=$(_xfs_get_dir_blocksize "$SCRATCH_MNT")
+echo testdata > $SCRATCH_MNT/a
+mkdir -p "$SCRATCH_MNT/some/victimdir"
+for ((i = 0; i < (dblksz / 255); i++)); do
+	fname="$(printf "%0255d" "$i")"
+	ln $SCRATCH_MNT/a $SCRATCH_MNT/some/victimdir/$fname
+done
+
+# Did we get at least two dir blocks?
+dirsize=$(stat -c '%s' $SCRATCH_MNT/some/victimdir)
+test "$dirsize" -gt "$dblksz" || echo "failed to create two-block directory"
+
+# Break the directory, remount filesystem
+_scratch_unmount
+_scratch_xfs_db -x \
+	-c 'path /some/victimdir' \
+	-c 'bmap' \
+	-c 'dblock 1' \
+	-c 'blocktrash -z -0 -o 0 -x 2048 -y 2048 -n 2048' >> $seqres.full
+_scratch_mount
+
+# If the system xfsprogs has self healing enabled, we need to shut down the
+# daemon before we try to capture things.
+if _systemd_is_running; then
+	svcname="xfs_scrubbed@$(systemd-escape --path "$SCRATCH_MNT")"
+	echo "$svcname: $(systemctl is-active "$svcname")" >> $seqres.full
+	_systemd_unit_stop "$svcname" &>> $seqres.full
+fi
+
+# Start the health monitor, have it repair everything reported corrupt
+_scratch_xfs_scrubbed --repair --log > $tmp.scrubbed &
+scrubbed_pid=$!
+sleep 1
+
+# Access the broken directory to trigger a repair, then poll the directory
+# for 5 seconds to see if it gets fixed without us needing to intervene.
+ls $SCRATCH_MNT/some/victimdir > /dev/null 2> $tmp.err
+_filter_scratch < $tmp.err
+try=0
+while [ $try -lt 50 ] && grep -q 'Structure needs cleaning' $tmp.err; do
+	echo "try $try saw corruption" >> $seqres.full
+	sleep 0.1
+	ls $SCRATCH_MNT/some/victimdir > /dev/null 2> $tmp.err
+	try=$((try + 1))
+done
+_filter_scratch < $tmp.err
+
+# Unmount fs to kill scrubbed, then wait for it to finish.
+while ! _scratch_unmount &>/dev/null; do
+	sleep 0.5
+done
+kill $scrubbed_pid
+wait
+cat $tmp.scrubbed >> $seqres.full
+
+status=0
+exit
diff --git a/tests/xfs/1884.out b/tests/xfs/1884.out
new file mode 100644
index 00000000000000..929e33da01f92c
--- /dev/null
+++ b/tests/xfs/1884.out
@@ -0,0 +1,2 @@
+QA output created by 1884
+ls: reading directory 'SCRATCH_MNT/some/victimdir': Structure needs cleaning


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCHSET 5/5] fstests: add difficult V5 features to filesystems
  2024-12-31 23:25 [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing Darrick J. Wong
                   ` (3 preceding siblings ...)
  2024-12-31 23:35 ` [PATCHSET 4/5] fstests: live health monitoring of filesystems Darrick J. Wong
@ 2024-12-31 23:35 ` Darrick J. Wong
  2024-12-31 23:58   ` [PATCH 1/3] xfs/1856: add metadir upgrade to test matrix Darrick J. Wong
                     ` (2 more replies)
  2025-01-02  1:37 ` [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing Stephen Zhang
  5 siblings, 3 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:35 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

Hi all,

This series enables xfs_repair to add select features to existing V5
filesystems.  Specifically, one can add free inode btrees, reflink
support, and reverse mapping.

If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.

This has been running on the djcloud for months with no problems.  Enjoy!
Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=upgrade-newer-features

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=upgrade-newer-features
---
Commits in this patchset:
 * xfs/1856: add metadir upgrade to test matrix
 * xfs/1856: add rtrmapbt upgrade to test matrix
 * xfs/1856: add rtreflink upgrade to test matrix
---
 tests/xfs/1856 |   42 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 41 insertions(+), 1 deletion(-)


^ permalink raw reply	[flat|nested] 24+ messages in thread

* [PATCH 1/3] xfs/1856: add metadir upgrade to test matrix
  2024-12-31 23:35 ` [PATCHSET 5/5] fstests: add difficult V5 features to filesystems Darrick J. Wong
@ 2024-12-31 23:58   ` Darrick J. Wong
  2024-12-31 23:58   ` [PATCH 2/3] xfs/1856: add rtrmapbt " Darrick J. Wong
  2024-12-31 23:59   ` [PATCH 3/3] xfs/1856: add rtreflink " Darrick J. Wong
  2 siblings, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:58 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

From: Darrick J. Wong <djwong@kernel.org>

Add metadata directory trees to the features that this test will try to
upgrade.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 tests/xfs/1856 |    1 +
 1 file changed, 1 insertion(+)


diff --git a/tests/xfs/1856 b/tests/xfs/1856
index 7524a449c3af00..fedeb157dbd9bb 100755
--- a/tests/xfs/1856
+++ b/tests/xfs/1856
@@ -188,6 +188,7 @@ else
 	check_repair_upgrade reflink && FEATURES+=("reflink")
 	check_repair_upgrade inobtcount && FEATURES+=("inobtcount")
 	check_repair_upgrade bigtime && FEATURES+=("bigtime")
+	check_repair_upgrade metadir && FEATURES+=("metadir")
 fi
 
 test "${#FEATURES[@]}" -eq 0 && \


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 2/3] xfs/1856: add rtrmapbt upgrade to test matrix
  2024-12-31 23:35 ` [PATCHSET 5/5] fstests: add difficult V5 features to filesystems Darrick J. Wong
  2024-12-31 23:58   ` [PATCH 1/3] xfs/1856: add metadir upgrade to test matrix Darrick J. Wong
@ 2024-12-31 23:58   ` Darrick J. Wong
  2024-12-31 23:59   ` [PATCH 3/3] xfs/1856: add rtreflink " Darrick J. Wong
  2 siblings, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:58 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

From: Darrick J. Wong <djwong@kernel.org>

Add realtime reverse mapping btrees to the features that this test will
try to upgrade.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 tests/xfs/1856 |   40 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 39 insertions(+), 1 deletion(-)


diff --git a/tests/xfs/1856 b/tests/xfs/1856
index fedeb157dbd9bb..8e3213da752348 100755
--- a/tests/xfs/1856
+++ b/tests/xfs/1856
@@ -30,11 +30,47 @@ rt_configured()
 	test "$USE_EXTERNAL" = "yes" && test -n "$SCRATCH_RTDEV"
 }
 
+# Does mkfs support metadir?
+supports_metadir()
+{
+	$MKFS_XFS_PROG 2>&1 | grep -q 'metadir='
+}
+
+# Do we need to enable metadir at mkfs time to support a feature upgrade test?
+need_metadir()
+{
+	local feat="$1"
+
+	# if realtime isn't configured, we don't need metadir
+	rt_configured || return 1
+
+	# If we don't even know what realtime rmap is, we don't need rt groups
+	# and hence don't need metadir.
+	test -z "${FEATURE_STATE["rmapbt"]}" && return 1
+
+	# rt rmap btrees require metadir, but metadir cannot be added to an
+	# existing rt filesystem.  Force it on at mkfs time.
+	test "${FEATURE_STATE["rmapbt"]}" -eq 1 && return 0
+	test "$feat" = "rmapbt" && return 0
+
+	return 1
+}
+
 # Compute the MKFS_OPTIONS string for a particular feature upgrade test
 compute_mkfs_options()
 {
+	local feat="$1"
 	local m_opts=""
 	local caller_options="$MKFS_OPTIONS"
+	local metadir
+
+	need_metadir "$feat" && metadir=1
+	if echo "$caller_options" | grep -q 'metadir='; then
+		test -z "$metadir" && metadir=0
+		caller_options="$(echo "$caller_options" | sed -e 's/metadir=*[0-9]*/metadir='$metadir'/g')"
+	elif [ -n "$metadir" ]; then
+		caller_options="$caller_options -m metadir=$metadir"
+	fi
 
 	for feat in "${FEATURES[@]}"; do
 		local feat_state="${FEATURE_STATE["${feat}"]}"
@@ -179,9 +215,11 @@ MKFS_OPTIONS="$(qerase_mkfs_options)"
 # upgrade don't spread failure to the rest of the tests.
 FEATURES=()
 if rt_configured; then
+	# rmap wasn't added to rt devices until after metadir
 	check_repair_upgrade finobt && FEATURES+=("finobt")
 	check_repair_upgrade inobtcount && FEATURES+=("inobtcount")
 	check_repair_upgrade bigtime && FEATURES+=("bigtime")
+	supports_metadir && check_repair_upgrade rmapbt && FEATURES+=("rmapbt")
 else
 	check_repair_upgrade finobt && FEATURES+=("finobt")
 	check_repair_upgrade rmapbt && FEATURES+=("rmapbt")
@@ -204,7 +242,7 @@ for feat in "${FEATURES[@]}"; do
 
 	upgrade_start_message "$feat" | _tee_kernlog $seqres.full > /dev/null
 
-	opts="$(compute_mkfs_options)"
+	opts="$(compute_mkfs_options "$feat")"
 	echo "mkfs.xfs $opts" >> $seqres.full
 
 	# Format filesystem


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH 3/3] xfs/1856: add rtreflink upgrade to test matrix
  2024-12-31 23:35 ` [PATCHSET 5/5] fstests: add difficult V5 features to filesystems Darrick J. Wong
  2024-12-31 23:58   ` [PATCH 1/3] xfs/1856: add metadir upgrade to test matrix Darrick J. Wong
  2024-12-31 23:58   ` [PATCH 2/3] xfs/1856: add rtrmapbt " Darrick J. Wong
@ 2024-12-31 23:59   ` Darrick J. Wong
  2 siblings, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2024-12-31 23:59 UTC (permalink / raw)
  To: zlang, djwong; +Cc: fstests, linux-xfs

From: Darrick J. Wong <djwong@kernel.org>

Add realtime reflink to the features that this test will try to
upgrade.

Signed-off-by: "Darrick J. Wong" <djwong@kernel.org>
---
 tests/xfs/1856 |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)


diff --git a/tests/xfs/1856 b/tests/xfs/1856
index 8e3213da752348..9b776493f0486f 100755
--- a/tests/xfs/1856
+++ b/tests/xfs/1856
@@ -215,11 +215,12 @@ MKFS_OPTIONS="$(qerase_mkfs_options)"
 # upgrade don't spread failure to the rest of the tests.
 FEATURES=()
 if rt_configured; then
-	# rmap wasn't added to rt devices until after metadir
+	# rmap & reflink weren't added to rt devices until after metadir
 	check_repair_upgrade finobt && FEATURES+=("finobt")
 	check_repair_upgrade inobtcount && FEATURES+=("inobtcount")
 	check_repair_upgrade bigtime && FEATURES+=("bigtime")
 	supports_metadir && check_repair_upgrade rmapbt && FEATURES+=("rmapbt")
+	supports_metadir && check_repair_upgrade reflink && FEATURES+=("reflink")
 else
 	check_repair_upgrade finobt && FEATURES+=("finobt")
 	check_repair_upgrade rmapbt && FEATURES+=("rmapbt")


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* Re: [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing
  2024-12-31 23:25 [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing Darrick J. Wong
                   ` (4 preceding siblings ...)
  2024-12-31 23:35 ` [PATCHSET 5/5] fstests: add difficult V5 features to filesystems Darrick J. Wong
@ 2025-01-02  1:37 ` Stephen Zhang
  2025-01-07  0:26   ` Darrick J. Wong
  5 siblings, 1 reply; 24+ messages in thread
From: Stephen Zhang @ 2025-01-02  1:37 UTC (permalink / raw)
  To: Darrick J. Wong
  Cc: Carlos Maiolino, Zorro Lang, Andrey Albershteyn,
	Christoph Hellwig, Dave Chinner, xfs, greg.marsden, shirley.ma,
	konrad.wilk, fstests

Darrick J. Wong <djwong@kernel.org> 于2025年1月1日周三 07:25写道：
>
> Hi everyone,
>
> Thank you all for helping get online repair, parent pointers, and
> metadata directory trees, and realtime allocation groups merged this
> year!  We got a lot done in 2024.
>
> Having sent pull requests to Carlos for the last pieces of the realtime
> modernization project, I have exactly two worthwhile projects left in my
> development trees!  The stuff here isn't necessarily in mergeable state
> yet, but I still believe everyone ought to know what I'm up to.
>
> The first project implements (somewhat buggily; I never quite got back
> to dealing with moving eof blocks) free space defragmentation so that we
> can meaningfully shrink filesystems; garbage collect regions of the
> filesystem; or prepare for large allocations.  There's not much new
> kernel code other than exporting refcounts and gaining the ability to
> map free space.
>
> The second project initiates filesystem self healing routines whenever
> problems start to crop up, which means that it can run fully
> autonomously in the background.  The monitoring system uses some
> pseudo-file and seqbuf tricks that I lifted from kmo last winter.
>
> Both of these projects are largely userspace code.
>
> Also I threw in some xfs_repair code to do dangerous fs upgrades.
> Nobody should use these, ever.
>
> Maintainers: please do not merge, this is a dog-and-pony show to attract
> developer attention.
>

[Add Dave to the list]

Hi, Darrick and all,

Recently, I have been considering implementing the XFS shrink feature based
on the AF concept, which was mentioned in this link:

https://lore.kernel.org/linux-xfs/20241104014439.3786609-1-zhangshida@kylinos.cn/

In the lore link, it stated:
The rules used by AG are more about extending outwards.
whilst
The rules used by AF are more about restricting inwards.

where the AF concept implicitly and naturally involves the semantics of
compressing/shrinking(restricting).

AG(for xfs extend) and AF(for xfs shrink) are constructed in a symmetrical way,
in which it is more elegant and easier to build more complex features on it.

To elaborate further, for example, AG should not be seen as
independent entities in
the shrink context. That means each AG requires separate
managements(flags or something to indicate the state of that
AG/region), which would increase the system complexity compared to the
idea behind AF. AF views several AGs as a whole.

And when it comes to growfs, things start to get a little more
complicated, and AF
can handle it easily and naturally.

However talk is too cheap, to validate our point, we truly hope to have the
opportunity to participate in developing these features by integrating
the existing
infrastructure you have already established with the AF concept.

Best regards,
Shida

> --D
>
> PS: I'll be back after the holidays to look at the zoned/atomic/fsverity
> patches.  And finally rebase fstests to 2024-12-08.
>

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing
  2025-01-02  1:37 ` [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing Stephen Zhang
@ 2025-01-07  0:26   ` Darrick J. Wong
  0 siblings, 0 replies; 24+ messages in thread
From: Darrick J. Wong @ 2025-01-07  0:26 UTC (permalink / raw)
  To: Stephen Zhang
  Cc: Carlos Maiolino, Zorro Lang, Andrey Albershteyn,
	Christoph Hellwig, Dave Chinner, xfs, greg.marsden, shirley.ma,
	konrad.wilk, fstests

On Thu, Jan 02, 2025 at 09:37:47AM +0800, Stephen Zhang wrote:
> Darrick J. Wong <djwong@kernel.org> 于2025年1月1日周三 07:25写道：
> >
> > Hi everyone,
> >
> > Thank you all for helping get online repair, parent pointers, and
> > metadata directory trees, and realtime allocation groups merged this
> > year!  We got a lot done in 2024.
> >
> > Having sent pull requests to Carlos for the last pieces of the realtime
> > modernization project, I have exactly two worthwhile projects left in my
> > development trees!  The stuff here isn't necessarily in mergeable state
> > yet, but I still believe everyone ought to know what I'm up to.
> >
> > The first project implements (somewhat buggily; I never quite got back
> > to dealing with moving eof blocks) free space defragmentation so that we
> > can meaningfully shrink filesystems; garbage collect regions of the
> > filesystem; or prepare for large allocations.  There's not much new
> > kernel code other than exporting refcounts and gaining the ability to
> > map free space.
> >
> > The second project initiates filesystem self healing routines whenever
> > problems start to crop up, which means that it can run fully
> > autonomously in the background.  The monitoring system uses some
> > pseudo-file and seqbuf tricks that I lifted from kmo last winter.
> >
> > Both of these projects are largely userspace code.
> >
> > Also I threw in some xfs_repair code to do dangerous fs upgrades.
> > Nobody should use these, ever.
> >
> > Maintainers: please do not merge, this is a dog-and-pony show to attract
> > developer attention.
> >
> 
> [Add Dave to the list]
> 
> Hi, Darrick and all,
> 
> Recently, I have been considering implementing the XFS shrink feature based
> on the AF concept, which was mentioned in this link:
> 
> https://lore.kernel.org/linux-xfs/20241104014439.3786609-1-zhangshida@kylinos.cn/
> 
> In the lore link, it stated:
> The rules used by AG are more about extending outwards.
> whilst
> The rules used by AF are more about restricting inwards.
> 
> where the AF concept implicitly and naturally involves the semantics of
> compressing/shrinking(restricting).
> 
> AG(for xfs extend) and AF(for xfs shrink) are constructed in a symmetrical way,
> in which it is more elegant and easier to build more complex features on it.
> 
> To elaborate further, for example, AG should not be seen as
> independent entities in
> the shrink context. That means each AG requires separate
> managements(flags or something to indicate the state of that
> AG/region), which would increase the system complexity compared to the
> idea behind AF. AF views several AGs as a whole.
> 
> And when it comes to growfs, things start to get a little more
> complicated, and AF
> can handle it easily and naturally.
> 
> However talk is too cheap, to validate our point, we truly hope to have the
> opportunity to participate in developing these features by integrating
> the existing
> infrastructure you have already established with the AF concept.

Hmm, now that's interesting -- using the AF ("allocation fencing"?)
capability to constrain allocations to a subset of AGs, and then slowly
rewriting files and whatnot to migrate data to other AGs.  Eventually
you end up with an AG that's empty and therefore ready for shrink.

That's definitely a different way to do that than what I did (add a
"mapfree" ioctl to pin space to a file).  I'll ponder these 2 approaches
a bit more.

--D

> Best regards,
> Shida
> 
> 
> 
> > --D
> >
> > PS: I'll be back after the holidays to look at the zoned/atomic/fsverity
> > patches.  And finally rebase fstests to 2024-12-08.
> >
> 

^ permalink raw reply	[flat|nested] 24+ messages in thread

end of thread, other threads:[~2025-01-13  5:55 UTC | newest]

Thread overview: 24+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-12-31 23:25 [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing Darrick J. Wong
2024-12-31 23:34 ` [PATCHSET 1/5] fstests: functional test for refcount reporting Darrick J. Wong
2024-12-31 23:56   ` [PATCH 1/1] xfs: test output of new FSREFCOUNTS ioctl Darrick J. Wong
2024-12-31 23:35 ` [PATCHSET 2/5] fstests: defragment free space Darrick J. Wong
2024-12-31 23:56   ` [PATCH 1/1] xfs: test clearing of " Darrick J. Wong
2024-12-31 23:35 ` [PATCHSET 3/5] fstests: capture logs from mount failures Darrick J. Wong
2024-12-31 23:56   ` [PATCH 1/2] treewide: convert all $MOUNT_PROG to _mount Darrick J. Wong
2024-12-31 23:56   ` [PATCH 2/2] check: capture dmesg of mount failures if test fails Darrick J. Wong
2025-01-06 11:18     ` Nirjhar Roy
2025-01-06 23:52       ` Darrick J. Wong
2025-01-13  5:55         ` Nirjhar Roy
2024-12-31 23:35 ` [PATCHSET 4/5] fstests: live health monitoring of filesystems Darrick J. Wong
2024-12-31 23:57   ` [PATCH 1/6] misc: convert all $UMOUNT_PROG to a _umount helper Darrick J. Wong
2024-12-31 23:57   ` [PATCH 2/6] misc: convert all umount(1) invocations to _umount Darrick J. Wong
2024-12-31 23:57   ` [PATCH 3/6] xfs: test health monitoring code Darrick J. Wong
2024-12-31 23:57   ` [PATCH 4/6] xfs: test for metadata corruption error reporting via healthmon Darrick J. Wong
2024-12-31 23:58   ` [PATCH 5/6] xfs: test io " Darrick J. Wong
2024-12-31 23:58   ` [PATCH 6/6] xfs: test new xfs_scrubbed daemon Darrick J. Wong
2024-12-31 23:35 ` [PATCHSET 5/5] fstests: add difficult V5 features to filesystems Darrick J. Wong
2024-12-31 23:58   ` [PATCH 1/3] xfs/1856: add metadir upgrade to test matrix Darrick J. Wong
2024-12-31 23:58   ` [PATCH 2/3] xfs/1856: add rtrmapbt " Darrick J. Wong
2024-12-31 23:59   ` [PATCH 3/3] xfs/1856: add rtreflink " Darrick J. Wong
2025-01-02  1:37 ` [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing Stephen Zhang
2025-01-07  0:26   ` Darrick J. Wong

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox