[PATCH 00/12] fstests: various fixes

[PATCH 00/12] fstests: various fixes

[Darrick J. Wong]

Hi all,

Here are a bunch of fixes and some new tests.

Patches 1-2 improve the random number algorithm that is used to shuffle
the test list when -r is used.

Patches 3-6 fix some minor problems with existing tests.

Patches 7-8 refactor the code that creates and maintains the cache of
populated filesystem images so that we can cache multiple different
configurations on the test filesystem, since it is advantageous to be
able to populate different configurations without discarding perfectly
good images.

Patch 9 implements a bunch more reflink corner cases when sharing into
or out from a bunch of different types of blocks.

Patch 10 ensures that xfs_copy and xfs_metadump can deal with all types
of xfs metadata.

Patch 11-12 wipe the scratch devices between tests and fix the tests
that erroneously expected the scratch devices to contain anything.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=djwong-devel

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=djwong-devel

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=djwong-devel

[PATCH 01/12] check: improve test list randomization

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

awk doesn't have a particularly good random number generator -- it seeds
from the Unix epoch time in seconds, which means that the run order
across a bunch of VMs started at exactly the same time are unsettlingly
predictable.  Therefore, at least try to seed it with bash's $RANDOM,
which is slightly less predictable.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 check         |   12 ++++++------
 randomize.awk |   12 ++++++++----
 2 files changed, 14 insertions(+), 10 deletions(-)


diff --git a/check b/check
index 20e95302..ecd1d39a 100755
--- a/check
+++ b/check
@@ -242,13 +242,13 @@ _prepare_test_list()
 	done
 
 	# sort the list of tests into numeric order
-	list=`sort -n $tmp.list | uniq`
-	rm -f $tmp.list
-
-	if $randomize
-	then
-		list=`echo $list | awk -f randomize.awk`
+	if $randomize; then
+		sorter="awk -v seed=$RANDOM -f randomize.awk"
+	else
+		sorter="cat"
 	fi
+	list=`sort -n $tmp.list | uniq | $sorter`
+	rm -f $tmp.list
 }
 
 # Process command arguments first.
diff --git a/randomize.awk b/randomize.awk
index 0a8ad71a..d979fb03 100644
--- a/randomize.awk
+++ b/randomize.awk
@@ -15,10 +15,14 @@ function randomize(array, N) {
 return
 }
 
+BEGIN {
+	srand(seed)
+}
 {
-    srand()
-    for (i = 0; i < NF; i++ ) array[i] = $(i+1)
-    randomize(array, NF)
-    for (i = 0; i < NF; i++) printf("%s ", array[i])
+	array[NR - 1] = $0
+}
+END {
+    randomize(array, NR)
+    for (i = 0; i < NR; i++) printf("%s ", array[i])
 }
 

[PATCH 02/12] check: really improve test list randomization

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

coreutils provides the shuf(1) utility that randomizes the order of a
list and seeds its random number generator with /dev/urandom.  It's a
bit speedier than awk, so use it if available.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 check |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)


diff --git a/check b/check
index ecd1d39a..a2c5ba21 100755
--- a/check
+++ b/check
@@ -243,7 +243,11 @@ _prepare_test_list()
 
 	# sort the list of tests into numeric order
 	if $randomize; then
-		sorter="awk -v seed=$RANDOM -f randomize.awk"
+		if type shuf >& /dev/null; then
+			sorter="shuf"
+		else
+			sorter="awk -v seed=$RANDOM -f randomize.awk"
+		fi
 	else
 		sorter="cat"
 	fi

[PATCH 03/12] generic/042: fix stale disk contents check

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

This test doesn't call fsync or sync to force writeback of the first 60k
of the file, which means that we could end up with a file full of
zeroes or an empty file.  Since this is a regression test that looks for
stale disk contents slipping through, change the test to look for the
stale bytes.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/generic/042 |   10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)


diff --git a/tests/generic/042 b/tests/generic/042
index 1aa18f9b..6c62eb63 100755
--- a/tests/generic/042
+++ b/tests/generic/042
@@ -44,7 +44,7 @@ _crashtest()
 
 	# Create an fs on a small, initialized image. The pattern is written to
 	# the image to detect stale data exposure.
-	$XFS_IO_PROG -f -c "truncate 0" -c "pwrite 0 25M" $img \
+	$XFS_IO_PROG -f -c "truncate 0" -c "pwrite -S 0xCD 0 25M" $img \
 		>> $seqres.full 2>&1
 	_mkfs_dev $img >> $seqres.full 2>&1
 
@@ -61,8 +61,12 @@ _crashtest()
 	$UMOUNT_PROG $mnt
 	_mount $img $mnt
 
-	# we generally expect a zero-sized file (this should be silent)
-	hexdump $file
+	# We should /never/ see 0xCD in the file, because we wrote that pattern
+	# to the filesystem image to expose stale data.
+	if hexdump -v -e '/1 "%02X "' $file | grep -q "CD"; then
+		echo "Saw stale data!!!"
+		hexdump $file
+	fi
 
 	$UMOUNT_PROG $mnt
 }

[PATCH 04/12] generic/032: fix unwritten extent checks

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Fix the unwritten extent detector in this test to ignore post-eof
allocations because those are harmless.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/generic/032 |   17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)


diff --git a/tests/generic/032 b/tests/generic/032
index affde4b7..38352a21 100755
--- a/tests/generic/032
+++ b/tests/generic/032
@@ -69,18 +69,21 @@ do
 
 	# preallocate the first 64k and overwite, writing past 64k to contend
 	# with writeback
+	file_len=0x100000
 	$XFS_IO_PROG \
 		-c "falloc 0 0x10000"	\
-		-c "pwrite 0 0x100000"	\
+		-c "pwrite 0 $file_len"	\
 		-c "fsync"		\
 		$SCRATCH_MNT/file >> $seqres.full 2>&1
 
-	# Check for unwritten extents. We should have none since we wrote over
-	# the entire preallocated region and ran fsync.
-	$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/file | \
-		tee -a $seqres.full | \
-		_filter_fiemap | grep unwritten
-	[ $? == 0 ] && _fail "Unwritten extents found!"
+	# Check for unwritten extents. We should have none before EOF since we
+	# wrote over the entire preallocated region and ran fsync.
+	eof_sector=$(( file_len / 512 ))
+	$XFS_IO_PROG -c 'fiemap -v' $SCRATCH_MNT/file | \
+		_filter_fiemap | \
+		tr '[.]:' '    ' | \
+		awk "{if (\$2 < $eof_sector) {print \$0}}" | \
+		grep -q unwritten && _fail "Unwritten extents found!"
 done
 
 echo $iters iterations

[PATCH 05/12] generic/454: stop the test if we run out of space

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Certain filesystems (ext4 w/ 1k block size) can run out of space while
running this test because they have very limited xattr storage
capabilities.  If we run out of space while setting an attr, don't
bother continuing the test.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/generic/454 |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)


diff --git a/tests/generic/454 b/tests/generic/454
index 4a0936c5..01986d6b 100755
--- a/tests/generic/454
+++ b/tests/generic/454
@@ -48,7 +48,12 @@ setf() {
 	key="$(echo -e "$1")"
 	value="$2"
 
-	$SETFATTR_PROG -n "user.${key}" -v "${value}" "${testfile}"
+	$SETFATTR_PROG -n "user.${key}" -v "${value}" "${testfile}" > $tmp.output 2>&1
+	if [ $? -ne 0 ]; then
+		grep -q 'No space left on device' $tmp.output && \
+			_notrun "ran out of space"
+		cat $tmp.output
+	fi
 	echo "Storing ${key} ($(hexbytes "${key}")) -> ${value}" >> $seqres.full
 }
 

[PATCH 06/12] ext4/023: don't require scrub for ext4 populated image creation

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Don't require scrub for ext4's populated fs creation test because there
is no general online scrub program for ext*.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/ext4/023 |    1 -
 1 file changed, 1 deletion(-)


diff --git a/tests/ext4/023 b/tests/ext4/023
index a598e2b4..1dadb29f 100755
--- a/tests/ext4/023
+++ b/tests/ext4/023
@@ -33,7 +33,6 @@ _cleanup()
 _supported_os Linux
 _supported_fs ext4
 _require_scratch
-_require_scrub
 
 echo "Format and populate"
 _scratch_populate_cached > $seqres.full 2>&1

[PATCH 07/12] common/populate: refactor _scratch_populate_cached

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Refactor _scratch_populate_cached into smaller helper functions.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 common/populate |   69 ++++++++++++++++++++++++++++++++++---------------------
 1 file changed, 43 insertions(+), 26 deletions(-)


diff --git a/common/populate b/common/populate
index 95953476..1f921ac5 100644
--- a/common/populate
+++ b/common/populate
@@ -765,20 +765,14 @@ _fill_fs()
 	done
 }
 
-# Populate a scratch FS from scratch or from a cached image.
-_scratch_populate_cached() {
-	POPULATE_METADUMP="${TEST_DIR}/__populate.${FSTYP}"
-	POPULATE_METADUMP_DESCR="${TEST_DIR}/__populate.${FSTYP}.txt"
-
-	# Don't keep metadata images cached for more 48 hours...
-	rm -rf "$(find "${POPULATE_METADUMP}" -mtime +2 2>/dev/null)"
+# Compute the fs geometry description of a populated filesystem
+_scratch_populate_cache_tag() {
+	local extra_descr=""
+	local size="$(blockdev --getsz "${SCRATCH_DEV}")"
 
-	# Throw away cached image if it doesn't match our spec.
 	case "${FSTYP}" in
 	"ext4")
 		extra_descr="LOGDEV ${SCRATCH_LOGDEV} USE_EXTERNAL ${USE_EXTERNAL}"
-		# ext4 cannot e2image external logs, so we cannot restore
-		test -n "${SCRATCH_LOGDEV}" && rm -f "${POPULATE_METADUMP}"
 		;;
 	"xfs")
 		extra_descr="LOGDEV ${SCRATCH_LOGDEV} USE_EXTERNAL ${USE_EXTERNAL} RTDEV ${SCRATCH_RTDEV}"
@@ -787,23 +781,46 @@ _scratch_populate_cached() {
 			extra_descr="${extra_descr} QUOTAS"
 		fi
 		;;
-	*)
-		extra_descr="";;
 	esac
-	meta_descr="FSTYP ${FSTYP} MKFS_OPTIONS ${MKFS_OPTIONS} SIZE $(blockdev --getsz "${SCRATCH_DEV}") ${extra_descr} ARGS $@"
-	cmp -s "${POPULATE_METADUMP_DESCR}" <(echo "${meta_descr}") || rm -rf "${POPULATE_METADUMP}"
-
-	# Do we have a cached image?
-	if [ -r "${POPULATE_METADUMP}" ]; then
-		case "${FSTYP}" in
-		"xfs")
-			xfs_mdrestore "${POPULATE_METADUMP}" "${SCRATCH_DEV}" && return
-			;;
-		"ext2"|"ext3"|"ext4")
-			e2image -r "${POPULATE_METADUMP}" "${SCRATCH_DEV}" && return
-			;;
-		esac
-	fi
+	echo "FSTYP ${FSTYP} MKFS_OPTIONS ${MKFS_OPTIONS} SIZE ${size} ${extra_descr} ARGS $@"
+}
+
+# Restore a cached populated fs from a metadata dump
+_scratch_populate_restore_cached() {
+	local metadump="$1"
+
+	case "${FSTYP}" in
+	"xfs")
+		xfs_mdrestore "${metadump}" "${SCRATCH_DEV}" && return 0
+		;;
+	"ext2"|"ext3"|"ext4")
+		# ext4 cannot e2image external logs, so we cannot restore
+		test -n "${SCRATCH_LOGDEV}" && return 1
+		e2image -r "${metadump}" "${SCRATCH_DEV}" && return 0
+		;;
+	esac
+	return 1
+}
+
+# Populate a scratch FS from scratch or from a cached image.
+_scratch_populate_cached() {
+	local meta_descr="$(_scratch_populate_cache_tag "$@")"
+
+	# These variables are shared outside this function
+	POPULATE_METADUMP="${TEST_DIR}/__populate.${FSTYP}"
+	POPULATE_METADUMP_DESCR="${TEST_DIR}/__populate.${FSTYP}.txt"
+
+	# Don't keep metadata images cached for more 48 hours...
+	rm -rf "$(find "${POPULATE_METADUMP}" -mtime +2 2>/dev/null)"
+
+	# Throw away cached image if it doesn't match our spec.
+	cmp -s "${POPULATE_METADUMP_DESCR}" <(echo "${meta_descr}") || \
+		rm -rf "${POPULATE_METADUMP}"
+
+	# Try to restore from the metadump
+	test -r "${POPULATE_METADUMP}" && \
+		_scratch_populate_restore_cached "${POPULATE_METADUMP}" && \
+		return
 
 	# Oh well, just create one from scratch
 	_scratch_mkfs

[PATCH 08/12] common/populate: support multiple cached images

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Enhance the populated fs metadump image cache to support multiple
configurations per filesystem so that we reduce the image creation
overhead even further.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 common/populate |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)


diff --git a/common/populate b/common/populate
index 1f921ac5..4fa118f0 100644
--- a/common/populate
+++ b/common/populate
@@ -805,10 +805,12 @@ _scratch_populate_restore_cached() {
 # Populate a scratch FS from scratch or from a cached image.
 _scratch_populate_cached() {
 	local meta_descr="$(_scratch_populate_cache_tag "$@")"
+	local meta_tag="$(echo "${meta_descr}" | md5sum - | cut -d ' ' -f 1)"
+	local metadump_stem="${TEST_DIR}/__populate.${FSTYP}.${meta_tag}"
 
 	# These variables are shared outside this function
-	POPULATE_METADUMP="${TEST_DIR}/__populate.${FSTYP}"
-	POPULATE_METADUMP_DESCR="${TEST_DIR}/__populate.${FSTYP}.txt"
+	POPULATE_METADUMP="${metadump_stem}.metadump"
+	POPULATE_METADUMP_DESCR="${metadump_stem}.txt"
 
 	# Don't keep metadata images cached for more 48 hours...
 	rm -rf "$(find "${POPULATE_METADUMP}" -mtime +2 2>/dev/null)"

[PATCH 09/12] clonerange: test remapping the rainbow

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Add some more clone range tests that missed various "wacky" combinations
of file state.  Specifically, we test reflinking into and out of rainbow
ranges (a mix of real, unwritten, hole, delalloc, and shared extents),
and also we test that we can correctly handle double-inode locking no
matter what order of inodes or the filesystem's locking rules.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 common/reflink        |    7 ++
 tests/generic/940     |   94 ++++++++++++++++++++++++
 tests/generic/940.out |   14 ++++
 tests/generic/941     |   99 +++++++++++++++++++++++++
 tests/generic/941.out |   16 ++++
 tests/generic/942     |   95 ++++++++++++++++++++++++
 tests/generic/942.out |   14 ++++
 tests/generic/943     |   99 +++++++++++++++++++++++++
 tests/generic/943.out |   16 ++++
 tests/generic/944     |  196 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/944.out |   62 ++++++++++++++++
 tests/generic/group   |    5 +
 12 files changed, 716 insertions(+), 1 deletion(-)
 create mode 100755 tests/generic/940
 create mode 100644 tests/generic/940.out
 create mode 100755 tests/generic/941
 create mode 100644 tests/generic/941.out
 create mode 100755 tests/generic/942
 create mode 100644 tests/generic/942.out
 create mode 100755 tests/generic/943
 create mode 100644 tests/generic/943.out
 create mode 100755 tests/generic/944
 create mode 100644 tests/generic/944.out


diff --git a/common/reflink b/common/reflink
index 11561a76..598f0877 100644
--- a/common/reflink
+++ b/common/reflink
@@ -14,9 +14,14 @@ _require_cp_reflink()
 # Can we reflink between arbitrary file sets?
 # i.e. if we reflink a->b and c->d, can we later share
 # blocks between b & c?
+_supports_arbitrary_fileset_reflink()
+{
+	test "$FSTYP" != "ocfs2"
+}
+
 _require_arbitrary_fileset_reflink()
 {
-	test "$FSTYP" = "ocfs2" && \
+	_supports_arbitrary_fileset_reflink ||
 		_notrun "reflink between arbitrary file groups not supported in $FSTYP"
 }
 
diff --git a/tests/generic/940 b/tests/generic/940
new file mode 100755
index 00000000..4573cbae
--- /dev/null
+++ b/tests/generic/940
@@ -0,0 +1,94 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (c) 2019, Oracle and/or its affiliates.  All Rights Reserved.
+#
+# FS QA Test No. 940
+#
+# Ensuring that reflinking works when the destination range covers multiple
+# extents, some unwritten, some not:
+#
+#   - Create a file with the following repeating sequence of blocks:
+#     1. reflinked
+#     2. unwritten
+#     3. hole
+#     4. regular block
+#     5. delalloc
+#   - reflink across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    cd /
+    rm -rf $tmp.* $testdir
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_supported_os Linux
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+_require_xfs_io_command "fpunch"
+_require_cp_reflink
+_require_odirect
+
+rm -f $seqres.full
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_pwrite_byte 0x64 0 $((blksz * nr)) $testdir/file2 >> $seqres.full
+_weave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file2 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "reflink across the transition"
+roff=$((filesize / 4))
+rsz=$((filesize / 2))
+_weave_reflink_rainbow_delalloc $blksz $nr $testdir/file3 >> $seqres.full
+
+# now reflink into the rainbow
+echo "before reflink" >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file2 >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file3 >> $seqres.full
+$XFS_IO_PROG -f -c "reflink $testdir/file2 $roff $roff $rsz" $testdir/file3 >> $seqres.full
+_pwrite_byte 0x64 $roff $rsz $testdir/file3.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+echo "after reflink" >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file2 >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file3 >> $seqres.full
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file2 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/940.out b/tests/generic/940.out
new file mode 100644
index 00000000..d34c7b50
--- /dev/null
+++ b/tests/generic/940.out
@@ -0,0 +1,14 @@
+QA output created by 940
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-940/file1
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-940/file2
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-940/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-940/file3.chk
+reflink across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-940/file1
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-940/file2
+5d47954e2336b2547afde6e44d2f13cc  SCRATCH_MNT/test-940/file3
+5d47954e2336b2547afde6e44d2f13cc  SCRATCH_MNT/test-940/file3.chk
diff --git a/tests/generic/941 b/tests/generic/941
new file mode 100755
index 00000000..145b8585
--- /dev/null
+++ b/tests/generic/941
@@ -0,0 +1,99 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (c) 2019, Oracle and/or its affiliates.  All Rights Reserved.
+#
+# FS QA Test No. 941
+#
+# Ensuring that reflinking works when the source range covers multiple
+# extents, some unwritten, some not:
+#
+#   - Create a file with the following repeating sequence of blocks:
+#     1. reflinked
+#     2. unwritten
+#     3. hole
+#     4. regular block
+#     5. delalloc
+#   - reflink across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    cd /
+    rm -rf $tmp.* $testdir
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_supported_os Linux
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+_require_xfs_io_command "fpunch"
+_require_cp_reflink
+_require_odirect
+
+rm -f $seqres.full
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_pwrite_byte 0x64 0 $((blksz * nr)) $testdir/file2 >> $seqres.full
+_pwrite_byte 0x64 0 $((blksz * nr)) $testdir/file2.chk >> $seqres.full
+_weave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file2 | _filter_scratch
+md5sum $testdir/file2.chk | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "reflink across the transition"
+roff=$((filesize / 4))
+rsz=$((filesize / 2))
+_weave_reflink_rainbow_delalloc $blksz $nr $testdir/file3 >> $seqres.full
+
+# now reflink the rainbow
+echo "before reflink" >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file2 >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file3 >> $seqres.full
+$XFS_IO_PROG -f -c "reflink $testdir/file3 $roff $roff $rsz" $testdir/file2 >> $seqres.full
+cp $testdir/file3.chk $testdir/file2.chk
+_pwrite_byte 0x64 0 $roff $testdir/file2.chk >> $seqres.full
+_pwrite_byte 0x64 $((roff + rsz)) $((filesize - (roff + rsz) )) $testdir/file2.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+echo "after reflink" >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file2 >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file3 >> $seqres.full
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file2 | _filter_scratch
+md5sum $testdir/file2.chk | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/941.out b/tests/generic/941.out
new file mode 100644
index 00000000..a76e6c62
--- /dev/null
+++ b/tests/generic/941.out
@@ -0,0 +1,16 @@
+QA output created by 941
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-941/file1
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-941/file2
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-941/file2.chk
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-941/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-941/file3.chk
+reflink across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-941/file1
+51a300aae3a4b4eaa023876a397e01ef  SCRATCH_MNT/test-941/file2
+51a300aae3a4b4eaa023876a397e01ef  SCRATCH_MNT/test-941/file2.chk
+7bf7a779a0a54647b41753206c5218b1  SCRATCH_MNT/test-941/file3
+7bf7a779a0a54647b41753206c5218b1  SCRATCH_MNT/test-941/file3.chk
diff --git a/tests/generic/942 b/tests/generic/942
new file mode 100755
index 00000000..92b0f298
--- /dev/null
+++ b/tests/generic/942
@@ -0,0 +1,95 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (c) 2019, Oracle and/or its affiliates.  All Rights Reserved.
+#
+# FS QA Test No. 942
+#
+# Ensuring that reflinking works when the destination range covers multiple
+# extents, some unwritten, some not:
+#
+#   - Create a file with the following repeating sequence of blocks:
+#     1. reflinked
+#     2. unwritten
+#     3. hole
+#     4. regular block
+#     5. delalloc
+#   - reflink across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    cd /
+    rm -rf $tmp.* $testdir
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_supported_os Linux
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+_require_xfs_io_command "fpunch"
+_require_cp_reflink
+_require_odirect
+
+rm -f $seqres.full
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_pwrite_byte 0x64 0 $((blksz * nr)) $testdir/file2 >> $seqres.full
+_weave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file2 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "reflink across the transition"
+soff=$((filesize / 4))
+doff=$((filesize / 2))
+rsz=$((filesize / 2))
+_weave_reflink_rainbow_delalloc $blksz $nr $testdir/file3 >> $seqres.full
+
+# now reflink into the rainbow
+echo "before reflink" >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file2 >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file3 >> $seqres.full
+$XFS_IO_PROG -f -c "reflink $testdir/file2 $soff $doff $rsz" $testdir/file3 >> $seqres.full
+_pwrite_byte 0x64 $doff $rsz $testdir/file3.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+echo "after reflink" >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file2 >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file3 >> $seqres.full
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file2 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/942.out b/tests/generic/942.out
new file mode 100644
index 00000000..529ecff2
--- /dev/null
+++ b/tests/generic/942.out
@@ -0,0 +1,14 @@
+QA output created by 942
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-942/file1
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-942/file2
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-942/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-942/file3.chk
+reflink across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-942/file1
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-942/file2
+52bb341f992de6ef4bf5e5d61177eddc  SCRATCH_MNT/test-942/file3
+52bb341f992de6ef4bf5e5d61177eddc  SCRATCH_MNT/test-942/file3.chk
diff --git a/tests/generic/943 b/tests/generic/943
new file mode 100755
index 00000000..91dfa75b
--- /dev/null
+++ b/tests/generic/943
@@ -0,0 +1,99 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (c) 2019, Oracle and/or its affiliates.  All Rights Reserved.
+#
+# FS QA Test No. 943
+#
+# Ensuring that reflinking works when the source range covers multiple
+# extents, some unwritten, some not:
+#
+#   - Create a file with the following repeating sequence of blocks:
+#     1. reflinked
+#     2. unwritten
+#     3. hole
+#     4. regular block
+#     5. delalloc
+#   - reflink across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    cd /
+    rm -rf $tmp.* $testdir
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_supported_os Linux
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+_require_xfs_io_command "fpunch"
+_require_cp_reflink
+_require_odirect
+
+rm -f $seqres.full
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_pwrite_byte 0x64 0 $((blksz * nr)) $testdir/file2 >> $seqres.full
+_pwrite_byte 0x64 0 $((blksz * nr)) $testdir/file2.chk >> $seqres.full
+_weave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file2 | _filter_scratch
+md5sum $testdir/file2.chk | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "reflink across the transition"
+soff=$((filesize / 4))
+doff=$((filesize / 2))
+rsz=$((filesize / 2))
+_weave_reflink_rainbow_delalloc $blksz $nr $testdir/file3 >> $seqres.full
+
+# now reflink the rainbow
+echo "before reflink" >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file2 >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file3 >> $seqres.full
+$XFS_IO_PROG -f -c "reflink $testdir/file3 $soff $doff $rsz" $testdir/file2 >> $seqres.full
+truncate -s $doff $testdir/file2.chk
+dd if=$testdir/file3.chk skip=$((soff / blksz)) count=$((rsz / blksz)) bs=$blksz >> $testdir/file2.chk 2> /dev/null
+_scratch_cycle_mount
+
+echo "Compare files"
+echo "after reflink" >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file2 >> $seqres.full
+$FILEFRAG_PROG -v $testdir/file3 >> $seqres.full
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file2 | _filter_scratch
+md5sum $testdir/file2.chk | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/943.out b/tests/generic/943.out
new file mode 100644
index 00000000..471c3f34
--- /dev/null
+++ b/tests/generic/943.out
@@ -0,0 +1,16 @@
+QA output created by 943
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-943/file1
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-943/file2
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-943/file2.chk
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-943/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-943/file3.chk
+reflink across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-943/file1
+d93123af536c8c012f866ea383a905ce  SCRATCH_MNT/test-943/file2
+d93123af536c8c012f866ea383a905ce  SCRATCH_MNT/test-943/file2.chk
+7bf7a779a0a54647b41753206c5218b1  SCRATCH_MNT/test-943/file3
+7bf7a779a0a54647b41753206c5218b1  SCRATCH_MNT/test-943/file3.chk
diff --git a/tests/generic/944 b/tests/generic/944
new file mode 100755
index 00000000..237cd2da
--- /dev/null
+++ b/tests/generic/944
@@ -0,0 +1,196 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (c) 2019, Oracle and/or its affiliates.  All Rights Reserved.
+#
+# FS QA Test No. 944
+#
+# Ensure that we can reflink from a file with a higher inode number to a lower
+# inode number and vice versa.  Mix it up by doing this test with inodes that
+# already share blocks and inodes that don't share blocks.  This tests both
+# double-inode locking order correctness as well as stressing things like ocfs2
+# which have per-inode sharing groups and therefore have to check that we don't
+# try to link data between disjoint sharing groups.
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    cd /
+    rm -rf $tmp.* $testdir
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_supported_os Linux
+_require_scratch_reflink
+_require_cp_reflink
+
+rm -f $seqres.full
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+blksz=65536
+nr=2
+filesize=$((blksz * nr))
+testdir=$SCRATCH_MNT/test-$seq
+dummy_file=$testdir/dummy
+low_file=$testdir/low
+high_file=$testdir/high
+scenario=1
+mkdir $testdir
+
+# Return inode number
+inum() {
+	stat -c '%i' $1
+}
+
+# Create two test files, make $low_file the file with the lower inode
+# number, and make $high_file the file with the higher inode number.
+create_files() {
+	_pwrite_byte 0x60 0 $filesize $testdir/file1 >> $seqres.full
+	_pwrite_byte 0x61 0 $filesize $testdir/file2 >> $seqres.full
+	if [ "$(inum $testdir/file1)" -lt "$(inum $testdir/file2)" ]; then
+		mv $testdir/file1 $low_file
+		mv $testdir/file2 $high_file
+	else
+		mv $testdir/file2 $low_file
+		mv $testdir/file1 $high_file
+	fi
+}
+
+# Check md5sum of both files, but keep results sorted by inode order
+check_files() {
+	md5sum $low_file | _filter_scratch
+	md5sum $high_file | _filter_scratch
+}
+
+# Test reflinking data from the first file to the second file
+test_files() {
+	local src="$1"
+	local dest="$2"
+	local off=$((filesize / 2))
+	local sz=$((filesize / 2))
+
+	check_files
+	_reflink_range $src $off $dest $off $sz >> $seqres.full
+	_scratch_cycle_mount
+	check_files
+}
+
+# Make a file shared with a dummy file
+dummy_share() {
+	local which="$2"
+	test -z "$which" && which=1
+	local dummy=$dummy_file.$which
+
+	rm -f $dummy
+	_cp_reflink $1 $dummy
+}
+
+# Make two files share (different ranges) with a dummy file
+mutual_dummy_share() {
+	rm -f $dummy_file
+	_cp_reflink $1 $dummy_file
+	_reflink_range $2 0 $dummy_file $blksz $blksz >> $seqres.full
+}
+
+# Announce ourselves, remembering which scenario we've tried
+ann() {
+	echo "$scenario: $@" | tee -a $seqres.full
+	scenario=$((scenario + 1))
+}
+
+# Scenario 1: low to high, neither file shares
+ann "low to high, neither share"
+create_files
+test_files $low_file $high_file
+
+# Scenario 2: high to low, neither file shares
+ann "high to low, neither share"
+create_files
+test_files $high_file $low_file
+
+# Scenario 3: low to high, only source file shares
+ann "low to high, only source shares"
+create_files
+dummy_share $low_file
+test_files $low_file $high_file
+
+# Scenario 4: high to low, only source file shares
+ann "high to low, only source shares"
+create_files
+dummy_share $high_file
+test_files $high_file $low_file
+
+# Scenario 5: low to high, only dest file shares
+ann "low to high, only dest shares"
+create_files
+dummy_share $high_file
+test_files $low_file $high_file
+
+# Scenario 6: high to low, only dest file shares
+ann "high to low, only dest shares"
+create_files
+dummy_share $low_file
+test_files $high_file $low_file
+
+# Scenario 7: low to high, both files share with each other
+ann "low to high, both files share with each other"
+create_files
+_reflink_range $low_file 0 $high_file 0 $blksz >> $seqres.full
+test_files $low_file $high_file
+
+# Scenario 8: high to low, both files share with each other
+ann "high to low, both files share with each other"
+create_files
+_reflink_range $low_file 0 $high_file 0 $blksz >> $seqres.full
+test_files $high_file $low_file
+
+# Scenario 9: low to high, both files share but not with each other
+ann "low to high, both files share but not with each other"
+create_files
+# ocfs2 can only reflink between files sharing a refcount tree, so for
+# this test (and #10) we skip the dummy file because we'd rather not split
+# the test code just to mask off the /one/ weird fs like this...
+if _supports_arbitrary_fileset_reflink; then
+	dummy_share $low_file 1
+	dummy_share $high_file 2
+fi
+test_files $low_file $high_file
+
+# Scenario 10: high to low, both files share but not with each other
+ann "high to low, both files share but not with each other"
+create_files
+if _supports_arbitrary_fileset_reflink; then
+	dummy_share $low_file 1
+	dummy_share $high_file 2
+fi
+test_files $high_file $low_file
+
+# Scenario 11: low to high, both files share mutually
+ann "low to high, both files share mutually"
+create_files
+mutual_dummy_share $low_file $high_file
+test_files $low_file $high_file
+
+# Scenario 12: high to low, both files share mutually
+ann "high to low, both files share mutually"
+create_files
+mutual_dummy_share $low_file $high_file
+test_files $high_file $low_file
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/944.out b/tests/generic/944.out
new file mode 100644
index 00000000..aa53fdce
--- /dev/null
+++ b/tests/generic/944.out
@@ -0,0 +1,62 @@
+QA output created by 944
+Format and mount
+1: low to high, neither share
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+de1d43fbed633326daed6f71912e09e1  SCRATCH_MNT/test-944/high
+2: high to low, neither share
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
+0d2ce48b6a4527783bd30ce21f09fec0  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
+3: low to high, only source shares
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+de1d43fbed633326daed6f71912e09e1  SCRATCH_MNT/test-944/high
+4: high to low, only source shares
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
+0d2ce48b6a4527783bd30ce21f09fec0  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
+5: low to high, only dest shares
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+de1d43fbed633326daed6f71912e09e1  SCRATCH_MNT/test-944/high
+6: high to low, only dest shares
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
+0d2ce48b6a4527783bd30ce21f09fec0  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
+7: low to high, both files share with each other
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+0d2ce48b6a4527783bd30ce21f09fec0  SCRATCH_MNT/test-944/high
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/high
+8: high to low, both files share with each other
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+0d2ce48b6a4527783bd30ce21f09fec0  SCRATCH_MNT/test-944/high
+0d2ce48b6a4527783bd30ce21f09fec0  SCRATCH_MNT/test-944/low
+0d2ce48b6a4527783bd30ce21f09fec0  SCRATCH_MNT/test-944/high
+9: low to high, both files share but not with each other
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+de1d43fbed633326daed6f71912e09e1  SCRATCH_MNT/test-944/high
+10: high to low, both files share but not with each other
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
+0d2ce48b6a4527783bd30ce21f09fec0  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
+11: low to high, both files share mutually
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+de1d43fbed633326daed6f71912e09e1  SCRATCH_MNT/test-944/high
+12: high to low, both files share mutually
+07d9f5b0e07f22bff26e39f929cfc460  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
+0d2ce48b6a4527783bd30ce21f09fec0  SCRATCH_MNT/test-944/low
+81615449a98aaaad8dc179b3bec87f38  SCRATCH_MNT/test-944/high
diff --git a/tests/generic/group b/tests/generic/group
index 78b9b45d..2e4341fb 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -538,3 +538,8 @@
 533 auto quick attr
 534 auto quick log
 535 auto quick log
+940 auto quick clone punch
+941 auto quick clone punch
+942 auto quick clone punch
+943 auto quick clone punch
+944 auto quick clone punch

[PATCH 10/12] xfs: test xfs_copy and xfs_mdrestore on the populate images

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Make sure that copy, metadump, and mdrestore work on a filesystem with
all known metadata types.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/xfs/740     |   97 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/740.out |   12 +++++++
 tests/xfs/group   |    1 +
 3 files changed, 110 insertions(+)
 create mode 100755 tests/xfs/740
 create mode 100644 tests/xfs/740.out


diff --git a/tests/xfs/740 b/tests/xfs/740
new file mode 100755
index 00000000..0ff548cb
--- /dev/null
+++ b/tests/xfs/740
@@ -0,0 +1,97 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (c) 2019 Oracle, Inc.  All Rights Reserved.
+#
+# FS QA Test No. 740
+#
+# Populate a XFS filesystem and ensure that metadump, mdrestore, and copy
+# all work properly.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.* $testdir
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/populate
+
+testdir=$TEST_DIR/test-$seq
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+
+_require_scratch_nocheck
+_require_populate_commands
+
+echo "Format and populate"
+_scratch_populate_cached nofill > $seqres.full 2>&1
+
+mkdir -p $testdir
+metadump_file=$testdir/scratch.md
+metadump_file_a=${metadump_file}.a
+metadump_file_g=${metadump_file}.g
+metadump_file_ag=${metadump_file}.ag
+copy_file=$testdir/copy.img
+
+echo metadump
+_scratch_metadump $metadump_file
+
+echo metadump a
+_scratch_metadump $metadump_file_a
+
+echo metadump g
+_scratch_metadump $metadump_file_g
+
+echo metadump ag
+_scratch_metadump $metadump_file_ag
+
+echo copy
+$XFS_COPY_PROG $SCRATCH_DEV $copy_file >> $seqres.full
+_check_scratch_fs $copy_file
+
+echo recopy
+$XFS_COPY_PROG $copy_file $SCRATCH_DEV >> $seqres.full
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore
+xfs_mdrestore $metadump_file $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore a
+xfs_mdrestore $metadump_file_a $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore g
+xfs_mdrestore $metadump_file_g $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore ag
+xfs_mdrestore $metadump_file_ag $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/740.out b/tests/xfs/740.out
new file mode 100644
index 00000000..0c979dc4
--- /dev/null
+++ b/tests/xfs/740.out
@@ -0,0 +1,12 @@
+QA output created by 740
+Format and populate
+metadump
+metadump a
+metadump g
+metadump ag
+copy
+recopy
+mdrestore
+mdrestore a
+mdrestore g
+mdrestore ag
diff --git a/tests/xfs/group b/tests/xfs/group
index 9d9458b8..b8bd1012 100644
--- a/tests/xfs/group
+++ b/tests/xfs/group
@@ -500,3 +500,4 @@
 500 auto quick mkfs prealloc mkfs
 501 auto quick unlink
 502 auto quick unlink
+740 auto copy metadump

Re: [PATCH 10/12] xfs: test xfs_copy and xfs_mdrestore on the populate images

[Eryu Guan]

On Tue, Mar 19, 2019 at 05:45:49PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Make sure that copy, metadump, and mdrestore work on a filesystem with
> all known metadata types.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  tests/xfs/740     |   97 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/xfs/740.out |   12 +++++++
>  tests/xfs/group   |    1 +
>  3 files changed, 110 insertions(+)
>  create mode 100755 tests/xfs/740
>  create mode 100644 tests/xfs/740.out
> 
> 
> diff --git a/tests/xfs/740 b/tests/xfs/740
> new file mode 100755
> index 00000000..0ff548cb
> --- /dev/null
> +++ b/tests/xfs/740
> @@ -0,0 +1,97 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0+
> +# Copyright (c) 2019 Oracle, Inc.  All Rights Reserved.
> +#
> +# FS QA Test No. 740
> +#
> +# Populate a XFS filesystem and ensure that metadump, mdrestore, and copy
> +# all work properly.
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 7 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -rf $tmp.* $testdir
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/populate
> +
> +testdir=$TEST_DIR/test-$seq
> +
> +# real QA test starts here
> +_supported_os Linux
> +_supported_fs xfs
> +
> +_require_scratch_nocheck
> +_require_populate_commands
> +
> +echo "Format and populate"
> +_scratch_populate_cached nofill > $seqres.full 2>&1
> +
> +mkdir -p $testdir
> +metadump_file=$testdir/scratch.md
> +metadump_file_a=${metadump_file}.a
> +metadump_file_g=${metadump_file}.g
> +metadump_file_ag=${metadump_file}.ag
> +copy_file=$testdir/copy.img
> +
> +echo metadump
> +_scratch_metadump $metadump_file
> +
> +echo metadump a
> +_scratch_metadump $metadump_file_a
> +
> +echo metadump g
> +_scratch_metadump $metadump_file_g
> +
> +echo metadump ag
> +_scratch_metadump $metadump_file_ag

I may miss something here, but I'm a bit confused, what's the difference
between all these metadump files? Looks like scratch device is just
dumped multiple times to different dump files? And later restored
multiple times?

Thanks,
Eryu

> +
> +echo copy
> +$XFS_COPY_PROG $SCRATCH_DEV $copy_file >> $seqres.full
> +_check_scratch_fs $copy_file
> +
> +echo recopy
> +$XFS_COPY_PROG $copy_file $SCRATCH_DEV >> $seqres.full
> +_scratch_mount
> +_check_scratch_fs
> +_scratch_unmount
> +
> +echo mdrestore
> +xfs_mdrestore $metadump_file $SCRATCH_DEV
> +_scratch_mount
> +_check_scratch_fs
> +_scratch_unmount
> +
> +echo mdrestore a
> +xfs_mdrestore $metadump_file_a $SCRATCH_DEV
> +_scratch_mount
> +_check_scratch_fs
> +_scratch_unmount
> +
> +echo mdrestore g
> +xfs_mdrestore $metadump_file_g $SCRATCH_DEV
> +_scratch_mount
> +_check_scratch_fs
> +_scratch_unmount
> +
> +echo mdrestore ag
> +xfs_mdrestore $metadump_file_ag $SCRATCH_DEV
> +_scratch_mount
> +_check_scratch_fs
> +_scratch_unmount
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/xfs/740.out b/tests/xfs/740.out
> new file mode 100644
> index 00000000..0c979dc4
> --- /dev/null
> +++ b/tests/xfs/740.out
> @@ -0,0 +1,12 @@
> +QA output created by 740
> +Format and populate
> +metadump
> +metadump a
> +metadump g
> +metadump ag
> +copy
> +recopy
> +mdrestore
> +mdrestore a
> +mdrestore g
> +mdrestore ag
> diff --git a/tests/xfs/group b/tests/xfs/group
> index 9d9458b8..b8bd1012 100644
> --- a/tests/xfs/group
> +++ b/tests/xfs/group
> @@ -500,3 +500,4 @@
>  500 auto quick mkfs prealloc mkfs
>  501 auto quick unlink
>  502 auto quick unlink
> +740 auto copy metadump
> 

Re: [PATCH 10/12] xfs: test xfs_copy and xfs_mdrestore on the populate images

[Darrick J. Wong]

On Sun, Mar 24, 2019 at 10:36:53AM +0800, Eryu Guan wrote:
> On Tue, Mar 19, 2019 at 05:45:49PM -0700, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Make sure that copy, metadump, and mdrestore work on a filesystem with
> > all known metadata types.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  tests/xfs/740     |   97 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/xfs/740.out |   12 +++++++
> >  tests/xfs/group   |    1 +
> >  3 files changed, 110 insertions(+)
> >  create mode 100755 tests/xfs/740
> >  create mode 100644 tests/xfs/740.out
> > 
> > 
> > diff --git a/tests/xfs/740 b/tests/xfs/740
> > new file mode 100755
> > index 00000000..0ff548cb
> > --- /dev/null
> > +++ b/tests/xfs/740
> > @@ -0,0 +1,97 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0+
> > +# Copyright (c) 2019 Oracle, Inc.  All Rights Reserved.
> > +#
> > +# FS QA Test No. 740
> > +#
> > +# Populate a XFS filesystem and ensure that metadump, mdrestore, and copy
> > +# all work properly.
> > +#
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1	# failure is the default!
> > +trap "_cleanup; exit \$status" 0 1 2 3 7 15
> > +
> > +_cleanup()
> > +{
> > +	cd /
> > +	rm -rf $tmp.* $testdir
> > +}
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +. ./common/populate
> > +
> > +testdir=$TEST_DIR/test-$seq
> > +
> > +# real QA test starts here
> > +_supported_os Linux
> > +_supported_fs xfs
> > +
> > +_require_scratch_nocheck
> > +_require_populate_commands
> > +
> > +echo "Format and populate"
> > +_scratch_populate_cached nofill > $seqres.full 2>&1
> > +
> > +mkdir -p $testdir
> > +metadump_file=$testdir/scratch.md
> > +metadump_file_a=${metadump_file}.a
> > +metadump_file_g=${metadump_file}.g
> > +metadump_file_ag=${metadump_file}.ag
> > +copy_file=$testdir/copy.img
> > +
> > +echo metadump
> > +_scratch_metadump $metadump_file
> > +
> > +echo metadump a
> > +_scratch_metadump $metadump_file_a
> > +
> > +echo metadump g
> > +_scratch_metadump $metadump_file_g
> > +
> > +echo metadump ag
> > +_scratch_metadump $metadump_file_ag
> 
> I may miss something here, but I'm a bit confused, what's the difference
> between all these metadump files? Looks like scratch device is just
> dumped multiple times to different dump files? And later restored
> multiple times?

Bah, that was *supposed* to be a test of the different flavors of
metadump (copy entire blocks or not; obfuscate names or not) but clearly
I thinko'd the metadump commands. :(

Where is this AI programming future where I can simply express a desire
to test all the major format-altering metadump variants and the computer
will figure out what needs to be done, rather than me having to write
that idea in triplicate in the form of a log message, a file name, and a
command line switch?

It's times like these I really hate programming. <grumble>

--D

> Thanks,
> Eryu
> 
> > +
> > +echo copy
> > +$XFS_COPY_PROG $SCRATCH_DEV $copy_file >> $seqres.full
> > +_check_scratch_fs $copy_file
> > +
> > +echo recopy
> > +$XFS_COPY_PROG $copy_file $SCRATCH_DEV >> $seqres.full
> > +_scratch_mount
> > +_check_scratch_fs
> > +_scratch_unmount
> > +
> > +echo mdrestore
> > +xfs_mdrestore $metadump_file $SCRATCH_DEV
> > +_scratch_mount
> > +_check_scratch_fs
> > +_scratch_unmount
> > +
> > +echo mdrestore a
> > +xfs_mdrestore $metadump_file_a $SCRATCH_DEV
> > +_scratch_mount
> > +_check_scratch_fs
> > +_scratch_unmount
> > +
> > +echo mdrestore g
> > +xfs_mdrestore $metadump_file_g $SCRATCH_DEV
> > +_scratch_mount
> > +_check_scratch_fs
> > +_scratch_unmount
> > +
> > +echo mdrestore ag
> > +xfs_mdrestore $metadump_file_ag $SCRATCH_DEV
> > +_scratch_mount
> > +_check_scratch_fs
> > +_scratch_unmount
> > +
> > +# success, all done
> > +status=0
> > +exit
> > diff --git a/tests/xfs/740.out b/tests/xfs/740.out
> > new file mode 100644
> > index 00000000..0c979dc4
> > --- /dev/null
> > +++ b/tests/xfs/740.out
> > @@ -0,0 +1,12 @@
> > +QA output created by 740
> > +Format and populate
> > +metadump
> > +metadump a
> > +metadump g
> > +metadump ag
> > +copy
> > +recopy
> > +mdrestore
> > +mdrestore a
> > +mdrestore g
> > +mdrestore ag
> > diff --git a/tests/xfs/group b/tests/xfs/group
> > index 9d9458b8..b8bd1012 100644
> > --- a/tests/xfs/group
> > +++ b/tests/xfs/group
> > @@ -500,3 +500,4 @@
> >  500 auto quick mkfs prealloc mkfs
> >  501 auto quick unlink
> >  502 auto quick unlink
> > +740 auto copy metadump
> > 

[PATCH v2 10/12] xfs: test xfs_copy and xfs_mdrestore on the populate images

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Make sure that copy, metadump, and mdrestore work on a filesystem with
all known metadata types.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/xfs/740     |   97 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/740.out |   12 +++++++
 tests/xfs/group   |    1 +
 3 files changed, 110 insertions(+)
 create mode 100755 tests/xfs/740
 create mode 100644 tests/xfs/740.out

diff --git a/tests/xfs/740 b/tests/xfs/740
new file mode 100755
index 00000000..bd0a4577
--- /dev/null
+++ b/tests/xfs/740
@@ -0,0 +1,97 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (c) 2019 Oracle, Inc.  All Rights Reserved.
+#
+# FS QA Test No. 740
+#
+# Populate a XFS filesystem and ensure that metadump, mdrestore, and copy
+# all work properly.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.* $testdir
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/populate
+
+testdir=$TEST_DIR/test-$seq
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+
+_require_scratch_nocheck
+_require_populate_commands
+
+echo "Format and populate"
+_scratch_populate_cached nofill > $seqres.full 2>&1
+
+mkdir -p $testdir
+metadump_file=$testdir/scratch.md
+metadump_file_a=${metadump_file}.a
+metadump_file_g=${metadump_file}.g
+metadump_file_ag=${metadump_file}.ag
+copy_file=$testdir/copy.img
+
+echo metadump
+_scratch_metadump $metadump_file >> $seqres.full
+
+echo metadump a
+_scratch_metadump $metadump_file_a -a >> $seqres.full
+
+echo metadump g
+_scratch_metadump $metadump_file_g -g >> $seqres.full
+
+echo metadump ag
+_scratch_metadump $metadump_file_ag -a -g >> $seqres.full
+
+echo copy
+$XFS_COPY_PROG $SCRATCH_DEV $copy_file >> $seqres.full
+_check_scratch_fs $copy_file
+
+echo recopy
+$XFS_COPY_PROG $copy_file $SCRATCH_DEV >> $seqres.full
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore
+xfs_mdrestore $metadump_file $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore a
+xfs_mdrestore $metadump_file_a $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore g
+xfs_mdrestore $metadump_file_g $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore ag
+xfs_mdrestore $metadump_file_ag $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/740.out b/tests/xfs/740.out
new file mode 100644
index 00000000..0c979dc4
--- /dev/null
+++ b/tests/xfs/740.out
@@ -0,0 +1,12 @@
+QA output created by 740
+Format and populate
+metadump
+metadump a
+metadump g
+metadump ag
+copy
+recopy
+mdrestore
+mdrestore a
+mdrestore g
+mdrestore ag
diff --git a/tests/xfs/group b/tests/xfs/group
index 9d9458b8..b8bd1012 100644
--- a/tests/xfs/group
+++ b/tests/xfs/group
@@ -500,3 +500,4 @@
 500 auto quick mkfs prealloc mkfs
 501 auto quick unlink
 502 auto quick unlink
+740 auto copy metadump

[PATCH 11/12] check: wipe scratch devices between tests

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Wipe the scratch devices in between each test to ensure that tests are
formatting them and not making assumptions about previous contents.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 check      |    1 +
 common/rc  |    6 ++++++
 common/xfs |    1 +
 3 files changed, 8 insertions(+)


diff --git a/check b/check
index a2c5ba21..bcf08dfe 100755
--- a/check
+++ b/check
@@ -737,6 +737,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 			# _check_dmesg depends on this log in dmesg
 			touch ${RESULT_DIR}/check_dmesg
 		fi
+		_try_wipe_scratch_devs > /dev/null 2>&1
 		if [ "$DUMP_OUTPUT" = true ]; then
 			./$seq 2>&1 | tee $tmp.out
 			# Because $? would get tee's return code
diff --git a/common/rc b/common/rc
index 1c42515f..40eef80f 100644
--- a/common/rc
+++ b/common/rc
@@ -3942,6 +3942,12 @@ _require_fibmap()
 	rm -f $file
 }
 
+_try_wipe_scratch_devs()
+{
+	_scratch_unmount
+	test -x "$WIPEFS_PROG" && $WIPEFS_PROG -a $SCRATCH_DEV_POOL $SCRATCH_DEV $SCRATCH_LOGDEV $SCRATCH_RTDEV
+}
+
 init_rc
 
 ################################################################################
diff --git a/common/xfs b/common/xfs
index 24065813..af2b62ba 100644
--- a/common/xfs
+++ b/common/xfs
@@ -295,6 +295,7 @@ _require_xfs_db_command()
 	fi
 	command=$1
 
+	_scratch_mkfs_xfs >/dev/null 2>&1
 	_scratch_xfs_db -x -c "help" | grep $command > /dev/null || \
 		_notrun "xfs_db $command support is missing"
 }

Re: [PATCH 11/12] check: wipe scratch devices between tests

[Amir Goldstein]

On Wed, Mar 20, 2019 at 2:46 AM Darrick J. Wong <darrick.wong@oracle.com> wrote:
>
> From: Darrick J. Wong <darrick.wong@oracle.com>
>
> Wipe the scratch devices in between each test to ensure that tests are
> formatting them and not making assumptions about previous contents.
>
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  check      |    1 +
>  common/rc  |    6 ++++++
>  common/xfs |    1 +
>  3 files changed, 8 insertions(+)
>
>
> diff --git a/check b/check
> index a2c5ba21..bcf08dfe 100755
> --- a/check
> +++ b/check
> @@ -737,6 +737,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
>                         # _check_dmesg depends on this log in dmesg
>                         touch ${RESULT_DIR}/check_dmesg
>                 fi
> +               _try_wipe_scratch_devs > /dev/null 2>&1
>                 if [ "$DUMP_OUTPUT" = true ]; then
>                         ./$seq 2>&1 | tee $tmp.out
>                         # Because $? would get tee's return code
> diff --git a/common/rc b/common/rc
> index 1c42515f..40eef80f 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -3942,6 +3942,12 @@ _require_fibmap()
>         rm -f $file
>  }
>
> +_try_wipe_scratch_devs()
> +{
> +       _scratch_unmount

So I find this change strange, not because it doesn't make sense
to start every test with scratch unmounted, but because today scratch *is*
mounted on test start and there is quite a bit of code to make it mounted
on test start, which seems backwards.

IOW, instead of unmounting scratch just before the test runs, seems
better to make sure it is unmounted at the end of each test and before
starting the loop.

Note that for a test with _require_scratch_nocheck, _check_filesystems()
will unmount scratch, but for a test with _require_scratch, _check_scratch_fs()
will unmount+fsck+mount scratch - inconsistent.
Or am I reading this wrong?


> +       test -x "$WIPEFS_PROG" && $WIPEFS_PROG -a $SCRATCH_DEV_POOL $SCRATCH_DEV $SCRATCH_LOGDEV $SCRATCH_RTDEV

Please check that $SCRATCH_DEV is not a network name (nfs|cifs) and that it
is really a blockdev (overlayfs) before trying to wipefs.

Thanks,
Amir.

Re: [PATCH 11/12] check: wipe scratch devices between tests

[Darrick J. Wong]

On Wed, Mar 20, 2019 at 09:06:08AM +0200, Amir Goldstein wrote:
> On Wed, Mar 20, 2019 at 2:46 AM Darrick J. Wong <darrick.wong@oracle.com> wrote:
> >
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> >
> > Wipe the scratch devices in between each test to ensure that tests are
> > formatting them and not making assumptions about previous contents.
> >
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  check      |    1 +
> >  common/rc  |    6 ++++++
> >  common/xfs |    1 +
> >  3 files changed, 8 insertions(+)
> >
> >
> > diff --git a/check b/check
> > index a2c5ba21..bcf08dfe 100755
> > --- a/check
> > +++ b/check
> > @@ -737,6 +737,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
> >                         # _check_dmesg depends on this log in dmesg
> >                         touch ${RESULT_DIR}/check_dmesg
> >                 fi
> > +               _try_wipe_scratch_devs > /dev/null 2>&1
> >                 if [ "$DUMP_OUTPUT" = true ]; then
> >                         ./$seq 2>&1 | tee $tmp.out
> >                         # Because $? would get tee's return code
> > diff --git a/common/rc b/common/rc
> > index 1c42515f..40eef80f 100644
> > --- a/common/rc
> > +++ b/common/rc
> > @@ -3942,6 +3942,12 @@ _require_fibmap()
> >         rm -f $file
> >  }
> >
> > +_try_wipe_scratch_devs()
> > +{
> > +       _scratch_unmount
> 
> So I find this change strange, not because it doesn't make sense
> to start every test with scratch unmounted, but because today scratch *is*
> mounted on test start and there is quite a bit of code to make it mounted
> on test start, which seems backwards.

Huh?  On XFS the test device is always mounted, but the scratch device
is never mounted before the test starts.

Hmmm, maybe this is one of those weird things that different with
nonblock filesystems...?

> IOW, instead of unmounting scratch just before the test runs, seems
> better to make sure it is unmounted at the end of each test and before
> starting the loop.
> 
> Note that for a test with _require_scratch_nocheck, _check_filesystems()
> will unmount scratch, but for a test with _require_scratch, _check_scratch_fs()
> will unmount+fsck+mount scratch - inconsistent.
> Or am I reading this wrong?
> 
> 
> > +       test -x "$WIPEFS_PROG" && $WIPEFS_PROG -a $SCRATCH_DEV_POOL $SCRATCH_DEV $SCRATCH_LOGDEV $SCRATCH_RTDEV
> 
> Please check that $SCRATCH_DEV is not a network name (nfs|cifs) and that it
> is really a blockdev (overlayfs) before trying to wipefs.

I wonder if maybe we only bother with wipefs for filesystems where
all tests have been fixed to not stumble over unformatted scratchdev
(i.e. xfs and ext* only)

--D

> Thanks,
> Amir.

Re: [PATCH 11/12] check: wipe scratch devices between tests

[Eryu Guan]

On Wed, Mar 20, 2019 at 09:15:44PM -0700, Darrick J. Wong wrote:
> On Wed, Mar 20, 2019 at 09:06:08AM +0200, Amir Goldstein wrote:
> > On Wed, Mar 20, 2019 at 2:46 AM Darrick J. Wong <darrick.wong@oracle.com> wrote:
> > >
> > > From: Darrick J. Wong <darrick.wong@oracle.com>
> > >
> > > Wipe the scratch devices in between each test to ensure that tests are
> > > formatting them and not making assumptions about previous contents.
> > >
> > > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > > ---
> > >  check      |    1 +
> > >  common/rc  |    6 ++++++
> > >  common/xfs |    1 +
> > >  3 files changed, 8 insertions(+)
> > >
> > >
> > > diff --git a/check b/check
> > > index a2c5ba21..bcf08dfe 100755
> > > --- a/check
> > > +++ b/check
> > > @@ -737,6 +737,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
> > >                         # _check_dmesg depends on this log in dmesg
> > >                         touch ${RESULT_DIR}/check_dmesg
> > >                 fi
> > > +               _try_wipe_scratch_devs > /dev/null 2>&1
> > >                 if [ "$DUMP_OUTPUT" = true ]; then
> > >                         ./$seq 2>&1 | tee $tmp.out
> > >                         # Because $? would get tee's return code
> > > diff --git a/common/rc b/common/rc
> > > index 1c42515f..40eef80f 100644
> > > --- a/common/rc
> > > +++ b/common/rc
> > > @@ -3942,6 +3942,12 @@ _require_fibmap()
> > >         rm -f $file
> > >  }
> > >
> > > +_try_wipe_scratch_devs()
> > > +{
> > > +       _scratch_unmount
> > 
> > So I find this change strange, not because it doesn't make sense
> > to start every test with scratch unmounted, but because today scratch *is*
> > mounted on test start and there is quite a bit of code to make it mounted
> > on test start, which seems backwards.
> 
> Huh?  On XFS the test device is always mounted, but the scratch device
> is never mounted before the test starts.

Yes, as _require_test always mounts TEST_DEV and
_require_scratch_nocheck always umounts SCRATCH_DEV.

> 
> Hmmm, maybe this is one of those weird things that different with
> nonblock filesystems...?
> 
> > IOW, instead of unmounting scratch just before the test runs, seems
> > better to make sure it is unmounted at the end of each test and before
> > starting the loop.
> > 
> > Note that for a test with _require_scratch_nocheck, _check_filesystems()
> > will unmount scratch, but for a test with _require_scratch, _check_scratch_fs()
> > will unmount+fsck+mount scratch - inconsistent.
> > Or am I reading this wrong?

For tests that don't call _require_scratch{_nocheck}, to make sure
SCRATCH_DEV is umounted I think we could just call _scratch_unmount
unconditionally in _check_filesystems(), then we could wipefs before
test safely.

> > 
> > 
> > > +       test -x "$WIPEFS_PROG" && $WIPEFS_PROG -a $SCRATCH_DEV_POOL $SCRATCH_DEV $SCRATCH_LOGDEV $SCRATCH_RTDEV
> > 
> > Please check that $SCRATCH_DEV is not a network name (nfs|cifs) and that it
> > is really a blockdev (overlayfs) before trying to wipefs.

Yes, good catch!

Thanks,
Eryu

> 
> I wonder if maybe we only bother with wipefs for filesystems where
> all tests have been fixed to not stumble over unformatted scratchdev
> (i.e. xfs and ext* only)
> 
> --D
> 
> > Thanks,
> > Amir.

Re: [PATCH 11/12] check: wipe scratch devices between tests

[Darrick J. Wong]

On Sat, Mar 23, 2019 at 09:29:48PM +0800, Eryu Guan wrote:
> On Wed, Mar 20, 2019 at 09:15:44PM -0700, Darrick J. Wong wrote:
> > On Wed, Mar 20, 2019 at 09:06:08AM +0200, Amir Goldstein wrote:
> > > On Wed, Mar 20, 2019 at 2:46 AM Darrick J. Wong <darrick.wong@oracle.com> wrote:
> > > >
> > > > From: Darrick J. Wong <darrick.wong@oracle.com>
> > > >
> > > > Wipe the scratch devices in between each test to ensure that tests are
> > > > formatting them and not making assumptions about previous contents.
> > > >
> > > > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > > > ---
> > > >  check      |    1 +
> > > >  common/rc  |    6 ++++++
> > > >  common/xfs |    1 +
> > > >  3 files changed, 8 insertions(+)
> > > >
> > > >
> > > > diff --git a/check b/check
> > > > index a2c5ba21..bcf08dfe 100755
> > > > --- a/check
> > > > +++ b/check
> > > > @@ -737,6 +737,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
> > > >                         # _check_dmesg depends on this log in dmesg
> > > >                         touch ${RESULT_DIR}/check_dmesg
> > > >                 fi
> > > > +               _try_wipe_scratch_devs > /dev/null 2>&1
> > > >                 if [ "$DUMP_OUTPUT" = true ]; then
> > > >                         ./$seq 2>&1 | tee $tmp.out
> > > >                         # Because $? would get tee's return code
> > > > diff --git a/common/rc b/common/rc
> > > > index 1c42515f..40eef80f 100644
> > > > --- a/common/rc
> > > > +++ b/common/rc
> > > > @@ -3942,6 +3942,12 @@ _require_fibmap()
> > > >         rm -f $file
> > > >  }
> > > >
> > > > +_try_wipe_scratch_devs()
> > > > +{
> > > > +       _scratch_unmount
> > > 
> > > So I find this change strange, not because it doesn't make sense
> > > to start every test with scratch unmounted, but because today scratch *is*
> > > mounted on test start and there is quite a bit of code to make it mounted
> > > on test start, which seems backwards.
> > 
> > Huh?  On XFS the test device is always mounted, but the scratch device
> > is never mounted before the test starts.
> 
> Yes, as _require_test always mounts TEST_DEV and
> _require_scratch_nocheck always umounts SCRATCH_DEV.

Heh, both of you were right and I was wrong, we format and mount the
scratch device before we start the loop.

> > 
> > Hmmm, maybe this is one of those weird things that different with
> > nonblock filesystems...?
> > 
> > > IOW, instead of unmounting scratch just before the test runs, seems
> > > better to make sure it is unmounted at the end of each test and before
> > > starting the loop.
> > > 
> > > Note that for a test with _require_scratch_nocheck, _check_filesystems()
> > > will unmount scratch, but for a test with _require_scratch, _check_scratch_fs()
> > > will unmount+fsck+mount scratch - inconsistent.
> > > Or am I reading this wrong?
> 
> For tests that don't call _require_scratch{_nocheck}, to make sure
> SCRATCH_DEV is umounted I think we could just call _scratch_unmount
> unconditionally in _check_filesystems(), then we could wipefs before
> test safely.

I think it does this already:

	if [ -f ${RESULT_DIR}/require_scratch ]; then
		_check_scratch_fs || err=true
		rm -f ${RESULT_DIR}/require_scratch*
	else
		_scratch_unmount 2> /dev/null
	fi

Right?  So I think we're covered for unmounting the scratch device after
each test, and it's just the first time through the loop where we have
to unmount the scratch device.

--D

> 
> > > 
> > > 
> > > > +       test -x "$WIPEFS_PROG" && $WIPEFS_PROG -a $SCRATCH_DEV_POOL $SCRATCH_DEV $SCRATCH_LOGDEV $SCRATCH_RTDEV
> > > 
> > > Please check that $SCRATCH_DEV is not a network name (nfs|cifs) and that it
> > > is really a blockdev (overlayfs) before trying to wipefs.
> 
> Yes, good catch!
> 
> Thanks,
> Eryu
> 
> > 
> > I wonder if maybe we only bother with wipefs for filesystems where
> > all tests have been fixed to not stumble over unformatted scratchdev
> > (i.e. xfs and ext* only)
> > 
> > --D
> > 
> > > Thanks,
> > > Amir.

Re: [PATCH 11/12] check: wipe scratch devices between tests

[Amir Goldstein]

On Tue, Mar 26, 2019 at 1:48 AM Darrick J. Wong <darrick.wong@oracle.com> wrote:
>
> On Sat, Mar 23, 2019 at 09:29:48PM +0800, Eryu Guan wrote:
> > On Wed, Mar 20, 2019 at 09:15:44PM -0700, Darrick J. Wong wrote:
> > > On Wed, Mar 20, 2019 at 09:06:08AM +0200, Amir Goldstein wrote:
> > > > On Wed, Mar 20, 2019 at 2:46 AM Darrick J. Wong <darrick.wong@oracle.com> wrote:
> > > > >
> > > > > From: Darrick J. Wong <darrick.wong@oracle.com>
> > > > >
> > > > > Wipe the scratch devices in between each test to ensure that tests are
> > > > > formatting them and not making assumptions about previous contents.
> > > > >
> > > > > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > > > > ---
> > > > >  check      |    1 +
> > > > >  common/rc  |    6 ++++++
> > > > >  common/xfs |    1 +
> > > > >  3 files changed, 8 insertions(+)
> > > > >
> > > > >
> > > > > diff --git a/check b/check
> > > > > index a2c5ba21..bcf08dfe 100755
> > > > > --- a/check
> > > > > +++ b/check
> > > > > @@ -737,6 +737,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
> > > > >                         # _check_dmesg depends on this log in dmesg
> > > > >                         touch ${RESULT_DIR}/check_dmesg
> > > > >                 fi
> > > > > +               _try_wipe_scratch_devs > /dev/null 2>&1
> > > > >                 if [ "$DUMP_OUTPUT" = true ]; then
> > > > >                         ./$seq 2>&1 | tee $tmp.out
> > > > >                         # Because $? would get tee's return code
> > > > > diff --git a/common/rc b/common/rc
> > > > > index 1c42515f..40eef80f 100644
> > > > > --- a/common/rc
> > > > > +++ b/common/rc
> > > > > @@ -3942,6 +3942,12 @@ _require_fibmap()
> > > > >         rm -f $file
> > > > >  }
> > > > >
> > > > > +_try_wipe_scratch_devs()
> > > > > +{
> > > > > +       _scratch_unmount
> > > >
> > > > So I find this change strange, not because it doesn't make sense
> > > > to start every test with scratch unmounted, but because today scratch *is*
> > > > mounted on test start and there is quite a bit of code to make it mounted
> > > > on test start, which seems backwards.
> > >
> > > Huh?  On XFS the test device is always mounted, but the scratch device
> > > is never mounted before the test starts.
> >
> > Yes, as _require_test always mounts TEST_DEV and
> > _require_scratch_nocheck always umounts SCRATCH_DEV.
>
> Heh, both of you were right and I was wrong, we format and mount the
> scratch device before we start the loop.
>
> > >
> > > Hmmm, maybe this is one of those weird things that different with
> > > nonblock filesystems...?
> > >
> > > > IOW, instead of unmounting scratch just before the test runs, seems
> > > > better to make sure it is unmounted at the end of each test and before
> > > > starting the loop.
> > > >
> > > > Note that for a test with _require_scratch_nocheck, _check_filesystems()
> > > > will unmount scratch, but for a test with _require_scratch, _check_scratch_fs()
> > > > will unmount+fsck+mount scratch - inconsistent.
> > > > Or am I reading this wrong?
> >
> > For tests that don't call _require_scratch{_nocheck}, to make sure
> > SCRATCH_DEV is umounted I think we could just call _scratch_unmount
> > unconditionally in _check_filesystems(), then we could wipefs before
> > test safely.
>
> I think it does this already:
>
>         if [ -f ${RESULT_DIR}/require_scratch ]; then
>                 _check_scratch_fs || err=true
>                 rm -f ${RESULT_DIR}/require_scratch*
>         else
>                 _scratch_unmount 2> /dev/null
>         fi
>
> Right?  So I think we're covered for unmounting the scratch device after
> each test, and it's just the first time through the loop where we have
> to unmount the scratch device.
>

Right about _require_scratch_nocheck and no _require_scratch.
Test with _require_scratch will call _check_filesystems().
_check_scratch_fs() will unmount scratch; fsck; mount scratch.
So as far as I can see, its not only the first time through the loop.

Thanks,
Amir.

[PATCH 12/12] misc: fix broken _require_scratch usage

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

_require_scratch doesn't actually format the scratch device with
anything, which means that tests are required to format them before
using them.  Fix tests that don't do this correctly.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/generic/026 |    1 -
 tests/generic/100 |    3 ---
 tests/generic/506 |    2 +-
 tests/xfs/065     |    2 +-
 tests/xfs/108     |    1 +
 tests/xfs/109     |    2 +-
 tests/xfs/194     |    4 +++-
 tests/xfs/261     |    2 ++
 tests/xfs/284     |    1 +
 9 files changed, 10 insertions(+), 8 deletions(-)


diff --git a/tests/generic/026 b/tests/generic/026
index 6cbd2829..31cd0827 100755
--- a/tests/generic/026
+++ b/tests/generic/026
@@ -32,7 +32,6 @@ _cleanup()
 _supported_fs generic
 _supported_os Linux
 _require_test
-_require_scratch
 _acl_setup_ids
 _require_acls
 _require_acl_get_max
diff --git a/tests/generic/100 b/tests/generic/100
index c2593333..927c4949 100755
--- a/tests/generic/100
+++ b/tests/generic/100
@@ -52,9 +52,6 @@ _populate_fs -n $NDIRS -f $NFILES -d $DEPTH -r $POPULATED_DIR -s $SIZE >>$seqres
 # Then tar up the directory structure
 tar -cvf $TEMP_DIR/$TAR_FILE $POPULATED_DIR >>$seqres.full 2>&1
 
-# create f/s
-_require_scratch
-
 # untar on f/s
 cd $TEST_DIR
 tar -xvf $TEMP_DIR/$TAR_FILE >>$seqres.full 2>&1
diff --git a/tests/generic/506 b/tests/generic/506
index 5d419cad..7002c00c 100755
--- a/tests/generic/506
+++ b/tests/generic/506
@@ -46,12 +46,12 @@ _supported_os Linux
 _require_command "$LSATTR_PROG" lsattr
 _require_command "$CHATTR_PROG" chattr
 
-_require_prjquota $SCRATCH_DEV
 _require_scratch
 _require_scratch_shutdown
 
 _scratch_mkfs >/dev/null 2>&1
 _require_metadata_journaling $SCRATCH_DEV
+_require_prjquota $SCRATCH_DEV
 
 testfile=$SCRATCH_MNT/testfile
 
diff --git a/tests/xfs/065 b/tests/xfs/065
index c3472486..c0d1ee93 100755
--- a/tests/xfs/065
+++ b/tests/xfs/065
@@ -55,7 +55,7 @@ _supported_os Linux
 # too much hassle to get output matching with quotas turned on
 # so don't run it
 #
-_scratch_unmount 2>/dev/null
+_scratch_mkfs_xfs >> $seqres.full
 _scratch_mount
 $here/src/feature -U $SCRATCH_DEV && \
 	_notrun "UQuota enabled, test needs controlled xfsdump output"
diff --git a/tests/xfs/108 b/tests/xfs/108
index c47f4f37..e70a1f9a 100755
--- a/tests/xfs/108
+++ b/tests/xfs/108
@@ -61,6 +61,7 @@ test_accounting()
 }
 
 export MOUNT_OPTIONS="-opquota"
+_scratch_mkfs_xfs >> $seqres.full
 _qmount
 _require_prjquota $SCRATCH_DEV
 
diff --git a/tests/xfs/109 b/tests/xfs/109
index a063dc3f..df4ec157 100755
--- a/tests/xfs/109
+++ b/tests/xfs/109
@@ -75,7 +75,7 @@ allocate()
 }
 
 # real QA test starts here
-_scratch_unmount 2>/dev/null
+_scratch_mkfs_xfs >> $seqres.full
 _scratch_mount
 rm -f $seqres.full
 
diff --git a/tests/xfs/194 b/tests/xfs/194
index 6c1eddba..3e186528 100755
--- a/tests/xfs/194
+++ b/tests/xfs/194
@@ -32,6 +32,9 @@ _supported_os Linux
 # real QA test starts here
 rm -f $seqres.full
 
+_require_scratch
+_scratch_mkfs_xfs >/dev/null 2>&1
+
 # For this test we use block size = 1/8 page size
 pgsize=`$here/src/feature -s`
 blksize=`expr $pgsize / 8`
@@ -70,7 +73,6 @@ _filter_od()
     sed -e "s/^[0-9A-Fa-f ]\{7,8\}//"
 }
 
-_require_scratch
 unset MKFS_OPTIONS
 unset XFS_MKFS_OPTIONS
 
diff --git a/tests/xfs/261 b/tests/xfs/261
index 0fabcd64..92152ace 100755
--- a/tests/xfs/261
+++ b/tests/xfs/261
@@ -47,6 +47,8 @@ _supported_os Linux
 _require_quota
 _require_scratch
 
+_scratch_mkfs >> $seqres.full 2>&1
+
 # Just use the current mount table as an example mtab file.  Odds
 # are good there's nothing wrong with it.
 _setup_my_mtab() {
diff --git a/tests/xfs/284 b/tests/xfs/284
index 07f71031..7af77634 100755
--- a/tests/xfs/284
+++ b/tests/xfs/284
@@ -50,6 +50,7 @@ COPY_FILE="${TEST_DIR}/${seq}_copyfile"
 
 # Test dump a mounted device
 # xfs_metadump should refuse to dump a mounted device
+_scratch_mkfs >> $seqres.full 2>&1
 _scratch_mount
 _scratch_metadump $METADUMP_FILE 2>&1 | filter_mounted
 _scratch_unmount