[PATCH 0/3] fstests: various fixes

[PATCH 0/3] fstests: various fixes

[Darrick J. Wong]

Hi all,

Here are three patches fixing various regressions in xfstests when
mkfs.xfs defaults to enabling reflink and/or rmap by default.  Most of
the changes deal with the change in minimum log size requirements.  They
weren't caught until now because there are a number of tests that call
mkfs on a loop device or a file without using MKFS_OPTIONS.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes

[PATCH 1/3] xfs: refactor minimum log size formatting code

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Create a new helper function to discover the minimum log size that will
work with the mkfs options provided, then remove all the hardcoded block
sizes from various xfs tests.  This will be necessary when we turn on
reflink or rmap by default and the minimum log size increases.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 common/xfs    |   36 ++++++++++++++++++++++++++++++++++++
 tests/xfs/104 |    3 ++-
 tests/xfs/119 |    3 ++-
 tests/xfs/291 |    3 ++-
 tests/xfs/295 |    5 +++--
 tests/xfs/297 |    3 ++-
 6 files changed, 47 insertions(+), 6 deletions(-)


diff --git a/common/xfs b/common/xfs
index af2b62ba..42f02ff7 100644
--- a/common/xfs
+++ b/common/xfs
@@ -77,6 +77,42 @@ _scratch_mkfs_xfs_supported()
 	return $mkfs_status
 }
 
+# Returns the minimum XFS log size, in units of log blocks.
+_scratch_find_xfs_min_logblocks()
+{
+	local mkfs_cmd="`_scratch_mkfs_xfs_opts`"
+
+	# The smallest log size we can specify is 2M (XFS_MIN_LOG_BYTES) so
+	# pass that in and see if mkfs succeeds or tells us what is the
+	# minimum log size.
+	local XFS_MIN_LOG_BYTES=2097152
+
+	_scratch_do_mkfs "$mkfs_cmd" "cat" $* -N -l size=$XFS_MIN_LOG_BYTES \
+		2>$tmp.mkfserr 1>$tmp.mkfsstd
+	local mkfs_status=$?
+
+	# mkfs suceeded, so we must pick out the log block size to do the
+	# unit conversion
+	if [ $mkfs_status -eq 0 ]; then
+		local blksz="$(grep '^log.*bsize' $tmp.mkfsstd | \
+			sed -e 's/log.*bsize=\([0-9]*\).*$/\1/g')"
+		echo $((XFS_MIN_LOG_BYTES / blksz))
+		return
+	fi
+
+	# Usually mkfs will tell us the minimum log size...
+	if grep -q 'minimum size is' $tmp.mkfserr; then
+		grep 'minimum size is' $tmp.mkfserr | \
+			sed -e 's/^.*minimum size is \([0-9]*\) blocks/\1/g'
+		return
+	fi
+
+	# Don't know what to do, so fail
+	echo "Cannot determine minimum log size" >&2
+	cat $tmp.mkfsstd >> $seqres.full
+	cat $tmp.mkfserr >> $seqres.full
+}
+
 _scratch_mkfs_xfs()
 {
 	local mkfs_cmd="`_scratch_mkfs_xfs_opts`"
diff --git a/tests/xfs/104 b/tests/xfs/104
index bc38f969..679aced4 100755
--- a/tests/xfs/104
+++ b/tests/xfs/104
@@ -71,7 +71,8 @@ nags=4
 size=`expr 125 \* 1048576`	# 120 megabytes initially
 sizeb=`expr $size / $dbsize`	# in data blocks
 echo "*** creating scratch filesystem"
-_create_scratch -lsize=10m -dsize=${size} -dagcount=${nags}
+logblks=$(_scratch_find_xfs_min_logblocks -dsize=${size} -dagcount=${nags})
+_create_scratch -lsize=${logblks}b -dsize=${size} -dagcount=${nags}
 
 echo "*** using some initial space on scratch filesystem"
 for i in `seq 125 -1 90`; do
diff --git a/tests/xfs/119 b/tests/xfs/119
index bf7f1ca8..8825a5c3 100755
--- a/tests/xfs/119
+++ b/tests/xfs/119
@@ -38,7 +38,8 @@ _require_scratch
 # this may hang
 sync
 
-export MKFS_OPTIONS="-l version=2,size=2560b,su=64k" 
+logblks=$(_scratch_find_xfs_min_logblocks -l version=2,su=64k)
+export MKFS_OPTIONS="-l version=2,size=${logblks}b,su=64k"
 export MOUNT_OPTIONS="-o logbsize=64k"
 _scratch_mkfs_xfs >/dev/null
 
diff --git a/tests/xfs/291 b/tests/xfs/291
index 349d0cd0..8a4b1354 100755
--- a/tests/xfs/291
+++ b/tests/xfs/291
@@ -31,7 +31,8 @@ _supported_os Linux
 # real QA test starts here
 rm -f $seqres.full
 _require_scratch
-_scratch_mkfs_xfs -n size=16k -l size=10m -d size=133m >> $seqres.full 2>&1
+logblks=$(_scratch_find_xfs_min_logblocks -n size=16k -d size=133m)
+_scratch_mkfs_xfs -n size=16k -l size=${logblks}b -d size=133m >> $seqres.full 2>&1
 _scratch_mount
 
 # First we cause very badly fragmented freespace, then
diff --git a/tests/xfs/295 b/tests/xfs/295
index 7d1c8faf..65da7d65 100755
--- a/tests/xfs/295
+++ b/tests/xfs/295
@@ -36,7 +36,8 @@ _require_attrs
 
 rm -f $seqres.full
 
-_scratch_mkfs -l size=2560b >/dev/null 2>&1
+logblks=$(_scratch_find_xfs_min_logblocks)
+_scratch_mkfs -l size=${logblks}b >/dev/null 2>&1
 
 # Should yield a multiply-logged inode, thanks to xattr
 # Old logprint says this, then coredumps:
@@ -53,7 +54,7 @@ _scratch_xfs_logprint 2>&1 >> $seqres.full
 # match, not as a continued transaction.  If that happens we'll see:
 #	xfs_logprint: unknown log operation type (494e)
 
-_scratch_mkfs -l size=2560b >/dev/null 2>&1
+_scratch_mkfs -l size=${logblks}b >/dev/null 2>&1
 _scratch_mount
 for I in `seq 0 8192`; do
         echo a >> $SCRATCH_MNT/cat
diff --git a/tests/xfs/297 b/tests/xfs/297
index 1a048b4b..4f564add 100755
--- a/tests/xfs/297
+++ b/tests/xfs/297
@@ -36,7 +36,8 @@ _require_freeze
 _require_command "$KILLALL_PROG" killall
 
 rm -f $seqres.full
-_scratch_mkfs_xfs -d agcount=16,su=256k,sw=12 -l su=256k,size=5120b >/dev/null 2>&1
+logblks=$(_scratch_find_xfs_min_logblocks -d agcount=16,su=256k,sw=12 -l su=256k)
+_scratch_mkfs_xfs -d agcount=16,su=256k,sw=12 -l su=256k,size=${logblks}b >/dev/null 2>&1
 _scratch_mount
 
 STRESS_DIR="$SCRATCH_MNT/testdir"

[PATCH 2/3] xfs/216: always disable rmap and reflink when creating log size test fs

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

This test seems to check that log sizes scale up properly with the size
of the filesystem, given a carefully controlled set of mkfs parameters.
Since turning on reflink or rmap will change the minimum log size,
change the test to detect their presence and ensure they're disabled.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/xfs/216 |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)


diff --git a/tests/xfs/216 b/tests/xfs/216
index b1fd8ecc..15ff9ae1 100755
--- a/tests/xfs/216
+++ b/tests/xfs/216
@@ -37,12 +37,18 @@ _require_loop
 LOOP_DEV=$SCRATCH_MNT/test_fs
 LOOP_MNT=$SCRATCH_MNT/test_fs_dir
 
+loop_mkfs_opts=
+$MKFS_XFS_PROG 2>&1 | grep -q rmapbt && \
+	loop_mkfs_opts="$loop_mkfs_opts -m rmapbt=0"
+$MKFS_XFS_PROG 2>&1 | grep -q reflink && \
+	loop_mkfs_opts="$loop_mkfs_opts -m reflink=0"
+
 _do_mkfs()
 {
 	for i in $*; do
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
-			-d name=$LOOP_DEV,size=${i}g |grep log
+			-d name=$LOOP_DEV,size=${i}g $loop_mkfs_opts |grep log
 		mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
 		umount $LOOP_MNT > /dev/null 2>&1

[PATCH 3/3] xfs/294: calculate space to reserve for fragmentation test

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

This test requires us to fragment free space, and in part accomplishes
this by fallocating 400M of a 512M filesystem, then fallocating another
70M, and then using dd to eat remaining space.  However, it's risky to
assume the 400M figure because new features such as reflink and rmap
have per-ag metadata reservations which add to overhead.  Therefore,
reserve the 70M fragment file first, then try to fallocate 95% of the
remaining free space.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/xfs/294 |   11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)


diff --git a/tests/xfs/294 b/tests/xfs/294
index bce4d07b..af0fc124 100755
--- a/tests/xfs/294
+++ b/tests/xfs/294
@@ -70,15 +70,16 @@ for I in `seq 1 100`; do
 	touch $SCRATCH_MNT/testdir/12345678901234567890$I;
 done
 
-# Now completely fragment freespace.
-# Consume most of it:
-$XFS_IO_PROG -f -c "falloc 0 400m" $SCRATCH_MNT/fillfile ||
-	_fail "Could not allocate space"
-
 # File to fragment:
 $XFS_IO_PROG -f -c "falloc 0 70m" $SCRATCH_MNT/fragfile ||
 	_fail "Could not allocate space"
 
+# Now completely fragment freespace.
+# Consume most of it:
+space=$(stat -f -c '%f * %S * 95 / 100' $SCRATCH_MNT | $BC_PROG)
+$XFS_IO_PROG -f -c "falloc 0 $space" $SCRATCH_MNT/fillfile ||
+	_fail "Could not allocate space"
+
 df -h $SCRATCH_MNT >> $seqres.full 2>&1
 
 # Fill remaining space; let this run to failure

Re: [PATCH 0/3] fstests: various fixes

[xuyang]

on 2019/05/08 0:56, Darrick J. Wong wrote:
> Hi all,
>
> Here are three patches fixing various regressions in xfstests when
> mkfs.xfs defaults to enabling reflink and/or rmap by default.  Most of
> the changes deal with the change in minimum log size requirements.  They
> weren't caught until now because there are a number of tests that call
> mkfs on a loop device or a file without using MKFS_OPTIONS.
>
> If you're going to start using this mess, you probably ought to just
> pull from my git trees, which are linked below.
>
> This is an extraordinary way to destroy everything.  Enjoy!
> Comments and questions are, as always, welcome.
>
> --D
>
> fstests git tree:
> https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes
>
>
>
Hi

Tested-by: Yang Xu<xuyang2018.jy@cn.fujitsu.com>

Re: [PATCH 0/3] fstests: various fixes

[Eryu Guan]

On Thu, May 09, 2019 at 10:21:12AM +0800, xuyang wrote:
> on 2019/05/08 0:56, Darrick J. Wong wrote:
> > Hi all,
> > 
> > Here are three patches fixing various regressions in xfstests when
> > mkfs.xfs defaults to enabling reflink and/or rmap by default.  Most of
> > the changes deal with the change in minimum log size requirements.  They
> > weren't caught until now because there are a number of tests that call
> > mkfs on a loop device or a file without using MKFS_OPTIONS.
> > 
> > If you're going to start using this mess, you probably ought to just
> > pull from my git trees, which are linked below.
> > 
> > This is an extraordinary way to destroy everything.  Enjoy!
> > Comments and questions are, as always, welcome.
> > 
> > --D
> > 
> > fstests git tree:
> > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes
> > 
> > 
> > 
> Hi
> 
> Tested-by: Yang Xu<xuyang2018.jy@cn.fujitsu.com>

Thanks for the testing! Just want to make sure that you tested all the
three patches so that I can add your Tested-by tag too all of them?

Thanks,
Eryu

Re: [PATCH 0/3] fstests: various fixes

[xuyang]

on 2019/05/10 16:47, Eryu Guan wrote:

> On Thu, May 09, 2019 at 10:21:12AM +0800, xuyang wrote:
>> on 2019/05/08 0:56, Darrick J. Wong wrote:
>>> Hi all,
>>>
>>> Here are three patches fixing various regressions in xfstests when
>>> mkfs.xfs defaults to enabling reflink and/or rmap by default.  Most of
>>> the changes deal with the change in minimum log size requirements.  They
>>> weren't caught until now because there are a number of tests that call
>>> mkfs on a loop device or a file without using MKFS_OPTIONS.
>>>
>>> If you're going to start using this mess, you probably ought to just
>>> pull from my git trees, which are linked below.
>>>
>>> This is an extraordinary way to destroy everything.  Enjoy!
>>> Comments and questions are, as always, welcome.
>>>
>>> --D
>>>
>>> fstests git tree:
>>> https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes
>>>
>>>
>>>
>> Hi
>>
>> Tested-by: Yang Xu<xuyang2018.jy@cn.fujitsu.com>
> Thanks for the testing! Just want to make sure that you tested all the
> three patches so that I can add your Tested-by tag too all of them?
>
> Thanks,
> Eryu
Hi Eryu

Yes.  I tested all the three patches.

>
>

[PATCH 0/3] fstests: various fixes

[Darrick J. Wong]

Hi all,

The first two patches fix t_open_tmpfiles to shut down the scratch
filesystem properly by reverting a broken fix and teaching xfstests to
pass the relevant handles around.  The final patch cleans up some open
coded src/godown calls against the scratch fs to use the wrapper.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes

[PATCH 0/3] fstests: various fixes

[Darrick J. Wong]

Hi all,

The first patch reduces the likelihood that the test control process
will get OOM killed by increasing the chance that the test processes
themselves will be targeted.

The second patch teaches xfs/122 about some new structures.

The third patch converts some typedef usage.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes

[PATCH 0/3] fstests: various fixes

[Darrick J. Wong]

Hi all,

Fix some problems with the xfs min log size calculation code.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes

[PATCH 0/3] fstests: various fixes

[Darrick J. Wong]

Hi all,

Fix the mixed buffered/directio filtering function to pick up the new
locations of the iomap directio code, fix the bogus description in
xfs/504, and reduce the time generic/561 waits for duperemove to finish.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes

[PATCH 0/3] fstests: various fixes

[Darrick J. Wong]

Hi all,

Fix broken cleanup in some tests.

Fix the mixed buffered/directio filtering function to pick up the new
locations of the iomap directio code, fix generic/081's broken lvm
cleanup, and reduce the time generic/561 waits for duperemove to finish.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes