[PATCH v5 1/2] common/populate: use _require_xfs_io_command() in right place

[PATCH v5 1/2] common/populate: use _require_xfs_io_command() in right place

[Wang Xiaoguang]

In original common/populate codes, we put _require_xfs_io_command "falloc" and
_require_xfs_io_command "fpunch" in the begin of common/populate, but it's
not appropriate, for fs, which does not support falloc and punch, will not
be able to use other helper functions in common/populate, so here I choose
to put _require_xfs_io_command "falloc" or "punch" in helper function which
really use falloc and fpunch.

And xfs/120 uses fpunch, add _require_xfs_io_command "fpunch".

Signed-off-by: Wang Xiaoguang <wangxg.fnst@cn.fujitsu.com>
---
 common/populate | 7 ++++---
 tests/xfs/120   | 1 +
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/common/populate b/common/populate
index d0003c5..3b9b531 100644
--- a/common/populate
+++ b/common/populate
@@ -22,9 +22,6 @@
 #  Mountain View, CA 94043, USA, or: http://www.sgi.com
 #-----------------------------------------------------------------------
 
-_require_xfs_io_command "falloc"
-_require_xfs_io_command "fpunch"
-
 _require_xfs_db_blocktrash_z_command() {
 	test "${FSTYP}" = "xfs" || _notrun "cannot run xfs_db on ${FSTYP}"
 	$XFS_DB_PROG -x -f -c 'blocktrash -z' "${TEST_DEV}" | grep -q 'nothing on stack' || _notrun "blocktrash -z not supported"
@@ -90,6 +87,8 @@ __populate_fill_fs() {
 # types of metadata block
 _scratch_xfs_populate() {
 	_scratch_mount
+	_require_xfs_io_command "fpunch"
+
 	blksz="$(stat -f -c '%s' "${SCRATCH_MNT}")"
 	dblksz="$(xfs_info "${SCRATCH_MNT}" | grep naming.*bsize | sed -e 's/^.*bsize=//g' -e 's/\([0-9]*\).*$/\1/g')"
 	leaf_lblk="$((32 * 1073741824 / blksz))"
@@ -192,6 +191,8 @@ _scratch_xfs_populate() {
 # types of metadata block
 _scratch_ext4_populate() {
 	_scratch_mount
+	_require_xfs_io_command "fpunch"
+
 	blksz="$(stat -f -c '%s' "${SCRATCH_MNT}")"
 	dblksz="${blksz}"
 	leaf_lblk="$((32 * 1073741824 / blksz))"
diff --git a/tests/xfs/120 b/tests/xfs/120
index 3deece6..631e2f2 100755
--- a/tests/xfs/120
+++ b/tests/xfs/120
@@ -47,6 +47,7 @@ _cleanup()
 _supported_fs xfs
 _supported_os Linux
 
+_require_xfs_io_command "fpunch"
 _require_scratch
 test -n "${FORCE_FUZZ}" || _require_scratch_xfs_crc
 _require_attrs
-- 
2.9.0

[PATCH v5 2/2] generic: make 17[1-4] work well when btrfs compression is enabled

[Wang Xiaoguang]

When enabling btrfs compression, original codes can not fill fs
correctly, here we introduce _fill_fs() in common/rc, which'll keep
creating and writing files until enospc error occurs. Note _fill_fs
is copied from tests/generic/256, but with some minor modifications.

Signed-off-by: Wang Xiaoguang <wangxg.fnst@cn.fujitsu.com>
---
V2: In common/, I did't find an existing function suitable for
    these 4 test cases to fill fs, so I still use _pwrite_byte() with
    a big enough file length fo fill fs. Note, for btrfs, metadata space
    still is not full, only data space is full, but it's OK for these
    4 test cases.

    All these 4 cases pass in xfs and btrfs(without compression), if
    btrfs has compression enabled, these 4 cases will fail for false
    enospc error, I have sent kernel patches to fix this bug.

V3: Introduce  _fill_fs in common/rc to fill fs.
V4: Fix some issues suggested by Eryu and Darrick.
V5: Put _fill_fs() in common/populate.
---
 common/populate   | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/171 |  4 ++--
 tests/generic/172 |  5 +++--
 tests/generic/173 |  4 ++--
 tests/generic/174 |  4 ++--
 tests/generic/256 | 66 +++++-------------------------------------------------
 6 files changed, 82 insertions(+), 68 deletions(-)

diff --git a/common/populate b/common/populate
index 3b9b531..78e9809 100644
--- a/common/populate
+++ b/common/populate
@@ -539,3 +539,70 @@ _scratch_fuzz_test() {
 	(find "${SCRATCH_MNT}/test.1/" -type f -size -1048576k -print0 | xargs -0 cat) >/dev/null 2>&1
 }
 
+# Fill a file system by repeatedly creating files in the given folder
+# starting with the given file size.  Files are reduced in size when
+# they can no longer fit until no more files can be created.
+_fill_fs()
+{
+	local file_size=$1
+	local dir=$2
+	local block_size=$3
+	local switch_user=$4
+	local file_count=1
+	local bytes_written=0
+	local use_falloc=1;
+
+	if [ $# -ne 4 ]; then
+		echo "Usage: _fill_fs filesize dir blocksize switch_user"
+		exit 1
+	fi
+
+	if [ $switch_user -eq 0 ]; then
+		mkdir -p $dir
+	else
+		_user_do "mkdir -p $dir"
+	fi
+	if [ ! -d $dir ]; then
+		return 0;
+	fi
+
+	testio=`$XFS_IO_PROG -F -fc "falloc 0 $block_size" $dir/$$.xfs_io 2>&1`
+	echo $testio | grep -q "not found" && use_falloc=0
+	echo $testio | grep -q "Operation not supported" && use_falloc=0
+
+	if [ $file_size -lt $block_size ]; then
+		$file_size = $block_size
+	fi
+
+	while [ $file_size -ge $block_size ]; do
+		bytes_written=0
+		if [ $switch_user -eq 0 ]; then
+			if [ $use_falloc -eq 0 ]; then
+				$XFS_IO_PROG -fc "pwrite -b 8388608 0 $file_size" \
+					$dir/$file_count
+			else
+				$XFS_IO_PROG -fc "falloc 0 $file_size" \
+					$dir/$file_count
+			fi
+		else
+			if [ $use_falloc -eq 0 ]; then
+				_user_do "$XFS_IO_PROG -f -c \"pwrite -b 8388608 0 \
+					$file_size\" $dir/$file_count"
+			else
+				_user_do "$XFS_IO_PROG -f -c \"falloc 0 \
+					$file_size\" $dir/$file_count"
+			fi
+		fi
+
+		if [ -f $dir/$file_count ]; then
+			bytes_written=$(stat -c '%s' $dir/$file_count)
+		fi
+
+		# If there was no room to make the file, then divide it in
+		# half, and keep going
+		if [ $bytes_written -lt $file_size ]; then
+			file_size=$((file_size / 2))
+		fi
+		file_count=$((file_count + 1))
+	done
+}
diff --git a/tests/generic/171 b/tests/generic/171
index a69f798..d0cd192 100755
--- a/tests/generic/171
+++ b/tests/generic/171
@@ -38,6 +38,7 @@ _cleanup()
 
 # get standard environment, filters and checks
 . ./common/rc
+. ./common/populate
 . ./common/filter
 . ./common/attr
 . ./common/reflink
@@ -76,8 +77,7 @@ sync
 
 echo "Allocate the rest of the space"
 nr_free=$(stat -f -c '%f' $testdir)
-touch $testdir/file0 $testdir/file1
-_pwrite_byte 0x61 0 $((blksz * nr_free)) $testdir/eat_my_space >> $seqres.full 2>&1
+_fill_fs $((nr_free * blksz)) $testdir/space $blksz 0 >> $seqres.full 2>&1
 sync
 
 echo "CoW the big file"
diff --git a/tests/generic/172 b/tests/generic/172
index 8192290..d943e64 100755
--- a/tests/generic/172
+++ b/tests/generic/172
@@ -38,6 +38,7 @@ _cleanup()
 
 # get standard environment, filters and checks
 . ./common/rc
+. ./common/populate
 . ./common/filter
 . ./common/attr
 . ./common/reflink
@@ -57,6 +58,7 @@ testdir=$SCRATCH_MNT/test-$seq
 mkdir $testdir
 
 echo "Reformat with appropriate size"
+blksz="$(get_block_size $testdir)"
 umount $SCRATCH_MNT
 
 file_size=$((168 * 1024 * 1024))
@@ -72,8 +74,7 @@ _cp_reflink $testdir/bigfile $testdir/clonefile
 sync
 
 echo "Allocate the rest of the space"
-touch $testdir/file0 $testdir/file1
-_pwrite_byte 0x61 0 $fs_size $testdir/eat_my_space >> $seqres.full 2>&1
+_fill_fs $fs_size  $testdir/space $blksz 0 >> $seqres.full 2>&1
 sync
 
 echo "CoW the big file"
diff --git a/tests/generic/173 b/tests/generic/173
index e35597f..a414c8e 100755
--- a/tests/generic/173
+++ b/tests/generic/173
@@ -38,6 +38,7 @@ _cleanup()
 
 # get standard environment, filters and checks
 . ./common/rc
+. ./common/populate
 . ./common/filter
 . ./common/attr
 . ./common/reflink
@@ -76,8 +77,7 @@ sync
 
 echo "Allocate the rest of the space"
 nr_free=$(stat -f -c '%f' $testdir)
-touch $testdir/file0 $testdir/file1
-_pwrite_byte 0x61 0 $((blksz * nr_free)) $testdir/eat_my_space >> $seqres.full 2>&1
+_fill_fs $((blksz * nr_free)) $testdir/space $blksz 0 >> $seqres.full 2>&1
 sync
 
 echo "mmap CoW the big file"
diff --git a/tests/generic/174 b/tests/generic/174
index e58d64b..b55cb72 100755
--- a/tests/generic/174
+++ b/tests/generic/174
@@ -38,6 +38,7 @@ _cleanup()
 
 # get standard environment, filters and checks
 . ./common/rc
+. ./common/populate
 . ./common/filter
 . ./common/attr
 . ./common/reflink
@@ -77,8 +78,7 @@ sync
 
 echo "Allocate the rest of the space"
 nr_free=$(stat -f -c '%f' $testdir)
-touch $testdir/file0 $testdir/file1
-_pwrite_byte 0x61 0 $((blksz * nr_free)) $testdir/eat_my_space >> $seqres.full 2>&1
+_fill_fs $((blksz * nr_free)) $testdir/space $blksz 0 >> $seqres.full 2>&1
 sync
 
 echo "CoW the big file"
diff --git a/tests/generic/256 b/tests/generic/256
index cfbf790..63f2d4f 100755
--- a/tests/generic/256
+++ b/tests/generic/256
@@ -39,6 +39,7 @@ trap "_cleanup ; exit \$status" 0 1 2 3 15
 
 # get standard environment, filters and checks
 . ./common/rc
+. ./common/populate
 . ./common/filter
 . ./common/punch
 
@@ -53,64 +54,6 @@ _require_test
 
 testfile=$TEST_DIR/256.$$
 
-# _fill_fs()
-#
-# Fills a file system by repeatedly creating files in the given folder
-# starting with the given file size.  Files are reduced in size when
-# they can no longer fit untill no more files can be created.
-#
-# This routine is used by _test_full_fs_punch to test that a hole may
-# still be punched when the disk is full by borrowing reserved blocks.
-# All files are created as a non root user to prevent reserved blocks
-# from being consumed.
-#
-_fill_fs() {
-	local file_size=$1
-	local dir=$2
-	local block_size=$3
-	local file_count=1
-	local bytes_written=0
-
-	if [ $# -ne 3 ]
-	then
-		echo "USAGE: _fill_fs filesize dir block size"
-		exit 1
-	fi
-
-	# Creation of files or folders
-	# must not be done as root or
-	# reserved blocks will be consumed
-	_user_do "mkdir -p $dir &> /dev/null"
-	if [ $? -ne 0 ] ; then
-		return 0
-	fi
-
-	if [ $file_size -lt $block_size ]
-	then
-		$file_size = $block_size
-	fi
-
-	while [ $file_size -ge $block_size ]
-	do
-		bytes_written=0
-		_user_do "$XFS_IO_PROG -f -c \"pwrite 0 $file_size\" $dir/$file_count.bin &> /dev/null"
-
-		if [ -f $dir/$file_count.bin ]
-		then
-			bytes_written=`$XFS_IO_PROG -c "stat"  $dir/$file_count.bin | grep stat.size | cut -d ' ' -f3`
-		fi
-
-		# If there was no room to make the file,
-		# then divide it in half, and keep going
-		if [ $bytes_written -lt $file_size ]
-		then
-			file_size=$(( $file_size / 2 ))
-		fi
-		file_count=$(( $file_count + 1 ))
-
-	done
-}
-
 # _test_full_fs_punch()
 #
 # This function will test that a hole may be punched
@@ -144,7 +87,10 @@ _test_full_fs_punch()
 		-c "fsync" $file_name &> /dev/null
 	chmod 666 $file_name
 
-	_fill_fs $(( 1024 * 1024 * 1024 )) $path/fill $block_size
+	# All files are created as a non root user to prevent reserved blocks
+	# from being consumed.
+	_fill_fs $(( 1024 * 1024 * 1024 )) $path/fill $block_size 1 \
+		> /dev/null 2>&1
 
 	for (( i=0; i<$iterations; i++ ))
 	do
@@ -159,7 +105,7 @@ _test_full_fs_punch()
 
 		hole_offset=$(( $hole_offset + $hole_len + $hole_interval ))
 
-		_fill_fs $hole_len $path/fill.$i $block_size
+		_fill_fs $hole_len $path/fill.$i $block_size 1 > /dev/null 2>&1
 
 	done
 }
-- 
2.9.0

Re: [PATCH v5 1/2] common/populate: use _require_xfs_io_command() in right place

[Eryu Guan]

On Wed, Nov 02, 2016 at 12:07:26PM +0800, Wang Xiaoguang wrote:
> In original common/populate codes, we put _require_xfs_io_command "falloc" and
> _require_xfs_io_command "fpunch" in the begin of common/populate, but it's
> not appropriate, for fs, which does not support falloc and punch, will not
> be able to use other helper functions in common/populate, so here I choose
> to put _require_xfs_io_command "falloc" or "punch" in helper function which
> really use falloc and fpunch.
> 
> And xfs/120 uses fpunch, add _require_xfs_io_command "fpunch".
> 
> Signed-off-by: Wang Xiaoguang <wangxg.fnst@cn.fujitsu.com>

Thanks for the patch! But I noticed that the new patchset posted by
Darrick recently removed "_require"s from common/populate too ([PATCH
2/9] populate: add _require_populate_commands to check for tools), and
his subsequent patches depend on this update. So I think it's better to
take Darrick's patch, rather than taking this one and forcing Darrick to
rebase his whole patchset again.

And patch 2 requires the movement of the "_require"s in common/populate,
so I'm going to merge it after reviewing & applying Darrick's patchset
(otherwise generic/256 won't run on filesystems that don't support
fallocate(2), e.g. ext2/3). That means patch 2 won't be in the pull
request this week either. Hopefully we can see it in next pr.

Thanks,
Eryu