* [PATCH v5 2/2] generic: make 17[1-4] work well when btrfs compression is enabled
2016-11-02 4:07 [PATCH v5 1/2] common/populate: use _require_xfs_io_command() in right place Wang Xiaoguang
@ 2016-11-02 4:07 ` Wang Xiaoguang
2016-11-06 4:58 ` [PATCH v5 1/2] common/populate: use _require_xfs_io_command() in right place Eryu Guan
1 sibling, 0 replies; 3+ messages in thread
From: Wang Xiaoguang @ 2016-11-02 4:07 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs, eguan, darrick.wong
When enabling btrfs compression, original codes can not fill fs
correctly, here we introduce _fill_fs() in common/rc, which'll keep
creating and writing files until enospc error occurs. Note _fill_fs
is copied from tests/generic/256, but with some minor modifications.
Signed-off-by: Wang Xiaoguang <wangxg.fnst@cn.fujitsu.com>
---
V2: In common/, I did't find an existing function suitable for
these 4 test cases to fill fs, so I still use _pwrite_byte() with
a big enough file length fo fill fs. Note, for btrfs, metadata space
still is not full, only data space is full, but it's OK for these
4 test cases.
All these 4 cases pass in xfs and btrfs(without compression), if
btrfs has compression enabled, these 4 cases will fail for false
enospc error, I have sent kernel patches to fix this bug.
V3: Introduce _fill_fs in common/rc to fill fs.
V4: Fix some issues suggested by Eryu and Darrick.
V5: Put _fill_fs() in common/populate.
---
common/populate | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/generic/171 | 4 ++--
tests/generic/172 | 5 +++--
tests/generic/173 | 4 ++--
tests/generic/174 | 4 ++--
tests/generic/256 | 66 +++++-------------------------------------------------
6 files changed, 82 insertions(+), 68 deletions(-)
diff --git a/common/populate b/common/populate
index 3b9b531..78e9809 100644
--- a/common/populate
+++ b/common/populate
@@ -539,3 +539,70 @@ _scratch_fuzz_test() {
(find "${SCRATCH_MNT}/test.1/" -type f -size -1048576k -print0 | xargs -0 cat) >/dev/null 2>&1
}
+# Fill a file system by repeatedly creating files in the given folder
+# starting with the given file size. Files are reduced in size when
+# they can no longer fit until no more files can be created.
+_fill_fs()
+{
+ local file_size=$1
+ local dir=$2
+ local block_size=$3
+ local switch_user=$4
+ local file_count=1
+ local bytes_written=0
+ local use_falloc=1;
+
+ if [ $# -ne 4 ]; then
+ echo "Usage: _fill_fs filesize dir blocksize switch_user"
+ exit 1
+ fi
+
+ if [ $switch_user -eq 0 ]; then
+ mkdir -p $dir
+ else
+ _user_do "mkdir -p $dir"
+ fi
+ if [ ! -d $dir ]; then
+ return 0;
+ fi
+
+ testio=`$XFS_IO_PROG -F -fc "falloc 0 $block_size" $dir/$$.xfs_io 2>&1`
+ echo $testio | grep -q "not found" && use_falloc=0
+ echo $testio | grep -q "Operation not supported" && use_falloc=0
+
+ if [ $file_size -lt $block_size ]; then
+ $file_size = $block_size
+ fi
+
+ while [ $file_size -ge $block_size ]; do
+ bytes_written=0
+ if [ $switch_user -eq 0 ]; then
+ if [ $use_falloc -eq 0 ]; then
+ $XFS_IO_PROG -fc "pwrite -b 8388608 0 $file_size" \
+ $dir/$file_count
+ else
+ $XFS_IO_PROG -fc "falloc 0 $file_size" \
+ $dir/$file_count
+ fi
+ else
+ if [ $use_falloc -eq 0 ]; then
+ _user_do "$XFS_IO_PROG -f -c \"pwrite -b 8388608 0 \
+ $file_size\" $dir/$file_count"
+ else
+ _user_do "$XFS_IO_PROG -f -c \"falloc 0 \
+ $file_size\" $dir/$file_count"
+ fi
+ fi
+
+ if [ -f $dir/$file_count ]; then
+ bytes_written=$(stat -c '%s' $dir/$file_count)
+ fi
+
+ # If there was no room to make the file, then divide it in
+ # half, and keep going
+ if [ $bytes_written -lt $file_size ]; then
+ file_size=$((file_size / 2))
+ fi
+ file_count=$((file_count + 1))
+ done
+}
diff --git a/tests/generic/171 b/tests/generic/171
index a69f798..d0cd192 100755
--- a/tests/generic/171
+++ b/tests/generic/171
@@ -38,6 +38,7 @@ _cleanup()
# get standard environment, filters and checks
. ./common/rc
+. ./common/populate
. ./common/filter
. ./common/attr
. ./common/reflink
@@ -76,8 +77,7 @@ sync
echo "Allocate the rest of the space"
nr_free=$(stat -f -c '%f' $testdir)
-touch $testdir/file0 $testdir/file1
-_pwrite_byte 0x61 0 $((blksz * nr_free)) $testdir/eat_my_space >> $seqres.full 2>&1
+_fill_fs $((nr_free * blksz)) $testdir/space $blksz 0 >> $seqres.full 2>&1
sync
echo "CoW the big file"
diff --git a/tests/generic/172 b/tests/generic/172
index 8192290..d943e64 100755
--- a/tests/generic/172
+++ b/tests/generic/172
@@ -38,6 +38,7 @@ _cleanup()
# get standard environment, filters and checks
. ./common/rc
+. ./common/populate
. ./common/filter
. ./common/attr
. ./common/reflink
@@ -57,6 +58,7 @@ testdir=$SCRATCH_MNT/test-$seq
mkdir $testdir
echo "Reformat with appropriate size"
+blksz="$(get_block_size $testdir)"
umount $SCRATCH_MNT
file_size=$((168 * 1024 * 1024))
@@ -72,8 +74,7 @@ _cp_reflink $testdir/bigfile $testdir/clonefile
sync
echo "Allocate the rest of the space"
-touch $testdir/file0 $testdir/file1
-_pwrite_byte 0x61 0 $fs_size $testdir/eat_my_space >> $seqres.full 2>&1
+_fill_fs $fs_size $testdir/space $blksz 0 >> $seqres.full 2>&1
sync
echo "CoW the big file"
diff --git a/tests/generic/173 b/tests/generic/173
index e35597f..a414c8e 100755
--- a/tests/generic/173
+++ b/tests/generic/173
@@ -38,6 +38,7 @@ _cleanup()
# get standard environment, filters and checks
. ./common/rc
+. ./common/populate
. ./common/filter
. ./common/attr
. ./common/reflink
@@ -76,8 +77,7 @@ sync
echo "Allocate the rest of the space"
nr_free=$(stat -f -c '%f' $testdir)
-touch $testdir/file0 $testdir/file1
-_pwrite_byte 0x61 0 $((blksz * nr_free)) $testdir/eat_my_space >> $seqres.full 2>&1
+_fill_fs $((blksz * nr_free)) $testdir/space $blksz 0 >> $seqres.full 2>&1
sync
echo "mmap CoW the big file"
diff --git a/tests/generic/174 b/tests/generic/174
index e58d64b..b55cb72 100755
--- a/tests/generic/174
+++ b/tests/generic/174
@@ -38,6 +38,7 @@ _cleanup()
# get standard environment, filters and checks
. ./common/rc
+. ./common/populate
. ./common/filter
. ./common/attr
. ./common/reflink
@@ -77,8 +78,7 @@ sync
echo "Allocate the rest of the space"
nr_free=$(stat -f -c '%f' $testdir)
-touch $testdir/file0 $testdir/file1
-_pwrite_byte 0x61 0 $((blksz * nr_free)) $testdir/eat_my_space >> $seqres.full 2>&1
+_fill_fs $((blksz * nr_free)) $testdir/space $blksz 0 >> $seqres.full 2>&1
sync
echo "CoW the big file"
diff --git a/tests/generic/256 b/tests/generic/256
index cfbf790..63f2d4f 100755
--- a/tests/generic/256
+++ b/tests/generic/256
@@ -39,6 +39,7 @@ trap "_cleanup ; exit \$status" 0 1 2 3 15
# get standard environment, filters and checks
. ./common/rc
+. ./common/populate
. ./common/filter
. ./common/punch
@@ -53,64 +54,6 @@ _require_test
testfile=$TEST_DIR/256.$$
-# _fill_fs()
-#
-# Fills a file system by repeatedly creating files in the given folder
-# starting with the given file size. Files are reduced in size when
-# they can no longer fit untill no more files can be created.
-#
-# This routine is used by _test_full_fs_punch to test that a hole may
-# still be punched when the disk is full by borrowing reserved blocks.
-# All files are created as a non root user to prevent reserved blocks
-# from being consumed.
-#
-_fill_fs() {
- local file_size=$1
- local dir=$2
- local block_size=$3
- local file_count=1
- local bytes_written=0
-
- if [ $# -ne 3 ]
- then
- echo "USAGE: _fill_fs filesize dir block size"
- exit 1
- fi
-
- # Creation of files or folders
- # must not be done as root or
- # reserved blocks will be consumed
- _user_do "mkdir -p $dir &> /dev/null"
- if [ $? -ne 0 ] ; then
- return 0
- fi
-
- if [ $file_size -lt $block_size ]
- then
- $file_size = $block_size
- fi
-
- while [ $file_size -ge $block_size ]
- do
- bytes_written=0
- _user_do "$XFS_IO_PROG -f -c \"pwrite 0 $file_size\" $dir/$file_count.bin &> /dev/null"
-
- if [ -f $dir/$file_count.bin ]
- then
- bytes_written=`$XFS_IO_PROG -c "stat" $dir/$file_count.bin | grep stat.size | cut -d ' ' -f3`
- fi
-
- # If there was no room to make the file,
- # then divide it in half, and keep going
- if [ $bytes_written -lt $file_size ]
- then
- file_size=$(( $file_size / 2 ))
- fi
- file_count=$(( $file_count + 1 ))
-
- done
-}
-
# _test_full_fs_punch()
#
# This function will test that a hole may be punched
@@ -144,7 +87,10 @@ _test_full_fs_punch()
-c "fsync" $file_name &> /dev/null
chmod 666 $file_name
- _fill_fs $(( 1024 * 1024 * 1024 )) $path/fill $block_size
+ # All files are created as a non root user to prevent reserved blocks
+ # from being consumed.
+ _fill_fs $(( 1024 * 1024 * 1024 )) $path/fill $block_size 1 \
+ > /dev/null 2>&1
for (( i=0; i<$iterations; i++ ))
do
@@ -159,7 +105,7 @@ _test_full_fs_punch()
hole_offset=$(( $hole_offset + $hole_len + $hole_interval ))
- _fill_fs $hole_len $path/fill.$i $block_size
+ _fill_fs $hole_len $path/fill.$i $block_size 1 > /dev/null 2>&1
done
}
--
2.9.0
^ permalink raw reply related [flat|nested] 3+ messages in thread* Re: [PATCH v5 1/2] common/populate: use _require_xfs_io_command() in right place
2016-11-02 4:07 [PATCH v5 1/2] common/populate: use _require_xfs_io_command() in right place Wang Xiaoguang
2016-11-02 4:07 ` [PATCH v5 2/2] generic: make 17[1-4] work well when btrfs compression is enabled Wang Xiaoguang
@ 2016-11-06 4:58 ` Eryu Guan
1 sibling, 0 replies; 3+ messages in thread
From: Eryu Guan @ 2016-11-06 4:58 UTC (permalink / raw)
To: Wang Xiaoguang; +Cc: fstests, linux-btrfs, darrick.wong
On Wed, Nov 02, 2016 at 12:07:26PM +0800, Wang Xiaoguang wrote:
> In original common/populate codes, we put _require_xfs_io_command "falloc" and
> _require_xfs_io_command "fpunch" in the begin of common/populate, but it's
> not appropriate, for fs, which does not support falloc and punch, will not
> be able to use other helper functions in common/populate, so here I choose
> to put _require_xfs_io_command "falloc" or "punch" in helper function which
> really use falloc and fpunch.
>
> And xfs/120 uses fpunch, add _require_xfs_io_command "fpunch".
>
> Signed-off-by: Wang Xiaoguang <wangxg.fnst@cn.fujitsu.com>
Thanks for the patch! But I noticed that the new patchset posted by
Darrick recently removed "_require"s from common/populate too ([PATCH
2/9] populate: add _require_populate_commands to check for tools), and
his subsequent patches depend on this update. So I think it's better to
take Darrick's patch, rather than taking this one and forcing Darrick to
rebase his whole patchset again.
And patch 2 requires the movement of the "_require"s in common/populate,
so I'm going to merge it after reviewing & applying Darrick's patchset
(otherwise generic/256 won't run on filesystems that don't support
fallocate(2), e.g. ext2/3). That means patch 2 won't be in the pull
request this week either. Hopefully we can see it in next pr.
Thanks,
Eryu
^ permalink raw reply [flat|nested] 3+ messages in thread