* [PATCH v6 1/6] fstests: common: Introduce _post_mount_hook for btrfs
2016-09-14 1:55 [PATCH v6 0/6] Btrfs in-band de-duplication test cases Qu Wenruo
@ 2016-09-14 1:55 ` Qu Wenruo
2016-09-15 4:24 ` Dave Chinner
2016-09-14 1:55 ` [PATCH v6 2/6] fstests: common: rename _require_btrfs to _require_btrfs_subcommand Qu Wenruo
` (4 subsequent siblings)
5 siblings, 1 reply; 9+ messages in thread
From: Qu Wenruo @ 2016-09-14 1:55 UTC (permalink / raw)
To: linux-btrfs, fstests; +Cc: fdmanana, mfasheh
Introduce _post_mount_hook(), which will be executed after mounting
scratch/test.
It's quite useful for fs(OK, only btrfs yet, again) which needs to
use ioctl other than mount option to enable some of its feature.
Now only btrfs quota needs this hook to allow enabling quota to be
enabled for *ALL* existing test cases.
This should dramatically improve the test coverage to expose quota
related bugs.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
common/rc | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/common/rc b/common/rc
index 23c007a..631397f 100644
--- a/common/rc
+++ b/common/rc
@@ -321,6 +321,27 @@ _overlay_scratch_unmount()
$UMOUNT_PROG $SCRATCH_MNT
}
+_run_btrfs_post_mount_hook()
+{
+ mnt_point=$1
+ for n in $ALWAYS_ENABLE_BTRFS_FEATURE; do
+ if [ $n == "quota" -o $n == "qgroup" ]; then
+ # Quota can be enabled for several times
+ # and won't cause bug
+ _run_btrfs_util_prog quota enable $mnt_point
+ fi
+ done
+}
+
+_post_mount_hook()
+{
+ mnt_point=$1
+
+ if [ $FSTYP == "btrfs" -a -v ALWAYS_ENABLE_BTRFS_FEATURE ]; then
+ _run_btrfs_post_mount_hook $mnt_point
+ fi
+}
+
_scratch_mount()
{
if [ "$FSTYP" == "overlay" ]; then
@@ -328,6 +349,7 @@ _scratch_mount()
return $?
fi
_mount -t $FSTYP `_scratch_mount_options $*`
+ _post_mount_hook $SCRATCH_MNT
}
_scratch_unmount()
@@ -377,6 +399,7 @@ _test_mount()
fi
_test_options mount
_mount -t $FSTYP $TEST_OPTIONS $TEST_FS_MOUNT_OPTS $SELINUX_MOUNT_OPTIONS $* $TEST_DEV $TEST_DIR
+ _post_mount_hook $TEST_DIR
}
_test_unmount()
--
2.7.4
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH v6 1/6] fstests: common: Introduce _post_mount_hook for btrfs
2016-09-14 1:55 ` [PATCH v6 1/6] fstests: common: Introduce _post_mount_hook for btrfs Qu Wenruo
@ 2016-09-15 4:24 ` Dave Chinner
2016-09-19 4:08 ` Qu Wenruo
0 siblings, 1 reply; 9+ messages in thread
From: Dave Chinner @ 2016-09-15 4:24 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs, fstests, fdmanana, mfasheh
On Wed, Sep 14, 2016 at 09:55:22AM +0800, Qu Wenruo wrote:
> Introduce _post_mount_hook(), which will be executed after mounting
> scratch/test.
>
> It's quite useful for fs(OK, only btrfs yet, again) which needs to
> use ioctl other than mount option to enable some of its feature.
Just implement a _btrfs_mount() function (similar to
_overlay_mount()) to do btrfs specific things at mount time.
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> ---
> common/rc | 23 +++++++++++++++++++++++
> 1 file changed, 23 insertions(+)
>
> diff --git a/common/rc b/common/rc
> index 23c007a..631397f 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -321,6 +321,27 @@ _overlay_scratch_unmount()
> $UMOUNT_PROG $SCRATCH_MNT
> }
>
> +_run_btrfs_post_mount_hook()
> +{
> + mnt_point=$1
> + for n in $ALWAYS_ENABLE_BTRFS_FEATURE; do
What's this magic, undefined, undocumented variable?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH v6 1/6] fstests: common: Introduce _post_mount_hook for btrfs
2016-09-15 4:24 ` Dave Chinner
@ 2016-09-19 4:08 ` Qu Wenruo
0 siblings, 0 replies; 9+ messages in thread
From: Qu Wenruo @ 2016-09-19 4:08 UTC (permalink / raw)
To: Dave Chinner; +Cc: linux-btrfs, fstests, fdmanana, mfasheh
At 09/15/2016 12:24 PM, Dave Chinner wrote:
> On Wed, Sep 14, 2016 at 09:55:22AM +0800, Qu Wenruo wrote:
>> Introduce _post_mount_hook(), which will be executed after mounting
>> scratch/test.
>>
>> It's quite useful for fs(OK, only btrfs yet, again) which needs to
>> use ioctl other than mount option to enable some of its feature.
>
> Just implement a _btrfs_mount() function (similar to
> _overlay_mount()) to do btrfs specific things at mount time.
>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> ---
>> common/rc | 23 +++++++++++++++++++++++
>> 1 file changed, 23 insertions(+)
>>
>> diff --git a/common/rc b/common/rc
>> index 23c007a..631397f 100644
>> --- a/common/rc
>> +++ b/common/rc
>> @@ -321,6 +321,27 @@ _overlay_scratch_unmount()
>> $UMOUNT_PROG $SCRATCH_MNT
>> }
>>
>> +_run_btrfs_post_mount_hook()
>> +{
>> + mnt_point=$1
>> + for n in $ALWAYS_ENABLE_BTRFS_FEATURE; do
>
> What's this magic, undefined, undocumented variable?
Yes this reminds me.
The biggest problem is, where is the document of all these variables?
common/config and config/examples all lacks variables like
TIME_FACTOR/LOAD_FACTOR.
Or it's the time to doc them all?
Thanks,
Qu
>
> Cheers,
>
> Dave.
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v6 2/6] fstests: common: rename _require_btrfs to _require_btrfs_subcommand
2016-09-14 1:55 [PATCH v6 0/6] Btrfs in-band de-duplication test cases Qu Wenruo
2016-09-14 1:55 ` [PATCH v6 1/6] fstests: common: Introduce _post_mount_hook for btrfs Qu Wenruo
@ 2016-09-14 1:55 ` Qu Wenruo
2016-09-14 1:55 ` [PATCH v6 3/6] fstests: Add btrfs dedupe post mount hook Qu Wenruo
` (3 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Qu Wenruo @ 2016-09-14 1:55 UTC (permalink / raw)
To: linux-btrfs, fstests; +Cc: fdmanana, mfasheh
Rename _require_btrfs() to _require_btrfs_subcommand() to avoid
confusion, as all other _require_btrfs_* has a quite clear suffix, like
_require_btrfs_mkfs_feature() or _require_btrfs_fs_feature().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
common/rc | 2 +-
tests/btrfs/004 | 2 +-
tests/btrfs/048 | 2 +-
tests/btrfs/059 | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/common/rc b/common/rc
index 631397f..636cba6 100644
--- a/common/rc
+++ b/common/rc
@@ -3001,7 +3001,7 @@ _require_deletable_scratch_dev_pool()
}
# We check for btrfs and (optionally) features of the btrfs command
-_require_btrfs()
+_require_btrfs_subcommand()
{
cmd=$1
_require_command "$BTRFS_UTIL_PROG" btrfs
diff --git a/tests/btrfs/004 b/tests/btrfs/004
index 905770a..2ce628e 100755
--- a/tests/btrfs/004
+++ b/tests/btrfs/004
@@ -51,7 +51,7 @@ _supported_fs btrfs
_supported_os Linux
_require_scratch
_require_no_large_scratch_dev
-_require_btrfs inspect-internal
+_require_btrfs_subcommand inspect-internal
_require_command "/usr/sbin/filefrag" filefrag
rm -f $seqres.full
diff --git a/tests/btrfs/048 b/tests/btrfs/048
index 0b907b0..ac731d1 100755
--- a/tests/btrfs/048
+++ b/tests/btrfs/048
@@ -48,7 +48,7 @@ _supported_fs btrfs
_supported_os Linux
_require_test
_require_scratch
-_require_btrfs "property"
+_require_btrfs_subcommand "property"
send_files_dir=$TEST_DIR/btrfs-test-$seq
diff --git a/tests/btrfs/059 b/tests/btrfs/059
index 8f106d2..fd67ebb 100755
--- a/tests/btrfs/059
+++ b/tests/btrfs/059
@@ -51,7 +51,7 @@ _supported_fs btrfs
_supported_os Linux
_require_test
_require_scratch
-_require_btrfs "property"
+_require_btrfs_subcommand "property"
rm -f $seqres.full
--
2.7.4
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH v6 3/6] fstests: Add btrfs dedupe post mount hook
2016-09-14 1:55 [PATCH v6 0/6] Btrfs in-band de-duplication test cases Qu Wenruo
2016-09-14 1:55 ` [PATCH v6 1/6] fstests: common: Introduce _post_mount_hook for btrfs Qu Wenruo
2016-09-14 1:55 ` [PATCH v6 2/6] fstests: common: rename _require_btrfs to _require_btrfs_subcommand Qu Wenruo
@ 2016-09-14 1:55 ` Qu Wenruo
2016-09-14 1:55 ` [PATCH v6 4/6] fstests: btrfs: Add basic test for btrfs in-band de-duplication Qu Wenruo
` (2 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Qu Wenruo @ 2016-09-14 1:55 UTC (permalink / raw)
To: linux-btrfs, fstests; +Cc: fdmanana, mfasheh
Now fstests can run any test cases with btrfs inband-dedupe enabled.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
common/rc | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
diff --git a/common/rc b/common/rc
index 636cba6..e0da69b 100644
--- a/common/rc
+++ b/common/rc
@@ -326,10 +326,26 @@ _run_btrfs_post_mount_hook()
mnt_point=$1
for n in $ALWAYS_ENABLE_BTRFS_FEATURE; do
if [ $n == "quota" -o $n == "qgroup" ]; then
- # Quota can be enabled for several times
- # and won't cause bug
+ # Re-enable quota won't cause anything wrong
_run_btrfs_util_prog quota enable $mnt_point
fi
+ if [ $n == "dedupe" -o $n == "inband-dedupe" -o $n == "dedupe-inband" ]; then
+ # Need to check if kernel/progs support dedupe
+ _require_btrfs_fs_feature dedupe
+ _require_btrfs_subcommand dedupe
+
+ # Check if dedupe is already enabled, or we will
+ # overwrite previous on-disk backend config
+ $BTRFS_UTIL_PROG dedupe status $mnt_point | grep -q "Enabled"
+ if [ $1 -eq 0 ]; then
+ continue
+ fi
+
+ # This will enable dedupe with default options, so
+ # btrfs dedupe test cases needs to use '-f' option to
+ # override default config.
+ _run_btrfs_util_prog dedupe enable $mnt_point
+ fi
done
}
--
2.7.4
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH v6 4/6] fstests: btrfs: Add basic test for btrfs in-band de-duplication
2016-09-14 1:55 [PATCH v6 0/6] Btrfs in-band de-duplication test cases Qu Wenruo
` (2 preceding siblings ...)
2016-09-14 1:55 ` [PATCH v6 3/6] fstests: Add btrfs dedupe post mount hook Qu Wenruo
@ 2016-09-14 1:55 ` Qu Wenruo
2016-09-14 1:55 ` [PATCH v6 5/6] fstests: btrfs: Add testcase for btrfs dedupe and metadata balance race test Qu Wenruo
2016-09-14 1:55 ` [PATCH v6 6/6] fstests: btrfs: Test inband dedupe with data balance Qu Wenruo
5 siblings, 0 replies; 9+ messages in thread
From: Qu Wenruo @ 2016-09-14 1:55 UTC (permalink / raw)
To: linux-btrfs, fstests; +Cc: fdmanana, mfasheh
Add basic test for btrfs in-band de-duplication(inmemory backend), including:
1) Enable
3) Dedup rate
4) File correctness
5) Disable
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
common/defrag | 13 ++++++
tests/btrfs/200 | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/200.out | 22 ++++++++++
tests/btrfs/group | 1 +
4 files changed, 152 insertions(+)
create mode 100755 tests/btrfs/200
create mode 100644 tests/btrfs/200.out
diff --git a/common/defrag b/common/defrag
index 986b4bf..73c7a99 100644
--- a/common/defrag
+++ b/common/defrag
@@ -59,6 +59,19 @@ _extent_count()
$XFS_IO_PROG -c "fiemap" $1 | tail -n +2 | grep -v hole | wc -l| $AWK_PROG '{print $1}'
}
+# Get the number of unique file extents
+# Unique file extents means they have different ondisk bytenr
+# Some filesystem supports reflinkat() or in-band de-dup can create
+# a file whose all file extents points to the same ondisk bytenr
+# this can be used to test if such reflinkat() or in-band de-dup works
+_extent_count_uniq()
+{
+ file=$1
+ $XFS_IO_PROG -c "fiemap" $file >> $seqres.full 2>&1
+ $XFS_IO_PROG -c "fiemap" $file | tail -n +2 | grep -v hole |\
+ $AWK_PROG '{print $3}' | sort | uniq | wc -l
+}
+
_check_extent_count()
{
min=$1
diff --git a/tests/btrfs/200 b/tests/btrfs/200
new file mode 100755
index 0000000..5a190dd
--- /dev/null
+++ b/tests/btrfs/200
@@ -0,0 +1,116 @@
+#! /bin/bash
+# FS QA Test 200
+#
+# Basic btrfs inband dedupe test for inmemory backend
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Fujitsu. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/defrag
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_btrfs_subcommand dedupe
+_require_btrfs_fs_feature dedupe
+
+# File size is twice the maximum file extent of btrfs
+# So even fallbacked to non-dedupe, it will have at least 2 extents
+file_size=256m
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+do_dedupe_test()
+{
+ dedupe_bs=$1
+
+ echo "Testing inmemory dedupe backend with block size $dedupe_bs"
+ _run_btrfs_util_prog dedupe enable -f -s inmemory -b $dedupe_bs \
+ $SCRATCH_MNT
+ # do sync write to ensure dedupe hash is added into dedupe pool
+ $XFS_IO_PROG -f -c "pwrite -b $dedupe_bs 0 $dedupe_bs" -c "fsync"\
+ $SCRATCH_MNT/initial_block | _filter_xfs_io
+
+ # do sync write to ensure we can get stable fiemap later
+ $XFS_IO_PROG -f -c "pwrite -b $dedupe_bs 0 $file_size" -c "fsync"\
+ $SCRATCH_MNT/real_file | _filter_xfs_io
+
+ # Test if real_file is de-duplicated
+ nr_uniq_extents=$(_extent_count_uniq $SCRATCH_MNT/real_file)
+ nr_total_extents=$(_extent_count $SCRATCH_MNT/real_file)
+ nr_deduped_extents=$(($nr_total_extents - $nr_uniq_extents))
+
+ echo "deduped/total: $nr_deduped_extents/$nr_total_extents" \
+ >> $seqres.full
+ # Allow a small amount of dedupe miss, as commit interval or
+ # memory pressure may break a dedupe_bs block and cause
+ # small extent which won't go through dedupe routine
+ _within_tolerance "number of deduped extents" $nr_deduped_extents \
+ $nr_total_extents 5% -v
+
+ # Also check the md5sum to ensure data is not corrupted
+ md5=$(_md5_checksum $SCRATCH_MNT/real_file)
+ echo "md5sum: $md5"
+}
+
+# Test inmemory dedupe first, use 64K dedupe bs to keep compatibility
+# with 64K page size
+do_dedupe_test 64K
+
+# Test 128K(default) dedupe bs
+do_dedupe_test 128K
+
+# Test 1M dedupe bs
+do_dedupe_test 1M
+
+# Check dedupe disable
+_run_btrfs_util_prog dedupe disable $SCRATCH_MNT
+
+# success, all done
+status=0
+exit
+# Check dedupe disable
+_run_btrfs_util_prog dedupe disable $SCRATCH_MNT
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/200.out b/tests/btrfs/200.out
new file mode 100644
index 0000000..e09e573
--- /dev/null
+++ b/tests/btrfs/200.out
@@ -0,0 +1,22 @@
+QA output created by 200
+Testing inmemory dedupe backend with block size 64K
+wrote 65536/65536 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+number of deduped extents is in range
+md5sum: a30e0f3f1b0884081de11d4357811c2e
+Testing inmemory dedupe backend with block size 128K
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+number of deduped extents is in range
+md5sum: a30e0f3f1b0884081de11d4357811c2e
+Testing inmemory dedupe backend with block size 1M
+wrote 1048576/1048576 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+number of deduped extents is in range
+md5sum: a30e0f3f1b0884081de11d4357811c2e
diff --git a/tests/btrfs/group b/tests/btrfs/group
index f3a7a4f..ea9c36b 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -133,3 +133,4 @@
128 auto quick send
129 auto quick send
130 auto clone send
+200 auto ib-dedupe
--
2.7.4
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH v6 5/6] fstests: btrfs: Add testcase for btrfs dedupe and metadata balance race test
2016-09-14 1:55 [PATCH v6 0/6] Btrfs in-band de-duplication test cases Qu Wenruo
` (3 preceding siblings ...)
2016-09-14 1:55 ` [PATCH v6 4/6] fstests: btrfs: Add basic test for btrfs in-band de-duplication Qu Wenruo
@ 2016-09-14 1:55 ` Qu Wenruo
2016-09-14 1:55 ` [PATCH v6 6/6] fstests: btrfs: Test inband dedupe with data balance Qu Wenruo
5 siblings, 0 replies; 9+ messages in thread
From: Qu Wenruo @ 2016-09-14 1:55 UTC (permalink / raw)
To: linux-btrfs, fstests; +Cc: fdmanana, mfasheh
Btrfs balance with inband dedupe enable/disable will expose a lot of
hidden dedupe bug:
1) Enable/disable race bug
2) Btrfs dedupe tree balance corrupted delayed_ref
3) Btrfs disable and balance will cause balance BUG_ON
Reported-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
tests/btrfs/201 | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/201.out | 2 +
tests/btrfs/group | 1 +
3 files changed, 115 insertions(+)
create mode 100755 tests/btrfs/201
create mode 100644 tests/btrfs/201.out
diff --git a/tests/btrfs/201 b/tests/btrfs/201
new file mode 100755
index 0000000..152e9f7
--- /dev/null
+++ b/tests/btrfs/201
@@ -0,0 +1,112 @@
+#! /bin/bash
+# FS QA Test 201
+#
+# Btrfs inband dedup enable/disable race with metadata balance
+#
+# This tests will test the following bugs exposed in development:
+# 1) enable/disable race
+# 2) tree balance cause delayed ref corruption
+# 3) disable and balance cause BUG_ON
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Fujitsu. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+ killall $FSSTRESS_PROG &> /dev/null
+ kill $trigger_pid &> /dev/null
+ kill $balance_pid &> /dev/null
+ wait
+
+ # See comment later
+ $BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_btrfs_subcommand dedupe
+_require_btrfs_fs_feature dedupe
+
+# Use 64K dedupe size to keep compatibility for 64K page size
+dedupe_bs=64K
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+mkdir -p $SCRATCH_MNT/stressdir
+
+runtime=$((60 * $TIME_FACTOR))
+
+trigger_work()
+{
+ while true; do
+ _run_btrfs_util_prog dedupe enable -f -s inmemory \
+ -b $dedupe_bs $SCRATCH_MNT
+ sleep 1
+ _run_btrfs_util_prog dedupe disable $SCRATCH_MNT
+ sleep 1
+ done
+}
+
+# redirect all output, as error output like 'balance cancelled by user'
+# will populuate the golden output.
+_btrfs_stress_balance -m $SCRATCH_MNT &> /dev/null &
+balance_pid=$!
+
+$FSSTRESS_PROG $(_scale_fsstress_args -p 1 -n 10000000) $FSSTRESS_AVOID \
+ -d $SCRATCH_MNT/stressdir > /dev/null 2>&1 &
+
+trigger_work &
+trigger_pid=$!
+
+sleep $runtime
+killall $FSSTRESS_PROG &> /dev/null
+kill $trigger_pid &> /dev/null
+kill $balance_pid &> /dev/null
+wait
+
+# Manually stop balance as it's possible balance is still running for a short
+# time. And we don't want to populate $seqres.full, so call $BTRFS_UTIL_PROG
+# directly
+$BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
+
+echo "Silence is golden"
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/201.out b/tests/btrfs/201.out
new file mode 100644
index 0000000..5ac973f
--- /dev/null
+++ b/tests/btrfs/201.out
@@ -0,0 +1,2 @@
+QA output created by 201
+Silence is golden
diff --git a/tests/btrfs/group b/tests/btrfs/group
index ea9c36b..2e539d0 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -134,3 +134,4 @@
129 auto quick send
130 auto clone send
200 auto ib-dedupe
+201 auto ib-dedupe
--
2.7.4
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH v6 6/6] fstests: btrfs: Test inband dedupe with data balance.
2016-09-14 1:55 [PATCH v6 0/6] Btrfs in-band de-duplication test cases Qu Wenruo
` (4 preceding siblings ...)
2016-09-14 1:55 ` [PATCH v6 5/6] fstests: btrfs: Add testcase for btrfs dedupe and metadata balance race test Qu Wenruo
@ 2016-09-14 1:55 ` Qu Wenruo
5 siblings, 0 replies; 9+ messages in thread
From: Qu Wenruo @ 2016-09-14 1:55 UTC (permalink / raw)
To: linux-btrfs, fstests; +Cc: fdmanana, mfasheh
Btrfs balance will reloate date extent, but its hash is removed too late
at run_delayed_ref() time, which will cause extent ref increased
during balance, cause either find_data_references() gives WARN_ON()
or even run_delayed_refs() fails and cause transaction abort.
Add such concurrency test for inband dedupe and data balance.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
tests/btrfs/203 | 110 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/203.out | 3 ++
tests/btrfs/group | 1 +
3 files changed, 114 insertions(+)
create mode 100755 tests/btrfs/203
create mode 100644 tests/btrfs/203.out
diff --git a/tests/btrfs/203 b/tests/btrfs/203
new file mode 100755
index 0000000..96af734
--- /dev/null
+++ b/tests/btrfs/203
@@ -0,0 +1,110 @@
+#! /bin/bash
+# FS QA Test 203
+#
+# Btrfs inband dedupe with balance concurrency test
+#
+# This can spot inband dedupe error which will increase delayed ref on
+# an data extent inside RO block group
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Fujitsu. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ kill $populate_pid &> /dev/null
+ kill $balance_pid &> /dev/null
+ wait
+ # Check later comment for reason
+ $BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_cp_reflink
+_require_btrfs_subcommand dedupe
+_require_btrfs_fs_feature dedupe
+
+dedupe_bs=128k
+file_size_in_kilo=4096
+init_file=$SCRATCH_MNT/foo
+run_time=$((60 * $TIME_FACTOR))
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+do_dedupe_balance_test()
+{
+ _run_btrfs_util_prog dedupe enable -f -b $dedupe_bs -s inmemory \
+ $SCRATCH_MNT
+
+ # create the initial file and fill hash pool
+ $XFS_IO_PROG -f -c "pwrite -S 0x0 -b $dedupe_bs 0 $dedupe_bs" -c "fsync" \
+ $init_file | _filter_xfs_io
+
+ _btrfs_stress_balance $SCRATCH_MNT >/dev/null 2>&1 &
+ balance_pid=$!
+
+ # Populate fs with all 0 data, to trigger enough in-band dedupe work
+ # to race with balance
+ _populate_fs -n 5 -f 10000000 -d 1 -r $SCRATCH_MNT \
+ -s $file_size_in_kilo &> /dev/null &
+ populate_pid=$!
+
+ sleep $run_time
+
+ kill $populate_pid
+ kill $balance_pid
+ wait
+
+ # Sometimes even we killed $balance_pid and wait returned,
+ # balance may still be running, use balance cancel to wait it.
+ # As this is just a workaround, we don't want it pollute seqres
+ # so call $BTRFS_UTIL_PROG directly
+ $BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
+
+ rm $SCRATCH_MNT/* -rf &> /dev/null
+ _run_btrfs_util_prog dedupe disable $SCRATCH_MNT
+}
+
+do_dedupe_balance_test
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/203.out b/tests/btrfs/203.out
new file mode 100644
index 0000000..404394c
--- /dev/null
+++ b/tests/btrfs/203.out
@@ -0,0 +1,3 @@
+QA output created by 203
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 2e539d0..c35b919 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -135,3 +135,4 @@
130 auto clone send
200 auto ib-dedupe
201 auto ib-dedupe
+203 auto ib-dedupe balance
--
2.7.4
^ permalink raw reply related [flat|nested] 9+ messages in thread