* [PATCH v2 RESEND 1/5] fstests: rename _require_btrfs to _require_btrfs_subcommand
2016-02-24 8:06 [PATCH v2 RESEND 0/5] Btrfs in-band de-duplication tests cases Qu Wenruo
@ 2016-02-24 8:06 ` Qu Wenruo
2016-02-24 8:06 ` [PATCH v2 RESEND 2/5] fstests: btrfs: Add basic test for btrfs in-band de-duplication Qu Wenruo
` (3 subsequent siblings)
4 siblings, 0 replies; 13+ messages in thread
From: Qu Wenruo @ 2016-02-24 8:06 UTC (permalink / raw)
To: fstests, linux-btrfs
Rename _require_btrfs() to _require_btrfs_subcommand() to avoid
confusion, as all other _require_btrfs_* has a quite clear suffix, like
_require_btrfs_mkfs_feature() or _require_btrfs_fs_feature().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
common/rc | 2 +-
tests/btrfs/004 | 2 +-
tests/btrfs/048 | 1 +
tests/btrfs/059 | 1 +
4 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/common/rc b/common/rc
index af16c81..ff57862 100644
--- a/common/rc
+++ b/common/rc
@@ -2706,7 +2706,7 @@ _require_deletable_scratch_dev_pool()
}
# We check for btrfs and (optionally) features of the btrfs command
-_require_btrfs()
+_require_btrfs_subcommand()
{
cmd=$1
_require_command "$BTRFS_UTIL_PROG" btrfs
diff --git a/tests/btrfs/004 b/tests/btrfs/004
index 905770a..2ce628e 100755
--- a/tests/btrfs/004
+++ b/tests/btrfs/004
@@ -51,7 +51,7 @@ _supported_fs btrfs
_supported_os Linux
_require_scratch
_require_no_large_scratch_dev
-_require_btrfs inspect-internal
+_require_btrfs_subcommand inspect-internal
_require_command "/usr/sbin/filefrag" filefrag
rm -f $seqres.full
diff --git a/tests/btrfs/048 b/tests/btrfs/048
index c2cb4a6..d15346a 100755
--- a/tests/btrfs/048
+++ b/tests/btrfs/048
@@ -48,6 +48,7 @@ _supported_os Linux
_require_test
_require_scratch
_require_btrfs "property"
+_require_btrfs_subcommand "property"
send_files_dir=$TEST_DIR/btrfs-test-$seq
diff --git a/tests/btrfs/059 b/tests/btrfs/059
index b9a6ef4..6e7f7ee 100755
--- a/tests/btrfs/059
+++ b/tests/btrfs/059
@@ -51,6 +51,7 @@ _supported_os Linux
_require_test
_require_scratch
_require_btrfs "property"
+_require_btrfs_subcommand "property"
rm -f $seqres.full
--
2.7.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH v2 RESEND 2/5] fstests: btrfs: Add basic test for btrfs in-band de-duplication
2016-02-24 8:06 [PATCH v2 RESEND 0/5] Btrfs in-band de-duplication tests cases Qu Wenruo
2016-02-24 8:06 ` [PATCH v2 RESEND 1/5] fstests: rename _require_btrfs to _require_btrfs_subcommand Qu Wenruo
@ 2016-02-24 8:06 ` Qu Wenruo
2016-02-28 22:26 ` Dave Chinner
2016-02-24 8:06 ` [PATCH v2 RESEND 3/5] fstests: btrfs: Add testcase for btrfs dedup enable disable race test Qu Wenruo
` (2 subsequent siblings)
4 siblings, 1 reply; 13+ messages in thread
From: Qu Wenruo @ 2016-02-24 8:06 UTC (permalink / raw)
To: fstests, linux-btrfs
Add basic test for btrfs in-band de-duplication, including:
1) Enable
2) Re-enable
3) On disk extents are refering to same bytenr
4) Disable
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
common/defrag | 8 ++++
tests/btrfs/200 | 125 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/200.out | 19 ++++++++
tests/btrfs/group | 1 +
4 files changed, 153 insertions(+)
create mode 100755 tests/btrfs/200
create mode 100644 tests/btrfs/200.out
diff --git a/common/defrag b/common/defrag
index 942593e..34cc822 100644
--- a/common/defrag
+++ b/common/defrag
@@ -47,6 +47,14 @@ _extent_count()
$XFS_IO_PROG -c "fiemap" $1 | tail -n +2 | grep -v hole | wc -l| $AWK_PROG '{print $1}'
}
+_uniq_extent_count()
+{
+ file=$1
+ $XFS_IO_PROG -c "fiemap" $file >> $seqres.full 2>&1
+ $XFS_IO_PROG -c "fiemap" $file | tail -n +2 | grep -v hole |\
+ $AWK_PROG '{print $3}' | sort | uniq | wc -l
+}
+
_check_extent_count()
{
min=$1
diff --git a/tests/btrfs/200 b/tests/btrfs/200
new file mode 100755
index 0000000..f2ff542
--- /dev/null
+++ b/tests/btrfs/200
@@ -0,0 +1,125 @@
+#! /bin/bash
+# FS QA Test 200
+#
+# Basic btrfs inband dedup test, including:
+# 1) Enable
+# 2) Uniq file extent number
+# 3) Re-enable
+# 4) Disable
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Fujitsu. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/defrag
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_btrfs_subcommand dedup
+_require_btrfs_fs_feature dedup
+_require_btrfs_mkfs_feature dedup
+
+# File size is twice the maximum file extent of btrfs
+# So even fallbacked to non-dedup, it will have at least 2 extents
+file_size=$(( 256 * 1024 * 1024 ))
+
+_scratch_mkfs "-O dedup" >> $seqres.full 2>&1
+_scratch_mount
+
+do_dedup_test()
+{
+ backend=$1
+ dedup_bs=$2
+
+ _run_btrfs_util_prog dedup enable -s $backend -b $dedup_bs $SCRATCH_MNT
+ $XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $dedup_bs" \
+ $SCRATCH_MNT/initial_block | _filter_xfs_io
+
+ # sync to ensure dedup hash is added into dedup pool
+ sync
+ $XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $file_size" \
+ $SCRATCH_MNT/real_file | _filter_xfs_io
+ # sync again to ensure data are all written to disk and
+ # we can get stable extent map
+ sync
+
+ # Test if real_file is de-duplicated
+ nr_uniq_extents=$(_uniq_extent_count $SCRATCH_MNT/real_file)
+ nr_total_extents=$(_extent_count $SCRATCH_MNT/real_file)
+
+ echo "uniq/total: $nr_uniq_extents/$nr_total_extents" >> $seqres.full
+ # Allow a small amount of dedup miss, as commit interval or
+ # memory pressure may break a dedup_bs block and cause
+ # smalll extent which won't go through dedup routine
+ if [ $nr_uniq_extents -ge $(( $nr_total_extents * 5 / 100 )) ]; then
+ echo "Too high dedup failure rate"
+ fi
+
+ # Also check the md5sum to ensure data is not corrupted
+ md5=$(_md5_checksum $SCRATCH_MNT/real_file)
+ if [ $md5 != $init_md5 ]; then
+ echo "File after in-band de-duplication is corrupted"
+ fi
+}
+
+# Create the initial file and calculate its checksum without dedup
+$XFS_IO_PROG -f -c "pwrite 0 $file_size" $SCRATCH_MNT/csum_file | \
+ _filter_xfs_io
+init_md5=$(_md5_checksum $SCRATCH_MNT/csum_file)
+echo "md5 of the initial file is $init_md5" >> $seqres.full
+
+# Test inmemory dedup first, use 64K dedup bs to keep compatibility
+# with 64K page size
+do_dedup_test inmemory 64K
+
+# Test ondisk backend, and re-enable function
+do_dedup_test ondisk 64K
+
+# Test 128K(default) dedup bs
+do_dedup_test inmemory 128K
+do_dedup_test ondisk 128K
+
+# Check dedup disable
+_run_btrfs_util_prog dedup disable $SCRATCH_MNT
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/200.out b/tests/btrfs/200.out
new file mode 100644
index 0000000..4b185be
--- /dev/null
+++ b/tests/btrfs/200.out
@@ -0,0 +1,19 @@
+QA output created by 200
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 65536/65536 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 65536/65536 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/btrfs/group b/tests/btrfs/group
index a2fa412..0b7354b 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -119,3 +119,4 @@
116 auto quick metadata
117 auto quick send clone
118 auto quick snapshot metadata
+200 auto dedup
--
2.7.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v2 RESEND 2/5] fstests: btrfs: Add basic test for btrfs in-band de-duplication
2016-02-24 8:06 ` [PATCH v2 RESEND 2/5] fstests: btrfs: Add basic test for btrfs in-band de-duplication Qu Wenruo
@ 2016-02-28 22:26 ` Dave Chinner
2016-02-29 2:04 ` Qu Wenruo
0 siblings, 1 reply; 13+ messages in thread
From: Dave Chinner @ 2016-02-28 22:26 UTC (permalink / raw)
To: Qu Wenruo; +Cc: fstests, linux-btrfs
On Wed, Feb 24, 2016 at 04:06:33PM +0800, Qu Wenruo wrote:
> Add basic test for btrfs in-band de-duplication, including:
> 1) Enable
> 2) Re-enable
> 3) On disk extents are refering to same bytenr
> 4) Disable
>
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> ---
> common/defrag | 8 ++++
> tests/btrfs/200 | 125 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> tests/btrfs/200.out | 19 ++++++++
> tests/btrfs/group | 1 +
> 4 files changed, 153 insertions(+)
> create mode 100755 tests/btrfs/200
> create mode 100644 tests/btrfs/200.out
>
> diff --git a/common/defrag b/common/defrag
> index 942593e..34cc822 100644
> --- a/common/defrag
> +++ b/common/defrag
> @@ -47,6 +47,14 @@ _extent_count()
> $XFS_IO_PROG -c "fiemap" $1 | tail -n +2 | grep -v hole | wc -l| $AWK_PROG '{print $1}'
> }
>
> +_uniq_extent_count()
> +{
> + file=$1
> + $XFS_IO_PROG -c "fiemap" $file >> $seqres.full 2>&1
> + $XFS_IO_PROG -c "fiemap" $file | tail -n +2 | grep -v hole |\
> + $AWK_PROG '{print $3}' | sort | uniq | wc -l
> +}
This needs comments ot explain how it is different to _extent_count.
Also should probably be named _extent_count_unique()
> +
> min=$1
> diff --git a/tests/btrfs/200 b/tests/btrfs/200
> new file mode 100755
> index 0000000..f2ff542
> --- /dev/null
> +++ b/tests/btrfs/200
> @@ -0,0 +1,125 @@
> +#! /bin/bash
> +# FS QA Test 200
> +#
> +# Basic btrfs inband dedup test, including:
> +# 1) Enable
> +# 2) Uniq file extent number
Unique.
> +# 3) Re-enable
> +# 4) Disable
I don't understand what 2-4 are describing. As a test summary,
"Basic btrfs inband dedup test" is sufficient.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +_require_btrfs_subcommand dedup
> +_require_btrfs_fs_feature dedup
> +_require_btrfs_mkfs_feature dedup
> +
> +# File size is twice the maximum file extent of btrfs
> +# So even fallbacked to non-dedup, it will have at least 2 extents
> +file_size=$(( 256 * 1024 * 1024 ))
Used for xfs_io, so "file_size=256m" is all that is needed here.
> +_scratch_mkfs "-O dedup" >> $seqres.full 2>&1
> +_scratch_mount
> +
> +do_dedup_test()
> +{
> + backend=$1
> + dedup_bs=$2
> +
> + _run_btrfs_util_prog dedup enable -s $backend -b $dedup_bs $SCRATCH_MNT
> + $XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $dedup_bs" \
> + $SCRATCH_MNT/initial_block | _filter_xfs_io
> +
> + # sync to ensure dedup hash is added into dedup pool
> + sync
xfs_io -fs or xfs_io ... -c "fsync" ... ?
> + $XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $file_size" \
> + $SCRATCH_MNT/real_file | _filter_xfs_io
> + # sync again to ensure data are all written to disk and
> + # we can get stable extent map
> + sync
Again, why now just do a sync write or fsync from the xfs?
> +
> + # Test if real_file is de-duplicated
> + nr_uniq_extents=$(_uniq_extent_count $SCRATCH_MNT/real_file)
> + nr_total_extents=$(_extent_count $SCRATCH_MNT/real_file)
> +
> + echo "uniq/total: $nr_uniq_extents/$nr_total_extents" >> $seqres.full
> + # Allow a small amount of dedup miss, as commit interval or
> + # memory pressure may break a dedup_bs block and cause
> + # smalll extent which won't go through dedup routine
> + if [ $nr_uniq_extents -ge $(( $nr_total_extents * 5 / 100 )) ]; then
> + echo "Too high dedup failure rate"
> + fi
_within_tolerance
> +
> + # Also check the md5sum to ensure data is not corrupted
> + md5=$(_md5_checksum $SCRATCH_MNT/real_file)
> + if [ $md5 != $init_md5 ]; then
> + echo "File after in-band de-duplication is corrupted"
> + fi
Nope. Just echo the md5sum to the golden output file.
> +}
> +
> +# Create the initial file and calculate its checksum without dedup
> +$XFS_IO_PROG -f -c "pwrite 0 $file_size" $SCRATCH_MNT/csum_file | \
> + _filter_xfs_io
> +init_md5=$(_md5_checksum $SCRATCH_MNT/csum_file)
> +echo "md5 of the initial file is $init_md5" >> $seqres.full
Just echo the md5sum to the golden output file.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: [PATCH v2 RESEND 2/5] fstests: btrfs: Add basic test for btrfs in-band de-duplication
2016-02-28 22:26 ` Dave Chinner
@ 2016-02-29 2:04 ` Qu Wenruo
2016-02-29 6:43 ` Dave Chinner
0 siblings, 1 reply; 13+ messages in thread
From: Qu Wenruo @ 2016-02-29 2:04 UTC (permalink / raw)
To: Dave Chinner; +Cc: fstests, linux-btrfs
Hi Dave,
Thanks for the review.
All comment are correct and I'll update the patchset soon.
Only one small question below
Dave Chinner wrote on 2016/02/29 09:26 +1100:
...
>> +# File size is twice the maximum file extent of btrfs
>> +# So even fallbacked to non-dedup, it will have at least 2 extents
>> +file_size=$(( 256 * 1024 * 1024 ))
>
> Used for xfs_io, so "file_size=256m" is all that is needed here.
Super nice feature for support unit suffix, I checked man page of xfs_io
but only value for extsize mentioned the support for such suffix.
I assume all offset/length/bsize/value support suffix, right?
Hope man page get updated.
Thanks,
Qu
>
>> +_scratch_mkfs "-O dedup" >> $seqres.full 2>&1
>> +_scratch_mount
>> +
>> +do_dedup_test()
>> +{
>> + backend=$1
>> + dedup_bs=$2
>> +
>> + _run_btrfs_util_prog dedup enable -s $backend -b $dedup_bs $SCRATCH_MNT
>> + $XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $dedup_bs" \
>> + $SCRATCH_MNT/initial_block | _filter_xfs_io
>> +
>> + # sync to ensure dedup hash is added into dedup pool
>> + sync
>
> xfs_io -fs or xfs_io ... -c "fsync" ... ?
>
>> + $XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $file_size" \
>> + $SCRATCH_MNT/real_file | _filter_xfs_io
>> + # sync again to ensure data are all written to disk and
>> + # we can get stable extent map
>> + sync
>
> Again, why now just do a sync write or fsync from the xfs?
>
>> +
>> + # Test if real_file is de-duplicated
>> + nr_uniq_extents=$(_uniq_extent_count $SCRATCH_MNT/real_file)
>> + nr_total_extents=$(_extent_count $SCRATCH_MNT/real_file)
>> +
>> + echo "uniq/total: $nr_uniq_extents/$nr_total_extents" >> $seqres.full
>> + # Allow a small amount of dedup miss, as commit interval or
>> + # memory pressure may break a dedup_bs block and cause
>> + # smalll extent which won't go through dedup routine
>> + if [ $nr_uniq_extents -ge $(( $nr_total_extents * 5 / 100 )) ]; then
>> + echo "Too high dedup failure rate"
>> + fi
>
> _within_tolerance
>
>> +
>> + # Also check the md5sum to ensure data is not corrupted
>> + md5=$(_md5_checksum $SCRATCH_MNT/real_file)
>> + if [ $md5 != $init_md5 ]; then
>> + echo "File after in-band de-duplication is corrupted"
>> + fi
>
> Nope. Just echo the md5sum to the golden output file.
>
>
>> +}
>> +
>> +# Create the initial file and calculate its checksum without dedup
>> +$XFS_IO_PROG -f -c "pwrite 0 $file_size" $SCRATCH_MNT/csum_file | \
>> + _filter_xfs_io
>> +init_md5=$(_md5_checksum $SCRATCH_MNT/csum_file)
>> +echo "md5 of the initial file is $init_md5" >> $seqres.full
>
> Just echo the md5sum to the golden output file.
>
> Cheers,
>
> Dave.
>
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: [PATCH v2 RESEND 2/5] fstests: btrfs: Add basic test for btrfs in-band de-duplication
2016-02-29 2:04 ` Qu Wenruo
@ 2016-02-29 6:43 ` Dave Chinner
2016-02-29 7:55 ` Qu Wenruo
0 siblings, 1 reply; 13+ messages in thread
From: Dave Chinner @ 2016-02-29 6:43 UTC (permalink / raw)
To: Qu Wenruo; +Cc: fstests, linux-btrfs
On Mon, Feb 29, 2016 at 10:04:35AM +0800, Qu Wenruo wrote:
> Hi Dave,
>
> Thanks for the review.
>
> All comment are correct and I'll update the patchset soon.
>
> Only one small question below
>
> Dave Chinner wrote on 2016/02/29 09:26 +1100:
> ...
> >>+# File size is twice the maximum file extent of btrfs
> >>+# So even fallbacked to non-dedup, it will have at least 2 extents
> >>+file_size=$(( 256 * 1024 * 1024 ))
> >
> >Used for xfs_io, so "file_size=256m" is all that is needed here.
>
> Super nice feature for support unit suffix, I checked man page of
> xfs_io but only value for extsize mentioned the support for such
> suffix.
>
> I assume all offset/length/bsize/value support suffix, right?
Yes, they do, always have, originally came from other XFS commands
(i.e see the mkfs.xfs for the "usual units suffixes" description).
> Hope man page get updated.
Can you send a patch?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2 RESEND 2/5] fstests: btrfs: Add basic test for btrfs in-band de-duplication
2016-02-29 6:43 ` Dave Chinner
@ 2016-02-29 7:55 ` Qu Wenruo
0 siblings, 0 replies; 13+ messages in thread
From: Qu Wenruo @ 2016-02-29 7:55 UTC (permalink / raw)
To: Dave Chinner; +Cc: fstests, linux-btrfs
Dave Chinner wrote on 2016/02/29 17:43 +1100:
> On Mon, Feb 29, 2016 at 10:04:35AM +0800, Qu Wenruo wrote:
>> Hi Dave,
>>
>> Thanks for the review.
>>
>> All comment are correct and I'll update the patchset soon.
>>
>> Only one small question below
>>
>> Dave Chinner wrote on 2016/02/29 09:26 +1100:
>> ...
>>>> +# File size is twice the maximum file extent of btrfs
>>>> +# So even fallbacked to non-dedup, it will have at least 2 extents
>>>> +file_size=$(( 256 * 1024 * 1024 ))
>>>
>>> Used for xfs_io, so "file_size=256m" is all that is needed here.
>>
>> Super nice feature for support unit suffix, I checked man page of
>> xfs_io but only value for extsize mentioned the support for such
>> suffix.
>>
>> I assume all offset/length/bsize/value support suffix, right?
>
> Yes, they do, always have, originally came from other XFS commands
> (i.e see the mkfs.xfs for the "usual units suffixes" description).
>
>> Hope man page get updated.
>
> Can you send a patch?
My pleasure.
Will send it soon.
Thanks,
Qu
>
> Cheers,
>
> Dave.
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v2 RESEND 3/5] fstests: btrfs: Add testcase for btrfs dedup enable disable race test
2016-02-24 8:06 [PATCH v2 RESEND 0/5] Btrfs in-band de-duplication tests cases Qu Wenruo
2016-02-24 8:06 ` [PATCH v2 RESEND 1/5] fstests: rename _require_btrfs to _require_btrfs_subcommand Qu Wenruo
2016-02-24 8:06 ` [PATCH v2 RESEND 2/5] fstests: btrfs: Add basic test for btrfs in-band de-duplication Qu Wenruo
@ 2016-02-24 8:06 ` Qu Wenruo
2016-02-28 22:29 ` Dave Chinner
2016-02-24 8:06 ` [PATCH v2 RESEND 4/5] fstests: btrfs: Add per inode dedup flag test Qu Wenruo
2016-02-24 8:06 ` [PATCH v2 RESEND 5/5] fstests: btrfs: Test inband dedup with balance Qu Wenruo
4 siblings, 1 reply; 13+ messages in thread
From: Qu Wenruo @ 2016-02-24 8:06 UTC (permalink / raw)
To: fstests, linux-btrfs
Add test case to check btrfs dedup enable/disable race.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
tests/btrfs/201 | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/201.out | 1 +
tests/btrfs/group | 1 +
3 files changed, 102 insertions(+)
create mode 100755 tests/btrfs/201
create mode 100644 tests/btrfs/201.out
diff --git a/tests/btrfs/201 b/tests/btrfs/201
new file mode 100755
index 0000000..4bcad13
--- /dev/null
+++ b/tests/btrfs/201
@@ -0,0 +1,100 @@
+#! /bin/bash
+# FS QA Test 201
+#
+# Basic btrfs inband dedup enable/disable race test
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Fujitsu. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+ kill $trigger_work &> /dev/null
+ kill $fsstress_work &> /dev/null
+ wait
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_btrfs_subcommand dedup
+_require_btrfs_fs_feature dedup
+_require_btrfs_mkfs_feature dedup
+
+# Use 64K dedup size to keep compatibility for 64K page size
+dedup_bs=64K
+
+_scratch_mkfs "-O dedup" >> $seqres.full 2>&1
+_scratch_mount
+
+mkdir -p $SCRATCH_MNT/stressdir
+
+fsstress_work()
+{
+ $FSSTRESS_PROG $(_scale_fsstress_args -p 8 -n 5000) $FSSTRESS_AVOID \
+ -d $SCRATCH_MNT/stressdir > /dev/null 2>&1
+}
+
+trigger_work()
+{
+ while true; do
+ _run_btrfs_util_prog dedup enable -s inmemory \
+ -b $dedup_bs $SCRATCH_MNT
+ sleep 5
+ _run_btrfs_util_prog dedup disable $SCRATCH_MNT
+ sleep 5
+ _run_btrfs_util_prog dedup enable -s ondisk \
+ -b $dedup_bs $SCRATCH_MNT
+ sleep 5
+ _run_btrfs_util_prog dedup disable $SCRATCH_MNT
+ sleep 5
+ done
+}
+
+fsstress_work &
+fsstress_pid=$!
+
+trigger_work &
+trigger_pid=$!
+
+wait $fsstress_pid
+kill $trigger_pid
+wait
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/201.out b/tests/btrfs/201.out
new file mode 100644
index 0000000..b8969af
--- /dev/null
+++ b/tests/btrfs/201.out
@@ -0,0 +1 @@
+QA output created by 201
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 0b7354b..76ebea7 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -120,3 +120,4 @@
117 auto quick send clone
118 auto quick snapshot metadata
200 auto dedup
+201 auto dedup
--
2.7.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH v2 RESEND 4/5] fstests: btrfs: Add per inode dedup flag test
2016-02-24 8:06 [PATCH v2 RESEND 0/5] Btrfs in-band de-duplication tests cases Qu Wenruo
` (2 preceding siblings ...)
2016-02-24 8:06 ` [PATCH v2 RESEND 3/5] fstests: btrfs: Add testcase for btrfs dedup enable disable race test Qu Wenruo
@ 2016-02-24 8:06 ` Qu Wenruo
2016-02-28 22:39 ` Dave Chinner
2016-02-24 8:06 ` [PATCH v2 RESEND 5/5] fstests: btrfs: Test inband dedup with balance Qu Wenruo
4 siblings, 1 reply; 13+ messages in thread
From: Qu Wenruo @ 2016-02-24 8:06 UTC (permalink / raw)
To: fstests, linux-btrfs
This test will check per inode dedup flag.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
tests/btrfs/202 | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/202.out | 15 +++++++
tests/btrfs/group | 1 +
3 files changed, 132 insertions(+)
create mode 100755 tests/btrfs/202
create mode 100644 tests/btrfs/202.out
diff --git a/tests/btrfs/202 b/tests/btrfs/202
new file mode 100755
index 0000000..3e5d470
--- /dev/null
+++ b/tests/btrfs/202
@@ -0,0 +1,116 @@
+#! /bin/bash
+# FS QA Test 202
+#
+# Btrfs per inode dedup flag test
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Fujitsu. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/defrag
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_btrfs_subcommand dedup
+_require_btrfs_subcommand property
+_require_btrfs_fs_feature dedup
+_require_btrfs_mkfs_feature dedup
+
+# File size is twice the maximum file extent of btrfs
+# So even fallbacked to non-dedup, it will have at least 2 extents
+file_size=$(( 256 * 1024 * 1024 ))
+dedup_bs=$(( 64 * 1024 ))
+
+_scratch_mkfs "-O dedup" >> $seqres.full 2>&1
+_scratch_mount
+
+# Return 0 for not deduped at all , return 1 for part or full deduped
+test_file_deduped () {
+ file=$1
+
+ nr_uniq_extents=$(_uniq_extent_count $file)
+ nr_total_extents=$(_extent_count $file)
+
+ if [ $nr_uniq_extents -eq $nr_total_extents ]; then
+ echo "not de-duplicated"
+ else
+ echo "de-duplicated"
+ fi
+}
+
+dedup_write_file () {
+ file=$1
+ size=$2
+
+ $XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $size" $file | _filter_xfs_io
+}
+
+print_result () {
+ file=$1
+
+ echo "$(basename $file): $(test_file_deduped $file)"
+}
+_run_btrfs_util_prog dedup enable -b $dedup_bs $SCRATCH_MNT
+touch $SCRATCH_MNT/dedup_file
+touch $SCRATCH_MNT/no_dedup_file
+mkdir $SCRATCH_MNT/dedup_dir
+mkdir $SCRATCH_MNT/no_dedup_dir
+
+_run_btrfs_util_prog property set $SCRATCH_MNT/no_dedup_file dedup disable
+_run_btrfs_util_prog property set $SCRATCH_MNT/no_dedup_dir dedup disable
+
+dedup_write_file $SCRATCH_MNT/tmp $dedup_bs
+# sync to ensure hash is added to dedup tree
+sync
+
+dedup_write_file $SCRATCH_MNT/dedup_file $file_size
+dedup_write_file $SCRATCH_MNT/no_dedup_file $file_size
+dedup_write_file $SCRATCH_MNT/dedup_dir/dedup_dir_default_file $file_size
+dedup_write_file $SCRATCH_MNT/no_dedup_dir/no_dedup_dir_default_file $file_size
+
+print_result $SCRATCH_MNT/dedup_file
+print_result $SCRATCH_MNT/no_dedup_file
+print_result $SCRATCH_MNT/dedup_dir/dedup_dir_default_file
+print_result $SCRATCH_MNT/no_dedup_dir/no_dedup_dir_default_file
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/202.out b/tests/btrfs/202.out
new file mode 100644
index 0000000..ced9e88
--- /dev/null
+++ b/tests/btrfs/202.out
@@ -0,0 +1,15 @@
+QA output created by 202
+wrote 65536/65536 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+dedup_file: de-duplicated
+no_dedup_file: not de-duplicated
+dedup_dir_default_file: de-duplicated
+no_dedup_dir_default_file: not de-duplicated
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 76ebea7..0c03cf1 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -121,3 +121,4 @@
118 auto quick snapshot metadata
200 auto dedup
201 auto dedup
+202 auto dedup
--
2.7.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v2 RESEND 4/5] fstests: btrfs: Add per inode dedup flag test
2016-02-24 8:06 ` [PATCH v2 RESEND 4/5] fstests: btrfs: Add per inode dedup flag test Qu Wenruo
@ 2016-02-28 22:39 ` Dave Chinner
0 siblings, 0 replies; 13+ messages in thread
From: Dave Chinner @ 2016-02-28 22:39 UTC (permalink / raw)
To: Qu Wenruo; +Cc: fstests, linux-btrfs
On Wed, Feb 24, 2016 at 04:06:35PM +0800, Qu Wenruo wrote:
> +# File size is twice the maximum file extent of btrfs
> +# So even fallbacked to non-dedup, it will have at least 2 extents
> +file_size=$(( 256 * 1024 * 1024 ))
> +dedup_bs=$(( 64 * 1024 ))
256m, 64k.
> +_scratch_mkfs "-O dedup" >> $seqres.full 2>&1
> +_scratch_mount
> +
> +# Return 0 for not deduped at all , return 1 for part or full deduped
> +test_file_deduped () {
> + file=$1
> +
echo -n "$file: "
> + nr_uniq_extents=$(_uniq_extent_count $file)
> + nr_total_extents=$(_extent_count $file)
> +
> + if [ $nr_uniq_extents -eq $nr_total_extents ]; then
> + echo "not de-duplicated"
> + else
> + echo "de-duplicated"
> + fi
> +}
> +
> +dedup_write_file () {
> + file=$1
> + size=$2
> +
> + $XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $size" $file | _filter_xfs_io
> +}
> +
> +print_result () {
> + file=$1
> +
> + echo "$(basename $file): $(test_file_deduped $file)"
> +}
> +_run_btrfs_util_prog dedup enable -b $dedup_bs $SCRATCH_MNT
> +touch $SCRATCH_MNT/dedup_file
> +touch $SCRATCH_MNT/no_dedup_file
> +mkdir $SCRATCH_MNT/dedup_dir
> +mkdir $SCRATCH_MNT/no_dedup_dir
> +
> +_run_btrfs_util_prog property set $SCRATCH_MNT/no_dedup_file dedup disable
> +_run_btrfs_util_prog property set $SCRATCH_MNT/no_dedup_dir dedup disable
> +
> +dedup_write_file $SCRATCH_MNT/tmp $dedup_bs
Just call xfs_io directly - wrapper is not necessary for a single
line command.
> +# sync to ensure hash is added to dedup tree
> +sync
> +
> +dedup_write_file $SCRATCH_MNT/dedup_file $file_size
> +dedup_write_file $SCRATCH_MNT/no_dedup_file $file_size
> +dedup_write_file $SCRATCH_MNT/dedup_dir/dedup_dir_default_file $file_size
> +dedup_write_file $SCRATCH_MNT/no_dedup_dir/no_dedup_dir_default_file $file_size
> +
> +print_result $SCRATCH_MNT/dedup_file
test_file_deduped $SCRATCH_MNT/dedup_file | _filter_scratch
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v2 RESEND 5/5] fstests: btrfs: Test inband dedup with balance.
2016-02-24 8:06 [PATCH v2 RESEND 0/5] Btrfs in-band de-duplication tests cases Qu Wenruo
` (3 preceding siblings ...)
2016-02-24 8:06 ` [PATCH v2 RESEND 4/5] fstests: btrfs: Add per inode dedup flag test Qu Wenruo
@ 2016-02-24 8:06 ` Qu Wenruo
2016-02-28 22:42 ` Dave Chinner
4 siblings, 1 reply; 13+ messages in thread
From: Qu Wenruo @ 2016-02-24 8:06 UTC (permalink / raw)
To: fstests, linux-btrfs
Btrfs balance will reloate date extent, but its hash is removed too late
at run_delayed_ref() time, which will cause extent ref increased
increased during balance, cause either find_data_references() gives
WARN_ON() or even run_delayed_refs() fails and cause transaction abort.
Add such concurrency test for inband dedup and balance.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
tests/btrfs/203 | 91 +++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/203.out | 3 ++
tests/btrfs/group | 1 +
3 files changed, 95 insertions(+)
create mode 100755 tests/btrfs/203
create mode 100644 tests/btrfs/203.out
diff --git a/tests/btrfs/203 b/tests/btrfs/203
new file mode 100755
index 0000000..19dc55c
--- /dev/null
+++ b/tests/btrfs/203
@@ -0,0 +1,91 @@
+#! /bin/bash
+# FS QA Test 203
+#
+# Btrfs reflink with balance concurrency test
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Fujitsu. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ kill $balance_pid &> /dev/null
+ wait
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_cp_reflink
+_require_btrfs_subcommand dedup
+_require_btrfs_fs_feature dedup
+_require_btrfs_mkfs_feature dedup
+
+dedup_bs=$(( 128 * 1024 ))
+file=$SCRATCH_MNT/foo
+nr=20000
+
+_scratch_mkfs "-O dedup" >> $seqres.full 2>&1
+_scratch_mount
+
+_run_btrfs_util_prog dedup enable -b $dedup_bs $SCRATCH_MNT
+
+# create the initial file
+$XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $dedup_bs" $file | _filter_xfs_io
+
+# make sure hash is added into hash pool
+sync
+
+_btrfs_stress_balance $SCRATCH_MNT >/dev/null 2>&1 &
+balance_pid=$!
+
+for n in $(seq 1 $nr); do
+ $XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $dedup_bs" \
+ ${file}_${n} > /dev/null 2>&1
+done
+
+kill $balance_pid &> /dev/null
+wait
+
+# Sometimes even we killed $balance_pid and wait returned,
+# balance may still be running, use balance cancel to wait it.
+_run_btrfs_util_prog balance cancel $SCRATCH_MNT &> /dev/null
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/203.out b/tests/btrfs/203.out
new file mode 100644
index 0000000..404394c
--- /dev/null
+++ b/tests/btrfs/203.out
@@ -0,0 +1,3 @@
+QA output created by 203
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 0c03cf1..fa90f33 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -122,3 +122,4 @@
200 auto dedup
201 auto dedup
202 auto dedup
+203 auto dedup balance
--
2.7.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v2 RESEND 5/5] fstests: btrfs: Test inband dedup with balance.
2016-02-24 8:06 ` [PATCH v2 RESEND 5/5] fstests: btrfs: Test inband dedup with balance Qu Wenruo
@ 2016-02-28 22:42 ` Dave Chinner
0 siblings, 0 replies; 13+ messages in thread
From: Dave Chinner @ 2016-02-28 22:42 UTC (permalink / raw)
To: Qu Wenruo; +Cc: fstests, linux-btrfs
On Wed, Feb 24, 2016 at 04:06:36PM +0800, Qu Wenruo wrote:
> Btrfs balance will reloate date extent, but its hash is removed too late
> at run_delayed_ref() time, which will cause extent ref increased
> increased during balance, cause either find_data_references() gives
> WARN_ON() or even run_delayed_refs() fails and cause transaction abort.
>
> Add such concurrency test for inband dedup and balance.
>
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> ---
...
> +for n in $(seq 1 $nr); do
> + $XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $dedup_bs" \
> + ${file}_${n} > /dev/null 2>&1
> +done
_populate_fs(), please.
> +
> +kill $balance_pid &> /dev/null
> +wait
> +
> +# Sometimes even we killed $balance_pid and wait returned,
> +# balance may still be running, use balance cancel to wait it.
> +_run_btrfs_util_prog balance cancel $SCRATCH_MNT &> /dev/null
This needs to be in the cleanup function.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 13+ messages in thread