[PATCH 1/3] xfstests/btrfs: add qgroup rescan stress test

[PATCH 1/3] xfstests/btrfs: add qgroup rescan stress test

[Wang Shilong]

Test flow is to run fsstress after triggering quota rescan.
the ruler is simple, we just remove all files and directories,
sync filesystem and see if qgroup's ref and excl are nodesize.

Signed-off-by: Wang Shilong <wangsl.fnst@cn.fujitsu.com>
---
 tests/btrfs/038     | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/038.out |  3 +++
 tests/btrfs/group   |  1 +
 3 files changed, 79 insertions(+)
 create mode 100644 tests/btrfs/038
 create mode 100644 tests/btrfs/038.out

diff --git a/tests/btrfs/038 b/tests/btrfs/038
new file mode 100644
index 0000000..f6bd872
--- /dev/null
+++ b/tests/btrfs/038
@@ -0,0 +1,75 @@
+#! /bin/bash
+# FSQA Test No. btrfs/038
+#
+# Quota rescan stress test, we run fsstress and quota rescan concurrently
+#
+#-----------------------------------------------------------------------
+# Copyright (C) 2014 Fujitsu.  All rights reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1
+
+_cleanup()
+{
+       rm -f $tmp.*
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_need_to_be_root
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+
+rm -f $seqres.full
+
+run_check _scratch_mkfs "-b 2g --nodesize 4096"
+run_check _scratch_mount
+
+# -w ensures that the only ops are ones which cause write I/O
+run_check $FSSTRESS_PROG -d $SCRATCH_MNT -w -p 5 -n 1000 \
+		$FSSTRESS_AVOID >&/dev/null
+
+run_check $BTRFS_UTIL_PROG subvolume snapshot $SCRATCH_MNT \
+       $SCRATCH_MNT/snap1 >>$seqres.full 2>&1
+
+run_check $FSSTRESS_PROG -d $SCRATCH_MNT/snap1 -w -p 5 -n 1000 \
+       $FSSTRESS_AVOID >&/dev/null
+
+run_check $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT
+run_check $BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT
+
+#ignore removing subvolume errors
+rm -rf $SCRATCH_MNT/* >& /dev/null
+$BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT >> $seqres.full 2>&1
+
+$BTRFS_UTIL_PROG qgroup show $SCRATCH_MNT | $SED_PROG -n '/[0-9]/p' \
+	| $AWK_PROG '{print $1" "$2" "$3 }'
+
+status=0
+exit
diff --git a/tests/btrfs/038.out b/tests/btrfs/038.out
new file mode 100644
index 0000000..1c24c55
--- /dev/null
+++ b/tests/btrfs/038.out
@@ -0,0 +1,3 @@
+QA output created by 038
+0/5 4096 4096
+0/258 4096 4096
diff --git a/tests/btrfs/group b/tests/btrfs/group
index f9f062f..7ac9f0a 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -37,3 +37,4 @@
 032 auto quick
 033 auto quick
 034 auto quick
+038 auto quick
-- 
1.8.3.1

[PATCH 2/3] xfstests/btrfs: add basic functional test for btrfs quota groups

[Wang Shilong]

Add missing test for btrfs quota groups feature,test idea is to create
a parent qgroup that groups some subvolume groups, we try to write
some data into every subvolume and then check if we exceed parent
qgroup's limit size.

Signed-off-by: Wang Shilong <wangsl.fnst@cn.fujitsu.com>
---
 tests/btrfs/039     | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/039.out |  2 ++
 tests/btrfs/group   |  1 +
 3 files changed, 97 insertions(+)
 create mode 100644 tests/btrfs/039
 create mode 100644 tests/btrfs/039.out

diff --git a/tests/btrfs/039 b/tests/btrfs/039
new file mode 100644
index 0000000..f8c9bee
--- /dev/null
+++ b/tests/btrfs/039
@@ -0,0 +1,94 @@
+#! /bin/bash
+# FS QA Test No. 039
+#
+# Test the basic functionality of Quota groups
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2014 Fujitsu.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+
+rm -f $seqres.full
+
+run_check _scratch_mkfs
+run_check _scratch_mount
+
+LIMIT_SIZE=$((10 * 1024 * 1024))
+
+run_check $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT
+run_check $BTRFS_UTIL_PROG qgroup create 1/1 $SCRATCH_MNT
+run_check $BTRFS_UTIL_PROG qgroup limit $LIMIT_SIZE 1/1 $SCRATCH_MNT
+
+for i in `seq 10 -1 1`; do
+	#add newly created subvolume qgroup to it's parent qgroup
+	run_check $BTRFS_UTIL_PROG subvolume create -i 1/1 \
+		$SCRATCH_MNT/subv_$i
+done
+
+#try to write data into every subvolume
+for i in `seq 10 -1 1`; do
+	dd if=/dev/zero of=$SCRATCH_MNT/subv_$i/data bs=4k \
+		>> /dev/null 2>&1 &
+done
+
+wait
+$BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT \
+	>>$seqres.full 2>&1
+
+total_written=0
+#calculate every subvolume's data.
+for i in `seq 10 -1 1`; do
+	#we may fail to create the file, skip this subvolume
+	test -f $SCRATCH_MNT/subv_$i || continue
+
+	filesize=`du -b $SCRATCH_MNT/subv_$i/data | $AWK_PROG '{print $1}'`
+	if [ $filesize -gt $LIMIT_SIZE ];then
+		_fail "subv_$i/data size should be less than $LIMIT_SIZE"
+	fi
+	total_written=$(($total_written+$filesize))
+done
+
+#check if total written exceeds limit
+if [ $total_written -gt $LIMIT_SIZE ];then
+	_fail "total written should be less than $LIMIT_SIZE"
+fi
+
+# success, all done
+echo "Silence is golden"
+status=0
+exit
diff --git a/tests/btrfs/039.out b/tests/btrfs/039.out
new file mode 100644
index 0000000..d4e7ef6
--- /dev/null
+++ b/tests/btrfs/039.out
@@ -0,0 +1,2 @@
+QA output created by 039
+Silence is golden
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 7ac9f0a..ba6b70c 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -38,3 +38,4 @@
 033 auto quick
 034 auto quick
 038 auto quick
+039 auto quick
-- 
1.8.3.1

[PATCH 3/3] xfstests/btrfs: add stress test for btrfs quota operations

[Wang Shilong]

So this is a stress test for btrfs quota operations. it can also
detect the following commit fixed problem:

4082bd3d73(Btrfs: fix oops when writting dirty qgroups to disk)

Signed-off-by: Wang Shilong <wangsl.fnst@cn.fujitsu.com>
---
 tests/btrfs/040     | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/040.out |  2 ++
 tests/btrfs/group   |  1 +
 3 files changed, 78 insertions(+)
 create mode 100644 tests/btrfs/040
 create mode 100644 tests/btrfs/040.out

diff --git a/tests/btrfs/040 b/tests/btrfs/040
new file mode 100644
index 0000000..d991304
--- /dev/null
+++ b/tests/btrfs/040
@@ -0,0 +1,75 @@
+#! /bin/bash
+# FS QA Test No. 040
+#
+# stresstest for btrfs quota operations. we run fsstress and quota
+# operations concurrently.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2014 Fujitsu.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+
+rm -f $seqres.full
+
+_quota_enabled_background()
+{
+	i=1
+	while [ $i -le 5 ]
+	do
+		run_check $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT
+		run_check $BTRFS_UTIL_PROG quota disable $SCRATCH_MNT
+		i=$(($i+1))
+		sleep 1
+	done
+}
+
+MKFS_SIZE=$((2 * 1024 * 1024 * 1024))
+run_check _scratch_mkfs_sized $MKFS_SIZE
+run_check _scratch_mount
+
+_quota_enabled_background &
+run_check $FSSTRESS_PROG -d $SCRATCH_MNT -w -p 5 -n 1000 \
+	$FSSTRESS_AVOID
+
+run_check _scratch_unmount
+_check_scratch_fs
+
+# success, all done
+echo "Silence is golden"
+status=0
+exit
diff --git a/tests/btrfs/040.out b/tests/btrfs/040.out
new file mode 100644
index 0000000..9e3d4d9
--- /dev/null
+++ b/tests/btrfs/040.out
@@ -0,0 +1,2 @@
+QA output created by 040
+Silence is golden
diff --git a/tests/btrfs/group b/tests/btrfs/group
index ba6b70c..3dc82e5 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -39,3 +39,4 @@
 034 auto quick
 038 auto quick
 039 auto quick
+040 auto quick
-- 
1.8.3.1

Re: [PATCH 1/3] xfstests/btrfs: add qgroup rescan stress test

[Dave Chinner]

On Thu, Feb 13, 2014 at 11:18:57AM +0800, Wang Shilong wrote:
> Test flow is to run fsstress after triggering quota rescan.
> the ruler is simple, we just remove all files and directories,
> sync filesystem and see if qgroup's ref and excl are nodesize.
> 
> Signed-off-by: Wang Shilong <wangsl.fnst@cn.fujitsu.com>
> ---
>  tests/btrfs/038     | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/btrfs/038.out |  3 +++
>  tests/btrfs/group   |  1 +
>  3 files changed, 79 insertions(+)
>  create mode 100644 tests/btrfs/038
>  create mode 100644 tests/btrfs/038.out
> 
> diff --git a/tests/btrfs/038 b/tests/btrfs/038
> new file mode 100644
> index 0000000..f6bd872
> --- /dev/null
> +++ b/tests/btrfs/038
> @@ -0,0 +1,75 @@
> +#! /bin/bash
> +# FSQA Test No. btrfs/038
> +#
> +# Quota rescan stress test, we run fsstress and quota rescan concurrently
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (C) 2014 Fujitsu.  All rights reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1
> +
> +_cleanup()
> +{
> +       rm -f $tmp.*
> +}
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# real QA test starts here
> +_need_to_be_root
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +
> +rm -f $seqres.full
> +
> +run_check _scratch_mkfs "-b 2g --nodesize 4096"
> +run_check _scratch_mount
> +
> +# -w ensures that the only ops are ones which cause write I/O
> +run_check $FSSTRESS_PROG -d $SCRATCH_MNT -w -p 5 -n 1000 \
> +		$FSSTRESS_AVOID >&/dev/null
> +
> +run_check $BTRFS_UTIL_PROG subvolume snapshot $SCRATCH_MNT \
> +       $SCRATCH_MNT/snap1 >>$seqres.full 2>&1
> +
> +run_check $FSSTRESS_PROG -d $SCRATCH_MNT/snap1 -w -p 5 -n 1000 \
> +       $FSSTRESS_AVOID >&/dev/null
> +
> +run_check $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT
> +run_check $BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT

"run_check considered harmful."

http://oss.sgi.com/archives/xfs/2014-02/msg00482.html

Once I've committed Filipe's run_btrfs_util_prog, can you update
this series to remove all the unnecessary run_check calls and
repost? Thanks!

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [PATCH 1/3] xfstests/btrfs: add qgroup rescan stress test

[Wang Shilong]

On 02/18/2014 02:46 PM, Dave Chinner wrote:
> On Thu, Feb 13, 2014 at 11:18:57AM +0800, Wang Shilong wrote:
>> Test flow is to run fsstress after triggering quota rescan.
>> the ruler is simple, we just remove all files and directories,
>> sync filesystem and see if qgroup's ref and excl are nodesize.
>>
>> Signed-off-by: Wang Shilong <wangsl.fnst@cn.fujitsu.com>
>> ---
>>   tests/btrfs/038     | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>   tests/btrfs/038.out |  3 +++
>>   tests/btrfs/group   |  1 +
>>   3 files changed, 79 insertions(+)
>>   create mode 100644 tests/btrfs/038
>>   create mode 100644 tests/btrfs/038.out
>>
>> diff --git a/tests/btrfs/038 b/tests/btrfs/038
>> new file mode 100644
>> index 0000000..f6bd872
>> --- /dev/null
>> +++ b/tests/btrfs/038
>> @@ -0,0 +1,75 @@
>> +#! /bin/bash
>> +# FSQA Test No. btrfs/038
>> +#
>> +# Quota rescan stress test, we run fsstress and quota rescan concurrently
>> +#
>> +#-----------------------------------------------------------------------
>> +# Copyright (C) 2014 Fujitsu.  All rights reserved.
>> +#
>> +# This program is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU General Public License as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it would be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write the Free Software Foundation,
>> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
>> +#
>> +#-----------------------------------------------------------------------
>> +#
>> +
>> +seq=`basename $0`
>> +seqres=$RESULT_DIR/$seq
>> +echo "QA output created by $seq"
>> +
>> +here=`pwd`
>> +tmp=/tmp/$$
>> +status=1
>> +
>> +_cleanup()
>> +{
>> +       rm -f $tmp.*
>> +}
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +# get standard environment, filters and checks
>> +. ./common/rc
>> +. ./common/filter
>> +
>> +# real QA test starts here
>> +_need_to_be_root
>> +_supported_fs btrfs
>> +_supported_os Linux
>> +_require_scratch
>> +
>> +rm -f $seqres.full
>> +
>> +run_check _scratch_mkfs "-b 2g --nodesize 4096"
>> +run_check _scratch_mount
>> +
>> +# -w ensures that the only ops are ones which cause write I/O
>> +run_check $FSSTRESS_PROG -d $SCRATCH_MNT -w -p 5 -n 1000 \
>> +		$FSSTRESS_AVOID >&/dev/null
>> +
>> +run_check $BTRFS_UTIL_PROG subvolume snapshot $SCRATCH_MNT \
>> +       $SCRATCH_MNT/snap1 >>$seqres.full 2>&1
>> +
>> +run_check $FSSTRESS_PROG -d $SCRATCH_MNT/snap1 -w -p 5 -n 1000 \
>> +       $FSSTRESS_AVOID >&/dev/null
>> +
>> +run_check $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT
>> +run_check $BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT
> "run_check considered harmful."
>
> http://oss.sgi.com/archives/xfs/2014-02/msg00482.html
>
> Once I've committed Filipe's run_btrfs_util_prog, can you update
> this series to remove all the unnecessary run_check calls and
> repost? Thanks!
No problem. ^_^

Thanks,
Wang
>
> Cheers,
>
> Dave.