* [PATCH v2] fstests: btrfs: Add regression test for reserved space leak.
@ 2015-08-05 1:08 ` Qu Wenruo
0 siblings, 0 replies; 12+ messages in thread
From: Qu Wenruo @ 2015-08-05 1:08 UTC (permalink / raw)
To: linux-btrfs, fstests
The regression is introduced in v4.2-rc1, with the big btrfs qgroup
change.
The problem is, qgroup reserved space is never freed, causing even we
increase the limit, we can still hit the EDQUOT much faster than it
should.
Reported-by: Tsutomu Itoh <t-itoh@jp.fujitsu.com>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
tests/btrfs/089 | 88 +++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/089.out | 5 +++
tests/btrfs/group | 1 +
3 files changed, 94 insertions(+)
create mode 100755 tests/btrfs/089
create mode 100644 tests/btrfs/089.out
diff --git a/tests/btrfs/089 b/tests/btrfs/089
new file mode 100755
index 0000000..82db96c
--- /dev/null
+++ b/tests/btrfs/089
@@ -0,0 +1,88 @@
+#! /bin/bash
+# FS QA Test 089
+#
+# Regression test for btrfs qgroup reserved space leak.
+#
+# Due to qgroup reserved space leak, EDQUOT can be trigged even it's not
+# over limit after previous write.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2015 Fujitsu. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_need_to_be_root
+
+# Use big blocksize to ensure there is still enough space left
+# for metadata reserve after hitting EDQUOT
+BLOCKSIZE=$(( 2 * 1024 * 1024 ))
+FILESIZE=$(( 128 * 1024 * 1024 )) # 128Mbytes
+
+# The last block won't be able to finish write, as metadata takes
+# $NODESIZE space, causing the last block triggering EDQUOT
+LENGTH=$(( $FILESIZE - $BLOCKSIZE ))
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+_require_fs_space $SCRATCH_MNT $(($FILESIZE * 2 / 1024))
+
+_run_btrfs_util_prog quota enable $SCRATCH_MNT
+_run_btrfs_util_prog qgroup limit $FILESIZE 5 $SCRATCH_MNT
+
+$XFS_IO_PROG -f -c "pwrite -b $BLOCKSIZE 0 $LENGTH" \
+ $SCRATCH_MNT/foo | _filter_xfs_io
+
+# A sync is needed to trigger a commit_transaction.
+# As the reserved space freeing happens at commit_transaction time,
+# without a transaction commit, no reserved space needs freeing and
+# won't trigger the bug.
+sync
+
+# Double the limit to allow further write
+_run_btrfs_util_prog qgroup limit $(($FILESIZE * 2)) 5 $SCRATCH_MNT
+
+# Test whether further write can succeed
+$XFS_IO_PROG -f -c "pwrite -b $BLOCKSIZE $LENGTH $LENGTH" \
+ $SCRATCH_MNT/foo | _filter_xfs_io
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/089.out b/tests/btrfs/089.out
new file mode 100644
index 0000000..396888f
--- /dev/null
+++ b/tests/btrfs/089.out
@@ -0,0 +1,5 @@
+QA output created by 089
+wrote 132120576/132120576 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 132120576/132120576 bytes at offset 132120576
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/btrfs/group b/tests/btrfs/group
index ffe18bf..225b532 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -91,6 +91,7 @@
086 auto quick clone
087 auto quick send
088 auto quick metadata
+089 auto quick qgroup
090 auto quick metadata
091 auto quick qgroup
092 auto quick send
--
1.8.3.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH v2] fstests: btrfs: Add regression test for reserved space leak.
2015-08-05 1:08 ` Qu Wenruo
(?)
@ 2015-08-05 1:26 ` Tsutomu Itoh
2015-08-05 1:39 ` Qu Wenruo
-1 siblings, 1 reply; 12+ messages in thread
From: Tsutomu Itoh @ 2015-08-05 1:26 UTC (permalink / raw)
To: linux-btrfs, fstests
On 2015/08/05 10:08, Qu Wenruo wrote:
> The regression is introduced in v4.2-rc1, with the big btrfs qgroup
> change.
> The problem is, qgroup reserved space is never freed, causing even we
> increase the limit, we can still hit the EDQUOT much faster than it
> should.
>
> Reported-by: Tsutomu Itoh <t-itoh@jp.fujitsu.com>
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> ---
> tests/btrfs/089 | 88 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> tests/btrfs/089.out | 5 +++
> tests/btrfs/group | 1 +
> 3 files changed, 94 insertions(+)
> create mode 100755 tests/btrfs/089
> create mode 100644 tests/btrfs/089.out
>
> diff --git a/tests/btrfs/089 b/tests/btrfs/089
> new file mode 100755
> index 0000000..82db96c
> --- /dev/null
> +++ b/tests/btrfs/089
> @@ -0,0 +1,88 @@
> +#! /bin/bash
> +# FS QA Test 089
> +#
> +# Regression test for btrfs qgroup reserved space leak.
> +#
> +# Due to qgroup reserved space leak, EDQUOT can be trigged even it's not
> +# over limit after previous write.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2015 Fujitsu. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +_need_to_be_root
> +
> +# Use big blocksize to ensure there is still enough space left
> +# for metadata reserve after hitting EDQUOT
> +BLOCKSIZE=$(( 2 * 1024 * 1024 ))
> +FILESIZE=$(( 128 * 1024 * 1024 )) # 128Mbytes
> +
> +# The last block won't be able to finish write, as metadata takes
> +# $NODESIZE space, causing the last block triggering EDQUOT
> +LENGTH=$(( $FILESIZE - $BLOCKSIZE ))
> +
> +_scratch_mkfs >>$seqres.full 2>&1
> +_scratch_mount
> +_require_fs_space $SCRATCH_MNT $(($FILESIZE * 2 / 1024))
> +
> +_run_btrfs_util_prog quota enable $SCRATCH_MNT
> +_run_btrfs_util_prog qgroup limit $FILESIZE 5 $SCRATCH_MNT
> +
> +$XFS_IO_PROG -f -c "pwrite -b $BLOCKSIZE 0 $LENGTH" \
> + $SCRATCH_MNT/foo | _filter_xfs_io
> +
> +# A sync is needed to trigger a commit_transaction.
> +# As the reserved space freeing happens at commit_transaction time,
> +# without a transaction commit, no reserved space needs freeing and
> +# won't trigger the bug.
> +sync
Isn't '$BTRFS_UTIL_PROG filesystem sync' better instead of 'sync'?
Thanks,
Tsutomu
> +
> +# Double the limit to allow further write
> +_run_btrfs_util_prog qgroup limit $(($FILESIZE * 2)) 5 $SCRATCH_MNT
> +
> +# Test whether further write can succeed
> +$XFS_IO_PROG -f -c "pwrite -b $BLOCKSIZE $LENGTH $LENGTH" \
> + $SCRATCH_MNT/foo | _filter_xfs_io
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/089.out b/tests/btrfs/089.out
> new file mode 100644
> index 0000000..396888f
> --- /dev/null
> +++ b/tests/btrfs/089.out
> @@ -0,0 +1,5 @@
> +QA output created by 089
> +wrote 132120576/132120576 bytes at offset 0
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +wrote 132120576/132120576 bytes at offset 132120576
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index ffe18bf..225b532 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -91,6 +91,7 @@
> 086 auto quick clone
> 087 auto quick send
> 088 auto quick metadata
> +089 auto quick qgroup
> 090 auto quick metadata
> 091 auto quick qgroup
> 092 auto quick send
>
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH v2] fstests: btrfs: Add regression test for reserved space leak.
2015-08-05 1:26 ` Tsutomu Itoh
@ 2015-08-05 1:39 ` Qu Wenruo
0 siblings, 0 replies; 12+ messages in thread
From: Qu Wenruo @ 2015-08-05 1:39 UTC (permalink / raw)
To: Tsutomu Itoh, linux-btrfs, fstests
Tsutomu Itoh wrote on 2015/08/05 10:26 +0900:
> On 2015/08/05 10:08, Qu Wenruo wrote:
>> The regression is introduced in v4.2-rc1, with the big btrfs qgroup
>> change.
>> The problem is, qgroup reserved space is never freed, causing even we
>> increase the limit, we can still hit the EDQUOT much faster than it
>> should.
>>
>> Reported-by: Tsutomu Itoh <t-itoh@jp.fujitsu.com>
>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> ---
>> tests/btrfs/089 | 88 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>> tests/btrfs/089.out | 5 +++
>> tests/btrfs/group | 1 +
>> 3 files changed, 94 insertions(+)
>> create mode 100755 tests/btrfs/089
>> create mode 100644 tests/btrfs/089.out
>>
>> diff --git a/tests/btrfs/089 b/tests/btrfs/089
>> new file mode 100755
>> index 0000000..82db96c
>> --- /dev/null
>> +++ b/tests/btrfs/089
>> @@ -0,0 +1,88 @@
>> +#! /bin/bash
>> +# FS QA Test 089
>> +#
>> +# Regression test for btrfs qgroup reserved space leak.
>> +#
>> +# Due to qgroup reserved space leak, EDQUOT can be trigged even it's not
>> +# over limit after previous write.
>> +#
>> +#-----------------------------------------------------------------------
>> +# Copyright (c) 2015 Fujitsu. All Rights Reserved.
>> +#
>> +# This program is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU General Public License as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it would be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write the Free Software Foundation,
>> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
>> +#-----------------------------------------------------------------------
>> +#
>> +
>> +seq=`basename $0`
>> +seqres=$RESULT_DIR/$seq
>> +echo "QA output created by $seq"
>> +
>> +here=`pwd`
>> +tmp=/tmp/$$
>> +status=1 # failure is the default!
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +_cleanup()
>> +{
>> + cd /
>> + rm -f $tmp.*
>> +}
>> +
>> +# get standard environment, filters and checks
>> +. ./common/rc
>> +. ./common/filter
>> +
>> +# real QA test starts here
>> +
>> +# Modify as appropriate.
>> +_supported_fs btrfs
>> +_supported_os Linux
>> +_require_scratch
>> +_need_to_be_root
>> +
>> +# Use big blocksize to ensure there is still enough space left
>> +# for metadata reserve after hitting EDQUOT
>> +BLOCKSIZE=$(( 2 * 1024 * 1024 ))
>> +FILESIZE=$(( 128 * 1024 * 1024 )) # 128Mbytes
>> +
>> +# The last block won't be able to finish write, as metadata takes
>> +# $NODESIZE space, causing the last block triggering EDQUOT
>> +LENGTH=$(( $FILESIZE - $BLOCKSIZE ))
>> +
>> +_scratch_mkfs >>$seqres.full 2>&1
>> +_scratch_mount
>> +_require_fs_space $SCRATCH_MNT $(($FILESIZE * 2 / 1024))
>> +
>> +_run_btrfs_util_prog quota enable $SCRATCH_MNT
>> +_run_btrfs_util_prog qgroup limit $FILESIZE 5 $SCRATCH_MNT
>> +
>> +$XFS_IO_PROG -f -c "pwrite -b $BLOCKSIZE 0 $LENGTH" \
>> + $SCRATCH_MNT/foo | _filter_xfs_io
>> +
>> +# A sync is needed to trigger a commit_transaction.
>> +# As the reserved space freeing happens at commit_transaction time,
>> +# without a transaction commit, no reserved space needs freeing and
>> +# won't trigger the bug.
>> +sync
>
> Isn't '$BTRFS_UTIL_PROG filesystem sync' better instead of 'sync'?
>
> Thanks,
> Tsutomu
Hi, Tsutomu-san,
Yes, I did use such method before, but Dave said it's better to use
unified interface to sync a filesystem other than the specialized one.
So I still use sync as Dave said.
Thanks,
Qu
>
>> +
>> +# Double the limit to allow further write
>> +_run_btrfs_util_prog qgroup limit $(($FILESIZE * 2)) 5 $SCRATCH_MNT
>> +
>> +# Test whether further write can succeed
>> +$XFS_IO_PROG -f -c "pwrite -b $BLOCKSIZE $LENGTH $LENGTH" \
>> + $SCRATCH_MNT/foo | _filter_xfs_io
>> +
>> +# success, all done
>> +status=0
>> +exit
>> diff --git a/tests/btrfs/089.out b/tests/btrfs/089.out
>> new file mode 100644
>> index 0000000..396888f
>> --- /dev/null
>> +++ b/tests/btrfs/089.out
>> @@ -0,0 +1,5 @@
>> +QA output created by 089
>> +wrote 132120576/132120576 bytes at offset 0
>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> +wrote 132120576/132120576 bytes at offset 132120576
>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> diff --git a/tests/btrfs/group b/tests/btrfs/group
>> index ffe18bf..225b532 100644
>> --- a/tests/btrfs/group
>> +++ b/tests/btrfs/group
>> @@ -91,6 +91,7 @@
>> 086 auto quick clone
>> 087 auto quick send
>> 088 auto quick metadata
>> +089 auto quick qgroup
>> 090 auto quick metadata
>> 091 auto quick qgroup
>> 092 auto quick send
>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH v2] fstests: btrfs: Add regression test for reserved space leak.
@ 2015-08-05 1:39 ` Qu Wenruo
0 siblings, 0 replies; 12+ messages in thread
From: Qu Wenruo @ 2015-08-05 1:39 UTC (permalink / raw)
To: Tsutomu Itoh, linux-btrfs, fstests
Tsutomu Itoh wrote on 2015/08/05 10:26 +0900:
> On 2015/08/05 10:08, Qu Wenruo wrote:
>> The regression is introduced in v4.2-rc1, with the big btrfs qgroup
>> change.
>> The problem is, qgroup reserved space is never freed, causing even we
>> increase the limit, we can still hit the EDQUOT much faster than it
>> should.
>>
>> Reported-by: Tsutomu Itoh <t-itoh@jp.fujitsu.com>
>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> ---
>> tests/btrfs/089 | 88 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>> tests/btrfs/089.out | 5 +++
>> tests/btrfs/group | 1 +
>> 3 files changed, 94 insertions(+)
>> create mode 100755 tests/btrfs/089
>> create mode 100644 tests/btrfs/089.out
>>
>> diff --git a/tests/btrfs/089 b/tests/btrfs/089
>> new file mode 100755
>> index 0000000..82db96c
>> --- /dev/null
>> +++ b/tests/btrfs/089
>> @@ -0,0 +1,88 @@
>> +#! /bin/bash
>> +# FS QA Test 089
>> +#
>> +# Regression test for btrfs qgroup reserved space leak.
>> +#
>> +# Due to qgroup reserved space leak, EDQUOT can be trigged even it's not
>> +# over limit after previous write.
>> +#
>> +#-----------------------------------------------------------------------
>> +# Copyright (c) 2015 Fujitsu. All Rights Reserved.
>> +#
>> +# This program is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU General Public License as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it would be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write the Free Software Foundation,
>> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
>> +#-----------------------------------------------------------------------
>> +#
>> +
>> +seq=`basename $0`
>> +seqres=$RESULT_DIR/$seq
>> +echo "QA output created by $seq"
>> +
>> +here=`pwd`
>> +tmp=/tmp/$$
>> +status=1 # failure is the default!
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +_cleanup()
>> +{
>> + cd /
>> + rm -f $tmp.*
>> +}
>> +
>> +# get standard environment, filters and checks
>> +. ./common/rc
>> +. ./common/filter
>> +
>> +# real QA test starts here
>> +
>> +# Modify as appropriate.
>> +_supported_fs btrfs
>> +_supported_os Linux
>> +_require_scratch
>> +_need_to_be_root
>> +
>> +# Use big blocksize to ensure there is still enough space left
>> +# for metadata reserve after hitting EDQUOT
>> +BLOCKSIZE=$(( 2 * 1024 * 1024 ))
>> +FILESIZE=$(( 128 * 1024 * 1024 )) # 128Mbytes
>> +
>> +# The last block won't be able to finish write, as metadata takes
>> +# $NODESIZE space, causing the last block triggering EDQUOT
>> +LENGTH=$(( $FILESIZE - $BLOCKSIZE ))
>> +
>> +_scratch_mkfs >>$seqres.full 2>&1
>> +_scratch_mount
>> +_require_fs_space $SCRATCH_MNT $(($FILESIZE * 2 / 1024))
>> +
>> +_run_btrfs_util_prog quota enable $SCRATCH_MNT
>> +_run_btrfs_util_prog qgroup limit $FILESIZE 5 $SCRATCH_MNT
>> +
>> +$XFS_IO_PROG -f -c "pwrite -b $BLOCKSIZE 0 $LENGTH" \
>> + $SCRATCH_MNT/foo | _filter_xfs_io
>> +
>> +# A sync is needed to trigger a commit_transaction.
>> +# As the reserved space freeing happens at commit_transaction time,
>> +# without a transaction commit, no reserved space needs freeing and
>> +# won't trigger the bug.
>> +sync
>
> Isn't '$BTRFS_UTIL_PROG filesystem sync' better instead of 'sync'?
>
> Thanks,
> Tsutomu
Hi, Tsutomu-san,
Yes, I did use such method before, but Dave said it's better to use
unified interface to sync a filesystem other than the specialized one.
So I still use sync as Dave said.
Thanks,
Qu
>
>> +
>> +# Double the limit to allow further write
>> +_run_btrfs_util_prog qgroup limit $(($FILESIZE * 2)) 5 $SCRATCH_MNT
>> +
>> +# Test whether further write can succeed
>> +$XFS_IO_PROG -f -c "pwrite -b $BLOCKSIZE $LENGTH $LENGTH" \
>> + $SCRATCH_MNT/foo | _filter_xfs_io
>> +
>> +# success, all done
>> +status=0
>> +exit
>> diff --git a/tests/btrfs/089.out b/tests/btrfs/089.out
>> new file mode 100644
>> index 0000000..396888f
>> --- /dev/null
>> +++ b/tests/btrfs/089.out
>> @@ -0,0 +1,5 @@
>> +QA output created by 089
>> +wrote 132120576/132120576 bytes at offset 0
>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> +wrote 132120576/132120576 bytes at offset 132120576
>> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> diff --git a/tests/btrfs/group b/tests/btrfs/group
>> index ffe18bf..225b532 100644
>> --- a/tests/btrfs/group
>> +++ b/tests/btrfs/group
>> @@ -91,6 +91,7 @@
>> 086 auto quick clone
>> 087 auto quick send
>> 088 auto quick metadata
>> +089 auto quick qgroup
>> 090 auto quick metadata
>> 091 auto quick qgroup
>> 092 auto quick send
>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH v2] fstests: btrfs: Add regression test for reserved space leak.
2015-08-05 1:39 ` Qu Wenruo
(?)
@ 2015-08-05 1:57 ` Dave Chinner
2015-08-05 2:52 ` Tsutomu Itoh
-1 siblings, 1 reply; 12+ messages in thread
From: Dave Chinner @ 2015-08-05 1:57 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Tsutomu Itoh, linux-btrfs, fstests
On Wed, Aug 05, 2015 at 09:39:35AM +0800, Qu Wenruo wrote:
> Tsutomu Itoh wrote on 2015/08/05 10:26 +0900:
> >On 2015/08/05 10:08, Qu Wenruo wrote:
> >>+# As the reserved space freeing happens at commit_transaction time,
> >>+# without a transaction commit, no reserved space needs freeing and
> >>+# won't trigger the bug.
> >>+sync
> >
> >Isn't '$BTRFS_UTIL_PROG filesystem sync' better instead of 'sync'?
> >
> >Thanks,
> >Tsutomu
> Hi, Tsutomu-san,
>
> Yes, I did use such method before, but Dave said it's better to use
> unified interface to sync a filesystem other than the specialized
> one.
>
> So I still use sync as Dave said.
Mainly because "sync" is what users will use to make sure their data
is safe. filesystem specific tools have a habit of doing "special
stuff" to sync a filesystem, so it may not reflect the way users
expect the system to behaviour when they run sync.
The other option is this:
_syncfs()
{
mntpt=$1
$XFS_IO_PROG -c syncfs $mntpt
}
_sync_test()
{
_syncfs $TEST_DIR
}
_sync_scratch()
{
_syncfs $SCRATCH_MNT
}
which only runs sync on the filesystem that needs syncing (via the
syncfs() syscall)
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH v2] fstests: btrfs: Add regression test for reserved space leak.
2015-08-05 1:57 ` Dave Chinner
@ 2015-08-05 2:52 ` Tsutomu Itoh
2015-08-05 2:59 ` Qu Wenruo
2015-08-05 3:10 ` Dave Chinner
0 siblings, 2 replies; 12+ messages in thread
From: Tsutomu Itoh @ 2015-08-05 2:52 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs, fstests
On 2015/08/05 10:57, Dave Chinner wrote:
> On Wed, Aug 05, 2015 at 09:39:35AM +0800, Qu Wenruo wrote:
>> Tsutomu Itoh wrote on 2015/08/05 10:26 +0900:
>>> On 2015/08/05 10:08, Qu Wenruo wrote:
>>>> +# As the reserved space freeing happens at commit_transaction time,
>>>> +# without a transaction commit, no reserved space needs freeing and
>>>> +# won't trigger the bug.
>>>> +sync
>>>
>>> Isn't '$BTRFS_UTIL_PROG filesystem sync' better instead of 'sync'?
>>>
>>> Thanks,
>>> Tsutomu
>> Hi, Tsutomu-san,
>>
>> Yes, I did use such method before, but Dave said it's better to use
>> unified interface to sync a filesystem other than the specialized
>> one.
>>
>> So I still use sync as Dave said.
>
> Mainly because "sync" is what users will use to make sure their data
> is safe. filesystem specific tools have a habit of doing "special
> stuff" to sync a filesystem, so it may not reflect the way users
> expect the system to behaviour when they run sync.
>
> The other option is this:
>
> _syncfs()
> {
> mntpt=$1
>
> $XFS_IO_PROG -c syncfs $mntpt
> }
>
> _sync_test()
> {
> _syncfs $TEST_DIR
> }
>
> _sync_scratch()
> {
> _syncfs $SCRATCH_MNT
> }
>
> which only runs sync on the filesystem that needs syncing (via the
> syncfs() syscall)
I think that syncfs is better instead of sync because the syncing of
the specified filesystem is necessary.
Thanks,
Tsutomu
>
> Cheers,
>
> Dave.
>
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH v2] fstests: btrfs: Add regression test for reserved space leak.
2015-08-05 2:52 ` Tsutomu Itoh
@ 2015-08-05 2:59 ` Qu Wenruo
2015-08-05 3:10 ` Dave Chinner
1 sibling, 0 replies; 12+ messages in thread
From: Qu Wenruo @ 2015-08-05 2:59 UTC (permalink / raw)
To: Tsutomu Itoh; +Cc: linux-btrfs, fstests
Tsutomu Itoh wrote on 2015/08/05 11:52 +0900:
> On 2015/08/05 10:57, Dave Chinner wrote:
>> On Wed, Aug 05, 2015 at 09:39:35AM +0800, Qu Wenruo wrote:
>>> Tsutomu Itoh wrote on 2015/08/05 10:26 +0900:
>>>> On 2015/08/05 10:08, Qu Wenruo wrote:
>>>>> +# As the reserved space freeing happens at commit_transaction time,
>>>>> +# without a transaction commit, no reserved space needs freeing and
>>>>> +# won't trigger the bug.
>>>>> +sync
>>>>
>>>> Isn't '$BTRFS_UTIL_PROG filesystem sync' better instead of 'sync'?
>>>>
>>>> Thanks,
>>>> Tsutomu
>>> Hi, Tsutomu-san,
>>>
>>> Yes, I did use such method before, but Dave said it's better to use
>>> unified interface to sync a filesystem other than the specialized
>>> one.
>>>
>>> So I still use sync as Dave said.
>>
>> Mainly because "sync" is what users will use to make sure their data
>> is safe. filesystem specific tools have a habit of doing "special
>> stuff" to sync a filesystem, so it may not reflect the way users
>> expect the system to behaviour when they run sync.
>>
>> The other option is this:
>>
>> _syncfs()
>> {
>> mntpt=$1
>>
>> $XFS_IO_PROG -c syncfs $mntpt
>> }
>>
>> _sync_test()
>> {
>> _syncfs $TEST_DIR
>> }
>>
>> _sync_scratch()
>> {
>> _syncfs $SCRATCH_MNT
>> }
>>
>> which only runs sync on the filesystem that needs syncing (via the
>> syncfs() syscall)
>
> I think that syncfs is better instead of sync because the syncing of
> the specified filesystem is necessary.
>
> Thanks,
> Tsutomu
>
I'm OK changing the sync to _sync_scratch().
But no one in btrfs testcase uses that function and I didn't see the
reason to specifically call _sync_scratch() instead of sync() BTW.
Thanks,
Qu
>>
>> Cheers,
>>
>> Dave.
>>
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH v2] fstests: btrfs: Add regression test for reserved space leak.
@ 2015-08-05 2:59 ` Qu Wenruo
0 siblings, 0 replies; 12+ messages in thread
From: Qu Wenruo @ 2015-08-05 2:59 UTC (permalink / raw)
To: Tsutomu Itoh; +Cc: linux-btrfs, fstests
Tsutomu Itoh wrote on 2015/08/05 11:52 +0900:
> On 2015/08/05 10:57, Dave Chinner wrote:
>> On Wed, Aug 05, 2015 at 09:39:35AM +0800, Qu Wenruo wrote:
>>> Tsutomu Itoh wrote on 2015/08/05 10:26 +0900:
>>>> On 2015/08/05 10:08, Qu Wenruo wrote:
>>>>> +# As the reserved space freeing happens at commit_transaction time,
>>>>> +# without a transaction commit, no reserved space needs freeing and
>>>>> +# won't trigger the bug.
>>>>> +sync
>>>>
>>>> Isn't '$BTRFS_UTIL_PROG filesystem sync' better instead of 'sync'?
>>>>
>>>> Thanks,
>>>> Tsutomu
>>> Hi, Tsutomu-san,
>>>
>>> Yes, I did use such method before, but Dave said it's better to use
>>> unified interface to sync a filesystem other than the specialized
>>> one.
>>>
>>> So I still use sync as Dave said.
>>
>> Mainly because "sync" is what users will use to make sure their data
>> is safe. filesystem specific tools have a habit of doing "special
>> stuff" to sync a filesystem, so it may not reflect the way users
>> expect the system to behaviour when they run sync.
>>
>> The other option is this:
>>
>> _syncfs()
>> {
>> mntpt=$1
>>
>> $XFS_IO_PROG -c syncfs $mntpt
>> }
>>
>> _sync_test()
>> {
>> _syncfs $TEST_DIR
>> }
>>
>> _sync_scratch()
>> {
>> _syncfs $SCRATCH_MNT
>> }
>>
>> which only runs sync on the filesystem that needs syncing (via the
>> syncfs() syscall)
>
> I think that syncfs is better instead of sync because the syncing of
> the specified filesystem is necessary.
>
> Thanks,
> Tsutomu
>
I'm OK changing the sync to _sync_scratch().
But no one in btrfs testcase uses that function and I didn't see the
reason to specifically call _sync_scratch() instead of sync() BTW.
Thanks,
Qu
>>
>> Cheers,
>>
>> Dave.
>>
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2] fstests: btrfs: Add regression test for reserved space leak.
2015-08-05 2:52 ` Tsutomu Itoh
2015-08-05 2:59 ` Qu Wenruo
@ 2015-08-05 3:10 ` Dave Chinner
2015-08-05 4:04 ` Tsutomu Itoh
1 sibling, 1 reply; 12+ messages in thread
From: Dave Chinner @ 2015-08-05 3:10 UTC (permalink / raw)
To: Tsutomu Itoh; +Cc: Qu Wenruo, linux-btrfs, fstests
On Wed, Aug 05, 2015 at 11:52:36AM +0900, Tsutomu Itoh wrote:
> On 2015/08/05 10:57, Dave Chinner wrote:
> >On Wed, Aug 05, 2015 at 09:39:35AM +0800, Qu Wenruo wrote:
> >>Tsutomu Itoh wrote on 2015/08/05 10:26 +0900:
> >>>On 2015/08/05 10:08, Qu Wenruo wrote:
> >>>>+# As the reserved space freeing happens at commit_transaction time,
> >>>>+# without a transaction commit, no reserved space needs freeing and
> >>>>+# won't trigger the bug.
> >>>>+sync
> >>>
> >>>Isn't '$BTRFS_UTIL_PROG filesystem sync' better instead of 'sync'?
> >>>
> >>>Thanks,
> >>>Tsutomu
> >>Hi, Tsutomu-san,
> >>
> >>Yes, I did use such method before, but Dave said it's better to use
> >>unified interface to sync a filesystem other than the specialized
> >>one.
> >>
> >>So I still use sync as Dave said.
> >
> >Mainly because "sync" is what users will use to make sure their data
> >is safe. filesystem specific tools have a habit of doing "special
> >stuff" to sync a filesystem, so it may not reflect the way users
> >expect the system to behaviour when they run sync.
> >
> >The other option is this:
> >
> >_syncfs()
> >{
> > mntpt=$1
> >
> > $XFS_IO_PROG -c syncfs $mntpt
> >}
> >
> >_sync_test()
> >{
> > _syncfs $TEST_DIR
> >}
> >
> >_sync_scratch()
> >{
> > _syncfs $SCRATCH_MNT
> >}
> >
> >which only runs sync on the filesystem that needs syncing (via the
> >syncfs() syscall)
>
> I think that syncfs is better instead of sync because the syncing of
> the specified filesystem is necessary.
sync guarantees that. IOWs, there's no difference between sync and
syncfs for the persepctive of this test and so the test does not
need really need changing.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH v2] fstests: btrfs: Add regression test for reserved space leak.
2015-08-05 3:10 ` Dave Chinner
@ 2015-08-05 4:04 ` Tsutomu Itoh
0 siblings, 0 replies; 12+ messages in thread
From: Tsutomu Itoh @ 2015-08-05 4:04 UTC (permalink / raw)
To: Dave Chinner, Qu Wenruo; +Cc: linux-btrfs, fstests
On 2015/08/05 12:10, Dave Chinner wrote:
> On Wed, Aug 05, 2015 at 11:52:36AM +0900, Tsutomu Itoh wrote:
>> On 2015/08/05 10:57, Dave Chinner wrote:
>>> On Wed, Aug 05, 2015 at 09:39:35AM +0800, Qu Wenruo wrote:
>>>> Tsutomu Itoh wrote on 2015/08/05 10:26 +0900:
>>>>> On 2015/08/05 10:08, Qu Wenruo wrote:
>>>>>> +# As the reserved space freeing happens at commit_transaction time,
>>>>>> +# without a transaction commit, no reserved space needs freeing and
>>>>>> +# won't trigger the bug.
>>>>>> +sync
>>>>>
>>>>> Isn't '$BTRFS_UTIL_PROG filesystem sync' better instead of 'sync'?
>>>>>
>>>>> Thanks,
>>>>> Tsutomu
>>>> Hi, Tsutomu-san,
>>>>
>>>> Yes, I did use such method before, but Dave said it's better to use
>>>> unified interface to sync a filesystem other than the specialized
>>>> one.
>>>>
>>>> So I still use sync as Dave said.
>>>
>>> Mainly because "sync" is what users will use to make sure their data
>>> is safe. filesystem specific tools have a habit of doing "special
>>> stuff" to sync a filesystem, so it may not reflect the way users
>>> expect the system to behaviour when they run sync.
>>>
>>> The other option is this:
>>>
>>> _syncfs()
>>> {
>>> mntpt=$1
>>>
>>> $XFS_IO_PROG -c syncfs $mntpt
>>> }
>>>
>>> _sync_test()
>>> {
>>> _syncfs $TEST_DIR
>>> }
>>>
>>> _sync_scratch()
>>> {
>>> _syncfs $SCRATCH_MNT
>>> }
>>>
>>> which only runs sync on the filesystem that needs syncing (via the
>>> syncfs() syscall)
>>
>> I think that syncfs is better instead of sync because the syncing of
>> the specified filesystem is necessary.
>
> sync guarantees that. IOWs, there's no difference between sync and
> syncfs for the persepctive of this test and so the test does not
> need really need changing.
I agree that sync and syncfs is not different for this test.
However, sync is syncing of all filesystems, and, syncfs is syncing
of the specified filesystem only.
Therefor, I think that it is better to use sync and syncfs properly
according to the necessity.
Thanks,
Tsutomu
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2] fstests: btrfs: Add regression test for reserved space leak.
2015-08-05 1:08 ` Qu Wenruo
(?)
(?)
@ 2015-08-05 7:46 ` Filipe David Manana
-1 siblings, 0 replies; 12+ messages in thread
From: Filipe David Manana @ 2015-08-05 7:46 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs@vger.kernel.org, fstests
On Wed, Aug 5, 2015 at 2:08 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
> The regression is introduced in v4.2-rc1, with the big btrfs qgroup
> change.
> The problem is, qgroup reserved space is never freed, causing even we
> increase the limit, we can still hit the EDQUOT much faster than it
> should.
>
> Reported-by: Tsutomu Itoh <t-itoh@jp.fujitsu.com>
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Reviewed-by: Filipe Manana <fdmanana@suse.com>
Thanks!
> ---
> tests/btrfs/089 | 88 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> tests/btrfs/089.out | 5 +++
> tests/btrfs/group | 1 +
> 3 files changed, 94 insertions(+)
> create mode 100755 tests/btrfs/089
> create mode 100644 tests/btrfs/089.out
>
> diff --git a/tests/btrfs/089 b/tests/btrfs/089
> new file mode 100755
> index 0000000..82db96c
> --- /dev/null
> +++ b/tests/btrfs/089
> @@ -0,0 +1,88 @@
> +#! /bin/bash
> +# FS QA Test 089
> +#
> +# Regression test for btrfs qgroup reserved space leak.
> +#
> +# Due to qgroup reserved space leak, EDQUOT can be trigged even it's not
> +# over limit after previous write.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2015 Fujitsu. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +_need_to_be_root
> +
> +# Use big blocksize to ensure there is still enough space left
> +# for metadata reserve after hitting EDQUOT
> +BLOCKSIZE=$(( 2 * 1024 * 1024 ))
> +FILESIZE=$(( 128 * 1024 * 1024 )) # 128Mbytes
> +
> +# The last block won't be able to finish write, as metadata takes
> +# $NODESIZE space, causing the last block triggering EDQUOT
> +LENGTH=$(( $FILESIZE - $BLOCKSIZE ))
> +
> +_scratch_mkfs >>$seqres.full 2>&1
> +_scratch_mount
> +_require_fs_space $SCRATCH_MNT $(($FILESIZE * 2 / 1024))
> +
> +_run_btrfs_util_prog quota enable $SCRATCH_MNT
> +_run_btrfs_util_prog qgroup limit $FILESIZE 5 $SCRATCH_MNT
> +
> +$XFS_IO_PROG -f -c "pwrite -b $BLOCKSIZE 0 $LENGTH" \
> + $SCRATCH_MNT/foo | _filter_xfs_io
> +
> +# A sync is needed to trigger a commit_transaction.
> +# As the reserved space freeing happens at commit_transaction time,
> +# without a transaction commit, no reserved space needs freeing and
> +# won't trigger the bug.
> +sync
> +
> +# Double the limit to allow further write
> +_run_btrfs_util_prog qgroup limit $(($FILESIZE * 2)) 5 $SCRATCH_MNT
> +
> +# Test whether further write can succeed
> +$XFS_IO_PROG -f -c "pwrite -b $BLOCKSIZE $LENGTH $LENGTH" \
> + $SCRATCH_MNT/foo | _filter_xfs_io
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/089.out b/tests/btrfs/089.out
> new file mode 100644
> index 0000000..396888f
> --- /dev/null
> +++ b/tests/btrfs/089.out
> @@ -0,0 +1,5 @@
> +QA output created by 089
> +wrote 132120576/132120576 bytes at offset 0
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +wrote 132120576/132120576 bytes at offset 132120576
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index ffe18bf..225b532 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -91,6 +91,7 @@
> 086 auto quick clone
> 087 auto quick send
> 088 auto quick metadata
> +089 auto quick qgroup
> 090 auto quick metadata
> 091 auto quick qgroup
> 092 auto quick send
> --
> 1.8.3.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Filipe David Manana,
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
^ permalink raw reply [flat|nested] 12+ messages in thread