* [PATCH] fstest: btrfs: test single 4k extent after subpagesize buffered writes
@ 2016-03-07 23:47 Liu Bo
2016-03-08 0:27 ` [PATCH v2] " Liu Bo
0 siblings, 1 reply; 16+ messages in thread
From: Liu Bo @ 2016-03-07 23:47 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs
This is to test if COW enabled btrfs can end up with single 4k extents
when doing subpagesize buffered writes.
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
---
tests/btrfs/027 | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/027.out | 2 ++
tests/btrfs/group | 1 +
3 files changed, 97 insertions(+)
create mode 100755 tests/btrfs/027
create mode 100644 tests/btrfs/027.out
diff --git a/tests/btrfs/027 b/tests/btrfs/027
new file mode 100755
index 0000000..37a9920
--- /dev/null
+++ b/tests/btrfs/027
@@ -0,0 +1,94 @@
+#! /bin/bash
+# FS QA Test 027
+#
+# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with
+# single 4k extents.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Liu Bo. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+
+ # restore expire
+ echo $default_expire > /proc/sys/vm/dirty_expire_centisecs
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_xfs_io_command "fiemap"
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+default_expire=`cat /proc/sys/vm/dirty_expire_centisecs`
+echo 50 > /proc/sys/vm/dirty_expire_centisecs
+tfile=$SCRATCH_MNT/testfile
+
+PAGE_SIZE=$(get_page_size)
+SUBLEN=$((RANDOM % PAGE_SIZE))
+
+$XFS_IO_PROG -f -c "pwrite 0 $PAGE_SIZE" $tfile > /dev/null 2>&1
+$XFS_IO_PROG -c "pwrite $PAGE_SIZE $SUBLEN" $tfile > /dev/null 2>&1
+
+toff=$((PAGE_SIZE + SUBLEN))
+for ((i=0; i<10000; i++))
+do
+ tlen=$PAGE_SIZE
+ if [ $((i % 2)) = 0 ]; then
+ tlen=$((PAGE_SIZE * 3))
+ fi
+ if [ $((i % 1000)) = 0 ]; then
+ tlen=$((RANDOM % PAGE_SIZE))
+ fi
+
+ $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1
+ toff=$((toff + tlen))
+done
+
+sync
+
+# check for single PAGESIZE extent
+$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1
+$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}'
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out
new file mode 100644
index 0000000..6050bdc
--- /dev/null
+++ b/tests/btrfs/027.out
@@ -0,0 +1,2 @@
+QA output created by 027
+8
diff --git a/tests/btrfs/group b/tests/btrfs/group
index a2fa412..f06d5fe 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -29,6 +29,7 @@
024 auto quick compress
025 auto quick send clone
026 auto quick compress prealloc
+027 auto
029 auto quick clone
030 auto quick send
031 auto quick subvol clone
--
2.5.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-07 23:47 [PATCH] fstest: btrfs: test single 4k extent after subpagesize buffered writes Liu Bo
@ 2016-03-08 0:27 ` Liu Bo
2016-03-15 3:39 ` Dave Chinner
2016-03-17 22:56 ` Liu Bo
0 siblings, 2 replies; 16+ messages in thread
From: Liu Bo @ 2016-03-08 0:27 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs
This is to test if COW enabled btrfs can end up with single 4k extents
when doing subpagesize buffered writes.
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
---
v2: - Fix 027.out to make sure we don't get single 4k extents.
- Add the original mail list discussion as the reference.
tests/btrfs/027 | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/027.out | 1 +
tests/btrfs/group | 1 +
3 files changed, 99 insertions(+)
create mode 100755 tests/btrfs/027
create mode 100644 tests/btrfs/027.out
diff --git a/tests/btrfs/027 b/tests/btrfs/027
new file mode 100755
index 0000000..820973d
--- /dev/null
+++ b/tests/btrfs/027
@@ -0,0 +1,97 @@
+#! /bin/bash
+# FS QA Test 027
+#
+# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with
+# single 4k extents.
+# Ref:
+# "Stray 4k extents with slow buffered writes"
+# https://www.spinics.net/lists/linux-btrfs/msg52628.html
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Liu Bo. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+
+ # restore expire
+ echo $default_expire > /proc/sys/vm/dirty_expire_centisecs
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_xfs_io_command "fiemap"
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+default_expire=`cat /proc/sys/vm/dirty_expire_centisecs`
+echo 50 > /proc/sys/vm/dirty_expire_centisecs
+tfile=$SCRATCH_MNT/testfile
+
+PAGE_SIZE=$(get_page_size)
+SUBLEN=$((RANDOM % PAGE_SIZE))
+
+$XFS_IO_PROG -f -c "pwrite 0 $PAGE_SIZE" $tfile > /dev/null 2>&1
+$XFS_IO_PROG -c "pwrite $PAGE_SIZE $SUBLEN" $tfile > /dev/null 2>&1
+
+toff=$((PAGE_SIZE + SUBLEN))
+for ((i=0; i<10000; i++))
+do
+ tlen=$PAGE_SIZE
+ if [ $((i % 2)) = 0 ]; then
+ tlen=$((PAGE_SIZE * 3))
+ fi
+ if [ $((i % 1000)) = 0 ]; then
+ tlen=$((RANDOM % PAGE_SIZE))
+ fi
+
+ $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1
+ toff=$((toff + tlen))
+done
+
+sync
+
+# check for single PAGESIZE extent
+$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1
+$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}'
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out
new file mode 100644
index 0000000..e8291ab
--- /dev/null
+++ b/tests/btrfs/027.out
@@ -0,0 +1 @@
+QA output created by 027
diff --git a/tests/btrfs/group b/tests/btrfs/group
index a2fa412..f06d5fe 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -29,6 +29,7 @@
024 auto quick compress
025 auto quick send clone
026 auto quick compress prealloc
+027 auto
029 auto quick clone
030 auto quick send
031 auto quick subvol clone
--
2.5.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-08 0:27 ` [PATCH v2] " Liu Bo
@ 2016-03-15 3:39 ` Dave Chinner
2016-03-17 4:13 ` Liu Bo
2016-03-17 22:56 ` Liu Bo
1 sibling, 1 reply; 16+ messages in thread
From: Dave Chinner @ 2016-03-15 3:39 UTC (permalink / raw)
To: Liu Bo; +Cc: fstests, linux-btrfs
On Mon, Mar 07, 2016 at 04:27:59PM -0800, Liu Bo wrote:
> This is to test if COW enabled btrfs can end up with single 4k extents
> when doing subpagesize buffered writes.
>
> Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
> ---
....
> +_scratch_mkfs >> $seqres.full 2>&1
> +_scratch_mount
> +
> +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs`
> +echo 50 > /proc/sys/vm/dirty_expire_centisecs
why?
> +tfile=$SCRATCH_MNT/testfile
> +
> +PAGE_SIZE=$(get_page_size)
> +SUBLEN=$((RANDOM % PAGE_SIZE))
local variables are lower case.
> +$XFS_IO_PROG -f -c "pwrite 0 $PAGE_SIZE" $tfile > /dev/null 2>&1
> +$XFS_IO_PROG -c "pwrite $PAGE_SIZE $SUBLEN" $tfile > /dev/null 2>&1
> +
> +toff=$((PAGE_SIZE + SUBLEN))
> +for ((i=0; i<10000; i++))
> +do
# Add comment on what the loop is trying to do to the layout of the
# file being written.
for ((i = 0; i < 10000; i++)); do
> + tlen=$PAGE_SIZE
> + if [ $((i % 2)) = 0 ]; then
> + tlen=$((PAGE_SIZE * 3))
> + fi
> + if [ $((i % 1000)) = 0 ]; then
> + tlen=$((RANDOM % PAGE_SIZE))
> + fi
> +
> + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1
> + toff=$((toff + tlen))
> +done
> +
> +sync
> +
> +# check for single PAGESIZE extent
> +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1
> +$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}'
Assumes page size is 4k. Also assumes that no individual extent in
the file is 4k. Likely broken.
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out
> new file mode 100644
> index 0000000..e8291ab
> --- /dev/null
> +++ b/tests/btrfs/027.out
> @@ -0,0 +1 @@
> +QA output created by 027
So are we expecting no output or not?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-15 3:39 ` Dave Chinner
@ 2016-03-17 4:13 ` Liu Bo
2016-03-17 4:24 ` Eryu Guan
2016-03-17 21:07 ` Dave Chinner
0 siblings, 2 replies; 16+ messages in thread
From: Liu Bo @ 2016-03-17 4:13 UTC (permalink / raw)
To: Dave Chinner; +Cc: fstests, linux-btrfs
On Tue, Mar 15, 2016 at 02:39:41PM +1100, Dave Chinner wrote:
> On Mon, Mar 07, 2016 at 04:27:59PM -0800, Liu Bo wrote:
> > This is to test if COW enabled btrfs can end up with single 4k extents
> > when doing subpagesize buffered writes.
> >
> > Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
> > ---
> ....
> > +_scratch_mkfs >> $seqres.full 2>&1
> > +_scratch_mount
> > +
> > +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs`
> > +echo 50 > /proc/sys/vm/dirty_expire_centisecs
>
> why?
Setting it to 50 is to flush dirty pages more frequently so that it's
more likely to reproduce this bug.
But I just realized that this may not be a good testcase for xfstests as
there're many factors impacting writeback flushing, which means this may
work well on my machine but may not trigger the bug on others' machines.
>
> > +tfile=$SCRATCH_MNT/testfile
> > +
> > +PAGE_SIZE=$(get_page_size)
> > +SUBLEN=$((RANDOM % PAGE_SIZE))
>
> local variables are lower case.
OK.
>
> > +$XFS_IO_PROG -f -c "pwrite 0 $PAGE_SIZE" $tfile > /dev/null 2>&1
> > +$XFS_IO_PROG -c "pwrite $PAGE_SIZE $SUBLEN" $tfile > /dev/null 2>&1
> > +
> > +toff=$((PAGE_SIZE + SUBLEN))
> > +for ((i=0; i<10000; i++))
> > +do
>
> # Add comment on what the loop is trying to do to the layout of the
> # file being written.
> for ((i = 0; i < 10000; i++)); do
OK.
>
> > + tlen=$PAGE_SIZE
> > + if [ $((i % 2)) = 0 ]; then
> > + tlen=$((PAGE_SIZE * 3))
> > + fi
> > + if [ $((i % 1000)) = 0 ]; then
> > + tlen=$((RANDOM % PAGE_SIZE))
> > + fi
> > +
> > + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1
> > + toff=$((toff + tlen))
> > +done
> > +
> > +sync
> > +
> > +# check for single PAGESIZE extent
> > +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1
> > +$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}'
>
> Assumes page size is 4k. Also assumes that no individual extent in
> the file is 4k. Likely broken.
My miss, will try to come up with a way to tell awk PAGE_SIZE, maybe call getpagesize() ?
(if you already how to do it, please do me a favor :-) )
>
> > +# success, all done
> > +status=0
> > +exit
> > diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out
> > new file mode 100644
> > index 0000000..e8291ab
> > --- /dev/null
> > +++ b/tests/btrfs/027.out
> > @@ -0,0 +1 @@
> > +QA output created by 027
>
> So are we expecting no output or not?
We don't expect any 4k single extent if btrfs is doing correctly, so I leave the output empty.
Thanks for the comments, Dave!
Thanks,
-liubo
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-17 4:13 ` Liu Bo
@ 2016-03-17 4:24 ` Eryu Guan
2016-03-17 22:23 ` Liu Bo
2016-03-17 21:07 ` Dave Chinner
1 sibling, 1 reply; 16+ messages in thread
From: Eryu Guan @ 2016-03-17 4:24 UTC (permalink / raw)
To: Liu Bo; +Cc: Dave Chinner, fstests, linux-btrfs
On Wed, Mar 16, 2016 at 09:13:57PM -0700, Liu Bo wrote:
> On Tue, Mar 15, 2016 at 02:39:41PM +1100, Dave Chinner wrote:
> > On Mon, Mar 07, 2016 at 04:27:59PM -0800, Liu Bo wrote:
> > > This is to test if COW enabled btrfs can end up with single 4k extents
> > > when doing subpagesize buffered writes.
> > >
> > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
...
> > > +# check for single PAGESIZE extent
> > > +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1
> > > +$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}'
> >
> > Assumes page size is 4k. Also assumes that no individual extent in
> > the file is 4k. Likely broken.
>
> My miss, will try to come up with a way to tell awk PAGE_SIZE, maybe call getpagesize() ?
> (if you already how to do it, please do me a favor :-) )
./src/feature is able to do this, e.g. pagesize=`$here/src/feature -s`
Thanks,
Eryu
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-17 4:13 ` Liu Bo
2016-03-17 4:24 ` Eryu Guan
@ 2016-03-17 21:07 ` Dave Chinner
2016-03-17 21:31 ` Liu Bo
1 sibling, 1 reply; 16+ messages in thread
From: Dave Chinner @ 2016-03-17 21:07 UTC (permalink / raw)
To: Liu Bo; +Cc: fstests, linux-btrfs
On Wed, Mar 16, 2016 at 09:13:57PM -0700, Liu Bo wrote:
> On Tue, Mar 15, 2016 at 02:39:41PM +1100, Dave Chinner wrote:
> > On Mon, Mar 07, 2016 at 04:27:59PM -0800, Liu Bo wrote:
> > > This is to test if COW enabled btrfs can end up with single 4k extents
> > > when doing subpagesize buffered writes.
> > >
> > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
> > > ---
> > ....
> > > +_scratch_mkfs >> $seqres.full 2>&1
> > > +_scratch_mount
> > > +
> > > +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs`
> > > +echo 50 > /proc/sys/vm/dirty_expire_centisecs
> >
> > why?
>
> Setting it to 50 is to flush dirty pages more frequently so that it's
> more likely to reproduce this bug.
Explainations should be in comments.
....
> > > + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1
> > > + toff=$((toff + tlen))
> > > +done
> > > +
> > > +sync
> > > +
> > > +# check for single PAGESIZE extent
> > > +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1
> > > +$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}'
> >
> > Assumes page size is 4k. Also assumes that no individual extent in
> > the file is 4k. Likely broken.
>
> My miss, will try to come up with a way to tell awk PAGE_SIZE, maybe call getpagesize() ?
> (if you already how to do it, please do me a favor :-) )
awk -v pgsize=$page_size \
'{ cnt = pgsize/512; if ($4 == pgsize) print $4; }
Assuming that fiemap is reporting in 512 byte block size, not sector
sizes...
> > > --- /dev/null
> > > +++ b/tests/btrfs/027.out
> > > @@ -0,0 +1 @@
> > > +QA output created by 027
> >
> > So are we expecting no output or not?
>
> We don't expect any 4k single extent if btrfs is doing correctly, so I leave the output empty.
Convention is that we echo a line to the golden output file to
document the test is expected to give no output. That's why you see
echo "Silence is golden"
in many tests...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-17 21:07 ` Dave Chinner
@ 2016-03-17 21:31 ` Liu Bo
0 siblings, 0 replies; 16+ messages in thread
From: Liu Bo @ 2016-03-17 21:31 UTC (permalink / raw)
To: Dave Chinner; +Cc: fstests, linux-btrfs
On Fri, Mar 18, 2016 at 08:07:15AM +1100, Dave Chinner wrote:
> On Wed, Mar 16, 2016 at 09:13:57PM -0700, Liu Bo wrote:
> > On Tue, Mar 15, 2016 at 02:39:41PM +1100, Dave Chinner wrote:
> > > On Mon, Mar 07, 2016 at 04:27:59PM -0800, Liu Bo wrote:
> > > > This is to test if COW enabled btrfs can end up with single 4k extents
> > > > when doing subpagesize buffered writes.
> > > >
> > > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
> > > > ---
> > > ....
> > > > +_scratch_mkfs >> $seqres.full 2>&1
> > > > +_scratch_mount
> > > > +
> > > > +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs`
> > > > +echo 50 > /proc/sys/vm/dirty_expire_centisecs
> > >
> > > why?
> >
> > Setting it to 50 is to flush dirty pages more frequently so that it's
> > more likely to reproduce this bug.
>
> Explainations should be in comments.
Got it, that makes sense.
>
> ....
> > > > + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1
> > > > + toff=$((toff + tlen))
> > > > +done
> > > > +
> > > > +sync
> > > > +
> > > > +# check for single PAGESIZE extent
> > > > +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1
> > > > +$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}'
> > >
> > > Assumes page size is 4k. Also assumes that no individual extent in
> > > the file is 4k. Likely broken.
> >
> > My miss, will try to come up with a way to tell awk PAGE_SIZE, maybe call getpagesize() ?
> > (if you already how to do it, please do me a favor :-) )
>
> awk -v pgsize=$page_size \
> '{ cnt = pgsize/512; if ($4 == pgsize) print $4; }
>
> Assuming that fiemap is reporting in 512 byte block size, not sector
> sizes...
Yeah, I noticed that.
>
> > > > --- /dev/null
> > > > +++ b/tests/btrfs/027.out
> > > > @@ -0,0 +1 @@
> > > > +QA output created by 027
> > >
> > > So are we expecting no output or not?
> >
> > We don't expect any 4k single extent if btrfs is doing correctly, so I leave the output empty.
>
> Convention is that we echo a line to the golden output file to
> document the test is expected to give no output. That's why you see
>
> echo "Silence is golden"
>
> in many tests...
I see, will update it.
Thank you for the nice explanation.
Thanks,
-liubo
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-17 4:24 ` Eryu Guan
@ 2016-03-17 22:23 ` Liu Bo
0 siblings, 0 replies; 16+ messages in thread
From: Liu Bo @ 2016-03-17 22:23 UTC (permalink / raw)
To: Eryu Guan; +Cc: Dave Chinner, fstests, linux-btrfs
On Thu, Mar 17, 2016 at 12:24:35PM +0800, Eryu Guan wrote:
> On Wed, Mar 16, 2016 at 09:13:57PM -0700, Liu Bo wrote:
> > On Tue, Mar 15, 2016 at 02:39:41PM +1100, Dave Chinner wrote:
> > > On Mon, Mar 07, 2016 at 04:27:59PM -0800, Liu Bo wrote:
> > > > This is to test if COW enabled btrfs can end up with single 4k extents
> > > > when doing subpagesize buffered writes.
> > > >
> > > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
> ...
> > > > +# check for single PAGESIZE extent
> > > > +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1
> > > > +$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}'
> > >
> > > Assumes page size is 4k. Also assumes that no individual extent in
> > > the file is 4k. Likely broken.
> >
> > My miss, will try to come up with a way to tell awk PAGE_SIZE, maybe call getpagesize() ?
> > (if you already how to do it, please do me a favor :-) )
>
> ./src/feature is able to do this, e.g. pagesize=`$here/src/feature -s`
It works, thanks for reminding me of that.
Thanks,
-liubo
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-08 0:27 ` [PATCH v2] " Liu Bo
2016-03-15 3:39 ` Dave Chinner
@ 2016-03-17 22:56 ` Liu Bo
2016-03-22 4:00 ` Eryu Guan
2016-03-24 4:55 ` [PATCH v3] " Liu Bo
1 sibling, 2 replies; 16+ messages in thread
From: Liu Bo @ 2016-03-17 22:56 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs, Dave Chinner, Eryu Guan
This is to test if COW enabled btrfs can end up with single 4k extents
when doing subpagesize buffered writes.
The patch to fix the problem is
https://patchwork.kernel.org/patch/8527991/
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
---
v2: - Teach awk to know system's pagesize.
- Add "Silence is golden" to output.
- Use local variables to lower case.
- Add comments to make code clear.
tests/btrfs/027 | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/027.out | 2 ++
tests/btrfs/group | 1 +
3 files changed, 105 insertions(+)
create mode 100755 tests/btrfs/027
create mode 100644 tests/btrfs/027.out
diff --git a/tests/btrfs/027 b/tests/btrfs/027
new file mode 100755
index 0000000..19d324b
--- /dev/null
+++ b/tests/btrfs/027
@@ -0,0 +1,102 @@
+#! /bin/bash
+# FS QA Test 027
+#
+# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with
+# single 4k extents.
+# Ref:
+# "Stray 4k extents with slow buffered writes"
+# https://www.spinics.net/lists/linux-btrfs/msg52628.html
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Liu Bo. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+
+ # restore expire
+ echo $default_expire > /proc/sys/vm/dirty_expire_centisecs
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+echo "Silence is golden"
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_xfs_io_command "fiemap"
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+default_expire=`cat /proc/sys/vm/dirty_expire_centisecs`
+# Make it flush dirty pages more frequently to make sure we reproduce the bug.
+echo 50 > /proc/sys/vm/dirty_expire_centisecs
+
+tfile=$SCRATCH_MNT/testfile
+pagesize=$(get_page_size)
+sublen=$((RANDOM % pagesize))
+
+$XFS_IO_PROG -f -c "pwrite 0 $pagesize" $tfile > /dev/null 2>&1
+# write some subpagesize data first.
+$XFS_IO_PROG -c "pwrite $pagesize $sublen" $tfile > /dev/null 2>&1
+
+# Mix up "abnormal" subpagesize writes with normal pagesize based writes
+toff=$((pagesize + sublen))
+for ((i = 0; i < 10000; i++))
+do
+ tlen=$pagesize
+ if [ $((i % 2)) = 0 ]; then
+ tlen=$((pagesize * 3))
+ fi
+ if [ $((i % 1000)) = 0 ]; then
+ tlen=$((RANDOM % pagesize))
+ fi
+
+ $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1
+ toff=$((toff + tlen))
+done
+
+sync
+
+# check for single PAGESIZE extent
+$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1
+$XFS_IO_PROG -c "fiemap -v" $tfile | \
+awk -v pgsize=$pagesize '{n = pgsize/512; if ($4 == n) print $4}'
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out
new file mode 100644
index 0000000..ef78cda
--- /dev/null
+++ b/tests/btrfs/027.out
@@ -0,0 +1,2 @@
+QA output created by 027
+Silence is golden
diff --git a/tests/btrfs/group b/tests/btrfs/group
index a2fa412..f06d5fe 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -29,6 +29,7 @@
024 auto quick compress
025 auto quick send clone
026 auto quick compress prealloc
+027 auto
029 auto quick clone
030 auto quick send
031 auto quick subvol clone
--
2.5.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-17 22:56 ` Liu Bo
@ 2016-03-22 4:00 ` Eryu Guan
2016-03-22 22:12 ` Liu Bo
2016-03-24 4:55 ` [PATCH v3] " Liu Bo
1 sibling, 1 reply; 16+ messages in thread
From: Eryu Guan @ 2016-03-22 4:00 UTC (permalink / raw)
To: Liu Bo; +Cc: fstests, linux-btrfs, Dave Chinner
On Thu, Mar 17, 2016 at 03:56:38PM -0700, Liu Bo wrote:
> This is to test if COW enabled btrfs can end up with single 4k extents
> when doing subpagesize buffered writes.
What happens if btrfs is mounted with "nodatacow" option? Does it need
to _notrun if cow is disabled?
>
> The patch to fix the problem is
> https://patchwork.kernel.org/patch/8527991/
>
> Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
> ---
> v2: - Teach awk to know system's pagesize.
> - Add "Silence is golden" to output.
> - Use local variables to lower case.
> - Add comments to make code clear.
This should be v3, and this patch was buried in the v2 thread :)
>
> tests/btrfs/027 | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> tests/btrfs/027.out | 2 ++
> tests/btrfs/group | 1 +
> 3 files changed, 105 insertions(+)
> create mode 100755 tests/btrfs/027
> create mode 100644 tests/btrfs/027.out
>
> diff --git a/tests/btrfs/027 b/tests/btrfs/027
> new file mode 100755
> index 0000000..19d324b
> --- /dev/null
> +++ b/tests/btrfs/027
> @@ -0,0 +1,102 @@
> +#! /bin/bash
> +# FS QA Test 027
> +#
> +# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with
> +# single 4k extents.
> +# Ref:
> +# "Stray 4k extents with slow buffered writes"
> +# https://www.spinics.net/lists/linux-btrfs/msg52628.html
After going through this thread, my understanding is that nodatacow
btrfs should pass this test even on unpatched kernel (e.g. v4.5). But
my test on v4.5 kernel failed with nodatacow mount option, pagesize
extent is still found.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2016 Liu Bo. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +
> + # restore expire
> + echo $default_expire > /proc/sys/vm/dirty_expire_centisecs
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +echo "Silence is golden"
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +_require_xfs_io_command "fiemap"
> +
> +_scratch_mkfs >> $seqres.full 2>&1
> +_scratch_mount
> +
> +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs`
> +# Make it flush dirty pages more frequently to make sure we reproduce the bug.
> +echo 50 > /proc/sys/vm/dirty_expire_centisecs
> +
> +tfile=$SCRATCH_MNT/testfile
> +pagesize=$(get_page_size)
> +sublen=$((RANDOM % pagesize))
> +
> +$XFS_IO_PROG -f -c "pwrite 0 $pagesize" $tfile > /dev/null 2>&1
> +# write some subpagesize data first.
> +$XFS_IO_PROG -c "pwrite $pagesize $sublen" $tfile > /dev/null 2>&1
> +
> +# Mix up "abnormal" subpagesize writes with normal pagesize based writes
> +toff=$((pagesize + sublen))
> +for ((i = 0; i < 10000; i++))
> +do
> + tlen=$pagesize
> + if [ $((i % 2)) = 0 ]; then
> + tlen=$((pagesize * 3))
> + fi
> + if [ $((i % 1000)) = 0 ]; then
> + tlen=$((RANDOM % pagesize))
> + fi
> +
> + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1
> + toff=$((toff + tlen))
> +done
fstests prefers this format:
for ...; do
...
done
Thanks,
Eryu
> +
> +sync
> +
> +# check for single PAGESIZE extent
> +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1
> +$XFS_IO_PROG -c "fiemap -v" $tfile | \
> +awk -v pgsize=$pagesize '{n = pgsize/512; if ($4 == n) print $4}'
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out
> new file mode 100644
> index 0000000..ef78cda
> --- /dev/null
> +++ b/tests/btrfs/027.out
> @@ -0,0 +1,2 @@
> +QA output created by 027
> +Silence is golden
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index a2fa412..f06d5fe 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -29,6 +29,7 @@
> 024 auto quick compress
> 025 auto quick send clone
> 026 auto quick compress prealloc
> +027 auto
> 029 auto quick clone
> 030 auto quick send
> 031 auto quick subvol clone
> --
> 2.5.0
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-22 4:00 ` Eryu Guan
@ 2016-03-22 22:12 ` Liu Bo
2016-03-23 11:53 ` Eryu Guan
0 siblings, 1 reply; 16+ messages in thread
From: Liu Bo @ 2016-03-22 22:12 UTC (permalink / raw)
To: Eryu Guan; +Cc: fstests, linux-btrfs, Dave Chinner
On Tue, Mar 22, 2016 at 12:00:13PM +0800, Eryu Guan wrote:
> On Thu, Mar 17, 2016 at 03:56:38PM -0700, Liu Bo wrote:
> > This is to test if COW enabled btrfs can end up with single 4k extents
> > when doing subpagesize buffered writes.
>
> What happens if btrfs is mounted with "nodatacow" option? Does it need
> to _notrun if cow is disabled?
In my test, the test passes if mounting with "nodatacow".
Yes, it makes sense to have a _notrun for nodatacow.
>
> >
> > The patch to fix the problem is
> > https://patchwork.kernel.org/patch/8527991/
> >
> > Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
> > ---
> > v2: - Teach awk to know system's pagesize.
> > - Add "Silence is golden" to output.
> > - Use local variables to lower case.
> > - Add comments to make code clear.
>
> This should be v3, and this patch was buried in the v2 thread :)
Oops, thanks for pointing it out.
>
> >
> > tests/btrfs/027 | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> > tests/btrfs/027.out | 2 ++
> > tests/btrfs/group | 1 +
> > 3 files changed, 105 insertions(+)
> > create mode 100755 tests/btrfs/027
> > create mode 100644 tests/btrfs/027.out
> >
> > diff --git a/tests/btrfs/027 b/tests/btrfs/027
> > new file mode 100755
> > index 0000000..19d324b
> > --- /dev/null
> > +++ b/tests/btrfs/027
> > @@ -0,0 +1,102 @@
> > +#! /bin/bash
> > +# FS QA Test 027
> > +#
> > +# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with
> > +# single 4k extents.
> > +# Ref:
> > +# "Stray 4k extents with slow buffered writes"
> > +# https://www.spinics.net/lists/linux-btrfs/msg52628.html
>
> After going through this thread, my understanding is that nodatacow
> btrfs should pass this test even on unpatched kernel (e.g. v4.5). But
> my test on v4.5 kernel failed with nodatacow mount option, pagesize
> extent is still found.
>
I verified it again on my kvm box and it passed with a unpatched v4.5 kernel.
Can you please show me the 027.full file?
I can't think of a reason for this..
> > +#
> > +#-----------------------------------------------------------------------
> > +# Copyright (c) 2016 Liu Bo. All Rights Reserved.
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> > +#-----------------------------------------------------------------------
> > +#
> > +
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1 # failure is the default!
> > +trap "_cleanup; exit \$status" 0 1 2 3 15
> > +
> > +_cleanup()
> > +{
> > + cd /
> > + rm -f $tmp.*
> > +
> > + # restore expire
> > + echo $default_expire > /proc/sys/vm/dirty_expire_centisecs
> > +}
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +
> > +# remove previous $seqres.full before test
> > +rm -f $seqres.full
> > +echo "Silence is golden"
> > +
> > +# real QA test starts here
> > +
> > +# Modify as appropriate.
> > +_supported_fs btrfs
> > +_supported_os Linux
> > +_require_scratch
> > +_require_xfs_io_command "fiemap"
> > +
> > +_scratch_mkfs >> $seqres.full 2>&1
> > +_scratch_mount
> > +
> > +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs`
> > +# Make it flush dirty pages more frequently to make sure we reproduce the bug.
> > +echo 50 > /proc/sys/vm/dirty_expire_centisecs
> > +
> > +tfile=$SCRATCH_MNT/testfile
> > +pagesize=$(get_page_size)
> > +sublen=$((RANDOM % pagesize))
> > +
> > +$XFS_IO_PROG -f -c "pwrite 0 $pagesize" $tfile > /dev/null 2>&1
> > +# write some subpagesize data first.
> > +$XFS_IO_PROG -c "pwrite $pagesize $sublen" $tfile > /dev/null 2>&1
> > +
> > +# Mix up "abnormal" subpagesize writes with normal pagesize based writes
> > +toff=$((pagesize + sublen))
> > +for ((i = 0; i < 10000; i++))
> > +do
> > + tlen=$pagesize
> > + if [ $((i % 2)) = 0 ]; then
> > + tlen=$((pagesize * 3))
> > + fi
> > + if [ $((i % 1000)) = 0 ]; then
> > + tlen=$((RANDOM % pagesize))
> > + fi
> > +
> > + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1
> > + toff=$((toff + tlen))
> > +done
>
> fstests prefers this format:
>
> for ...; do
> ...
> done
OK, thank you very much, Eryu!
Thanks,
-liubo
>
> Thanks,
> Eryu
>
> > +
> > +sync
> > +
> > +# check for single PAGESIZE extent
> > +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1
> > +$XFS_IO_PROG -c "fiemap -v" $tfile | \
> > +awk -v pgsize=$pagesize '{n = pgsize/512; if ($4 == n) print $4}'
> > +
> > +# success, all done
> > +status=0
> > +exit
> > diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out
> > new file mode 100644
> > index 0000000..ef78cda
> > --- /dev/null
> > +++ b/tests/btrfs/027.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 027
> > +Silence is golden
> > diff --git a/tests/btrfs/group b/tests/btrfs/group
> > index a2fa412..f06d5fe 100644
> > --- a/tests/btrfs/group
> > +++ b/tests/btrfs/group
> > @@ -29,6 +29,7 @@
> > 024 auto quick compress
> > 025 auto quick send clone
> > 026 auto quick compress prealloc
> > +027 auto
> > 029 auto quick clone
> > 030 auto quick send
> > 031 auto quick subvol clone
> > --
> > 2.5.0
> >
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-22 22:12 ` Liu Bo
@ 2016-03-23 11:53 ` Eryu Guan
2016-03-24 4:52 ` Liu Bo
0 siblings, 1 reply; 16+ messages in thread
From: Eryu Guan @ 2016-03-23 11:53 UTC (permalink / raw)
To: Liu Bo; +Cc: fstests, linux-btrfs, Dave Chinner
On Tue, Mar 22, 2016 at 03:12:25PM -0700, Liu Bo wrote:
> On Tue, Mar 22, 2016 at 12:00:13PM +0800, Eryu Guan wrote:
> > On Thu, Mar 17, 2016 at 03:56:38PM -0700, Liu Bo wrote:
> > > This is to test if COW enabled btrfs can end up with single 4k extents
> > > when doing subpagesize buffered writes.
> >
> > What happens if btrfs is mounted with "nodatacow" option? Does it need
> > to _notrun if cow is disabled?
>
> In my test, the test passes if mounting with "nodatacow".
> Yes, it makes sense to have a _notrun for nodatacow.
If "nodatacow" btrfs should pass the test as well, then I don't think
_notrun is needed, so when it failed, something went wrong.
>
> >
> > >
> > > The patch to fix the problem is
> > > https://patchwork.kernel.org/patch/8527991/
> > >
> > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
> > > ---
> > > v2: - Teach awk to know system's pagesize.
> > > - Add "Silence is golden" to output.
> > > - Use local variables to lower case.
> > > - Add comments to make code clear.
> >
> > This should be v3, and this patch was buried in the v2 thread :)
>
> Oops, thanks for pointing it out.
>
> >
> > >
> > > tests/btrfs/027 | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > tests/btrfs/027.out | 2 ++
> > > tests/btrfs/group | 1 +
> > > 3 files changed, 105 insertions(+)
> > > create mode 100755 tests/btrfs/027
> > > create mode 100644 tests/btrfs/027.out
> > >
> > > diff --git a/tests/btrfs/027 b/tests/btrfs/027
> > > new file mode 100755
> > > index 0000000..19d324b
> > > --- /dev/null
> > > +++ b/tests/btrfs/027
> > > @@ -0,0 +1,102 @@
> > > +#! /bin/bash
> > > +# FS QA Test 027
> > > +#
> > > +# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with
> > > +# single 4k extents.
> > > +# Ref:
> > > +# "Stray 4k extents with slow buffered writes"
> > > +# https://www.spinics.net/lists/linux-btrfs/msg52628.html
> >
> > After going through this thread, my understanding is that nodatacow
> > btrfs should pass this test even on unpatched kernel (e.g. v4.5). But
> > my test on v4.5 kernel failed with nodatacow mount option, pagesize
> > extent is still found.
> >
>
> I verified it again on my kvm box and it passed with a unpatched v4.5 kernel.
>
> Can you please show me the 027.full file?
>
> I can't think of a reason for this..
I'm using v4.5 kernel and v4.4 btrfs-progs, and it's not reproduced
everytime.
SECTION -- btrfs_nodatacow
RECREATING -- btrfs on /dev/sda5
FSTYP -- btrfs
PLATFORM -- Linux/x86_64 dhcp-66-86-11 4.5.0
MKFS_OPTIONS -- /dev/sda6
MOUNT_OPTIONS -- -o nodatacow -o context=system_u:object_r:nfs_t:s0 /dev/sda6 /mnt/testarea/scratch
btrfs/027 28s ... - output mismatch (see /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad)
--- tests/btrfs/027.out 2016-03-23 15:39:41.562000000 +0800
+++ /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad 2016-03-23 19:37:38.962000000 +0800
@@ -1,2 +1,3 @@
QA output created by 027
Silence is golden
+8
...
(Run 'diff -u tests/btrfs/027.out /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad' to see the entire diff)
Ran: btrfs/027
Failures: btrfs/027
Failed 1 of 1 tests
And btrfs/027.full shows:
/mnt/testarea/scratch/testfile:
EXT: FILE-OFFSET BLOCK-RANGE TOTAL FLAGS
0: [0..28863]: 2154496..2183359 28864 0x0
1: [28864..57751]: 2183360..2212247 28888 0x0
2: [57752..85543]: 2212248..2240039 27792 0x0
3: [85544..113239]: 2240040..2267735 27696 0x0
4: [113240..113247]: 2267736..2267743 8 0x0
5: [113248..141999]: 2267744..2296495 28752 0x0
6: [142000..142023]: 2296496..2296519 24 0x0
7: [142024..159799]: 2296520..2314295 17776 0x1
Thanks,
Eryu
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-23 11:53 ` Eryu Guan
@ 2016-03-24 4:52 ` Liu Bo
2016-03-29 2:13 ` Liu Bo
0 siblings, 1 reply; 16+ messages in thread
From: Liu Bo @ 2016-03-24 4:52 UTC (permalink / raw)
To: Eryu Guan; +Cc: fstests, linux-btrfs, Dave Chinner
On Wed, Mar 23, 2016 at 07:53:38PM +0800, Eryu Guan wrote:
> On Tue, Mar 22, 2016 at 03:12:25PM -0700, Liu Bo wrote:
> > On Tue, Mar 22, 2016 at 12:00:13PM +0800, Eryu Guan wrote:
> > > On Thu, Mar 17, 2016 at 03:56:38PM -0700, Liu Bo wrote:
> > > > This is to test if COW enabled btrfs can end up with single 4k extents
> > > > when doing subpagesize buffered writes.
> > >
> > > What happens if btrfs is mounted with "nodatacow" option? Does it need
> > > to _notrun if cow is disabled?
> >
> > In my test, the test passes if mounting with "nodatacow".
> > Yes, it makes sense to have a _notrun for nodatacow.
>
> If "nodatacow" btrfs should pass the test as well, then I don't think
> _notrun is needed, so when it failed, something went wrong.
Ok, and it should pass in theory.
>
> >
> > >
> > > >
> > > > The patch to fix the problem is
> > > > https://patchwork.kernel.org/patch/8527991/
> > > >
> > > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
> > > > ---
> > > > v2: - Teach awk to know system's pagesize.
> > > > - Add "Silence is golden" to output.
> > > > - Use local variables to lower case.
> > > > - Add comments to make code clear.
> > >
> > > This should be v3, and this patch was buried in the v2 thread :)
> >
> > Oops, thanks for pointing it out.
> >
> > >
> > > >
> > > > tests/btrfs/027 | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > tests/btrfs/027.out | 2 ++
> > > > tests/btrfs/group | 1 +
> > > > 3 files changed, 105 insertions(+)
> > > > create mode 100755 tests/btrfs/027
> > > > create mode 100644 tests/btrfs/027.out
> > > >
> > > > diff --git a/tests/btrfs/027 b/tests/btrfs/027
> > > > new file mode 100755
> > > > index 0000000..19d324b
> > > > --- /dev/null
> > > > +++ b/tests/btrfs/027
> > > > @@ -0,0 +1,102 @@
> > > > +#! /bin/bash
> > > > +# FS QA Test 027
> > > > +#
> > > > +# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with
> > > > +# single 4k extents.
> > > > +# Ref:
> > > > +# "Stray 4k extents with slow buffered writes"
> > > > +# https://www.spinics.net/lists/linux-btrfs/msg52628.html
> > >
> > > After going through this thread, my understanding is that nodatacow
> > > btrfs should pass this test even on unpatched kernel (e.g. v4.5). But
> > > my test on v4.5 kernel failed with nodatacow mount option, pagesize
> > > extent is still found.
> > >
> >
> > I verified it again on my kvm box and it passed with a unpatched v4.5 kernel.
> >
> > Can you please show me the 027.full file?
> >
> > I can't think of a reason for this..
>
> I'm using v4.5 kernel and v4.4 btrfs-progs, and it's not reproduced
> everytime.
>
> SECTION -- btrfs_nodatacow
> RECREATING -- btrfs on /dev/sda5
> FSTYP -- btrfs
> PLATFORM -- Linux/x86_64 dhcp-66-86-11 4.5.0
> MKFS_OPTIONS -- /dev/sda6
> MOUNT_OPTIONS -- -o nodatacow -o context=system_u:object_r:nfs_t:s0 /dev/sda6 /mnt/testarea/scratch
>
> btrfs/027 28s ... - output mismatch (see /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad)
> --- tests/btrfs/027.out 2016-03-23 15:39:41.562000000 +0800
> +++ /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad 2016-03-23 19:37:38.962000000 +0800
> @@ -1,2 +1,3 @@
> QA output created by 027
> Silence is golden
> +8
> ...
> (Run 'diff -u tests/btrfs/027.out /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad' to see the entire diff)
> Ran: btrfs/027
> Failures: btrfs/027
> Failed 1 of 1 tests
>
> And btrfs/027.full shows:
>
> /mnt/testarea/scratch/testfile:
> EXT: FILE-OFFSET BLOCK-RANGE TOTAL FLAGS
> 0: [0..28863]: 2154496..2183359 28864 0x0
> 1: [28864..57751]: 2183360..2212247 28888 0x0
> 2: [57752..85543]: 2212248..2240039 27792 0x0
> 3: [85544..113239]: 2240040..2267735 27696 0x0
> 4: [113240..113247]: 2267736..2267743 8 0x0
> 5: [113248..141999]: 2267744..2296495 28752 0x0
> 6: [142000..142023]: 2296496..2296519 24 0x0
> 7: [142024..159799]: 2296520..2314295 17776 0x1
I can barely reproduce one in 100 runs... but anyway if it is a bug,
it's not a problem in this test case, I'll send a v3 version patch and
work on this nocow case.
Thanks,
-liubo
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v3] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-17 22:56 ` Liu Bo
2016-03-22 4:00 ` Eryu Guan
@ 2016-03-24 4:55 ` Liu Bo
2016-03-24 8:31 ` Eryu Guan
1 sibling, 1 reply; 16+ messages in thread
From: Liu Bo @ 2016-03-24 4:55 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs, Dave Chinner, Eryu Guan
This is to test if COW enabled btrfs can end up with single 4k extents
when doing subpagesize buffered writes.
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
---
v3: - fix style of 'for' loop.
v2: - Teach awk to know system's pagesize.
- Add "Silence is golden" to output.
- Use local variables to lower case.
- Add comments to make code clear.
tests/btrfs/027 | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/027.out | 2 ++
tests/btrfs/group | 1 +
3 files changed, 104 insertions(+)
create mode 100755 tests/btrfs/027
create mode 100644 tests/btrfs/027.out
diff --git a/tests/btrfs/027 b/tests/btrfs/027
new file mode 100755
index 0000000..bfed502
--- /dev/null
+++ b/tests/btrfs/027
@@ -0,0 +1,101 @@
+#! /bin/bash
+# FS QA Test 027
+#
+# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with
+# single 4k extents.
+# Ref:
+# "Stray 4k extents with slow buffered writes"
+# https://www.spinics.net/lists/linux-btrfs/msg52628.html
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Liu Bo. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+
+ # restore expire
+ echo $default_expire > /proc/sys/vm/dirty_expire_centisecs
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+echo "Silence is golden"
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_xfs_io_command "fiemap"
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+default_expire=`cat /proc/sys/vm/dirty_expire_centisecs`
+# Make it flush dirty pages more frequently to make sure we reproduce the bug.
+echo 50 > /proc/sys/vm/dirty_expire_centisecs
+
+tfile=$SCRATCH_MNT/testfile
+pagesize=$(get_page_size)
+sublen=$((RANDOM % pagesize))
+
+$XFS_IO_PROG -f -c "pwrite 0 $pagesize" $tfile > /dev/null 2>&1
+# write some subpagesize data first.
+$XFS_IO_PROG -c "pwrite $pagesize $sublen" $tfile > /dev/null 2>&1
+
+# Mix up "abnormal" subpagesize writes with normal pagesize based writes
+toff=$((pagesize + sublen))
+for ((i = 0; i < 10000; i++)); do
+ tlen=$pagesize
+ if [ $((i % 2)) = 0 ]; then
+ tlen=$((pagesize * 3))
+ fi
+ if [ $((i % 1000)) = 0 ]; then
+ tlen=$((RANDOM % pagesize))
+ fi
+
+ $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1
+ toff=$((toff + tlen))
+done
+
+sync
+
+# check for single PAGESIZE extent
+$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1
+$XFS_IO_PROG -c "fiemap -v" $tfile | \
+awk -v pgsize=$pagesize '{n = pgsize/512; if ($4 == n) print $4}'
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out
new file mode 100644
index 0000000..ef78cda
--- /dev/null
+++ b/tests/btrfs/027.out
@@ -0,0 +1,2 @@
+QA output created by 027
+Silence is golden
diff --git a/tests/btrfs/group b/tests/btrfs/group
index a2fa412..f06d5fe 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -29,6 +29,7 @@
024 auto quick compress
025 auto quick send clone
026 auto quick compress prealloc
+027 auto
029 auto quick clone
030 auto quick send
031 auto quick subvol clone
--
2.5.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH v3] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-24 4:55 ` [PATCH v3] " Liu Bo
@ 2016-03-24 8:31 ` Eryu Guan
0 siblings, 0 replies; 16+ messages in thread
From: Eryu Guan @ 2016-03-24 8:31 UTC (permalink / raw)
To: Liu Bo; +Cc: fstests, linux-btrfs, Dave Chinner
On Wed, Mar 23, 2016 at 09:55:52PM -0700, Liu Bo wrote:
> This is to test if COW enabled btrfs can end up with single 4k extents
> when doing subpagesize buffered writes.
>
> Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
Looks good to me.
Reviewed-by: Eryu Guan <eguan@redhat.com>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes
2016-03-24 4:52 ` Liu Bo
@ 2016-03-29 2:13 ` Liu Bo
0 siblings, 0 replies; 16+ messages in thread
From: Liu Bo @ 2016-03-29 2:13 UTC (permalink / raw)
To: Eryu Guan; +Cc: linux-btrfs
On Wed, Mar 23, 2016 at 09:52:21PM -0700, Liu Bo wrote:
> On Wed, Mar 23, 2016 at 07:53:38PM +0800, Eryu Guan wrote:
> > On Tue, Mar 22, 2016 at 03:12:25PM -0700, Liu Bo wrote:
> > > On Tue, Mar 22, 2016 at 12:00:13PM +0800, Eryu Guan wrote:
> > > > On Thu, Mar 17, 2016 at 03:56:38PM -0700, Liu Bo wrote:
> > > > > This is to test if COW enabled btrfs can end up with single 4k extents
> > > > > when doing subpagesize buffered writes.
> > > >
> > > > What happens if btrfs is mounted with "nodatacow" option? Does it need
> > > > to _notrun if cow is disabled?
> > >
> > > In my test, the test passes if mounting with "nodatacow".
> > > Yes, it makes sense to have a _notrun for nodatacow.
> >
> > If "nodatacow" btrfs should pass the test as well, then I don't think
> > _notrun is needed, so when it failed, something went wrong.
>
> Ok, and it should pass in theory.
>
> >
> > >
> > > >
> > > > >
> > > > > The patch to fix the problem is
> > > > > https://patchwork.kernel.org/patch/8527991/
> > > > >
> > > > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
> > > > > ---
> > > > > v2: - Teach awk to know system's pagesize.
> > > > > - Add "Silence is golden" to output.
> > > > > - Use local variables to lower case.
> > > > > - Add comments to make code clear.
> > > >
> > > > This should be v3, and this patch was buried in the v2 thread :)
> > >
> > > Oops, thanks for pointing it out.
> > >
> > > >
> > > > >
> > > > > tests/btrfs/027 | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > > tests/btrfs/027.out | 2 ++
> > > > > tests/btrfs/group | 1 +
> > > > > 3 files changed, 105 insertions(+)
> > > > > create mode 100755 tests/btrfs/027
> > > > > create mode 100644 tests/btrfs/027.out
> > > > >
> > > > > diff --git a/tests/btrfs/027 b/tests/btrfs/027
> > > > > new file mode 100755
> > > > > index 0000000..19d324b
> > > > > --- /dev/null
> > > > > +++ b/tests/btrfs/027
> > > > > @@ -0,0 +1,102 @@
> > > > > +#! /bin/bash
> > > > > +# FS QA Test 027
> > > > > +#
> > > > > +# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with
> > > > > +# single 4k extents.
> > > > > +# Ref:
> > > > > +# "Stray 4k extents with slow buffered writes"
> > > > > +# https://www.spinics.net/lists/linux-btrfs/msg52628.html
> > > >
> > > > After going through this thread, my understanding is that nodatacow
> > > > btrfs should pass this test even on unpatched kernel (e.g. v4.5). But
> > > > my test on v4.5 kernel failed with nodatacow mount option, pagesize
> > > > extent is still found.
> > > >
> > >
> > > I verified it again on my kvm box and it passed with a unpatched v4.5 kernel.
> > >
> > > Can you please show me the 027.full file?
> > >
> > > I can't think of a reason for this..
> >
> > I'm using v4.5 kernel and v4.4 btrfs-progs, and it's not reproduced
> > everytime.
> >
> > SECTION -- btrfs_nodatacow
> > RECREATING -- btrfs on /dev/sda5
> > FSTYP -- btrfs
> > PLATFORM -- Linux/x86_64 dhcp-66-86-11 4.5.0
> > MKFS_OPTIONS -- /dev/sda6
> > MOUNT_OPTIONS -- -o nodatacow -o context=system_u:object_r:nfs_t:s0 /dev/sda6 /mnt/testarea/scratch
> >
> > btrfs/027 28s ... - output mismatch (see /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad)
> > --- tests/btrfs/027.out 2016-03-23 15:39:41.562000000 +0800
> > +++ /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad 2016-03-23 19:37:38.962000000 +0800
> > @@ -1,2 +1,3 @@
> > QA output created by 027
> > Silence is golden
> > +8
> > ...
> > (Run 'diff -u tests/btrfs/027.out /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad' to see the entire diff)
> > Ran: btrfs/027
> > Failures: btrfs/027
> > Failed 1 of 1 tests
> >
> > And btrfs/027.full shows:
> >
> > /mnt/testarea/scratch/testfile:
> > EXT: FILE-OFFSET BLOCK-RANGE TOTAL FLAGS
> > 0: [0..28863]: 2154496..2183359 28864 0x0
> > 1: [28864..57751]: 2183360..2212247 28888 0x0
> > 2: [57752..85543]: 2212248..2240039 27792 0x0
> > 3: [85544..113239]: 2240040..2267735 27696 0x0
> > 4: [113240..113247]: 2267736..2267743 8 0x0
> > 5: [113248..141999]: 2267744..2296495 28752 0x0
> > 6: [142000..142023]: 2296496..2296519 24 0x0
> > 7: [142024..159799]: 2296520..2314295 17776 0x1
>
> I can barely reproduce one in 100 runs... but anyway if it is a bug,
> it's not a problem in this test case, I'll send a v3 version patch and
> work on this nocow case.
My trace results show that it's not a bug.
[0, 4096]
[4096, 8192]
...
[N-4096, N]
[N, N+4096]
[N+4096, N+8192]
...
There could be some latencies between writes against [N, N+4096] and writes against [N+4096, N+8192],
so when writeback starts between [N-4096, N] and [N, N+4096], btrfs will
find delayed allocation range ending at extent [N-4096, N], and then it
creates a extent to cover that range. Later [N, N+4096] is dirtied and
writeback thread will continue pick up the newly dirtied page and
allocate an extent for it, at which time the next [N+4096, N+8192] write
has not performed. This is how we get the single 4k extent for
nodatacow, but it's rare.
So I assume that even after fixing writeback index, we can still experience
this case in case of cow, but that should be quite rare, too.
Thanks,
-liubo
>
> Thanks,
>
> -liubo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2016-03-29 2:11 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-03-07 23:47 [PATCH] fstest: btrfs: test single 4k extent after subpagesize buffered writes Liu Bo
2016-03-08 0:27 ` [PATCH v2] " Liu Bo
2016-03-15 3:39 ` Dave Chinner
2016-03-17 4:13 ` Liu Bo
2016-03-17 4:24 ` Eryu Guan
2016-03-17 22:23 ` Liu Bo
2016-03-17 21:07 ` Dave Chinner
2016-03-17 21:31 ` Liu Bo
2016-03-17 22:56 ` Liu Bo
2016-03-22 4:00 ` Eryu Guan
2016-03-22 22:12 ` Liu Bo
2016-03-23 11:53 ` Eryu Guan
2016-03-24 4:52 ` Liu Bo
2016-03-29 2:13 ` Liu Bo
2016-03-24 4:55 ` [PATCH v3] " Liu Bo
2016-03-24 8:31 ` Eryu Guan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).