* [PATCH] fstest: btrfs: test single 4k extent after subpagesize buffered writes @ 2016-03-07 23:47 Liu Bo 2016-03-08 0:27 ` [PATCH v2] " Liu Bo 0 siblings, 1 reply; 16+ messages in thread From: Liu Bo @ 2016-03-07 23:47 UTC (permalink / raw) To: fstests; +Cc: linux-btrfs This is to test if COW enabled btrfs can end up with single 4k extents when doing subpagesize buffered writes. Signed-off-by: Liu Bo <bo.li.liu@oracle.com> --- tests/btrfs/027 | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/027.out | 2 ++ tests/btrfs/group | 1 + 3 files changed, 97 insertions(+) create mode 100755 tests/btrfs/027 create mode 100644 tests/btrfs/027.out diff --git a/tests/btrfs/027 b/tests/btrfs/027 new file mode 100755 index 0000000..37a9920 --- /dev/null +++ b/tests/btrfs/027 @@ -0,0 +1,94 @@ +#! /bin/bash +# FS QA Test 027 +# +# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with +# single 4k extents. +# +#----------------------------------------------------------------------- +# Copyright (c) 2016 Liu Bo. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +#----------------------------------------------------------------------- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* + + # restore expire + echo $default_expire > /proc/sys/vm/dirty_expire_centisecs +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# remove previous $seqres.full before test +rm -f $seqres.full + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_xfs_io_command "fiemap" + +_scratch_mkfs >> $seqres.full 2>&1 +_scratch_mount + +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs` +echo 50 > /proc/sys/vm/dirty_expire_centisecs +tfile=$SCRATCH_MNT/testfile + +PAGE_SIZE=$(get_page_size) +SUBLEN=$((RANDOM % PAGE_SIZE)) + +$XFS_IO_PROG -f -c "pwrite 0 $PAGE_SIZE" $tfile > /dev/null 2>&1 +$XFS_IO_PROG -c "pwrite $PAGE_SIZE $SUBLEN" $tfile > /dev/null 2>&1 + +toff=$((PAGE_SIZE + SUBLEN)) +for ((i=0; i<10000; i++)) +do + tlen=$PAGE_SIZE + if [ $((i % 2)) = 0 ]; then + tlen=$((PAGE_SIZE * 3)) + fi + if [ $((i % 1000)) = 0 ]; then + tlen=$((RANDOM % PAGE_SIZE)) + fi + + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1 + toff=$((toff + tlen)) +done + +sync + +# check for single PAGESIZE extent +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1 +$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}' + +# success, all done +status=0 +exit diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out new file mode 100644 index 0000000..6050bdc --- /dev/null +++ b/tests/btrfs/027.out @@ -0,0 +1,2 @@ +QA output created by 027 +8 diff --git a/tests/btrfs/group b/tests/btrfs/group index a2fa412..f06d5fe 100644 --- a/tests/btrfs/group +++ b/tests/btrfs/group @@ -29,6 +29,7 @@ 024 auto quick compress 025 auto quick send clone 026 auto quick compress prealloc +027 auto 029 auto quick clone 030 auto quick send 031 auto quick subvol clone -- 2.5.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-07 23:47 [PATCH] fstest: btrfs: test single 4k extent after subpagesize buffered writes Liu Bo @ 2016-03-08 0:27 ` Liu Bo 2016-03-15 3:39 ` Dave Chinner 2016-03-17 22:56 ` Liu Bo 0 siblings, 2 replies; 16+ messages in thread From: Liu Bo @ 2016-03-08 0:27 UTC (permalink / raw) To: fstests; +Cc: linux-btrfs This is to test if COW enabled btrfs can end up with single 4k extents when doing subpagesize buffered writes. Signed-off-by: Liu Bo <bo.li.liu@oracle.com> --- v2: - Fix 027.out to make sure we don't get single 4k extents. - Add the original mail list discussion as the reference. tests/btrfs/027 | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/027.out | 1 + tests/btrfs/group | 1 + 3 files changed, 99 insertions(+) create mode 100755 tests/btrfs/027 create mode 100644 tests/btrfs/027.out diff --git a/tests/btrfs/027 b/tests/btrfs/027 new file mode 100755 index 0000000..820973d --- /dev/null +++ b/tests/btrfs/027 @@ -0,0 +1,97 @@ +#! /bin/bash +# FS QA Test 027 +# +# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with +# single 4k extents. +# Ref: +# "Stray 4k extents with slow buffered writes" +# https://www.spinics.net/lists/linux-btrfs/msg52628.html +# +#----------------------------------------------------------------------- +# Copyright (c) 2016 Liu Bo. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +#----------------------------------------------------------------------- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* + + # restore expire + echo $default_expire > /proc/sys/vm/dirty_expire_centisecs +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# remove previous $seqres.full before test +rm -f $seqres.full + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_xfs_io_command "fiemap" + +_scratch_mkfs >> $seqres.full 2>&1 +_scratch_mount + +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs` +echo 50 > /proc/sys/vm/dirty_expire_centisecs +tfile=$SCRATCH_MNT/testfile + +PAGE_SIZE=$(get_page_size) +SUBLEN=$((RANDOM % PAGE_SIZE)) + +$XFS_IO_PROG -f -c "pwrite 0 $PAGE_SIZE" $tfile > /dev/null 2>&1 +$XFS_IO_PROG -c "pwrite $PAGE_SIZE $SUBLEN" $tfile > /dev/null 2>&1 + +toff=$((PAGE_SIZE + SUBLEN)) +for ((i=0; i<10000; i++)) +do + tlen=$PAGE_SIZE + if [ $((i % 2)) = 0 ]; then + tlen=$((PAGE_SIZE * 3)) + fi + if [ $((i % 1000)) = 0 ]; then + tlen=$((RANDOM % PAGE_SIZE)) + fi + + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1 + toff=$((toff + tlen)) +done + +sync + +# check for single PAGESIZE extent +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1 +$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}' + +# success, all done +status=0 +exit diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out new file mode 100644 index 0000000..e8291ab --- /dev/null +++ b/tests/btrfs/027.out @@ -0,0 +1 @@ +QA output created by 027 diff --git a/tests/btrfs/group b/tests/btrfs/group index a2fa412..f06d5fe 100644 --- a/tests/btrfs/group +++ b/tests/btrfs/group @@ -29,6 +29,7 @@ 024 auto quick compress 025 auto quick send clone 026 auto quick compress prealloc +027 auto 029 auto quick clone 030 auto quick send 031 auto quick subvol clone -- 2.5.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-08 0:27 ` [PATCH v2] " Liu Bo @ 2016-03-15 3:39 ` Dave Chinner 2016-03-17 4:13 ` Liu Bo 2016-03-17 22:56 ` Liu Bo 1 sibling, 1 reply; 16+ messages in thread From: Dave Chinner @ 2016-03-15 3:39 UTC (permalink / raw) To: Liu Bo; +Cc: fstests, linux-btrfs On Mon, Mar 07, 2016 at 04:27:59PM -0800, Liu Bo wrote: > This is to test if COW enabled btrfs can end up with single 4k extents > when doing subpagesize buffered writes. > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com> > --- .... > +_scratch_mkfs >> $seqres.full 2>&1 > +_scratch_mount > + > +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs` > +echo 50 > /proc/sys/vm/dirty_expire_centisecs why? > +tfile=$SCRATCH_MNT/testfile > + > +PAGE_SIZE=$(get_page_size) > +SUBLEN=$((RANDOM % PAGE_SIZE)) local variables are lower case. > +$XFS_IO_PROG -f -c "pwrite 0 $PAGE_SIZE" $tfile > /dev/null 2>&1 > +$XFS_IO_PROG -c "pwrite $PAGE_SIZE $SUBLEN" $tfile > /dev/null 2>&1 > + > +toff=$((PAGE_SIZE + SUBLEN)) > +for ((i=0; i<10000; i++)) > +do # Add comment on what the loop is trying to do to the layout of the # file being written. for ((i = 0; i < 10000; i++)); do > + tlen=$PAGE_SIZE > + if [ $((i % 2)) = 0 ]; then > + tlen=$((PAGE_SIZE * 3)) > + fi > + if [ $((i % 1000)) = 0 ]; then > + tlen=$((RANDOM % PAGE_SIZE)) > + fi > + > + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1 > + toff=$((toff + tlen)) > +done > + > +sync > + > +# check for single PAGESIZE extent > +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1 > +$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}' Assumes page size is 4k. Also assumes that no individual extent in the file is 4k. Likely broken. > +# success, all done > +status=0 > +exit > diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out > new file mode 100644 > index 0000000..e8291ab > --- /dev/null > +++ b/tests/btrfs/027.out > @@ -0,0 +1 @@ > +QA output created by 027 So are we expecting no output or not? Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-15 3:39 ` Dave Chinner @ 2016-03-17 4:13 ` Liu Bo 2016-03-17 4:24 ` Eryu Guan 2016-03-17 21:07 ` Dave Chinner 0 siblings, 2 replies; 16+ messages in thread From: Liu Bo @ 2016-03-17 4:13 UTC (permalink / raw) To: Dave Chinner; +Cc: fstests, linux-btrfs On Tue, Mar 15, 2016 at 02:39:41PM +1100, Dave Chinner wrote: > On Mon, Mar 07, 2016 at 04:27:59PM -0800, Liu Bo wrote: > > This is to test if COW enabled btrfs can end up with single 4k extents > > when doing subpagesize buffered writes. > > > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com> > > --- > .... > > +_scratch_mkfs >> $seqres.full 2>&1 > > +_scratch_mount > > + > > +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs` > > +echo 50 > /proc/sys/vm/dirty_expire_centisecs > > why? Setting it to 50 is to flush dirty pages more frequently so that it's more likely to reproduce this bug. But I just realized that this may not be a good testcase for xfstests as there're many factors impacting writeback flushing, which means this may work well on my machine but may not trigger the bug on others' machines. > > > +tfile=$SCRATCH_MNT/testfile > > + > > +PAGE_SIZE=$(get_page_size) > > +SUBLEN=$((RANDOM % PAGE_SIZE)) > > local variables are lower case. OK. > > > +$XFS_IO_PROG -f -c "pwrite 0 $PAGE_SIZE" $tfile > /dev/null 2>&1 > > +$XFS_IO_PROG -c "pwrite $PAGE_SIZE $SUBLEN" $tfile > /dev/null 2>&1 > > + > > +toff=$((PAGE_SIZE + SUBLEN)) > > +for ((i=0; i<10000; i++)) > > +do > > # Add comment on what the loop is trying to do to the layout of the > # file being written. > for ((i = 0; i < 10000; i++)); do OK. > > > + tlen=$PAGE_SIZE > > + if [ $((i % 2)) = 0 ]; then > > + tlen=$((PAGE_SIZE * 3)) > > + fi > > + if [ $((i % 1000)) = 0 ]; then > > + tlen=$((RANDOM % PAGE_SIZE)) > > + fi > > + > > + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1 > > + toff=$((toff + tlen)) > > +done > > + > > +sync > > + > > +# check for single PAGESIZE extent > > +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1 > > +$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}' > > Assumes page size is 4k. Also assumes that no individual extent in > the file is 4k. Likely broken. My miss, will try to come up with a way to tell awk PAGE_SIZE, maybe call getpagesize() ? (if you already how to do it, please do me a favor :-) ) > > > +# success, all done > > +status=0 > > +exit > > diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out > > new file mode 100644 > > index 0000000..e8291ab > > --- /dev/null > > +++ b/tests/btrfs/027.out > > @@ -0,0 +1 @@ > > +QA output created by 027 > > So are we expecting no output or not? We don't expect any 4k single extent if btrfs is doing correctly, so I leave the output empty. Thanks for the comments, Dave! Thanks, -liubo > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-17 4:13 ` Liu Bo @ 2016-03-17 4:24 ` Eryu Guan 2016-03-17 22:23 ` Liu Bo 2016-03-17 21:07 ` Dave Chinner 1 sibling, 1 reply; 16+ messages in thread From: Eryu Guan @ 2016-03-17 4:24 UTC (permalink / raw) To: Liu Bo; +Cc: Dave Chinner, fstests, linux-btrfs On Wed, Mar 16, 2016 at 09:13:57PM -0700, Liu Bo wrote: > On Tue, Mar 15, 2016 at 02:39:41PM +1100, Dave Chinner wrote: > > On Mon, Mar 07, 2016 at 04:27:59PM -0800, Liu Bo wrote: > > > This is to test if COW enabled btrfs can end up with single 4k extents > > > when doing subpagesize buffered writes. > > > > > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com> ... > > > +# check for single PAGESIZE extent > > > +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1 > > > +$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}' > > > > Assumes page size is 4k. Also assumes that no individual extent in > > the file is 4k. Likely broken. > > My miss, will try to come up with a way to tell awk PAGE_SIZE, maybe call getpagesize() ? > (if you already how to do it, please do me a favor :-) ) ./src/feature is able to do this, e.g. pagesize=`$here/src/feature -s` Thanks, Eryu ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-17 4:24 ` Eryu Guan @ 2016-03-17 22:23 ` Liu Bo 0 siblings, 0 replies; 16+ messages in thread From: Liu Bo @ 2016-03-17 22:23 UTC (permalink / raw) To: Eryu Guan; +Cc: Dave Chinner, fstests, linux-btrfs On Thu, Mar 17, 2016 at 12:24:35PM +0800, Eryu Guan wrote: > On Wed, Mar 16, 2016 at 09:13:57PM -0700, Liu Bo wrote: > > On Tue, Mar 15, 2016 at 02:39:41PM +1100, Dave Chinner wrote: > > > On Mon, Mar 07, 2016 at 04:27:59PM -0800, Liu Bo wrote: > > > > This is to test if COW enabled btrfs can end up with single 4k extents > > > > when doing subpagesize buffered writes. > > > > > > > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com> > ... > > > > +# check for single PAGESIZE extent > > > > +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1 > > > > +$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}' > > > > > > Assumes page size is 4k. Also assumes that no individual extent in > > > the file is 4k. Likely broken. > > > > My miss, will try to come up with a way to tell awk PAGE_SIZE, maybe call getpagesize() ? > > (if you already how to do it, please do me a favor :-) ) > > ./src/feature is able to do this, e.g. pagesize=`$here/src/feature -s` It works, thanks for reminding me of that. Thanks, -liubo ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-17 4:13 ` Liu Bo 2016-03-17 4:24 ` Eryu Guan @ 2016-03-17 21:07 ` Dave Chinner 2016-03-17 21:31 ` Liu Bo 1 sibling, 1 reply; 16+ messages in thread From: Dave Chinner @ 2016-03-17 21:07 UTC (permalink / raw) To: Liu Bo; +Cc: fstests, linux-btrfs On Wed, Mar 16, 2016 at 09:13:57PM -0700, Liu Bo wrote: > On Tue, Mar 15, 2016 at 02:39:41PM +1100, Dave Chinner wrote: > > On Mon, Mar 07, 2016 at 04:27:59PM -0800, Liu Bo wrote: > > > This is to test if COW enabled btrfs can end up with single 4k extents > > > when doing subpagesize buffered writes. > > > > > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com> > > > --- > > .... > > > +_scratch_mkfs >> $seqres.full 2>&1 > > > +_scratch_mount > > > + > > > +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs` > > > +echo 50 > /proc/sys/vm/dirty_expire_centisecs > > > > why? > > Setting it to 50 is to flush dirty pages more frequently so that it's > more likely to reproduce this bug. Explainations should be in comments. .... > > > + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1 > > > + toff=$((toff + tlen)) > > > +done > > > + > > > +sync > > > + > > > +# check for single PAGESIZE extent > > > +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1 > > > +$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}' > > > > Assumes page size is 4k. Also assumes that no individual extent in > > the file is 4k. Likely broken. > > My miss, will try to come up with a way to tell awk PAGE_SIZE, maybe call getpagesize() ? > (if you already how to do it, please do me a favor :-) ) awk -v pgsize=$page_size \ '{ cnt = pgsize/512; if ($4 == pgsize) print $4; } Assuming that fiemap is reporting in 512 byte block size, not sector sizes... > > > --- /dev/null > > > +++ b/tests/btrfs/027.out > > > @@ -0,0 +1 @@ > > > +QA output created by 027 > > > > So are we expecting no output or not? > > We don't expect any 4k single extent if btrfs is doing correctly, so I leave the output empty. Convention is that we echo a line to the golden output file to document the test is expected to give no output. That's why you see echo "Silence is golden" in many tests... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-17 21:07 ` Dave Chinner @ 2016-03-17 21:31 ` Liu Bo 0 siblings, 0 replies; 16+ messages in thread From: Liu Bo @ 2016-03-17 21:31 UTC (permalink / raw) To: Dave Chinner; +Cc: fstests, linux-btrfs On Fri, Mar 18, 2016 at 08:07:15AM +1100, Dave Chinner wrote: > On Wed, Mar 16, 2016 at 09:13:57PM -0700, Liu Bo wrote: > > On Tue, Mar 15, 2016 at 02:39:41PM +1100, Dave Chinner wrote: > > > On Mon, Mar 07, 2016 at 04:27:59PM -0800, Liu Bo wrote: > > > > This is to test if COW enabled btrfs can end up with single 4k extents > > > > when doing subpagesize buffered writes. > > > > > > > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com> > > > > --- > > > .... > > > > +_scratch_mkfs >> $seqres.full 2>&1 > > > > +_scratch_mount > > > > + > > > > +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs` > > > > +echo 50 > /proc/sys/vm/dirty_expire_centisecs > > > > > > why? > > > > Setting it to 50 is to flush dirty pages more frequently so that it's > > more likely to reproduce this bug. > > Explainations should be in comments. Got it, that makes sense. > > .... > > > > + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1 > > > > + toff=$((toff + tlen)) > > > > +done > > > > + > > > > +sync > > > > + > > > > +# check for single PAGESIZE extent > > > > +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1 > > > > +$XFS_IO_PROG -c "fiemap -v" $tfile | awk '{if ($4 == 8) print $4}' > > > > > > Assumes page size is 4k. Also assumes that no individual extent in > > > the file is 4k. Likely broken. > > > > My miss, will try to come up with a way to tell awk PAGE_SIZE, maybe call getpagesize() ? > > (if you already how to do it, please do me a favor :-) ) > > awk -v pgsize=$page_size \ > '{ cnt = pgsize/512; if ($4 == pgsize) print $4; } > > Assuming that fiemap is reporting in 512 byte block size, not sector > sizes... Yeah, I noticed that. > > > > > --- /dev/null > > > > +++ b/tests/btrfs/027.out > > > > @@ -0,0 +1 @@ > > > > +QA output created by 027 > > > > > > So are we expecting no output or not? > > > > We don't expect any 4k single extent if btrfs is doing correctly, so I leave the output empty. > > Convention is that we echo a line to the golden output file to > document the test is expected to give no output. That's why you see > > echo "Silence is golden" > > in many tests... I see, will update it. Thank you for the nice explanation. Thanks, -liubo ^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-08 0:27 ` [PATCH v2] " Liu Bo 2016-03-15 3:39 ` Dave Chinner @ 2016-03-17 22:56 ` Liu Bo 2016-03-22 4:00 ` Eryu Guan 2016-03-24 4:55 ` [PATCH v3] " Liu Bo 1 sibling, 2 replies; 16+ messages in thread From: Liu Bo @ 2016-03-17 22:56 UTC (permalink / raw) To: fstests; +Cc: linux-btrfs, Dave Chinner, Eryu Guan This is to test if COW enabled btrfs can end up with single 4k extents when doing subpagesize buffered writes. The patch to fix the problem is https://patchwork.kernel.org/patch/8527991/ Signed-off-by: Liu Bo <bo.li.liu@oracle.com> --- v2: - Teach awk to know system's pagesize. - Add "Silence is golden" to output. - Use local variables to lower case. - Add comments to make code clear. tests/btrfs/027 | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/027.out | 2 ++ tests/btrfs/group | 1 + 3 files changed, 105 insertions(+) create mode 100755 tests/btrfs/027 create mode 100644 tests/btrfs/027.out diff --git a/tests/btrfs/027 b/tests/btrfs/027 new file mode 100755 index 0000000..19d324b --- /dev/null +++ b/tests/btrfs/027 @@ -0,0 +1,102 @@ +#! /bin/bash +# FS QA Test 027 +# +# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with +# single 4k extents. +# Ref: +# "Stray 4k extents with slow buffered writes" +# https://www.spinics.net/lists/linux-btrfs/msg52628.html +# +#----------------------------------------------------------------------- +# Copyright (c) 2016 Liu Bo. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +#----------------------------------------------------------------------- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* + + # restore expire + echo $default_expire > /proc/sys/vm/dirty_expire_centisecs +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# remove previous $seqres.full before test +rm -f $seqres.full +echo "Silence is golden" + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_xfs_io_command "fiemap" + +_scratch_mkfs >> $seqres.full 2>&1 +_scratch_mount + +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs` +# Make it flush dirty pages more frequently to make sure we reproduce the bug. +echo 50 > /proc/sys/vm/dirty_expire_centisecs + +tfile=$SCRATCH_MNT/testfile +pagesize=$(get_page_size) +sublen=$((RANDOM % pagesize)) + +$XFS_IO_PROG -f -c "pwrite 0 $pagesize" $tfile > /dev/null 2>&1 +# write some subpagesize data first. +$XFS_IO_PROG -c "pwrite $pagesize $sublen" $tfile > /dev/null 2>&1 + +# Mix up "abnormal" subpagesize writes with normal pagesize based writes +toff=$((pagesize + sublen)) +for ((i = 0; i < 10000; i++)) +do + tlen=$pagesize + if [ $((i % 2)) = 0 ]; then + tlen=$((pagesize * 3)) + fi + if [ $((i % 1000)) = 0 ]; then + tlen=$((RANDOM % pagesize)) + fi + + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1 + toff=$((toff + tlen)) +done + +sync + +# check for single PAGESIZE extent +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1 +$XFS_IO_PROG -c "fiemap -v" $tfile | \ +awk -v pgsize=$pagesize '{n = pgsize/512; if ($4 == n) print $4}' + +# success, all done +status=0 +exit diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out new file mode 100644 index 0000000..ef78cda --- /dev/null +++ b/tests/btrfs/027.out @@ -0,0 +1,2 @@ +QA output created by 027 +Silence is golden diff --git a/tests/btrfs/group b/tests/btrfs/group index a2fa412..f06d5fe 100644 --- a/tests/btrfs/group +++ b/tests/btrfs/group @@ -29,6 +29,7 @@ 024 auto quick compress 025 auto quick send clone 026 auto quick compress prealloc +027 auto 029 auto quick clone 030 auto quick send 031 auto quick subvol clone -- 2.5.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-17 22:56 ` Liu Bo @ 2016-03-22 4:00 ` Eryu Guan 2016-03-22 22:12 ` Liu Bo 2016-03-24 4:55 ` [PATCH v3] " Liu Bo 1 sibling, 1 reply; 16+ messages in thread From: Eryu Guan @ 2016-03-22 4:00 UTC (permalink / raw) To: Liu Bo; +Cc: fstests, linux-btrfs, Dave Chinner On Thu, Mar 17, 2016 at 03:56:38PM -0700, Liu Bo wrote: > This is to test if COW enabled btrfs can end up with single 4k extents > when doing subpagesize buffered writes. What happens if btrfs is mounted with "nodatacow" option? Does it need to _notrun if cow is disabled? > > The patch to fix the problem is > https://patchwork.kernel.org/patch/8527991/ > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com> > --- > v2: - Teach awk to know system's pagesize. > - Add "Silence is golden" to output. > - Use local variables to lower case. > - Add comments to make code clear. This should be v3, and this patch was buried in the v2 thread :) > > tests/btrfs/027 | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > tests/btrfs/027.out | 2 ++ > tests/btrfs/group | 1 + > 3 files changed, 105 insertions(+) > create mode 100755 tests/btrfs/027 > create mode 100644 tests/btrfs/027.out > > diff --git a/tests/btrfs/027 b/tests/btrfs/027 > new file mode 100755 > index 0000000..19d324b > --- /dev/null > +++ b/tests/btrfs/027 > @@ -0,0 +1,102 @@ > +#! /bin/bash > +# FS QA Test 027 > +# > +# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with > +# single 4k extents. > +# Ref: > +# "Stray 4k extents with slow buffered writes" > +# https://www.spinics.net/lists/linux-btrfs/msg52628.html After going through this thread, my understanding is that nodatacow btrfs should pass this test even on unpatched kernel (e.g. v4.5). But my test on v4.5 kernel failed with nodatacow mount option, pagesize extent is still found. > +# > +#----------------------------------------------------------------------- > +# Copyright (c) 2016 Liu Bo. All Rights Reserved. > +# > +# This program is free software; you can redistribute it and/or > +# modify it under the terms of the GNU General Public License as > +# published by the Free Software Foundation. > +# > +# This program is distributed in the hope that it would be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program; if not, write the Free Software Foundation, > +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA > +#----------------------------------------------------------------------- > +# > + > +seq=`basename $0` > +seqres=$RESULT_DIR/$seq > +echo "QA output created by $seq" > + > +here=`pwd` > +tmp=/tmp/$$ > +status=1 # failure is the default! > +trap "_cleanup; exit \$status" 0 1 2 3 15 > + > +_cleanup() > +{ > + cd / > + rm -f $tmp.* > + > + # restore expire > + echo $default_expire > /proc/sys/vm/dirty_expire_centisecs > +} > + > +# get standard environment, filters and checks > +. ./common/rc > +. ./common/filter > + > +# remove previous $seqres.full before test > +rm -f $seqres.full > +echo "Silence is golden" > + > +# real QA test starts here > + > +# Modify as appropriate. > +_supported_fs btrfs > +_supported_os Linux > +_require_scratch > +_require_xfs_io_command "fiemap" > + > +_scratch_mkfs >> $seqres.full 2>&1 > +_scratch_mount > + > +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs` > +# Make it flush dirty pages more frequently to make sure we reproduce the bug. > +echo 50 > /proc/sys/vm/dirty_expire_centisecs > + > +tfile=$SCRATCH_MNT/testfile > +pagesize=$(get_page_size) > +sublen=$((RANDOM % pagesize)) > + > +$XFS_IO_PROG -f -c "pwrite 0 $pagesize" $tfile > /dev/null 2>&1 > +# write some subpagesize data first. > +$XFS_IO_PROG -c "pwrite $pagesize $sublen" $tfile > /dev/null 2>&1 > + > +# Mix up "abnormal" subpagesize writes with normal pagesize based writes > +toff=$((pagesize + sublen)) > +for ((i = 0; i < 10000; i++)) > +do > + tlen=$pagesize > + if [ $((i % 2)) = 0 ]; then > + tlen=$((pagesize * 3)) > + fi > + if [ $((i % 1000)) = 0 ]; then > + tlen=$((RANDOM % pagesize)) > + fi > + > + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1 > + toff=$((toff + tlen)) > +done fstests prefers this format: for ...; do ... done Thanks, Eryu > + > +sync > + > +# check for single PAGESIZE extent > +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1 > +$XFS_IO_PROG -c "fiemap -v" $tfile | \ > +awk -v pgsize=$pagesize '{n = pgsize/512; if ($4 == n) print $4}' > + > +# success, all done > +status=0 > +exit > diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out > new file mode 100644 > index 0000000..ef78cda > --- /dev/null > +++ b/tests/btrfs/027.out > @@ -0,0 +1,2 @@ > +QA output created by 027 > +Silence is golden > diff --git a/tests/btrfs/group b/tests/btrfs/group > index a2fa412..f06d5fe 100644 > --- a/tests/btrfs/group > +++ b/tests/btrfs/group > @@ -29,6 +29,7 @@ > 024 auto quick compress > 025 auto quick send clone > 026 auto quick compress prealloc > +027 auto > 029 auto quick clone > 030 auto quick send > 031 auto quick subvol clone > -- > 2.5.0 > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-22 4:00 ` Eryu Guan @ 2016-03-22 22:12 ` Liu Bo 2016-03-23 11:53 ` Eryu Guan 0 siblings, 1 reply; 16+ messages in thread From: Liu Bo @ 2016-03-22 22:12 UTC (permalink / raw) To: Eryu Guan; +Cc: fstests, linux-btrfs, Dave Chinner On Tue, Mar 22, 2016 at 12:00:13PM +0800, Eryu Guan wrote: > On Thu, Mar 17, 2016 at 03:56:38PM -0700, Liu Bo wrote: > > This is to test if COW enabled btrfs can end up with single 4k extents > > when doing subpagesize buffered writes. > > What happens if btrfs is mounted with "nodatacow" option? Does it need > to _notrun if cow is disabled? In my test, the test passes if mounting with "nodatacow". Yes, it makes sense to have a _notrun for nodatacow. > > > > > The patch to fix the problem is > > https://patchwork.kernel.org/patch/8527991/ > > > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com> > > --- > > v2: - Teach awk to know system's pagesize. > > - Add "Silence is golden" to output. > > - Use local variables to lower case. > > - Add comments to make code clear. > > This should be v3, and this patch was buried in the v2 thread :) Oops, thanks for pointing it out. > > > > > tests/btrfs/027 | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > > tests/btrfs/027.out | 2 ++ > > tests/btrfs/group | 1 + > > 3 files changed, 105 insertions(+) > > create mode 100755 tests/btrfs/027 > > create mode 100644 tests/btrfs/027.out > > > > diff --git a/tests/btrfs/027 b/tests/btrfs/027 > > new file mode 100755 > > index 0000000..19d324b > > --- /dev/null > > +++ b/tests/btrfs/027 > > @@ -0,0 +1,102 @@ > > +#! /bin/bash > > +# FS QA Test 027 > > +# > > +# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with > > +# single 4k extents. > > +# Ref: > > +# "Stray 4k extents with slow buffered writes" > > +# https://www.spinics.net/lists/linux-btrfs/msg52628.html > > After going through this thread, my understanding is that nodatacow > btrfs should pass this test even on unpatched kernel (e.g. v4.5). But > my test on v4.5 kernel failed with nodatacow mount option, pagesize > extent is still found. > I verified it again on my kvm box and it passed with a unpatched v4.5 kernel. Can you please show me the 027.full file? I can't think of a reason for this.. > > +# > > +#----------------------------------------------------------------------- > > +# Copyright (c) 2016 Liu Bo. All Rights Reserved. > > +# > > +# This program is free software; you can redistribute it and/or > > +# modify it under the terms of the GNU General Public License as > > +# published by the Free Software Foundation. > > +# > > +# This program is distributed in the hope that it would be useful, > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > +# GNU General Public License for more details. > > +# > > +# You should have received a copy of the GNU General Public License > > +# along with this program; if not, write the Free Software Foundation, > > +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA > > +#----------------------------------------------------------------------- > > +# > > + > > +seq=`basename $0` > > +seqres=$RESULT_DIR/$seq > > +echo "QA output created by $seq" > > + > > +here=`pwd` > > +tmp=/tmp/$$ > > +status=1 # failure is the default! > > +trap "_cleanup; exit \$status" 0 1 2 3 15 > > + > > +_cleanup() > > +{ > > + cd / > > + rm -f $tmp.* > > + > > + # restore expire > > + echo $default_expire > /proc/sys/vm/dirty_expire_centisecs > > +} > > + > > +# get standard environment, filters and checks > > +. ./common/rc > > +. ./common/filter > > + > > +# remove previous $seqres.full before test > > +rm -f $seqres.full > > +echo "Silence is golden" > > + > > +# real QA test starts here > > + > > +# Modify as appropriate. > > +_supported_fs btrfs > > +_supported_os Linux > > +_require_scratch > > +_require_xfs_io_command "fiemap" > > + > > +_scratch_mkfs >> $seqres.full 2>&1 > > +_scratch_mount > > + > > +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs` > > +# Make it flush dirty pages more frequently to make sure we reproduce the bug. > > +echo 50 > /proc/sys/vm/dirty_expire_centisecs > > + > > +tfile=$SCRATCH_MNT/testfile > > +pagesize=$(get_page_size) > > +sublen=$((RANDOM % pagesize)) > > + > > +$XFS_IO_PROG -f -c "pwrite 0 $pagesize" $tfile > /dev/null 2>&1 > > +# write some subpagesize data first. > > +$XFS_IO_PROG -c "pwrite $pagesize $sublen" $tfile > /dev/null 2>&1 > > + > > +# Mix up "abnormal" subpagesize writes with normal pagesize based writes > > +toff=$((pagesize + sublen)) > > +for ((i = 0; i < 10000; i++)) > > +do > > + tlen=$pagesize > > + if [ $((i % 2)) = 0 ]; then > > + tlen=$((pagesize * 3)) > > + fi > > + if [ $((i % 1000)) = 0 ]; then > > + tlen=$((RANDOM % pagesize)) > > + fi > > + > > + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1 > > + toff=$((toff + tlen)) > > +done > > fstests prefers this format: > > for ...; do > ... > done OK, thank you very much, Eryu! Thanks, -liubo > > Thanks, > Eryu > > > + > > +sync > > + > > +# check for single PAGESIZE extent > > +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1 > > +$XFS_IO_PROG -c "fiemap -v" $tfile | \ > > +awk -v pgsize=$pagesize '{n = pgsize/512; if ($4 == n) print $4}' > > + > > +# success, all done > > +status=0 > > +exit > > diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out > > new file mode 100644 > > index 0000000..ef78cda > > --- /dev/null > > +++ b/tests/btrfs/027.out > > @@ -0,0 +1,2 @@ > > +QA output created by 027 > > +Silence is golden > > diff --git a/tests/btrfs/group b/tests/btrfs/group > > index a2fa412..f06d5fe 100644 > > --- a/tests/btrfs/group > > +++ b/tests/btrfs/group > > @@ -29,6 +29,7 @@ > > 024 auto quick compress > > 025 auto quick send clone > > 026 auto quick compress prealloc > > +027 auto > > 029 auto quick clone > > 030 auto quick send > > 031 auto quick subvol clone > > -- > > 2.5.0 > > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-22 22:12 ` Liu Bo @ 2016-03-23 11:53 ` Eryu Guan 2016-03-24 4:52 ` Liu Bo 0 siblings, 1 reply; 16+ messages in thread From: Eryu Guan @ 2016-03-23 11:53 UTC (permalink / raw) To: Liu Bo; +Cc: fstests, linux-btrfs, Dave Chinner On Tue, Mar 22, 2016 at 03:12:25PM -0700, Liu Bo wrote: > On Tue, Mar 22, 2016 at 12:00:13PM +0800, Eryu Guan wrote: > > On Thu, Mar 17, 2016 at 03:56:38PM -0700, Liu Bo wrote: > > > This is to test if COW enabled btrfs can end up with single 4k extents > > > when doing subpagesize buffered writes. > > > > What happens if btrfs is mounted with "nodatacow" option? Does it need > > to _notrun if cow is disabled? > > In my test, the test passes if mounting with "nodatacow". > Yes, it makes sense to have a _notrun for nodatacow. If "nodatacow" btrfs should pass the test as well, then I don't think _notrun is needed, so when it failed, something went wrong. > > > > > > > > > The patch to fix the problem is > > > https://patchwork.kernel.org/patch/8527991/ > > > > > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com> > > > --- > > > v2: - Teach awk to know system's pagesize. > > > - Add "Silence is golden" to output. > > > - Use local variables to lower case. > > > - Add comments to make code clear. > > > > This should be v3, and this patch was buried in the v2 thread :) > > Oops, thanks for pointing it out. > > > > > > > > > tests/btrfs/027 | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > tests/btrfs/027.out | 2 ++ > > > tests/btrfs/group | 1 + > > > 3 files changed, 105 insertions(+) > > > create mode 100755 tests/btrfs/027 > > > create mode 100644 tests/btrfs/027.out > > > > > > diff --git a/tests/btrfs/027 b/tests/btrfs/027 > > > new file mode 100755 > > > index 0000000..19d324b > > > --- /dev/null > > > +++ b/tests/btrfs/027 > > > @@ -0,0 +1,102 @@ > > > +#! /bin/bash > > > +# FS QA Test 027 > > > +# > > > +# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with > > > +# single 4k extents. > > > +# Ref: > > > +# "Stray 4k extents with slow buffered writes" > > > +# https://www.spinics.net/lists/linux-btrfs/msg52628.html > > > > After going through this thread, my understanding is that nodatacow > > btrfs should pass this test even on unpatched kernel (e.g. v4.5). But > > my test on v4.5 kernel failed with nodatacow mount option, pagesize > > extent is still found. > > > > I verified it again on my kvm box and it passed with a unpatched v4.5 kernel. > > Can you please show me the 027.full file? > > I can't think of a reason for this.. I'm using v4.5 kernel and v4.4 btrfs-progs, and it's not reproduced everytime. SECTION -- btrfs_nodatacow RECREATING -- btrfs on /dev/sda5 FSTYP -- btrfs PLATFORM -- Linux/x86_64 dhcp-66-86-11 4.5.0 MKFS_OPTIONS -- /dev/sda6 MOUNT_OPTIONS -- -o nodatacow -o context=system_u:object_r:nfs_t:s0 /dev/sda6 /mnt/testarea/scratch btrfs/027 28s ... - output mismatch (see /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad) --- tests/btrfs/027.out 2016-03-23 15:39:41.562000000 +0800 +++ /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad 2016-03-23 19:37:38.962000000 +0800 @@ -1,2 +1,3 @@ QA output created by 027 Silence is golden +8 ... (Run 'diff -u tests/btrfs/027.out /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad' to see the entire diff) Ran: btrfs/027 Failures: btrfs/027 Failed 1 of 1 tests And btrfs/027.full shows: /mnt/testarea/scratch/testfile: EXT: FILE-OFFSET BLOCK-RANGE TOTAL FLAGS 0: [0..28863]: 2154496..2183359 28864 0x0 1: [28864..57751]: 2183360..2212247 28888 0x0 2: [57752..85543]: 2212248..2240039 27792 0x0 3: [85544..113239]: 2240040..2267735 27696 0x0 4: [113240..113247]: 2267736..2267743 8 0x0 5: [113248..141999]: 2267744..2296495 28752 0x0 6: [142000..142023]: 2296496..2296519 24 0x0 7: [142024..159799]: 2296520..2314295 17776 0x1 Thanks, Eryu ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-23 11:53 ` Eryu Guan @ 2016-03-24 4:52 ` Liu Bo 2016-03-29 2:13 ` Liu Bo 0 siblings, 1 reply; 16+ messages in thread From: Liu Bo @ 2016-03-24 4:52 UTC (permalink / raw) To: Eryu Guan; +Cc: fstests, linux-btrfs, Dave Chinner On Wed, Mar 23, 2016 at 07:53:38PM +0800, Eryu Guan wrote: > On Tue, Mar 22, 2016 at 03:12:25PM -0700, Liu Bo wrote: > > On Tue, Mar 22, 2016 at 12:00:13PM +0800, Eryu Guan wrote: > > > On Thu, Mar 17, 2016 at 03:56:38PM -0700, Liu Bo wrote: > > > > This is to test if COW enabled btrfs can end up with single 4k extents > > > > when doing subpagesize buffered writes. > > > > > > What happens if btrfs is mounted with "nodatacow" option? Does it need > > > to _notrun if cow is disabled? > > > > In my test, the test passes if mounting with "nodatacow". > > Yes, it makes sense to have a _notrun for nodatacow. > > If "nodatacow" btrfs should pass the test as well, then I don't think > _notrun is needed, so when it failed, something went wrong. Ok, and it should pass in theory. > > > > > > > > > > > > > > The patch to fix the problem is > > > > https://patchwork.kernel.org/patch/8527991/ > > > > > > > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com> > > > > --- > > > > v2: - Teach awk to know system's pagesize. > > > > - Add "Silence is golden" to output. > > > > - Use local variables to lower case. > > > > - Add comments to make code clear. > > > > > > This should be v3, and this patch was buried in the v2 thread :) > > > > Oops, thanks for pointing it out. > > > > > > > > > > > > > tests/btrfs/027 | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > tests/btrfs/027.out | 2 ++ > > > > tests/btrfs/group | 1 + > > > > 3 files changed, 105 insertions(+) > > > > create mode 100755 tests/btrfs/027 > > > > create mode 100644 tests/btrfs/027.out > > > > > > > > diff --git a/tests/btrfs/027 b/tests/btrfs/027 > > > > new file mode 100755 > > > > index 0000000..19d324b > > > > --- /dev/null > > > > +++ b/tests/btrfs/027 > > > > @@ -0,0 +1,102 @@ > > > > +#! /bin/bash > > > > +# FS QA Test 027 > > > > +# > > > > +# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with > > > > +# single 4k extents. > > > > +# Ref: > > > > +# "Stray 4k extents with slow buffered writes" > > > > +# https://www.spinics.net/lists/linux-btrfs/msg52628.html > > > > > > After going through this thread, my understanding is that nodatacow > > > btrfs should pass this test even on unpatched kernel (e.g. v4.5). But > > > my test on v4.5 kernel failed with nodatacow mount option, pagesize > > > extent is still found. > > > > > > > I verified it again on my kvm box and it passed with a unpatched v4.5 kernel. > > > > Can you please show me the 027.full file? > > > > I can't think of a reason for this.. > > I'm using v4.5 kernel and v4.4 btrfs-progs, and it's not reproduced > everytime. > > SECTION -- btrfs_nodatacow > RECREATING -- btrfs on /dev/sda5 > FSTYP -- btrfs > PLATFORM -- Linux/x86_64 dhcp-66-86-11 4.5.0 > MKFS_OPTIONS -- /dev/sda6 > MOUNT_OPTIONS -- -o nodatacow -o context=system_u:object_r:nfs_t:s0 /dev/sda6 /mnt/testarea/scratch > > btrfs/027 28s ... - output mismatch (see /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad) > --- tests/btrfs/027.out 2016-03-23 15:39:41.562000000 +0800 > +++ /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad 2016-03-23 19:37:38.962000000 +0800 > @@ -1,2 +1,3 @@ > QA output created by 027 > Silence is golden > +8 > ... > (Run 'diff -u tests/btrfs/027.out /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad' to see the entire diff) > Ran: btrfs/027 > Failures: btrfs/027 > Failed 1 of 1 tests > > And btrfs/027.full shows: > > /mnt/testarea/scratch/testfile: > EXT: FILE-OFFSET BLOCK-RANGE TOTAL FLAGS > 0: [0..28863]: 2154496..2183359 28864 0x0 > 1: [28864..57751]: 2183360..2212247 28888 0x0 > 2: [57752..85543]: 2212248..2240039 27792 0x0 > 3: [85544..113239]: 2240040..2267735 27696 0x0 > 4: [113240..113247]: 2267736..2267743 8 0x0 > 5: [113248..141999]: 2267744..2296495 28752 0x0 > 6: [142000..142023]: 2296496..2296519 24 0x0 > 7: [142024..159799]: 2296520..2314295 17776 0x1 I can barely reproduce one in 100 runs... but anyway if it is a bug, it's not a problem in this test case, I'll send a v3 version patch and work on this nocow case. Thanks, -liubo ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-24 4:52 ` Liu Bo @ 2016-03-29 2:13 ` Liu Bo 0 siblings, 0 replies; 16+ messages in thread From: Liu Bo @ 2016-03-29 2:13 UTC (permalink / raw) To: Eryu Guan; +Cc: linux-btrfs On Wed, Mar 23, 2016 at 09:52:21PM -0700, Liu Bo wrote: > On Wed, Mar 23, 2016 at 07:53:38PM +0800, Eryu Guan wrote: > > On Tue, Mar 22, 2016 at 03:12:25PM -0700, Liu Bo wrote: > > > On Tue, Mar 22, 2016 at 12:00:13PM +0800, Eryu Guan wrote: > > > > On Thu, Mar 17, 2016 at 03:56:38PM -0700, Liu Bo wrote: > > > > > This is to test if COW enabled btrfs can end up with single 4k extents > > > > > when doing subpagesize buffered writes. > > > > > > > > What happens if btrfs is mounted with "nodatacow" option? Does it need > > > > to _notrun if cow is disabled? > > > > > > In my test, the test passes if mounting with "nodatacow". > > > Yes, it makes sense to have a _notrun for nodatacow. > > > > If "nodatacow" btrfs should pass the test as well, then I don't think > > _notrun is needed, so when it failed, something went wrong. > > Ok, and it should pass in theory. > > > > > > > > > > > > > > > > > > > > The patch to fix the problem is > > > > > https://patchwork.kernel.org/patch/8527991/ > > > > > > > > > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com> > > > > > --- > > > > > v2: - Teach awk to know system's pagesize. > > > > > - Add "Silence is golden" to output. > > > > > - Use local variables to lower case. > > > > > - Add comments to make code clear. > > > > > > > > This should be v3, and this patch was buried in the v2 thread :) > > > > > > Oops, thanks for pointing it out. > > > > > > > > > > > > > > > > > tests/btrfs/027 | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > tests/btrfs/027.out | 2 ++ > > > > > tests/btrfs/group | 1 + > > > > > 3 files changed, 105 insertions(+) > > > > > create mode 100755 tests/btrfs/027 > > > > > create mode 100644 tests/btrfs/027.out > > > > > > > > > > diff --git a/tests/btrfs/027 b/tests/btrfs/027 > > > > > new file mode 100755 > > > > > index 0000000..19d324b > > > > > --- /dev/null > > > > > +++ b/tests/btrfs/027 > > > > > @@ -0,0 +1,102 @@ > > > > > +#! /bin/bash > > > > > +# FS QA Test 027 > > > > > +# > > > > > +# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with > > > > > +# single 4k extents. > > > > > +# Ref: > > > > > +# "Stray 4k extents with slow buffered writes" > > > > > +# https://www.spinics.net/lists/linux-btrfs/msg52628.html > > > > > > > > After going through this thread, my understanding is that nodatacow > > > > btrfs should pass this test even on unpatched kernel (e.g. v4.5). But > > > > my test on v4.5 kernel failed with nodatacow mount option, pagesize > > > > extent is still found. > > > > > > > > > > I verified it again on my kvm box and it passed with a unpatched v4.5 kernel. > > > > > > Can you please show me the 027.full file? > > > > > > I can't think of a reason for this.. > > > > I'm using v4.5 kernel and v4.4 btrfs-progs, and it's not reproduced > > everytime. > > > > SECTION -- btrfs_nodatacow > > RECREATING -- btrfs on /dev/sda5 > > FSTYP -- btrfs > > PLATFORM -- Linux/x86_64 dhcp-66-86-11 4.5.0 > > MKFS_OPTIONS -- /dev/sda6 > > MOUNT_OPTIONS -- -o nodatacow -o context=system_u:object_r:nfs_t:s0 /dev/sda6 /mnt/testarea/scratch > > > > btrfs/027 28s ... - output mismatch (see /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad) > > --- tests/btrfs/027.out 2016-03-23 15:39:41.562000000 +0800 > > +++ /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad 2016-03-23 19:37:38.962000000 +0800 > > @@ -1,2 +1,3 @@ > > QA output created by 027 > > Silence is golden > > +8 > > ... > > (Run 'diff -u tests/btrfs/027.out /root/xfstests/results//btrfs_nodatacow/btrfs/027.out.bad' to see the entire diff) > > Ran: btrfs/027 > > Failures: btrfs/027 > > Failed 1 of 1 tests > > > > And btrfs/027.full shows: > > > > /mnt/testarea/scratch/testfile: > > EXT: FILE-OFFSET BLOCK-RANGE TOTAL FLAGS > > 0: [0..28863]: 2154496..2183359 28864 0x0 > > 1: [28864..57751]: 2183360..2212247 28888 0x0 > > 2: [57752..85543]: 2212248..2240039 27792 0x0 > > 3: [85544..113239]: 2240040..2267735 27696 0x0 > > 4: [113240..113247]: 2267736..2267743 8 0x0 > > 5: [113248..141999]: 2267744..2296495 28752 0x0 > > 6: [142000..142023]: 2296496..2296519 24 0x0 > > 7: [142024..159799]: 2296520..2314295 17776 0x1 > > I can barely reproduce one in 100 runs... but anyway if it is a bug, > it's not a problem in this test case, I'll send a v3 version patch and > work on this nocow case. My trace results show that it's not a bug. [0, 4096] [4096, 8192] ... [N-4096, N] [N, N+4096] [N+4096, N+8192] ... There could be some latencies between writes against [N, N+4096] and writes against [N+4096, N+8192], so when writeback starts between [N-4096, N] and [N, N+4096], btrfs will find delayed allocation range ending at extent [N-4096, N], and then it creates a extent to cover that range. Later [N, N+4096] is dirtied and writeback thread will continue pick up the newly dirtied page and allocate an extent for it, at which time the next [N+4096, N+8192] write has not performed. This is how we get the single 4k extent for nodatacow, but it's rare. So I assume that even after fixing writeback index, we can still experience this case in case of cow, but that should be quite rare, too. Thanks, -liubo > > Thanks, > > -liubo > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v3] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-17 22:56 ` Liu Bo 2016-03-22 4:00 ` Eryu Guan @ 2016-03-24 4:55 ` Liu Bo 2016-03-24 8:31 ` Eryu Guan 1 sibling, 1 reply; 16+ messages in thread From: Liu Bo @ 2016-03-24 4:55 UTC (permalink / raw) To: fstests; +Cc: linux-btrfs, Dave Chinner, Eryu Guan This is to test if COW enabled btrfs can end up with single 4k extents when doing subpagesize buffered writes. Signed-off-by: Liu Bo <bo.li.liu@oracle.com> --- v3: - fix style of 'for' loop. v2: - Teach awk to know system's pagesize. - Add "Silence is golden" to output. - Use local variables to lower case. - Add comments to make code clear. tests/btrfs/027 | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/btrfs/027.out | 2 ++ tests/btrfs/group | 1 + 3 files changed, 104 insertions(+) create mode 100755 tests/btrfs/027 create mode 100644 tests/btrfs/027.out diff --git a/tests/btrfs/027 b/tests/btrfs/027 new file mode 100755 index 0000000..bfed502 --- /dev/null +++ b/tests/btrfs/027 @@ -0,0 +1,101 @@ +#! /bin/bash +# FS QA Test 027 +# +# When btrfs is using cow mode, buffered writes of sub-pagesize can end up with +# single 4k extents. +# Ref: +# "Stray 4k extents with slow buffered writes" +# https://www.spinics.net/lists/linux-btrfs/msg52628.html +# +#----------------------------------------------------------------------- +# Copyright (c) 2016 Liu Bo. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +#----------------------------------------------------------------------- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +_cleanup() +{ + cd / + rm -f $tmp.* + + # restore expire + echo $default_expire > /proc/sys/vm/dirty_expire_centisecs +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter + +# remove previous $seqres.full before test +rm -f $seqres.full +echo "Silence is golden" + +# real QA test starts here + +# Modify as appropriate. +_supported_fs btrfs +_supported_os Linux +_require_scratch +_require_xfs_io_command "fiemap" + +_scratch_mkfs >> $seqres.full 2>&1 +_scratch_mount + +default_expire=`cat /proc/sys/vm/dirty_expire_centisecs` +# Make it flush dirty pages more frequently to make sure we reproduce the bug. +echo 50 > /proc/sys/vm/dirty_expire_centisecs + +tfile=$SCRATCH_MNT/testfile +pagesize=$(get_page_size) +sublen=$((RANDOM % pagesize)) + +$XFS_IO_PROG -f -c "pwrite 0 $pagesize" $tfile > /dev/null 2>&1 +# write some subpagesize data first. +$XFS_IO_PROG -c "pwrite $pagesize $sublen" $tfile > /dev/null 2>&1 + +# Mix up "abnormal" subpagesize writes with normal pagesize based writes +toff=$((pagesize + sublen)) +for ((i = 0; i < 10000; i++)); do + tlen=$pagesize + if [ $((i % 2)) = 0 ]; then + tlen=$((pagesize * 3)) + fi + if [ $((i % 1000)) = 0 ]; then + tlen=$((RANDOM % pagesize)) + fi + + $XFS_IO_PROG -c "pwrite $toff $tlen" $tfile > /dev/null 2>&1 + toff=$((toff + tlen)) +done + +sync + +# check for single PAGESIZE extent +$XFS_IO_PROG -c "fiemap -v" $tfile >> $seqres.full 2>&1 +$XFS_IO_PROG -c "fiemap -v" $tfile | \ +awk -v pgsize=$pagesize '{n = pgsize/512; if ($4 == n) print $4}' + +# success, all done +status=0 +exit diff --git a/tests/btrfs/027.out b/tests/btrfs/027.out new file mode 100644 index 0000000..ef78cda --- /dev/null +++ b/tests/btrfs/027.out @@ -0,0 +1,2 @@ +QA output created by 027 +Silence is golden diff --git a/tests/btrfs/group b/tests/btrfs/group index a2fa412..f06d5fe 100644 --- a/tests/btrfs/group +++ b/tests/btrfs/group @@ -29,6 +29,7 @@ 024 auto quick compress 025 auto quick send clone 026 auto quick compress prealloc +027 auto 029 auto quick clone 030 auto quick send 031 auto quick subvol clone -- 2.5.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH v3] fstest: btrfs: test single 4k extent after subpagesize buffered writes 2016-03-24 4:55 ` [PATCH v3] " Liu Bo @ 2016-03-24 8:31 ` Eryu Guan 0 siblings, 0 replies; 16+ messages in thread From: Eryu Guan @ 2016-03-24 8:31 UTC (permalink / raw) To: Liu Bo; +Cc: fstests, linux-btrfs, Dave Chinner On Wed, Mar 23, 2016 at 09:55:52PM -0700, Liu Bo wrote: > This is to test if COW enabled btrfs can end up with single 4k extents > when doing subpagesize buffered writes. > > Signed-off-by: Liu Bo <bo.li.liu@oracle.com> Looks good to me. Reviewed-by: Eryu Guan <eguan@redhat.com> ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2016-03-29 2:11 UTC | newest] Thread overview: 16+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-03-07 23:47 [PATCH] fstest: btrfs: test single 4k extent after subpagesize buffered writes Liu Bo 2016-03-08 0:27 ` [PATCH v2] " Liu Bo 2016-03-15 3:39 ` Dave Chinner 2016-03-17 4:13 ` Liu Bo 2016-03-17 4:24 ` Eryu Guan 2016-03-17 22:23 ` Liu Bo 2016-03-17 21:07 ` Dave Chinner 2016-03-17 21:31 ` Liu Bo 2016-03-17 22:56 ` Liu Bo 2016-03-22 4:00 ` Eryu Guan 2016-03-22 22:12 ` Liu Bo 2016-03-23 11:53 ` Eryu Guan 2016-03-24 4:52 ` Liu Bo 2016-03-29 2:13 ` Liu Bo 2016-03-24 4:55 ` [PATCH v3] " Liu Bo 2016-03-24 8:31 ` Eryu Guan
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).