[PATCH 0/4] xfstests: new tests and fixes

[PATCH 0/4] xfstests: new tests and fixes

[Dave Chinner]

Pushing out my current stack:

	- Fix test 016 to work with delaylog.
	- Speed up 042 by using preallocation instead of writing
	  data and files.
	- Add test to exercise bmap btree corruption caused by
	  multilevel split.
	- Add simple test to exercise splice writes to catch locking
	  issues.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 1/4] xfstests: add simple splice test

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

We don't have any coverage of the splice functionality provided by
the kernel in xfstests. Add a simple test that uses the sendfile
operation built into xfs_io to copy a file ensure we at least
execute the code path in xfstests.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 249     |   62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 249.out |    2 ++
 group   |    1 +
 3 files changed, 65 insertions(+), 0 deletions(-)
 create mode 100644 249
 create mode 100644 249.out

diff --git a/249 b/249
new file mode 100644
index 0000000..6fc972e
--- /dev/null
+++ b/249
@@ -0,0 +1,62 @@
+#! /bin/bash
+# FS QA Test No. 249
+#
+# simple splice(2) test.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2011 Red Hat, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dchinner@redhat.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=tmp/$$
+status=1  # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+	rm -f $SRC $DST
+	_cleanup_testdir
+}
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_setup_testdir
+
+echo "Feel the serenity."
+
+SRC=$TEST_DIR/$seq.src
+DST=$TEST_DIR/$seq.dst
+
+$XFS_IO_PROG -f -c "pwrite -S 0xa5a55a5a 0 32768k" -c fsync $SRC > /dev/null 2>&1
+$XFS_IO_PROG -f -c "sendfile -i $SRC 0 32768k" -c fsync $DST > /dev/null 2>&1
+
+diff $SRC $DST
+
+status=$?
+exit
diff --git a/249.out b/249.out
new file mode 100644
index 0000000..592696f
--- /dev/null
+++ b/249.out
@@ -0,0 +1,2 @@
+QA output created by 249
+Feel the serenity.
diff --git a/group b/group
index a40c98f..b94f543 100644
--- a/group
+++ b/group
@@ -362,3 +362,4 @@ deprecated
 246 auto quick rw
 247 auto quick rw
 248 auto quick rw 
+249 auto quick rw 
-- 
1.7.2.3

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 2/4] xfstests: add test to reproduce bmap btree corruption.

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

The problem was reprted here:

https://bugzilla.redhat.com/show_bug.cgi?id=626244

With the simple test case:

# mkfs.xfs -f -d agsize=16m,size=50g <dev>
# mount <dev> /mnt
# xfs_io -f -c 'resvsp 0 40G' /mnt/foo

Triggering the problem. Turn this into a new xfsqa test so that we
exercise the problem code and prevent future regressions.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 250   |   97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 group |    1 +
 2 files changed, 98 insertions(+), 0 deletions(-)
 create mode 100644 250

diff --git a/250 b/250
new file mode 100644
index 0000000..cb0fa9c
--- /dev/null
+++ b/250
@@ -0,0 +1,97 @@
+#! /bin/bash
+# FS QA Test No. 250
+#
+# Bmap btree corruption regression test
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2011 Red Hat, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dchinner@redhat.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=tmp/$$
+status=1  # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	umount $LOOP_MNT 2>/dev/null
+	rm -f $LOOP_DEV
+	rmdir $LOOP_MNT
+	_cleanup_testdir
+}
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+# real QA test starts here
+_supported_fs xfs
+_supported_os Linux
+_require_loop
+
+LOOP_DEV=$TEST_DIR/$seq.fs
+LOOP_MNT=$TEST_DIR/$seq.mnt
+
+_filter_io()
+{
+	sed -e '/.* ops; /d'
+}
+
+# real QA test starts here
+
+echo "*** create loop mount point"
+rmdir $LOOP_MNT 2>/dev/null
+mkdir -p $LOOP_MNT || _fail "cannot create loopback mount point"
+
+_test_loop()
+{
+	size=$1
+	agsize=$2
+	fsize=$3
+
+	dparam="file,name=$LOOP_DEV,size=$size"
+	if [ -n "$agsize" ]; then
+		dparam="$dparam,agsize=$agsize"
+	fi
+
+	echo "*** mkfs loop file (size=$size)"
+	$MKFS_XFS_PROG -d $dparam \
+		| _filter_mkfs # 2>/dev/null
+
+	echo "*** mount loop filesystem"
+	mount -t xfs -o loop $LOOP_DEV $LOOP_MNT
+
+	echo "*** preallocate large file"
+	xfs_io -f -c "resvsp 0 $fsize" $LOOP_MNT/foo | _filter_io
+
+	echo "*** unmount loop filesystem"
+	umount $LOOP_MNT
+
+	echo "*** check loop filesystem"
+	 _check_xfs_filesystem $LOOP_DEV none none
+}
+
+_test_loop 50g 16m 40G
+echo "*** done"
+status=0
+exit
diff --git a/group b/group
index b94f543..b18f2e6 100644
--- a/group
+++ b/group
@@ -363,3 +363,4 @@ deprecated
 247 auto quick rw
 248 auto quick rw 
 249 auto quick rw 
+250 auto quick rw prealloc metadata
-- 
1.7.2.3

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 3/4] xfstests: make 016 work with delaylog

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

Test 016 fails with delaylog because it measures log traffic to disk
and delaylog writes almost nothing to the log for the given test. TO
make it work, add sync calls to the work loop to cause the log to be
flushed reliably for both delaylog and nodelaylog and hence contain
the same nuumber of log records.

As a result, the log space consumed by the test is not changed by
the delaylog option and the test passes. The test is not
significantly slowed down by the addition of the sync calls (takes
15s to run on a single SATA drive).

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 016 |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/016 b/016
index 980eb0b..32d8b19 100755
--- a/016
+++ b/016
@@ -101,7 +101,9 @@ _log_traffic()
     while [ $count -ge 0 ]
     do
         touch $out
+	sync
         rm $out
+	sync
         let "count = count - 1"
     done
 
-- 
1.7.2.3

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 4/4] xfstests: Speed up test 042

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

test 042 generates a worst-case fragmented filesystem and uses it to
test xfs_fsr. It uses small 4k files to generate the hole-space-hole
pattern that fragments free space badly. It is much faster to
generate the same pattern by creating a single large file and
punching holes in it.  Also, instead of writing large files to
create unfragmented space, just use preallocation so we don't have
to write the data to disk.

These changes reduce the runtime of the test on a single SATA drive
from 106s to 27s.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 042     |   38 ++++++++++++++++++++++++++++++--------
 042.out |    6 +++---
 2 files changed, 33 insertions(+), 11 deletions(-)

diff --git a/042 b/042
index 7ac5492..c583746 100755
--- a/042
+++ b/042
@@ -100,16 +100,19 @@ echo "done"
 echo -n "Reserve 16 1Mb unfragmented regions... "
 for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
 do
-    _do "dd if=/dev/zero of=$SCRATCH_MNT/hole$i bs=4096 count=256"
-    _do "dd if=/dev/zero of=$SCRATCH_MNT/space$i bs=4096 count=1"
-    _do "xfs_bmap -v $SCRATCH_MNT/hole$i"
+	_do "$XFS_IO_PROG -f -c \"resvsp 0 1m\" $SCRATCH_MNT/hole$i"
+	_do "$XFS_IO_PROG -f -c \"resvsp 0 4k\" $SCRATCH_MNT/space$i"
+	_do "$XFS_IO_PROG -f -c \"resvsp 0 1m\" $SCRATCH_MNT/hole$i"
+	_do "xfs_bmap -vp $SCRATCH_MNT/hole$i"
 done
 echo "done" 
 
 # set up filesystem
-echo -n "Fill filesystem with 4k files, generate manifest... "
-fill_options="--verbose --seed=0 --filesize=4096 --stddev=0 --sync=1000000"
-_do "src/fill2fs $fill_options --dir=$SCRATCH_MNT/fill --list=- > $tmp.manifest"
+echo -n "Fill filesystem with fill file... "
+for i in `seq 0 1 31`; do
+	_do "$XFS_IO_PROG -fs -c \"pwrite -S$i ${i}m 1m\" $SCRATCH_MNT/fill"
+done
+_do "xfs_bmap -vp $SCRATCH_MNT/fill"
 echo "done"
 # flush the filesystem - make sure there is no space "lost" to pre-allocation
 _do "umount $SCRATCH_MNT"
@@ -119,7 +122,18 @@ _do "dd if=/dev/zero of=$SCRATCH_MNT/pad bs=4096"
 echo "done"
 
 # create fragmented file
-_do "Delete every second file" "_cull_files"
+#_do "Delete every second file" "_cull_files"
+echo -n "Punch every second 4k block... "
+for i in `seq 0 8 32768`; do
+	# This generates excessive output that significantly slows down the
+	# test. It's not necessary for debug, so just bin it.
+	$XFS_IO_PROG -f -c "unresvsp ${i}k 4k" $SCRATCH_MNT/fill \
+								> /dev/null 2>&1
+done
+_do "xfs_bmap -vp $SCRATCH_MNT/fill"
+_do "sum $SCRATCH_MNT/fill >$tmp.fillsum1"
+echo "done"
+
 echo -n "Create one very large file... "
 _do "src/fill2 -d nbytes=16000000,file=$SCRATCH_MNT/fragmented"
 echo "done"
@@ -130,7 +144,15 @@ _do "Remove other files" "rm -rf $SCRATCH_MNT/{pad,hole*}"
 # defragment
 _do "Run xfs_fsr on filesystem" "$XFS_FSR_PROG -v $SCRATCH_MNT/fragmented"
 _do "xfs_bmap -v $SCRATCH_MNT/fragmented"
-_do "Check 4k files" "src/fill2fs_check $tmp.manifest"
+
+echo -n "Check fill file... "
+_do "sum $SCRATCH_MNT/fill >$tmp.fillsum2"
+if ! _do "diff $tmp.fillsum1 $tmp.fillsum2"; then
+    echo "fail"
+    echo "Fill file is corrupt/missing after fsr. Test failed see $seq.full"
+    status=1; exit
+fi
+echo "done"
 
 # check
 echo -n "Check large file... "
diff --git a/042.out b/042.out
index 7f2e837..f5b37b3 100644
--- a/042.out
+++ b/042.out
@@ -1,13 +1,13 @@
 QA output created by 042
 Make a 48 megabyte filesystem on SCRATCH_DEV and mount... done
 Reserve 16 1Mb unfragmented regions... done
-Fill filesystem with 4k files, generate manifest... done
+Fill filesystem with fill file... done
 Use up any further available space using dd... done
-Delete every second file... done
+Punch every second 4k block... done
 Create one very large file... done
 Remove other files... done
 Run xfs_fsr on filesystem... done
-Check 4k files... done
+Check fill file... done
 Check large file... done
 Checking filesystem... done
 xfs_fsr tests passed.
-- 
1.7.2.3

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 1/4] xfstests: add simple splice test

[Christoph Hellwig]

On Tue, Jan 25, 2011 at 02:58:54PM +1100, Dave Chinner wrote:
> From: Dave Chinner <dchinner@redhat.com>
> 
> We don't have any coverage of the splice functionality provided by
> the kernel in xfstests. Add a simple test that uses the sendfile
> operation built into xfs_io to copy a file ensure we at least
> execute the code path in xfstests.

I'm pretty sure I reviewed this one before, but in case it's needed here
it is again:

Reviewed-by: Christoph Hellwig <hch@lst.de>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 2/4] xfstests: add test to reproduce bmap btree corruption.

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 3/4] xfstests: make 016 work with delaylog

[Christoph Hellwig]

> @@ -101,7 +101,9 @@ _log_traffic()
>      while [ $count -ge 0 ]
>      do
>          touch $out
> +	sync
>          rm $out
> +	sync
>          let "count = count - 1"
>      done

Looks good, although it might be good to make the indentation fit the
surrounding code.

Reviewed-by: Christoph Hellwig <hch@lst.de>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 4/4] xfstests: Speed up test 042

[Christoph Hellwig]

On Tue, Jan 25, 2011 at 02:58:57PM +1100, Dave Chinner wrote:
> From: Dave Chinner <dchinner@redhat.com>
> 
> test 042 generates a worst-case fragmented filesystem and uses it to
> test xfs_fsr. It uses small 4k files to generate the hole-space-hole
> pattern that fragments free space badly. It is much faster to
> generate the same pattern by creating a single large file and
> punching holes in it.  Also, instead of writing large files to
> create unfragmented space, just use preallocation so we don't have
> to write the data to disk.
> 
> These changes reduce the runtime of the test on a single SATA drive
> from 106s to 27s.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 1/4] xfstests: add simple splice test

[Alex Elder]

On Tue, 2011-01-25 at 14:58 +1100, Dave Chinner wrote:
> From: Dave Chinner <dchinner@redhat.com>
> 
> We don't have any coverage of the splice functionality provided by
> the kernel in xfstests. Add a simple test that uses the sendfile
> operation built into xfs_io to copy a file ensure we at least
> execute the code path in xfstests.
> 
> Signed-off-by: Dave Chinner <dchinner@redhat.com>

Looks good.

Reviewed-by: Alex Elder <aelder@sgi.com>


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 2/4] xfstests: add test to reproduce bmap btree corruption.

[Alex Elder]

On Tue, 2011-01-25 at 14:58 +1100, Dave Chinner wrote:
> From: Dave Chinner <dchinner@redhat.com>
> 
> The problem was reprted here:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=626244
> 
> With the simple test case:
> 
> # mkfs.xfs -f -d agsize=16m,size=50g <dev>
> # mount <dev> /mnt
> # xfs_io -f -c 'resvsp 0 40G' /mnt/foo
> 
> Triggering the problem. Turn this into a new xfsqa test so that we
> exercise the problem code and prevent future regressions.
> 
> Signed-off-by: Dave Chinner <dchinner@redhat.com>

This looks good, HOWEVER, since the test crashes
the target system please remove it from (at least)
the "auto" group before you commit.

Reviewed-by: Alex Elder <aelder@sgi.com>



_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 3/4] xfstests: make 016 work with delaylog

[Alex Elder]

On Tue, 2011-01-25 at 14:58 +1100, Dave Chinner wrote:
> From: Dave Chinner <dchinner@redhat.com>
> 
> Test 016 fails with delaylog because it measures log traffic to disk
> and delaylog writes almost nothing to the log for the given test. TO
> make it work, add sync calls to the work loop to cause the log to be
> flushed reliably for both delaylog and nodelaylog and hence contain
> the same nuumber of log records.
> 
> As a result, the log space consumed by the test is not changed by
> the delaylog option and the test passes. The test is not
> significantly slowed down by the addition of the sync calls (takes
> 15s to run on a single SATA drive).
> 
> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> ---

Looks good.

Reviewed-by: Alex Elder <aelder@sgi.com>


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 4/4] xfstests: Speed up test 042

[Alex Elder]

On Tue, 2011-01-25 at 14:58 +1100, Dave Chinner wrote:
> From: Dave Chinner <dchinner@redhat.com>
> 
> test 042 generates a worst-case fragmented filesystem and uses it to
> test xfs_fsr. It uses small 4k files to generate the hole-space-hole
> pattern that fragments free space badly. It is much faster to
> generate the same pattern by creating a single large file and
> punching holes in it.  Also, instead of writing large files to
> create unfragmented space, just use preallocation so we don't have
> to write the data to disk.
> 
> These changes reduce the runtime of the test on a single SATA drive
> from 106s to 27s.
> 
> Signed-off-by: Dave Chinner <dchinner@redhat.com>

Perhaps your test system is configured different
from mine, but I don't see the speedup you see.
In fact, it might have slowed it down.  I really
haven't experimented much though and will report
back if I find anything more constructive to say.

Other than that I see no problem with the change.
And the difference either way isn't a big deal
to me, so:

Reviewed-by: Alex Elder <aelder@sgi.com>


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 4/4] xfstests: Speed up test 042

[Dave Chinner]

On Tue, Feb 01, 2011 at 07:30:09PM -0600, Alex Elder wrote:
> On Tue, 2011-01-25 at 14:58 +1100, Dave Chinner wrote:
> > From: Dave Chinner <dchinner@redhat.com>
> > 
> > test 042 generates a worst-case fragmented filesystem and uses it to
> > test xfs_fsr. It uses small 4k files to generate the hole-space-hole
> > pattern that fragments free space badly. It is much faster to
> > generate the same pattern by creating a single large file and
> > punching holes in it.  Also, instead of writing large files to
> > create unfragmented space, just use preallocation so we don't have
> > to write the data to disk.
> > 
> > These changes reduce the runtime of the test on a single SATA drive
> > from 106s to 27s.
> > 
> > Signed-off-by: Dave Chinner <dchinner@redhat.com>
> 
> Perhaps your test system is configured different
> from mine, but I don't see the speedup you see.
> In fact, it might have slowed it down.  I really
> haven't experimented much though and will report
> back if I find anything more constructive to say.

That may be because the resvsp call at the moment is a sync
transaction. I've got a patch to make that async which should solve
the problem for you. Mind you, the above numbers come from a Vm that
wasn't running that patch, so it probably depends mostly on how
efficient your storage is a processing barriers to the difference
you see right now...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 2/4] xfstests: add test to reproduce bmap btree corruption.

[Dave Chinner]

On Tue, Feb 01, 2011 at 07:29:54PM -0600, Alex Elder wrote:
> On Tue, 2011-01-25 at 14:58 +1100, Dave Chinner wrote:
> > From: Dave Chinner <dchinner@redhat.com>
> > 
> > The problem was reprted here:
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=626244
> > 
> > With the simple test case:
> > 
> > # mkfs.xfs -f -d agsize=16m,size=50g <dev>
> > # mount <dev> /mnt
> > # xfs_io -f -c 'resvsp 0 40G' /mnt/foo
> > 
> > Triggering the problem. Turn this into a new xfsqa test so that we
> > exercise the problem code and prevent future regressions.
> > 
> > Signed-off-by: Dave Chinner <dchinner@redhat.com>
> 
> This looks good, HOWEVER, since the test crashes
> the target system please remove it from (at least)
> the "auto" group before you commit.

If causes an assert failure on a DEBUG kernel, a shutdown on a
non-debug kernel. Either way, I'd prefer to leave it enabled as a
good reminder that we need to fix the damn bug (*) sooner rather
than later...

(*) This patch prevents the assert failure, but, like Lachlan, I'm
not sure it is the right fix.

http://oss.sgi.com/archives/xfs/2010-09/msg00337.html

I'm working on it at the moment, along with all the other allocation
things...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs