public inbox for linux-xfs@vger.kernel.org
 help / color / mirror / Atom feed
* [PATCH 00/13] xfstests: generic --large-fs support
@ 2013-03-15 11:53 Dave Chinner
  2013-03-15 11:53 ` [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs Dave Chinner
                   ` (13 more replies)
  0 siblings, 14 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
  To: xfs

Hi folks,

This is a repost of my current version of the large filesystem
testing support for xfstests. I've been using it for more than a
year, and I think the ext4 guys would like to have it available to
them.

Last posted here:

http://oss.sgi.com/archives/xfs/2012-07/msg00361.html

There are changes from last time, but it's months since i've made
them, so there's no fine grained changelog. :/

Cheers,

Dave.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 23+ messages in thread
* [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs
  2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
  2013-03-19 19:44   ` Rich Johnston
  2013-03-19 20:58   ` Ben Myers
  2013-03-15 11:53 ` [PATCH 02/13] xfstests: add --largefs check option Dave Chinner
                   ` (12 subsequent siblings)
  13 siblings, 2 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
  To: xfs

From: Dave Chinner <dchinner@redhat.com>

This patch it to ensure that xfstests passes on non-crc filesystems
with a CRC enabled userspace.

Filter out the mkfs/xfs_info CRC line from tests that capture the
output of these commands.

Filter out new error noise from xfs_repair that occurs for
xfs_repair as a result of the CRC changes.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 096           |    1 +
 206           |    9 ++++++---
 common.repair |    3 +++
 3 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/096 b/096
index 237edad..5a2273d 100755
--- a/096
+++ b/096
@@ -85,6 +85,7 @@ _mkfs_filter()
 	-e '/inode-paths/d' \
 	-e 's/\(log[ 	]*=\).*bsize/\1LOG                    bsize/' \
 	-e 's/\(realtime[ 	]*=\).*extsz/\1REALTIME               extsz/' \
+	-e '/.*crc=/d' \
 	-e 's/ *$//' \
    | grep -v parent
 }
diff --git a/206 b/206
index e8bc922..e46e6a3 100755
--- a/206
+++ b/206
@@ -77,7 +77,8 @@ echo "=== mkfs.xfs ==="
 mkfs.xfs -f -bsize=4096 -dagsize=76288719b,size=3905982455b -llazy-count=0 $tmpfile \
 	| sed -e "s,^meta-data=.*isize,meta-data=FILE                   isize,g" \
 		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
-		-e "s/, projid32bit=[0-9]//"
+		-e "s/, projid32bit=[0-9]//" \
+		-e "/.*crc=/d"
 
 mount -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount"
 
@@ -85,13 +86,15 @@ mount -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount"
 echo "=== xfs_growfs ==="
 xfs_growfs $tmpdir \
 	| sed -e "s,^meta-data=.*isize,meta-data=FILE                   isize,g" \
-		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/"
+		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
+		-e "/.*crc=/d"
 
 # and double-check the new geometry
 echo "=== xfs_info ==="
 xfs_info $tmpdir \
 	| sed -e "s,^meta-data=.*isize,meta-data=FILE                   isize,g" \
-		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/"
+		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
+		-e "/.*crc=/d"
 
 # _cleanup cleans up for us
 
diff --git a/common.repair b/common.repair
index 23a1ed6..db160bb 100644
--- a/common.repair
+++ b/common.repair
@@ -77,6 +77,9 @@ s/\s+- \d+:\d\d:\d\d:.*\n//g;
 /^agi_/ && next; # remove agi counts
 /^sb_/ && next; # remove sb counts
 /^agi unlinked/ && next; # remove agi unlinked bucket warning
+# crc enabled filesystem output
+/XFS_CORRUPTION_ERROR/ && next;
+/^bad uuid/ && next;
 	print;'
 }
 
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 02/13] xfstests: add --largefs check option
  2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
  2013-03-15 11:53 ` [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
  2013-03-15 11:53 ` [PATCH 03/13] xfstests: rename USE_BIG_LOOPFS to be more generic Dave Chinner
                   ` (11 subsequent siblings)
  13 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
  To: xfs

From: Dave Chinner <dchinner@redhat.com>

Make it easier to check large filesystems quickly by adding a
--large-fs option to check to turn on shortcuts for large scratch
device filesystem testing.

Also, reject invalid command line options with a usage message.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
 common |   63 +++++++++++++++++++++++++++++++++++++++------------------------
 1 file changed, 39 insertions(+), 24 deletions(-)

diff --git a/common b/common
index 0723224..7d13078 100644
--- a/common
+++ b/common
@@ -27,6 +27,35 @@ _setenvironment()
     export MSGVERB
 }
 
+usage()
+{
+    echo "Usage: $0 [options] [testlist]"'
+
+common options
+    -v			verbose
+
+check options
+    -xfs                test XFS (default)
+    -udf                test UDF
+    -nfs                test NFS
+    -l			line mode diff
+    -xdiff		graphical mode diff
+    -udiff		show unified diff (default)
+    -n			show me, do not run tests
+    -q			quick [deprecated]
+    -T			output timestamps
+    -r			randomize test order
+    --large-fs		optimise scratch device for large filesystems
+
+testlist options
+    -g group[,group...]	include tests from these groups
+    -x group[,group...]	exclude tests from these groups
+    NNN			include test NNN
+    NNN-NNN		include test range (eg. 012-021)
+'
+	    exit 0
+}
+
 here=`pwd`
 rm -f $here/$iam.out
 _setenvironment
@@ -117,30 +146,7 @@ s/ .*//p
     in
 
 	-\? | -h | --help)	# usage
-	    echo "Usage: $0 [options] [testlist]"'
-
-common options
-    -v			verbose
-
-check options
-    -xfs                test XFS (default)
-    -udf                test UDF
-    -nfs                test NFS
-    -l			line mode diff
-    -xdiff		graphical mode diff
-    -udiff		show unified diff (default)
-    -n			show me, do not run tests
-    -q			quick [deprecated]
-    -T			output timestamps
-    -r 			randomize test order
-    
-testlist options
-    -g group[,group...]	include tests from these groups
-    -x group[,group...]	exclude tests from these groups
-    NNN			include test NNN
-    NNN-NNN		include test range (eg. 012-021)
-'
-	    exit 0
+	    usage
 	    ;;
 
 	-udf)	# -udf ... set FSTYP to udf
@@ -232,6 +238,15 @@ testlist options
 	    fi
 	    ;;
 
+	--large-fs)
+	    export USE_BIG_LOOPFS=yes
+	    xpand=false
+	    ;;
+
+	-*)
+	    usage
+	    ;;
+
 	*)
 	    start=$r
 	    end=$r
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 03/13] xfstests: rename USE_BIG_LOOPFS to be more generic
  2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
  2013-03-15 11:53 ` [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs Dave Chinner
  2013-03-15 11:53 ` [PATCH 02/13] xfstests: add --largefs check option Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
  2013-03-19 19:46   ` Rich Johnston
  2013-03-15 11:53 ` [PATCH 04/13] xfstests: rename RETAIN_AG_BYTES Dave Chinner
                   ` (10 subsequent siblings)
  13 siblings, 1 reply; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
  To: xfs

From: Dave Chinner <dchinner@redhat.com>

USE_BIG_LOOPFS is really misnamed - it can be used on real devices just as
easily as loop devices. It really means we are testing a large scratch device
and that we should enable the special filesystem filling and checking options
that enable xfstests to be run sanely on large XFS filesystems.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 004       |    2 +-
 015       |    2 +-
 030       |    2 +-
 031       |    2 +-
 032       |    2 +-
 033       |    4 ++--
 041       |    2 +-
 049       |    2 +-
 083       |    2 +-
 092       |    2 +-
 148       |    2 +-
 149       |    2 +-
 common    |    2 +-
 common.rc |   12 ++++++------
 setup     |    5 +++--
 15 files changed, 23 insertions(+), 22 deletions(-)

diff --git a/004 b/004
index 9f28e17..23729da 100755
--- a/004
+++ b/004
@@ -64,7 +64,7 @@ _supported_os IRIX Linux
 
 _need_to_be_root
 _require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
 
 rm -f $seq.full
 
diff --git a/015 b/015
index 4206b93..a99f1ed 100755
--- a/015
+++ b/015
@@ -52,7 +52,7 @@ _supported_fs generic
 _supported_os IRIX Linux
 
 _require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
 
 _scratch_mkfs_sized `expr 50 \* 1024 \* 1024` >/dev/null 2>&1 \
     || _fail "mkfs failed"
diff --git a/030 b/030
index fe57372..a8fb92c 100755
--- a/030
+++ b/030
@@ -64,8 +64,8 @@ _check_ag()
 _supported_fs xfs
 _supported_os IRIX Linux
 
-_require_nobigloopfs
 _require_scratch
+_require_no_large_scratch_dev
 
 DSIZE="-dsize=100m,agcount=6"
 
diff --git a/031 b/031
index b062277..fb6f15b 100755
--- a/031
+++ b/031
@@ -97,8 +97,8 @@ EOF
 _supported_fs xfs
 _supported_os IRIX Linux
 
-_require_nobigloopfs
 _require_scratch
+_require_no_large_scratch_dev
 
 # sanity test - default + one root directory entry
 # Note: must do this proto/mkfs now for later inode size calcs
diff --git a/032 b/032
index bf88492..09f08c3 100755
--- a/032
+++ b/032
@@ -41,8 +41,8 @@ rm -f $seq.full
 _supported_fs xfs
 _supported_os Linux
 
-_require_nobigloopfs
 _require_scratch
+_require_no_large_scratch_dev
 
 echo "Silence is golden."
 for fs in `echo ${MKFS_PROG}.* | sed -e 's/.sbin.mkfs.//g'`
diff --git a/033 b/033
index 9651f26..68a688e 100755
--- a/033
+++ b/033
@@ -76,9 +76,9 @@ _filter_bad_ids()
 # real QA test starts here
 _supported_fs xfs
 _supported_os IRIX Linux
- 
-_require_nobigloopfs
+
 _require_scratch
+_require_no_large_scratch_dev
 
 # devzero blows away 512byte blocks, so make 512byte inodes (at least)
 _scratch_mkfs_xfs | _filter_mkfs 2>$tmp.mkfs
diff --git a/041 b/041
index 2800811..28dcb33 100755
--- a/041
+++ b/041
@@ -50,7 +50,7 @@ _supported_fs xfs
 _supported_os IRIX Linux
 
 _require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
 umount $SCRATCH_DEV 2>/dev/null
 
 _fill()
diff --git a/049 b/049
index b68821a..8593814 100755
--- a/049
+++ b/049
@@ -60,9 +60,9 @@ _log()
     echo "--- $*" >> $seq.full
 }
 
-_require_nobigloopfs
 _require_nonexternal
 _require_scratch
+_require_no_large_scratch_dev
 _require_loop
 _require_ext2
 
diff --git a/083 b/083
index f5349a9..a977ffd 100755
--- a/083
+++ b/083
@@ -58,7 +58,7 @@ _supported_fs generic
 _supported_os IRIX Linux
 
 _require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
 
 rm -f $seq.full
 
diff --git a/092 b/092
index 429fa80..02ccc71 100755
--- a/092
+++ b/092
@@ -48,7 +48,7 @@ _cleanup()
 _supported_fs xfs
 _supported_os IRIX Linux
 _require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
 
 MOUNT_OPTIONS="$MOUNT_OPTIONS -o inode64"
 _scratch_mkfs_xfs | _filter_mkfs 2>/dev/null
diff --git a/148 b/148
index 76cbf37..7bb1722 100755
--- a/148
+++ b/148
@@ -66,8 +66,8 @@ _check_ag()
 _supported_fs xfs
 _supported_os IRIX Linux
 
-_require_nobigloopfs
 _require_scratch
+_require_no_large_scratch_dev
 
 DSIZE="-dsize=100m"
 
diff --git a/149 b/149
index 5131a45..193e6d7 100755
--- a/149
+++ b/149
@@ -100,8 +100,8 @@ EOF
 _supported_fs xfs
 _supported_os IRIX Linux
 
-_require_nobigloopfs
 _require_scratch
+_require_no_large_scratch_dev
 
 # sanity test - default + one root directory entry
 # Note: must do this proto/mkfs now for later inode size calcs
diff --git a/common b/common
index 7d13078..da86cd9 100644
--- a/common
+++ b/common
@@ -239,7 +239,7 @@ s/ .*//p
 	    ;;
 
 	--large-fs)
-	    export USE_BIG_LOOPFS=yes
+	    export LARGE_SCRATCH_DEV=yes
 	    xpand=false
 	    ;;
 
diff --git a/common.rc b/common.rc
index ef41cbf..edf0318 100644
--- a/common.rc
+++ b/common.rc
@@ -310,7 +310,7 @@ _scratch_mkfs_xfs()
     cat $tmp_dir.mkfsstd
     rm -f $tmp_dir.mkfserr $tmp_dir.mkfsstd
 
-    if [ "$USE_BIG_LOOPFS" = yes ]; then
+    if [ "$LARGE_SCRATCH_DEV" = yes ]; then
         [ -z "$RETAIN_AG_BYTES" ] && RETAIN_AG_BYTES=0
         ./tools/ag-wipe -q -r $RETAIN_AG_BYTES $SCRATCH_DEV
     fi
@@ -437,7 +437,7 @@ _scratch_xfs_repair()
         SCRATCH_OPTIONS="-l$SCRATCH_LOGDEV"
     [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_RTDEV" ] && \
         SCRATCH_OPTIONS=$SCRATCH_OPTIONS" -r$SCRATCH_RTDEV"
-    [ "$USE_BIG_LOOPFS" = yes ] && SCRATCH_OPTIONS=$SCRATCH_OPTIONS" -t"
+    [ "$LARGE_SCRATCH_DEV" = yes ] && SCRATCH_OPTIONS=$SCRATCH_OPTIONS" -t"
     $XFS_REPAIR_PROG $SCRATCH_OPTIONS $* $SCRATCH_DEV
 }
 
@@ -826,9 +826,9 @@ _require_ext2()
 
 # this test requires that (large) loopback device files are not in use
 #
-_require_nobigloopfs()
+_require_no_large_scratch_dev()
 {
-    [ "$USE_BIG_LOOPFS" = yes ] && \
+    [ "$LARGE_SCRATCH_DEV" = yes ] && \
 	_notrun "Large filesystem testing in progress, skipped this test"
 }
 
@@ -1237,7 +1237,7 @@ _check_xfs_filesystem()
 
     [ "$FSTYP" != xfs ] && return 0
     testoption=""
-    [ "$USE_BIG_LOOPFS" = yes ] && testoption=-t
+    [ "$LARGE_SCRATCH_DEV" = yes ] && testoption=-t
 
     type=`_fs_type $device`
     ok=1
@@ -1276,7 +1276,7 @@ _check_xfs_filesystem()
         ok=0
     fi
     # repair doesn't scale massively at this stage, optionally skip it for now
-    [ "$USE_BIG_LOOPFS" = yes ] || \
+    [ "$LARGE_SCRATCH_DEV" = yes ] || \
     $XFS_REPAIR_PROG -n $extra_log_options $extra_rt_options $device >$tmp.repair 2>&1
     if [ $? -ne 0 ]
     then
diff --git a/setup b/setup
index 62798cc..5225951 100755
--- a/setup
+++ b/setup
@@ -23,7 +23,7 @@ fi
 
 [ "$USE_EXTERNAL"    = yes ] || USE_EXTERNAL=no
 [ "$USE_LBD_PATCH"   = yes ] || USE_LBD_PATCH=no
-[ "$USE_BIG_LOOPFS"  = yes ] || USE_BIG_LOOPFS=no
+[ "$LARGE_SCRATCH_DEV"  = yes ] || LARGE_SCRATCH_DEV=no
 [ "$USE_ATTR_SECURE" = yes ] || USE_ATTR_SECURE=no
 [ -z "$FSTYP" ] && FSTYP="xfs"
 
@@ -31,5 +31,6 @@ cat <<EOF
 TEST: DIR=$TEST_DIR DEV=$TEST_DEV rt=[$TEST_RTDEV] log=[$TEST_LOGDEV]
 TAPE: dev=[$TAPE_DEV] rmt=[$RMT_TAPE_DEV] rmtirix=[$RMT_TAPE_USER@$RMT_IRIXTAPE_DEV]
 SCRATCH: MNT=$SCRATCH_MNT DEV=$SCRATCH_DEV rt=[$SCRATCH_RTDEV] log=[$SCRATCH_LOGDEV]
-VARIABLES: external=$USE_EXTERNAL largeblk=$USE_LBD_PATCH fstyp=$FSTYP bigloopfs=$USE_BIG_LOOPFS attrsecure=$USE_ATTR_SECURE
+VARIABLES: external=$USE_EXTERNAL largeblk=$USE_LBD_PATCH fstyp=$FSTYP
+	   large_scratch_dev=$LARGE_SCRATCH_DEV attrsecure=$USE_ATTR_SECURE
 EOF
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 04/13] xfstests: rename RETAIN_AG_BYTES
  2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
                   ` (2 preceding siblings ...)
  2013-03-15 11:53 ` [PATCH 03/13] xfstests: rename USE_BIG_LOOPFS to be more generic Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
  2013-03-15 11:53 ` [PATCH 05/13] xfstests: use preallocation for ag-wiper Dave Chinner
                   ` (9 subsequent siblings)
  13 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
  To: xfs

From: Dave Chinner <dchinner@redhat.com>

Rename the $RETAIN_AG_BYTES variable to be more generic so that it
reflects the fact that it is designed to retain a certain amount of
extra free space above the default amount in the filesystem when
doing large scratch device testing.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
 common.rc |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/common.rc b/common.rc
index edf0318..42d2ddc 100644
--- a/common.rc
+++ b/common.rc
@@ -311,8 +311,8 @@ _scratch_mkfs_xfs()
     rm -f $tmp_dir.mkfserr $tmp_dir.mkfsstd
 
     if [ "$LARGE_SCRATCH_DEV" = yes ]; then
-        [ -z "$RETAIN_AG_BYTES" ] && RETAIN_AG_BYTES=0
-        ./tools/ag-wipe -q -r $RETAIN_AG_BYTES $SCRATCH_DEV
+        [ -z "$SCRATCH_DEV_EMPTY_SPACE" ] && SCRATCH_DEV_EMPTY_SPACE=0
+        ./tools/ag-wipe -q -r $SCRATCH_DEV_EMPTY_SPACE $SCRATCH_DEV
     fi
 
     return $mkfs_status
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 05/13] xfstests: use preallocation for ag-wiper
  2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
                   ` (3 preceding siblings ...)
  2013-03-15 11:53 ` [PATCH 04/13] xfstests: rename RETAIN_AG_BYTES Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
  2013-03-19 19:47   ` Rich Johnston
  2013-03-15 11:53 ` [PATCH 06/13] xfstests: use command line option for setting extra space Dave Chinner
                   ` (8 subsequent siblings)
  13 siblings, 1 reply; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
  To: xfs

From: Dave Chinner <dchinner@redhat.com>

To enable sane testing of large scale filesystems, the --large-fs
test option uses xfs_db magic to mark AGs full without doing any IO.
This leaves only a small amount of free space left in the filesystem
to stress the high AGs of the filesystem rather than the low AGs.

This method requires us to have special filesystem check options to
avoid free space checking in xfs_check, and we cannot current run
xfs_repair on such a filesystem at all. As it is, free space
checking on xfs_check does not scale, so we still need to avoid this
checking regardless of how we fill the filesystem.

We can acheive exactly the same fill behaviour by preallocating a
single large file in the filesystem immediately after creating it.
This is a filesystem independent manner of filling the filesystem,
and allows us to do large filesystem testing on more than just XFS.

Further, this preallocation method effectively adds a new "very
large file" test. It also enables us to run an unmodified xfs_repair
or filesystem specific fsck program to check the filesystem for
sanity, so we can now do full sanity checking of such large
filesystems.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 common.rc |   58 +++++++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 53 insertions(+), 5 deletions(-)

diff --git a/common.rc b/common.rc
index 42d2ddc..d221be1 100644
--- a/common.rc
+++ b/common.rc
@@ -276,6 +276,47 @@ _scratch_mkfs_options()
     echo $SCRATCH_OPTIONS $MKFS_OPTIONS $* $SCRATCH_DEV
 }
 
+
+_setup_large_xfs_fs()
+{
+	fs_size=$1
+	local tmp_dir=/tmp/
+
+	[ "$LARGE_SCRATCH_DEV" != yes ] && return 0
+	[ -z "$SCRATCH_DEV_EMPTY_SPACE" ] && SCRATCH_DEV_EMPTY_SPACE=0
+	[ $SCRATCH_DEV_EMPTY_SPACE -ge $fs_size ] && return 0
+
+	# calculate the size of the file we need to allocate.
+	# Default free space in the FS is 50GB, but you can specify more via
+	# SCRATCH_DEV_EMPTY_SPACE
+	file_size=$(($fs_size - 50*1024*1024*1024))
+	file_size=$(($file_size - $SCRATCH_DEV_EMPTY_SPACE))
+
+	# mount the filesystem, create the file, unmount it
+	_scratch_mount 2>&1 >$tmp_dir/mnt.err
+	local status=$?
+	if [ $status -ne 0 ]; then
+		echo "mount failed"
+		cat $tmp_dir/mnt.err >&2
+		rm -f $tmp_dir/mnt.err
+		return $status
+	fi
+	rm -f $tmp_dir/mnt.err
+
+	xfs_io -F -f \
+		-c "truncate $file_size" \
+		-c "falloc -k 0 $file_size" \
+		$SCRATCH_MNT/.use_space 2>&1 > /dev/null
+	status=$?
+	umount $SCRATCH_MNT
+	if [ $status -ne 0 ]; then
+		echo "large file prealloc failed"
+		cat $tmp_dir/mnt.err >&2
+		return $status
+	fi
+	return 0
+}
+
 _scratch_mkfs_xfs()
 {
     # extra mkfs options can be added by tests
@@ -305,16 +346,23 @@ _scratch_mkfs_xfs()
         mkfs_status=$?
     fi
 
+    if [ $mkfs_status -eq 0 -a "$LARGE_SCRATCH_DEV" = yes ]; then
+	# manually parse the mkfs output to get the fs size in bytes
+	local fs_size
+	fs_size=`cat $tmp_dir.mkfsstd | perl -ne '
+	    if (/^data\s+=\s+bsize=(\d+)\s+blocks=(\d+)/) {
+		my $size = $1 * $2;
+		print STDOUT "$size\n";
+	    }'`
+	_setup_large_xfs_fs $fs_size
+	mkfs_status=$?
+    fi
+
     # output stored mkfs output
     cat $tmp_dir.mkfserr >&2
     cat $tmp_dir.mkfsstd
     rm -f $tmp_dir.mkfserr $tmp_dir.mkfsstd
 
-    if [ "$LARGE_SCRATCH_DEV" = yes ]; then
-        [ -z "$SCRATCH_DEV_EMPTY_SPACE" ] && SCRATCH_DEV_EMPTY_SPACE=0
-        ./tools/ag-wipe -q -r $SCRATCH_DEV_EMPTY_SPACE $SCRATCH_DEV
-    fi
-
     return $mkfs_status
 }
 
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 06/13] xfstests: use command line option for setting extra space
  2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
                   ` (4 preceding siblings ...)
  2013-03-15 11:53 ` [PATCH 05/13] xfstests: use preallocation for ag-wiper Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
  2013-03-15 11:53 ` [PATCH 07/13] xfstest: enable xfs_repair for large filesystem testing Dave Chinner
                   ` (7 subsequent siblings)
  13 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
  To: xfs

From: Dave Chinner <dchinner@redhat.com>

Allow the extra free space to leave in large scratch filesystems to
be specified by a command line option rather than just via an
environment variable.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
 common |    5 +++++
 1 file changed, 5 insertions(+)

diff --git a/common b/common
index da86cd9..78ac654 100644
--- a/common
+++ b/common
@@ -247,6 +247,11 @@ s/ .*//p
 	    usage
 	    ;;
 
+	--extra-space=*)
+	    export SCRATCH_DEV_EMPTY_SPACE=${r#*=}
+	    xpand=false
+	    ;;
+
 	*)
 	    start=$r
 	    end=$r
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 07/13] xfstest: enable xfs_repair for large filesystem testing
  2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
                   ` (5 preceding siblings ...)
  2013-03-15 11:53 ` [PATCH 06/13] xfstests: use command line option for setting extra space Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
  2013-03-15 11:53 ` [PATCH 08/13] xfstests: always use test option when checking large scratch device Dave Chinner
                   ` (6 subsequent siblings)
  13 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
  To: xfs

From: Dave Chinner <dchinner@redhat.com>

Now that large filesystem testing does not play free space games to
fill the space without IO, we can enable xfs_repair when running in
this mode. xfs_repair has had it's scalability problems solved, too,
so this is a safe thing to do.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
 common.rc |   18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/common.rc b/common.rc
index d221be1..ee4f522 100644
--- a/common.rc
+++ b/common.rc
@@ -1273,19 +1273,17 @@ _check_xfs_filesystem()
     extra_mount_options=""
     device=$1
     if [ "$2" != "none" ]; then
-       	extra_log_options="-l$2"
+	extra_log_options="-l$2"
         extra_mount_options="-ologdev=$2"
     fi
 
     if [ "$3" != "none" ]; then
-       	extra_rt_options="-r$3"
+	extra_rt_options="-r$3"
         extra_mount_options=$extra_mount_options" -ortdev=$3"
     fi
     extra_mount_options=$extra_mount_options" $MOUNT_OPTIONS"
 
     [ "$FSTYP" != xfs ] && return 0
-    testoption=""
-    [ "$LARGE_SCRATCH_DEV" = yes ] && testoption=-t
 
     type=`_fs_type $device`
     ok=1
@@ -1310,8 +1308,13 @@ _check_xfs_filesystem()
         ok=0
     fi
 
-    $XFS_CHECK_PROG $testoption $extra_log_options $device 2>&1 |\
-	 _fix_malloc >$tmp.fs_check
+    # xfs_check runs out of memory on large files, so even providing the test
+    # option (-t) to avoid indexing the free space trees doesn't make it pass on
+    # large filesystems. Avoid it.
+    if [ "$LARGE_SCRATCH_DEV" != yes ]; then
+	    $XFS_CHECK_PROG $extra_log_options $device 2>&1 |\
+		 _fix_malloc >$tmp.fs_check
+    fi
     if [ -s $tmp.fs_check ]
     then
         echo "_check_xfs_filesystem: filesystem on $device is inconsistent (c) (see $seq.full)"
@@ -1323,8 +1326,7 @@ _check_xfs_filesystem()
 
         ok=0
     fi
-    # repair doesn't scale massively at this stage, optionally skip it for now
-    [ "$LARGE_SCRATCH_DEV" = yes ] || \
+
     $XFS_REPAIR_PROG -n $extra_log_options $extra_rt_options $device >$tmp.repair 2>&1
     if [ $? -ne 0 ]
     then
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 08/13] xfstests: always use test option when checking large scratch device
  2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
                   ` (6 preceding siblings ...)
  2013-03-15 11:53 ` [PATCH 07/13] xfstest: enable xfs_repair for large filesystem testing Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
  2013-03-15 11:53 ` [PATCH 09/13] xfstests: enable large fs testing on ext4 Dave Chinner
                   ` (5 subsequent siblings)
  13 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
  To: xfs

From: Dave Chinner <dchinner@redhat.com>

Some tests call _check_scratch_device directly and when using a
large filesystem this needs to run with a -t option to avoid
consuming large amounts of memory. Make this happen in all cases
that the scratch device is checked.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
 017       |    7 ++-----
 common.rc |    2 ++
 2 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/017 b/017
index 7db4667..fb92375 100755
--- a/017
+++ b/017
@@ -48,9 +48,6 @@ _supported_os Linux
 
 _require_scratch
 
-checkopts=""
-[ "$USE_BIG_LOOPFS" = yes ] && checkopts=-t
-
 echo "*** init FS"
 
 rm -f $seq.full
@@ -82,8 +79,8 @@ do
         echo ""                             >>$seq.full
         echo "*** XFS_CHECK ***"            >>$seq.full
         echo ""                             >>$seq.full
-        _scratch_xfs_check $checkopts       >>$seq.full 2>&1 \
-            || _fail "xfs_check $checkopts failed"
+        _scratch_xfs_check                  >>$seq.full 2>&1 \
+            || _fail "xfs_check failed"
         _scratch_mount -o remount,rw \
             || _fail "remount rw failed"
 done
diff --git a/common.rc b/common.rc
index ee4f522..1026572 100644
--- a/common.rc
+++ b/common.rc
@@ -475,6 +475,8 @@ _scratch_xfs_check()
     SCRATCH_OPTIONS=""
     [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ] && \
         SCRATCH_OPTIONS="-l $SCRATCH_LOGDEV"
+    [ "$LARGE_SCRATCH_DEV" = yes ] && \
+        SCRATCH_OPTIONS=$SCRATCH_OPTIONS" -t"
     $XFS_CHECK_PROG $SCRATCH_OPTIONS $* $SCRATCH_DEV
 }
 
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 09/13] xfstests: enable large fs testing on ext4
  2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
                   ` (7 preceding siblings ...)
  2013-03-15 11:53 ` [PATCH 08/13] xfstests: always use test option when checking large scratch device Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
  2013-03-15 11:53 ` [PATCH 10/13] xfstests: disable tests that typically fail on large filesystems Dave Chinner
                   ` (4 subsequent siblings)
  13 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
  To: xfs

From: Dave Chinner <dchinner@redhat.com>

Now that setting up large filesystem testing on sparse loopback
devices uses a generic method for filling the filesystem, extent
support to ext4 filesystems.

ext4 is slightly more complex to fill as it does not support files
larger than 16TB. Hence a slightly more complex method of using
multiple smaller files to fill the space is necessary.

WARNING: be prepeared for ext4 to take forever to prepare large
filesystems as allocation of large amounts of space (especially as
it approaches ENOSPC) is can take minutes rather than milliseconds

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
 common.rc |   90 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 90 insertions(+)

diff --git a/common.rc b/common.rc
index 1026572..690c6a3 100644
--- a/common.rc
+++ b/common.rc
@@ -366,6 +366,93 @@ _scratch_mkfs_xfs()
     return $mkfs_status
 }
 
+_setup_large_ext4_fs()
+{
+	fs_size=$1
+	local tmp_dir=/tmp/
+
+	[ "$LARGE_SCRATCH_DEV" != yes ] && return 0
+	[ -z "$SCRATCH_DEV_EMPTY_SPACE" ] && SCRATCH_DEV_EMPTY_SPACE=0
+	[ $SCRATCH_DEV_EMPTY_SPACE -ge $fs_size ] && return 0
+
+	# Default free space in the FS is 50GB, but you can specify more via
+	# SCRATCH_DEV_EMPTY_SPACE
+	space_to_consume=$(($fs_size - 50*1024*1024*1024 - $SCRATCH_DEV_EMPTY_SPACE))
+
+	# mount the filesystem and create 16TB - 4KB files until we consume
+	# all the necessary space.
+	_scratch_mount 2>&1 >$tmp_dir/mnt.err
+	local status=$?
+	if [ $status -ne 0 ]; then
+		echo "mount failed"
+		cat $tmp_dir/mnt.err >&2
+		rm -f $tmp_dir/mnt.err
+		return $status
+	fi
+	rm -f $tmp_dir/mnt.err
+
+	file_size=$((16*1024*1024*1024*1024 - 4096))
+	nfiles=0
+	while [ $space_to_consume -gt $file_size ]; do
+
+		xfs_io -F -f \
+			-c "truncate $file_size" \
+			-c "falloc -k 0 $file_size" \
+			$SCRATCH_MNT/.use_space.$nfiles 2>&1
+		status=$?
+		if [ $status -ne 0 ]; then
+			break;
+		fi
+
+		space_to_consume=$(( $space_to_consume - $file_size ))
+		nfiles=$(($nfiles + 1))
+	done
+
+	# consume the remaining space.
+	if [ $space_to_consume -gt 0 ]; then
+		xfs_io -F -f \
+			-c "truncate $space_to_consume" \
+			-c "falloc -k 0 $space_to_consume" \
+			$SCRATCH_MNT/.use_space.$nfiles 2>&1
+		status=$?
+	fi
+
+	umount $SCRATCH_MNT
+	if [ $status -ne 0 ]; then
+		echo "large file prealloc failed"
+		cat $tmp_dir/mnt.err >&2
+		return $status
+	fi
+	return 0
+}
+_scratch_mkfs_ext4()
+{
+	local tmp_dir=/tmp/
+
+	/sbin/mkfs -t $FSTYP -- $MKFS_OPTIONS $* $SCRATCH_DEV \
+			2>$tmp_dir.mkfserr 1>$tmp_dir.mkfsstd
+	local mkfs_status=$?
+
+	if [ $mkfs_status -eq 0 -a "$LARGE_SCRATCH_DEV" = yes ]; then
+		# manually parse the mkfs output to get the fs size in bytes
+		fs_size=`cat $tmp_dir.mkfsstd | awk ' \
+			/^Block size/ { split($2, a, "="); bs = a[2] ; } \
+			/ inodes, / { blks = $3 } \
+			/reserved for the super user/ { resv = $1 } \
+			END { fssize = bs * blks - resv; print fssize }'`
+
+		_setup_large_ext4_fs $fs_size
+		mkfs_status=$?
+	fi
+
+	# output stored mkfs output
+	cat $tmp_dir.mkfserr >&2
+	cat $tmp_dir.mkfsstd
+	rm -f $tmp_dir.mkfserr $tmp_dir.mkfsstd
+
+	return $mkfs_status
+}
+
 _scratch_mkfs()
 {
     case $FSTYP in
@@ -381,6 +468,9 @@ _scratch_mkfs()
     btrfs)
         $MKFS_BTRFS_PROG $MKFS_OPTIONS $* $SCRATCH_DEV > /dev/null
 	;;
+    ext4)
+	_scratch_mkfs_ext4 $*
+	;;
     *)
 	yes | $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS $* $SCRATCH_DEV
 	;;
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 10/13] xfstests: disable tests that typically fail on large filesystems
  2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
                   ` (8 preceding siblings ...)
  2013-03-15 11:53 ` [PATCH 09/13] xfstests: enable large fs testing on ext4 Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
  2013-03-15 11:53 ` [PATCH 11/13] xfstests: exclude largefs fill files from dump tests Dave Chinner
                   ` (3 subsequent siblings)
  13 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
  To: xfs

From: Dave Chinner <dchinner@redhat.com>

Test 017 typically fails due to the OOM killer kicking in and
killing everything but xfs_db so panics the machine.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
 017 |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/017 b/017
index fb92375..65ddce1 100755
--- a/017
+++ b/017
@@ -48,6 +48,10 @@ _supported_os Linux
 
 _require_scratch
 
+# xfs_db will OOM kill the machine if you don't have huge amounts of RAM, so
+# don't run this on large filesystems.
+_require_no_large_scratch_dev
+
 echo "*** init FS"
 
 rm -f $seq.full
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 11/13] xfstests: exclude largefs fill files from dump tests
  2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
                   ` (9 preceding siblings ...)
  2013-03-15 11:53 ` [PATCH 10/13] xfstests: disable tests that typically fail on large filesystems Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
  2013-03-19 19:49   ` Rich Johnston
  2013-03-15 11:53 ` [PATCH 12/13] xfstests: filter files in 019 for --large-fs Dave Chinner
                   ` (2 subsequent siblings)
  13 siblings, 1 reply; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
  To: xfs

From: Dave Chinner <dchinner@redhat.com>

The xfsdump/restore tests will see the space filling files and may
try to backthem up and restore them, consuming huge amounts of time
to do (espcially when diffing the results). Exclude the space
filling files by setting the no dump attributes on them and ensure
that xfsdump runs with the -e flag to exclude such files.

This also needs a dump filter addition to remove the output that
files were skipped, and to decrement the count of files processed by
xfsrestore because the inventory still includes excluded files.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 common.config |    3 ++-
 common.dump   |   34 ++++++++++++++++++++++++++--------
 common.filter |    4 +++-
 common.rc     |    1 +
 4 files changed, 32 insertions(+), 10 deletions(-)

diff --git a/common.config b/common.config
index 5ac58bf..ab19772 100644
--- a/common.config
+++ b/common.config
@@ -160,7 +160,8 @@ export XFS_GROWFS_PROG=`set_prog_path xfs_growfs`
 export XFS_IO_PROG="`set_prog_path xfs_io`"
 export XFS_PARALLEL_REPAIR_PROG="`set_prog_path xfs_prepair`"
 export XFS_PARALLEL_REPAIR64_PROG="`set_prog_path xfs_prepair64`"
-export XFSDUMP_PROG="`set_prog_path xfsdump`"
+export __XFSDUMP_PROG="`set_prog_path xfsdump`"
+export XFSDUMP_PROG="$__XFSDUMP_PROG -e"
 export XFSRESTORE_PROG="`set_prog_path xfsrestore`"
 export XFSINVUTIL_PROG="`set_prog_path xfsinvutil`"
 export DBENCH_PROG="`set_prog_path dbench`"
diff --git a/common.dump b/common.dump
index e89dedc..be55249 100644
--- a/common.dump
+++ b/common.dump
@@ -29,7 +29,8 @@ if [ -n "$DEBUGDUMP" ]; then
 
 	# Use dump/restore in qa directory (copy them here) for debugging
 	export PATH="$here:$PATH"
-	export XFSDUMP_PROG="`set_prog_path xfsdump`"
+	export __XFSDUMP_PROG="`set_prog_path xfsdump`"
+	export XFSDUMP_PROG="$__XFSDUMP_PROG -e"
 	export XFSRESTORE_PROG="`set_prog_path xfsrestore`"
 	export XFSINVUTIL_PROG="`set_prog_path xfsinvutil`"
 	[ -x $here/xfsdump ]    && echo "Using xfstests' xfsdump for debug"
@@ -829,7 +830,7 @@ _dump_filter_main()
 {
   _filter_devchar |\
   sed \
-      -e "s#$XFSDUMP_PROG#xfsdump#"			\
+      -e "s#$__XFSDUMP_PROG#xfsdump#"			\
       -e "s#$XFSRESTORE_PROG#xfsrestore#"		\
       -e "s#$XFSINVUTIL_PROG#xfsinvutil#"		\
       -e "s/`hostname`/HOSTNAME/"			\
@@ -872,13 +873,29 @@ _dump_filter_main()
       -e '/xfsdump: ino map phase [5]/,1d'		\
       -e 's/id:[[:space:]]*[0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}/ID: ID/'						\
       -e 's/\[y\/n\][- ]----------------------*/\[y\/n\]/'		\
+      -e '/skip attribute set/d'				\
   | perl -ne '
-      if ($_ =~ /(?:Dump|Restore) Summary/) {
-        $skip = 1;
-      } elsif ($_ =~ /(?:Dump|Restore) Status/) {
-        $skip = 0;
-      }
-      print if (! $skip);'
+	# filter out all the output between the lines "Dump Summary:"
+	# and "Dump Status:"
+	if ($_ =~ /(?:Dump|Restore) Summary/) {
+		$skip = 1;
+	} elsif ($_ =~ /(?:Dump|Restore) Status/) {
+		$skip = 0;
+	}
+	print if (! $skip);' \
+  | perl -ne '
+	# correct the file count if large scratch devices are being used
+	$skip = 0;
+	if ($_ =~ /(\S+) directories and (\S+) entries/) {
+		$foo = $2;
+		if ($ENV{'LARGE_SCRATCH_DEV'} && $foo > 0) {
+			$foo -= 1;
+		}
+		printf("xfsrestore: %u directories and %u entries processed\n",
+						$1, $foo);
+		$skip = 1;
+	}
+	print if (! $skip);'
 }
 
 _dump_filter()
@@ -916,6 +933,7 @@ _dir_filter()
     -e "s#$dump_sdir#DUMP_SUBDIR#g"   \
     -e "s#$restore_sdir#RESTORE_SUBDIR#g" \
     -e "s#$$#PID#g" \
+    -e "/Only in SCRATCH_MNT: .use_space/d" \
 
 }
 
diff --git a/common.filter b/common.filter
index c34809f..da5675f 100644
--- a/common.filter
+++ b/common.filter
@@ -223,7 +223,9 @@ _filter_test_dir()
 
 _filter_scratch()
 {
-	sed -e "s,$SCRATCH_DEV,SCRATCH_DEV,g" -e "s,$SCRATCH_MNT,SCRATCH_MNT,g"
+	sed -e "s,$SCRATCH_DEV,SCRATCH_DEV,g" \
+	    -e "s,$SCRATCH_MNT,SCRATCH_MNT,g" \
+	    -e "/.use_space/d"
 }
 
 # Turn any device in the scratch pool into SCRATCH_DEV
diff --git a/common.rc b/common.rc
index 690c6a3..96d5077 100644
--- a/common.rc
+++ b/common.rc
@@ -306,6 +306,7 @@ _setup_large_xfs_fs()
 	xfs_io -F -f \
 		-c "truncate $file_size" \
 		-c "falloc -k 0 $file_size" \
+		-c "chattr +d" \
 		$SCRATCH_MNT/.use_space 2>&1 > /dev/null
 	status=$?
 	umount $SCRATCH_MNT
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 12/13] xfstests: filter files in 019 for --large-fs
  2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
                   ` (10 preceding siblings ...)
  2013-03-15 11:53 ` [PATCH 11/13] xfstests: exclude largefs fill files from dump tests Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
  2013-03-19 19:52   ` Rich Johnston
  2013-03-15 11:53 ` [PATCH 13/13] xfstests: correct quota file count " Dave Chinner
  2013-03-19 20:20 ` [PATCH 00/13] xfstests: generic --large-fs support Rich Johnston
  13 siblings, 1 reply; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
  To: xfs

From: Dave Chinner <dchinner@redhat.com>

Make sure that the .use_space files don't appear in the files dumped
to the output files.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 019 |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/019 b/019
index 42e3c22..88be464 100755
--- a/019
+++ b/019
@@ -128,15 +128,16 @@ _verify_fs()
 
 	echo "*** verify FS"
 	(cd $SCRATCH_MNT ; find . | LC_COLLATE=POSIX sort \
+		| grep -v ".use_space" \
 		| xargs $here/src/lstat64 | _filter_stat)
 	diff -q $SCRATCH_MNT/bigfile $tempfile.2 \
 		|| _fail "bigfile corrupted"
-    
+
 	echo "*** unmount FS"
 	_full "umount"
 	umount $SCRATCH_DEV >>$seqfull 2>&1 \
 		|| _fail "umount failed"
-	    
+
 	echo "*** check FS"
 	_check_scratch_fs
 }
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 13/13] xfstests: correct quota file count for --large-fs
  2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
                   ` (11 preceding siblings ...)
  2013-03-15 11:53 ` [PATCH 12/13] xfstests: filter files in 019 for --large-fs Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
  2013-03-19 19:52   ` Rich Johnston
  2013-03-19 20:20 ` [PATCH 00/13] xfstests: generic --large-fs support Rich Johnston
  13 siblings, 1 reply; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
  To: xfs

From: Dave Chinner <dchinner@redhat.com>

The files used to fill space when using --large-fs contribute
towards the quota of uid 0. Modify the quota output filter in test
050 to "hide" these files from the test output.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 050       |    8 +++++++-
 common.rc |    2 ++
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/050 b/050
index cf70b5a..acca654 100755
--- a/050
+++ b/050
@@ -69,7 +69,13 @@ _filter_report()
 	perl -npe '
 		s/^\#'$id' /[NAME] /g;
 		s/^\#0 \d+ /[ROOT] 0 /g;
-		s/6 days/7 days/g'
+		s/6 days/7 days/g' |
+	perl -npe '
+		$val = 0;
+		if ($ENV{'LARGE_SCRATCH_DEV'}) {
+			$val = $ENV{'NUM_SPACE_FILES'};
+		}
+		s/(^\[ROOT\] \S+ \S+ \S+ \S+ \[--------\] )(\S+)/$1@{[$2 - $val]}/g'
 }
 
 # The actual point at which limit enforcement takes place for the
diff --git a/common.rc b/common.rc
index 96d5077..319680d 100644
--- a/common.rc
+++ b/common.rc
@@ -308,6 +308,7 @@ _setup_large_xfs_fs()
 		-c "falloc -k 0 $file_size" \
 		-c "chattr +d" \
 		$SCRATCH_MNT/.use_space 2>&1 > /dev/null
+	export NUM_SPACE_FILES=1
 	status=$?
 	umount $SCRATCH_MNT
 	if [ $status -ne 0 ]; then
@@ -417,6 +418,7 @@ _setup_large_ext4_fs()
 			$SCRATCH_MNT/.use_space.$nfiles 2>&1
 		status=$?
 	fi
+	export NUM_SPACE_FILES=$nfiles
 
 	umount $SCRATCH_MNT
 	if [ $status -ne 0 ]; then
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* Re: [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs
  2013-03-15 11:53 ` [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs Dave Chinner
@ 2013-03-19 19:44   ` Rich Johnston
  2013-03-19 20:58   ` Ben Myers
  1 sibling, 0 replies; 23+ messages in thread
From: Rich Johnston @ 2013-03-19 19:44 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs


Looks good.
Reviewed-by: Rich Johnston <rjohnston@sgi.com>


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 03/13] xfstests: rename USE_BIG_LOOPFS to be more generic
  2013-03-15 11:53 ` [PATCH 03/13] xfstests: rename USE_BIG_LOOPFS to be more generic Dave Chinner
@ 2013-03-19 19:46   ` Rich Johnston
  0 siblings, 0 replies; 23+ messages in thread
From: Rich Johnston @ 2013-03-19 19:46 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs


Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 05/13] xfstests: use preallocation for ag-wiper
  2013-03-15 11:53 ` [PATCH 05/13] xfstests: use preallocation for ag-wiper Dave Chinner
@ 2013-03-19 19:47   ` Rich Johnston
  0 siblings, 0 replies; 23+ messages in thread
From: Rich Johnston @ 2013-03-19 19:47 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs

Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 11/13] xfstests: exclude largefs fill files from dump tests
  2013-03-15 11:53 ` [PATCH 11/13] xfstests: exclude largefs fill files from dump tests Dave Chinner
@ 2013-03-19 19:49   ` Rich Johnston
  0 siblings, 0 replies; 23+ messages in thread
From: Rich Johnston @ 2013-03-19 19:49 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs


Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 13/13] xfstests: correct quota file count for --large-fs
  2013-03-15 11:53 ` [PATCH 13/13] xfstests: correct quota file count " Dave Chinner
@ 2013-03-19 19:52   ` Rich Johnston
  0 siblings, 0 replies; 23+ messages in thread
From: Rich Johnston @ 2013-03-19 19:52 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs


Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 12/13] xfstests: filter files in 019 for --large-fs
  2013-03-15 11:53 ` [PATCH 12/13] xfstests: filter files in 019 for --large-fs Dave Chinner
@ 2013-03-19 19:52   ` Rich Johnston
  0 siblings, 0 replies; 23+ messages in thread
From: Rich Johnston @ 2013-03-19 19:52 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs

Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 00/13] xfstests: generic --large-fs support
  2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
                   ` (12 preceding siblings ...)
  2013-03-15 11:53 ` [PATCH 13/13] xfstests: correct quota file count " Dave Chinner
@ 2013-03-19 20:20 ` Rich Johnston
  13 siblings, 0 replies; 23+ messages in thread
From: Rich Johnston @ 2013-03-19 20:20 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs

This patchset has been committed, thanks for submitting it Dave.

Thanks
--Rich

On 03/15/2013 06:53 AM, Dave Chinner wrote:
> Hi folks,
>
> This is a repost of my current version of the large filesystem
> testing support for xfstests. I've been using it for more than a
> year, and I think the ext4 guys would like to have it available to
> them.
>
> Last posted here:
>
> http://oss.sgi.com/archives/xfs/2012-07/msg00361.html
>
> There are changes from last time, but it's months since i've made
> them, so there's no fine grained changelog. :/
>
> Cheers,
>
> Dave.
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs
  2013-03-15 11:53 ` [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs Dave Chinner
  2013-03-19 19:44   ` Rich Johnston
@ 2013-03-19 20:58   ` Ben Myers
  2013-03-19 22:52     ` Dave Chinner
  1 sibling, 1 reply; 23+ messages in thread
From: Ben Myers @ 2013-03-19 20:58 UTC (permalink / raw)
  To: Rich Johnston; +Cc: xfs

Hey Rich,
   
On Fri, Mar 15, 2013 at 10:53:19PM +1100, Dave Chinner wrote:
> From: Dave Chinner <dchinner@redhat.com>
> 
> This patch it to ensure that xfstests passes on non-crc filesystems
> with a CRC enabled userspace.
> 
> Filter out the mkfs/xfs_info CRC line from tests that capture the
> output of these commands.
> 
> Filter out new error noise from xfs_repair that occurs for
> xfs_repair as a result of the CRC changes.

IIRC this change was a temporary one so that xfstests could be run with crc
enabled filesystems.

> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> ---
>  096           |    1 +
>  206           |    9 ++++++---
>  common.repair |    3 +++
>  3 files changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/096 b/096
> index 237edad..5a2273d 100755
> --- a/096
> +++ b/096
> @@ -85,6 +85,7 @@ _mkfs_filter()
>  	-e '/inode-paths/d' \
>  	-e 's/\(log[ 	]*=\).*bsize/\1LOG                    bsize/' \
>  	-e 's/\(realtime[ 	]*=\).*extsz/\1REALTIME               extsz/' \
> +	-e '/.*crc=/d' \
>  	-e 's/ *$//' \
>     | grep -v parent
>  }
> diff --git a/206 b/206
> index e8bc922..e46e6a3 100755
> --- a/206
> +++ b/206
> @@ -77,7 +77,8 @@ echo "=== mkfs.xfs ==="
>  mkfs.xfs -f -bsize=4096 -dagsize=76288719b,size=3905982455b -llazy-count=0 $tmpfile \
>  	| sed -e "s,^meta-data=.*isize,meta-data=FILE                   isize,g" \
>  		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
> -		-e "s/, projid32bit=[0-9]//"
> +		-e "s/, projid32bit=[0-9]//" \
> +		-e "/.*crc=/d"
>  
>  mount -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount"
>  
> @@ -85,13 +86,15 @@ mount -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount"
>  echo "=== xfs_growfs ==="
>  xfs_growfs $tmpdir \
>  	| sed -e "s,^meta-data=.*isize,meta-data=FILE                   isize,g" \
> -		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/"
> +		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
> +		-e "/.*crc=/d"
>  
>  # and double-check the new geometry
>  echo "=== xfs_info ==="
>  xfs_info $tmpdir \
>  	| sed -e "s,^meta-data=.*isize,meta-data=FILE                   isize,g" \
> -		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/"
> +		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
> +		-e "/.*crc=/d"
>  
>  # _cleanup cleans up for us
>  
> diff --git a/common.repair b/common.repair
> index 23a1ed6..db160bb 100644
> --- a/common.repair
> +++ b/common.repair
> @@ -77,6 +77,9 @@ s/\s+- \d+:\d\d:\d\d:.*\n//g;
>  /^agi_/ && next; # remove agi counts
>  /^sb_/ && next; # remove sb counts
>  /^agi unlinked/ && next; # remove agi unlinked bucket warning
> +# crc enabled filesystem output
> +/XFS_CORRUPTION_ERROR/ && next;
> +/^bad uuid/ && next;

Doesn't this filter out xfs repair failures which should be reported?

Thanks,
	Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs
  2013-03-19 20:58   ` Ben Myers
@ 2013-03-19 22:52     ` Dave Chinner
  0 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-19 22:52 UTC (permalink / raw)
  To: Ben Myers; +Cc: Rich Johnston, xfs

On Tue, Mar 19, 2013 at 03:58:01PM -0500, Ben Myers wrote:
> Hey Rich,
>    
> On Fri, Mar 15, 2013 at 10:53:19PM +1100, Dave Chinner wrote:
> > From: Dave Chinner <dchinner@redhat.com>
> > 
> > This patch it to ensure that xfstests passes on non-crc filesystems
> > with a CRC enabled userspace.
> > 
> > Filter out the mkfs/xfs_info CRC line from tests that capture the
> > output of these commands.
> > 
> > Filter out new error noise from xfs_repair that occurs for
> > xfs_repair as a result of the CRC changes.
> 
> IIRC this change was a temporary one so that xfstests could be run with crc
> enabled filesystems.

It's going to be a permanent change because we dump mkfs.xfs output
into golden output files and that means the "crc=X" line will always
need to be filtered out

> > diff --git a/common.repair b/common.repair
> > index 23a1ed6..db160bb 100644
> > --- a/common.repair
> > +++ b/common.repair
> > @@ -77,6 +77,9 @@ s/\s+- \d+:\d\d:\d\d:.*\n//g;
> >  /^agi_/ && next; # remove agi counts
> >  /^sb_/ && next; # remove sb counts
> >  /^agi unlinked/ && next; # remove agi unlinked bucket warning
> > +# crc enabled filesystem output
> > +/XFS_CORRUPTION_ERROR/ && next;
> > +/^bad uuid/ && next;
> 
> Doesn't this filter out xfs repair failures which should be reported?

No, because repair will issue errors when bad metadata is read off
disk now, and as such they are brand new output introduced by the
CRC patches. Over time this might get cleaned up, but at least for
initial testing and debugging this noise will remain in the repair
code....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 23+ messages in thread
end of thread, other threads:[~2013-03-19 22:53 UTC | newest]

Thread overview: 23+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
2013-03-15 11:53 ` [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs Dave Chinner
2013-03-19 19:44   ` Rich Johnston
2013-03-19 20:58   ` Ben Myers
2013-03-19 22:52     ` Dave Chinner
2013-03-15 11:53 ` [PATCH 02/13] xfstests: add --largefs check option Dave Chinner
2013-03-15 11:53 ` [PATCH 03/13] xfstests: rename USE_BIG_LOOPFS to be more generic Dave Chinner
2013-03-19 19:46   ` Rich Johnston
2013-03-15 11:53 ` [PATCH 04/13] xfstests: rename RETAIN_AG_BYTES Dave Chinner
2013-03-15 11:53 ` [PATCH 05/13] xfstests: use preallocation for ag-wiper Dave Chinner
2013-03-19 19:47   ` Rich Johnston
2013-03-15 11:53 ` [PATCH 06/13] xfstests: use command line option for setting extra space Dave Chinner
2013-03-15 11:53 ` [PATCH 07/13] xfstest: enable xfs_repair for large filesystem testing Dave Chinner
2013-03-15 11:53 ` [PATCH 08/13] xfstests: always use test option when checking large scratch device Dave Chinner
2013-03-15 11:53 ` [PATCH 09/13] xfstests: enable large fs testing on ext4 Dave Chinner
2013-03-15 11:53 ` [PATCH 10/13] xfstests: disable tests that typically fail on large filesystems Dave Chinner
2013-03-15 11:53 ` [PATCH 11/13] xfstests: exclude largefs fill files from dump tests Dave Chinner
2013-03-19 19:49   ` Rich Johnston
2013-03-15 11:53 ` [PATCH 12/13] xfstests: filter files in 019 for --large-fs Dave Chinner
2013-03-19 19:52   ` Rich Johnston
2013-03-15 11:53 ` [PATCH 13/13] xfstests: correct quota file count " Dave Chinner
2013-03-19 19:52   ` Rich Johnston
2013-03-19 20:20 ` [PATCH 00/13] xfstests: generic --large-fs support Rich Johnston

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox