* [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
2013-03-19 19:44 ` Rich Johnston
2013-03-19 20:58 ` Ben Myers
2013-03-15 11:53 ` [PATCH 02/13] xfstests: add --largefs check option Dave Chinner
` (12 subsequent siblings)
13 siblings, 2 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
To: xfs
From: Dave Chinner <dchinner@redhat.com>
This patch it to ensure that xfstests passes on non-crc filesystems
with a CRC enabled userspace.
Filter out the mkfs/xfs_info CRC line from tests that capture the
output of these commands.
Filter out new error noise from xfs_repair that occurs for
xfs_repair as a result of the CRC changes.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
096 | 1 +
206 | 9 ++++++---
common.repair | 3 +++
3 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/096 b/096
index 237edad..5a2273d 100755
--- a/096
+++ b/096
@@ -85,6 +85,7 @@ _mkfs_filter()
-e '/inode-paths/d' \
-e 's/\(log[ ]*=\).*bsize/\1LOG bsize/' \
-e 's/\(realtime[ ]*=\).*extsz/\1REALTIME extsz/' \
+ -e '/.*crc=/d' \
-e 's/ *$//' \
| grep -v parent
}
diff --git a/206 b/206
index e8bc922..e46e6a3 100755
--- a/206
+++ b/206
@@ -77,7 +77,8 @@ echo "=== mkfs.xfs ==="
mkfs.xfs -f -bsize=4096 -dagsize=76288719b,size=3905982455b -llazy-count=0 $tmpfile \
| sed -e "s,^meta-data=.*isize,meta-data=FILE isize,g" \
-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
- -e "s/, projid32bit=[0-9]//"
+ -e "s/, projid32bit=[0-9]//" \
+ -e "/.*crc=/d"
mount -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount"
@@ -85,13 +86,15 @@ mount -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount"
echo "=== xfs_growfs ==="
xfs_growfs $tmpdir \
| sed -e "s,^meta-data=.*isize,meta-data=FILE isize,g" \
- -e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/"
+ -e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
+ -e "/.*crc=/d"
# and double-check the new geometry
echo "=== xfs_info ==="
xfs_info $tmpdir \
| sed -e "s,^meta-data=.*isize,meta-data=FILE isize,g" \
- -e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/"
+ -e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
+ -e "/.*crc=/d"
# _cleanup cleans up for us
diff --git a/common.repair b/common.repair
index 23a1ed6..db160bb 100644
--- a/common.repair
+++ b/common.repair
@@ -77,6 +77,9 @@ s/\s+- \d+:\d\d:\d\d:.*\n//g;
/^agi_/ && next; # remove agi counts
/^sb_/ && next; # remove sb counts
/^agi unlinked/ && next; # remove agi unlinked bucket warning
+# crc enabled filesystem output
+/XFS_CORRUPTION_ERROR/ && next;
+/^bad uuid/ && next;
print;'
}
--
1.7.10.4
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs
2013-03-15 11:53 ` [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs Dave Chinner
@ 2013-03-19 19:44 ` Rich Johnston
2013-03-19 20:58 ` Ben Myers
1 sibling, 0 replies; 23+ messages in thread
From: Rich Johnston @ 2013-03-19 19:44 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
Looks good.
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs
2013-03-15 11:53 ` [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs Dave Chinner
2013-03-19 19:44 ` Rich Johnston
@ 2013-03-19 20:58 ` Ben Myers
2013-03-19 22:52 ` Dave Chinner
1 sibling, 1 reply; 23+ messages in thread
From: Ben Myers @ 2013-03-19 20:58 UTC (permalink / raw)
To: Rich Johnston; +Cc: xfs
Hey Rich,
On Fri, Mar 15, 2013 at 10:53:19PM +1100, Dave Chinner wrote:
> From: Dave Chinner <dchinner@redhat.com>
>
> This patch it to ensure that xfstests passes on non-crc filesystems
> with a CRC enabled userspace.
>
> Filter out the mkfs/xfs_info CRC line from tests that capture the
> output of these commands.
>
> Filter out new error noise from xfs_repair that occurs for
> xfs_repair as a result of the CRC changes.
IIRC this change was a temporary one so that xfstests could be run with crc
enabled filesystems.
> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> ---
> 096 | 1 +
> 206 | 9 ++++++---
> common.repair | 3 +++
> 3 files changed, 10 insertions(+), 3 deletions(-)
>
> diff --git a/096 b/096
> index 237edad..5a2273d 100755
> --- a/096
> +++ b/096
> @@ -85,6 +85,7 @@ _mkfs_filter()
> -e '/inode-paths/d' \
> -e 's/\(log[ ]*=\).*bsize/\1LOG bsize/' \
> -e 's/\(realtime[ ]*=\).*extsz/\1REALTIME extsz/' \
> + -e '/.*crc=/d' \
> -e 's/ *$//' \
> | grep -v parent
> }
> diff --git a/206 b/206
> index e8bc922..e46e6a3 100755
> --- a/206
> +++ b/206
> @@ -77,7 +77,8 @@ echo "=== mkfs.xfs ==="
> mkfs.xfs -f -bsize=4096 -dagsize=76288719b,size=3905982455b -llazy-count=0 $tmpfile \
> | sed -e "s,^meta-data=.*isize,meta-data=FILE isize,g" \
> -e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
> - -e "s/, projid32bit=[0-9]//"
> + -e "s/, projid32bit=[0-9]//" \
> + -e "/.*crc=/d"
>
> mount -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount"
>
> @@ -85,13 +86,15 @@ mount -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount"
> echo "=== xfs_growfs ==="
> xfs_growfs $tmpdir \
> | sed -e "s,^meta-data=.*isize,meta-data=FILE isize,g" \
> - -e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/"
> + -e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
> + -e "/.*crc=/d"
>
> # and double-check the new geometry
> echo "=== xfs_info ==="
> xfs_info $tmpdir \
> | sed -e "s,^meta-data=.*isize,meta-data=FILE isize,g" \
> - -e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/"
> + -e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
> + -e "/.*crc=/d"
>
> # _cleanup cleans up for us
>
> diff --git a/common.repair b/common.repair
> index 23a1ed6..db160bb 100644
> --- a/common.repair
> +++ b/common.repair
> @@ -77,6 +77,9 @@ s/\s+- \d+:\d\d:\d\d:.*\n//g;
> /^agi_/ && next; # remove agi counts
> /^sb_/ && next; # remove sb counts
> /^agi unlinked/ && next; # remove agi unlinked bucket warning
> +# crc enabled filesystem output
> +/XFS_CORRUPTION_ERROR/ && next;
> +/^bad uuid/ && next;
Doesn't this filter out xfs repair failures which should be reported?
Thanks,
Ben
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs
2013-03-19 20:58 ` Ben Myers
@ 2013-03-19 22:52 ` Dave Chinner
0 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-19 22:52 UTC (permalink / raw)
To: Ben Myers; +Cc: Rich Johnston, xfs
On Tue, Mar 19, 2013 at 03:58:01PM -0500, Ben Myers wrote:
> Hey Rich,
>
> On Fri, Mar 15, 2013 at 10:53:19PM +1100, Dave Chinner wrote:
> > From: Dave Chinner <dchinner@redhat.com>
> >
> > This patch it to ensure that xfstests passes on non-crc filesystems
> > with a CRC enabled userspace.
> >
> > Filter out the mkfs/xfs_info CRC line from tests that capture the
> > output of these commands.
> >
> > Filter out new error noise from xfs_repair that occurs for
> > xfs_repair as a result of the CRC changes.
>
> IIRC this change was a temporary one so that xfstests could be run with crc
> enabled filesystems.
It's going to be a permanent change because we dump mkfs.xfs output
into golden output files and that means the "crc=X" line will always
need to be filtered out
> > diff --git a/common.repair b/common.repair
> > index 23a1ed6..db160bb 100644
> > --- a/common.repair
> > +++ b/common.repair
> > @@ -77,6 +77,9 @@ s/\s+- \d+:\d\d:\d\d:.*\n//g;
> > /^agi_/ && next; # remove agi counts
> > /^sb_/ && next; # remove sb counts
> > /^agi unlinked/ && next; # remove agi unlinked bucket warning
> > +# crc enabled filesystem output
> > +/XFS_CORRUPTION_ERROR/ && next;
> > +/^bad uuid/ && next;
>
> Doesn't this filter out xfs repair failures which should be reported?
No, because repair will issue errors when bad metadata is read off
disk now, and as such they are brand new output introduced by the
CRC patches. Over time this might get cleaned up, but at least for
initial testing and debugging this noise will remain in the repair
code....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 02/13] xfstests: add --largefs check option
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
2013-03-15 11:53 ` [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
2013-03-15 11:53 ` [PATCH 03/13] xfstests: rename USE_BIG_LOOPFS to be more generic Dave Chinner
` (11 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
To: xfs
From: Dave Chinner <dchinner@redhat.com>
Make it easier to check large filesystems quickly by adding a
--large-fs option to check to turn on shortcuts for large scratch
device filesystem testing.
Also, reject invalid command line options with a usage message.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
common | 63 +++++++++++++++++++++++++++++++++++++++------------------------
1 file changed, 39 insertions(+), 24 deletions(-)
diff --git a/common b/common
index 0723224..7d13078 100644
--- a/common
+++ b/common
@@ -27,6 +27,35 @@ _setenvironment()
export MSGVERB
}
+usage()
+{
+ echo "Usage: $0 [options] [testlist]"'
+
+common options
+ -v verbose
+
+check options
+ -xfs test XFS (default)
+ -udf test UDF
+ -nfs test NFS
+ -l line mode diff
+ -xdiff graphical mode diff
+ -udiff show unified diff (default)
+ -n show me, do not run tests
+ -q quick [deprecated]
+ -T output timestamps
+ -r randomize test order
+ --large-fs optimise scratch device for large filesystems
+
+testlist options
+ -g group[,group...] include tests from these groups
+ -x group[,group...] exclude tests from these groups
+ NNN include test NNN
+ NNN-NNN include test range (eg. 012-021)
+'
+ exit 0
+}
+
here=`pwd`
rm -f $here/$iam.out
_setenvironment
@@ -117,30 +146,7 @@ s/ .*//p
in
-\? | -h | --help) # usage
- echo "Usage: $0 [options] [testlist]"'
-
-common options
- -v verbose
-
-check options
- -xfs test XFS (default)
- -udf test UDF
- -nfs test NFS
- -l line mode diff
- -xdiff graphical mode diff
- -udiff show unified diff (default)
- -n show me, do not run tests
- -q quick [deprecated]
- -T output timestamps
- -r randomize test order
-
-testlist options
- -g group[,group...] include tests from these groups
- -x group[,group...] exclude tests from these groups
- NNN include test NNN
- NNN-NNN include test range (eg. 012-021)
-'
- exit 0
+ usage
;;
-udf) # -udf ... set FSTYP to udf
@@ -232,6 +238,15 @@ testlist options
fi
;;
+ --large-fs)
+ export USE_BIG_LOOPFS=yes
+ xpand=false
+ ;;
+
+ -*)
+ usage
+ ;;
+
*)
start=$r
end=$r
--
1.7.10.4
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH 03/13] xfstests: rename USE_BIG_LOOPFS to be more generic
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
2013-03-15 11:53 ` [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs Dave Chinner
2013-03-15 11:53 ` [PATCH 02/13] xfstests: add --largefs check option Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
2013-03-19 19:46 ` Rich Johnston
2013-03-15 11:53 ` [PATCH 04/13] xfstests: rename RETAIN_AG_BYTES Dave Chinner
` (10 subsequent siblings)
13 siblings, 1 reply; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
To: xfs
From: Dave Chinner <dchinner@redhat.com>
USE_BIG_LOOPFS is really misnamed - it can be used on real devices just as
easily as loop devices. It really means we are testing a large scratch device
and that we should enable the special filesystem filling and checking options
that enable xfstests to be run sanely on large XFS filesystems.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
004 | 2 +-
015 | 2 +-
030 | 2 +-
031 | 2 +-
032 | 2 +-
033 | 4 ++--
041 | 2 +-
049 | 2 +-
083 | 2 +-
092 | 2 +-
148 | 2 +-
149 | 2 +-
common | 2 +-
common.rc | 12 ++++++------
setup | 5 +++--
15 files changed, 23 insertions(+), 22 deletions(-)
diff --git a/004 b/004
index 9f28e17..23729da 100755
--- a/004
+++ b/004
@@ -64,7 +64,7 @@ _supported_os IRIX Linux
_need_to_be_root
_require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
rm -f $seq.full
diff --git a/015 b/015
index 4206b93..a99f1ed 100755
--- a/015
+++ b/015
@@ -52,7 +52,7 @@ _supported_fs generic
_supported_os IRIX Linux
_require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
_scratch_mkfs_sized `expr 50 \* 1024 \* 1024` >/dev/null 2>&1 \
|| _fail "mkfs failed"
diff --git a/030 b/030
index fe57372..a8fb92c 100755
--- a/030
+++ b/030
@@ -64,8 +64,8 @@ _check_ag()
_supported_fs xfs
_supported_os IRIX Linux
-_require_nobigloopfs
_require_scratch
+_require_no_large_scratch_dev
DSIZE="-dsize=100m,agcount=6"
diff --git a/031 b/031
index b062277..fb6f15b 100755
--- a/031
+++ b/031
@@ -97,8 +97,8 @@ EOF
_supported_fs xfs
_supported_os IRIX Linux
-_require_nobigloopfs
_require_scratch
+_require_no_large_scratch_dev
# sanity test - default + one root directory entry
# Note: must do this proto/mkfs now for later inode size calcs
diff --git a/032 b/032
index bf88492..09f08c3 100755
--- a/032
+++ b/032
@@ -41,8 +41,8 @@ rm -f $seq.full
_supported_fs xfs
_supported_os Linux
-_require_nobigloopfs
_require_scratch
+_require_no_large_scratch_dev
echo "Silence is golden."
for fs in `echo ${MKFS_PROG}.* | sed -e 's/.sbin.mkfs.//g'`
diff --git a/033 b/033
index 9651f26..68a688e 100755
--- a/033
+++ b/033
@@ -76,9 +76,9 @@ _filter_bad_ids()
# real QA test starts here
_supported_fs xfs
_supported_os IRIX Linux
-
-_require_nobigloopfs
+
_require_scratch
+_require_no_large_scratch_dev
# devzero blows away 512byte blocks, so make 512byte inodes (at least)
_scratch_mkfs_xfs | _filter_mkfs 2>$tmp.mkfs
diff --git a/041 b/041
index 2800811..28dcb33 100755
--- a/041
+++ b/041
@@ -50,7 +50,7 @@ _supported_fs xfs
_supported_os IRIX Linux
_require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
umount $SCRATCH_DEV 2>/dev/null
_fill()
diff --git a/049 b/049
index b68821a..8593814 100755
--- a/049
+++ b/049
@@ -60,9 +60,9 @@ _log()
echo "--- $*" >> $seq.full
}
-_require_nobigloopfs
_require_nonexternal
_require_scratch
+_require_no_large_scratch_dev
_require_loop
_require_ext2
diff --git a/083 b/083
index f5349a9..a977ffd 100755
--- a/083
+++ b/083
@@ -58,7 +58,7 @@ _supported_fs generic
_supported_os IRIX Linux
_require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
rm -f $seq.full
diff --git a/092 b/092
index 429fa80..02ccc71 100755
--- a/092
+++ b/092
@@ -48,7 +48,7 @@ _cleanup()
_supported_fs xfs
_supported_os IRIX Linux
_require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
MOUNT_OPTIONS="$MOUNT_OPTIONS -o inode64"
_scratch_mkfs_xfs | _filter_mkfs 2>/dev/null
diff --git a/148 b/148
index 76cbf37..7bb1722 100755
--- a/148
+++ b/148
@@ -66,8 +66,8 @@ _check_ag()
_supported_fs xfs
_supported_os IRIX Linux
-_require_nobigloopfs
_require_scratch
+_require_no_large_scratch_dev
DSIZE="-dsize=100m"
diff --git a/149 b/149
index 5131a45..193e6d7 100755
--- a/149
+++ b/149
@@ -100,8 +100,8 @@ EOF
_supported_fs xfs
_supported_os IRIX Linux
-_require_nobigloopfs
_require_scratch
+_require_no_large_scratch_dev
# sanity test - default + one root directory entry
# Note: must do this proto/mkfs now for later inode size calcs
diff --git a/common b/common
index 7d13078..da86cd9 100644
--- a/common
+++ b/common
@@ -239,7 +239,7 @@ s/ .*//p
;;
--large-fs)
- export USE_BIG_LOOPFS=yes
+ export LARGE_SCRATCH_DEV=yes
xpand=false
;;
diff --git a/common.rc b/common.rc
index ef41cbf..edf0318 100644
--- a/common.rc
+++ b/common.rc
@@ -310,7 +310,7 @@ _scratch_mkfs_xfs()
cat $tmp_dir.mkfsstd
rm -f $tmp_dir.mkfserr $tmp_dir.mkfsstd
- if [ "$USE_BIG_LOOPFS" = yes ]; then
+ if [ "$LARGE_SCRATCH_DEV" = yes ]; then
[ -z "$RETAIN_AG_BYTES" ] && RETAIN_AG_BYTES=0
./tools/ag-wipe -q -r $RETAIN_AG_BYTES $SCRATCH_DEV
fi
@@ -437,7 +437,7 @@ _scratch_xfs_repair()
SCRATCH_OPTIONS="-l$SCRATCH_LOGDEV"
[ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_RTDEV" ] && \
SCRATCH_OPTIONS=$SCRATCH_OPTIONS" -r$SCRATCH_RTDEV"
- [ "$USE_BIG_LOOPFS" = yes ] && SCRATCH_OPTIONS=$SCRATCH_OPTIONS" -t"
+ [ "$LARGE_SCRATCH_DEV" = yes ] && SCRATCH_OPTIONS=$SCRATCH_OPTIONS" -t"
$XFS_REPAIR_PROG $SCRATCH_OPTIONS $* $SCRATCH_DEV
}
@@ -826,9 +826,9 @@ _require_ext2()
# this test requires that (large) loopback device files are not in use
#
-_require_nobigloopfs()
+_require_no_large_scratch_dev()
{
- [ "$USE_BIG_LOOPFS" = yes ] && \
+ [ "$LARGE_SCRATCH_DEV" = yes ] && \
_notrun "Large filesystem testing in progress, skipped this test"
}
@@ -1237,7 +1237,7 @@ _check_xfs_filesystem()
[ "$FSTYP" != xfs ] && return 0
testoption=""
- [ "$USE_BIG_LOOPFS" = yes ] && testoption=-t
+ [ "$LARGE_SCRATCH_DEV" = yes ] && testoption=-t
type=`_fs_type $device`
ok=1
@@ -1276,7 +1276,7 @@ _check_xfs_filesystem()
ok=0
fi
# repair doesn't scale massively at this stage, optionally skip it for now
- [ "$USE_BIG_LOOPFS" = yes ] || \
+ [ "$LARGE_SCRATCH_DEV" = yes ] || \
$XFS_REPAIR_PROG -n $extra_log_options $extra_rt_options $device >$tmp.repair 2>&1
if [ $? -ne 0 ]
then
diff --git a/setup b/setup
index 62798cc..5225951 100755
--- a/setup
+++ b/setup
@@ -23,7 +23,7 @@ fi
[ "$USE_EXTERNAL" = yes ] || USE_EXTERNAL=no
[ "$USE_LBD_PATCH" = yes ] || USE_LBD_PATCH=no
-[ "$USE_BIG_LOOPFS" = yes ] || USE_BIG_LOOPFS=no
+[ "$LARGE_SCRATCH_DEV" = yes ] || LARGE_SCRATCH_DEV=no
[ "$USE_ATTR_SECURE" = yes ] || USE_ATTR_SECURE=no
[ -z "$FSTYP" ] && FSTYP="xfs"
@@ -31,5 +31,6 @@ cat <<EOF
TEST: DIR=$TEST_DIR DEV=$TEST_DEV rt=[$TEST_RTDEV] log=[$TEST_LOGDEV]
TAPE: dev=[$TAPE_DEV] rmt=[$RMT_TAPE_DEV] rmtirix=[$RMT_TAPE_USER@$RMT_IRIXTAPE_DEV]
SCRATCH: MNT=$SCRATCH_MNT DEV=$SCRATCH_DEV rt=[$SCRATCH_RTDEV] log=[$SCRATCH_LOGDEV]
-VARIABLES: external=$USE_EXTERNAL largeblk=$USE_LBD_PATCH fstyp=$FSTYP bigloopfs=$USE_BIG_LOOPFS attrsecure=$USE_ATTR_SECURE
+VARIABLES: external=$USE_EXTERNAL largeblk=$USE_LBD_PATCH fstyp=$FSTYP
+ large_scratch_dev=$LARGE_SCRATCH_DEV attrsecure=$USE_ATTR_SECURE
EOF
--
1.7.10.4
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH 04/13] xfstests: rename RETAIN_AG_BYTES
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
` (2 preceding siblings ...)
2013-03-15 11:53 ` [PATCH 03/13] xfstests: rename USE_BIG_LOOPFS to be more generic Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
2013-03-15 11:53 ` [PATCH 05/13] xfstests: use preallocation for ag-wiper Dave Chinner
` (9 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
To: xfs
From: Dave Chinner <dchinner@redhat.com>
Rename the $RETAIN_AG_BYTES variable to be more generic so that it
reflects the fact that it is designed to retain a certain amount of
extra free space above the default amount in the filesystem when
doing large scratch device testing.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
common.rc | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/common.rc b/common.rc
index edf0318..42d2ddc 100644
--- a/common.rc
+++ b/common.rc
@@ -311,8 +311,8 @@ _scratch_mkfs_xfs()
rm -f $tmp_dir.mkfserr $tmp_dir.mkfsstd
if [ "$LARGE_SCRATCH_DEV" = yes ]; then
- [ -z "$RETAIN_AG_BYTES" ] && RETAIN_AG_BYTES=0
- ./tools/ag-wipe -q -r $RETAIN_AG_BYTES $SCRATCH_DEV
+ [ -z "$SCRATCH_DEV_EMPTY_SPACE" ] && SCRATCH_DEV_EMPTY_SPACE=0
+ ./tools/ag-wipe -q -r $SCRATCH_DEV_EMPTY_SPACE $SCRATCH_DEV
fi
return $mkfs_status
--
1.7.10.4
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH 05/13] xfstests: use preallocation for ag-wiper
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
` (3 preceding siblings ...)
2013-03-15 11:53 ` [PATCH 04/13] xfstests: rename RETAIN_AG_BYTES Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
2013-03-19 19:47 ` Rich Johnston
2013-03-15 11:53 ` [PATCH 06/13] xfstests: use command line option for setting extra space Dave Chinner
` (8 subsequent siblings)
13 siblings, 1 reply; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
To: xfs
From: Dave Chinner <dchinner@redhat.com>
To enable sane testing of large scale filesystems, the --large-fs
test option uses xfs_db magic to mark AGs full without doing any IO.
This leaves only a small amount of free space left in the filesystem
to stress the high AGs of the filesystem rather than the low AGs.
This method requires us to have special filesystem check options to
avoid free space checking in xfs_check, and we cannot current run
xfs_repair on such a filesystem at all. As it is, free space
checking on xfs_check does not scale, so we still need to avoid this
checking regardless of how we fill the filesystem.
We can acheive exactly the same fill behaviour by preallocating a
single large file in the filesystem immediately after creating it.
This is a filesystem independent manner of filling the filesystem,
and allows us to do large filesystem testing on more than just XFS.
Further, this preallocation method effectively adds a new "very
large file" test. It also enables us to run an unmodified xfs_repair
or filesystem specific fsck program to check the filesystem for
sanity, so we can now do full sanity checking of such large
filesystems.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
common.rc | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 53 insertions(+), 5 deletions(-)
diff --git a/common.rc b/common.rc
index 42d2ddc..d221be1 100644
--- a/common.rc
+++ b/common.rc
@@ -276,6 +276,47 @@ _scratch_mkfs_options()
echo $SCRATCH_OPTIONS $MKFS_OPTIONS $* $SCRATCH_DEV
}
+
+_setup_large_xfs_fs()
+{
+ fs_size=$1
+ local tmp_dir=/tmp/
+
+ [ "$LARGE_SCRATCH_DEV" != yes ] && return 0
+ [ -z "$SCRATCH_DEV_EMPTY_SPACE" ] && SCRATCH_DEV_EMPTY_SPACE=0
+ [ $SCRATCH_DEV_EMPTY_SPACE -ge $fs_size ] && return 0
+
+ # calculate the size of the file we need to allocate.
+ # Default free space in the FS is 50GB, but you can specify more via
+ # SCRATCH_DEV_EMPTY_SPACE
+ file_size=$(($fs_size - 50*1024*1024*1024))
+ file_size=$(($file_size - $SCRATCH_DEV_EMPTY_SPACE))
+
+ # mount the filesystem, create the file, unmount it
+ _scratch_mount 2>&1 >$tmp_dir/mnt.err
+ local status=$?
+ if [ $status -ne 0 ]; then
+ echo "mount failed"
+ cat $tmp_dir/mnt.err >&2
+ rm -f $tmp_dir/mnt.err
+ return $status
+ fi
+ rm -f $tmp_dir/mnt.err
+
+ xfs_io -F -f \
+ -c "truncate $file_size" \
+ -c "falloc -k 0 $file_size" \
+ $SCRATCH_MNT/.use_space 2>&1 > /dev/null
+ status=$?
+ umount $SCRATCH_MNT
+ if [ $status -ne 0 ]; then
+ echo "large file prealloc failed"
+ cat $tmp_dir/mnt.err >&2
+ return $status
+ fi
+ return 0
+}
+
_scratch_mkfs_xfs()
{
# extra mkfs options can be added by tests
@@ -305,16 +346,23 @@ _scratch_mkfs_xfs()
mkfs_status=$?
fi
+ if [ $mkfs_status -eq 0 -a "$LARGE_SCRATCH_DEV" = yes ]; then
+ # manually parse the mkfs output to get the fs size in bytes
+ local fs_size
+ fs_size=`cat $tmp_dir.mkfsstd | perl -ne '
+ if (/^data\s+=\s+bsize=(\d+)\s+blocks=(\d+)/) {
+ my $size = $1 * $2;
+ print STDOUT "$size\n";
+ }'`
+ _setup_large_xfs_fs $fs_size
+ mkfs_status=$?
+ fi
+
# output stored mkfs output
cat $tmp_dir.mkfserr >&2
cat $tmp_dir.mkfsstd
rm -f $tmp_dir.mkfserr $tmp_dir.mkfsstd
- if [ "$LARGE_SCRATCH_DEV" = yes ]; then
- [ -z "$SCRATCH_DEV_EMPTY_SPACE" ] && SCRATCH_DEV_EMPTY_SPACE=0
- ./tools/ag-wipe -q -r $SCRATCH_DEV_EMPTY_SPACE $SCRATCH_DEV
- fi
-
return $mkfs_status
}
--
1.7.10.4
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH 06/13] xfstests: use command line option for setting extra space
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
` (4 preceding siblings ...)
2013-03-15 11:53 ` [PATCH 05/13] xfstests: use preallocation for ag-wiper Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
2013-03-15 11:53 ` [PATCH 07/13] xfstest: enable xfs_repair for large filesystem testing Dave Chinner
` (7 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
To: xfs
From: Dave Chinner <dchinner@redhat.com>
Allow the extra free space to leave in large scratch filesystems to
be specified by a command line option rather than just via an
environment variable.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
common | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/common b/common
index da86cd9..78ac654 100644
--- a/common
+++ b/common
@@ -247,6 +247,11 @@ s/ .*//p
usage
;;
+ --extra-space=*)
+ export SCRATCH_DEV_EMPTY_SPACE=${r#*=}
+ xpand=false
+ ;;
+
*)
start=$r
end=$r
--
1.7.10.4
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH 07/13] xfstest: enable xfs_repair for large filesystem testing
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
` (5 preceding siblings ...)
2013-03-15 11:53 ` [PATCH 06/13] xfstests: use command line option for setting extra space Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
2013-03-15 11:53 ` [PATCH 08/13] xfstests: always use test option when checking large scratch device Dave Chinner
` (6 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
To: xfs
From: Dave Chinner <dchinner@redhat.com>
Now that large filesystem testing does not play free space games to
fill the space without IO, we can enable xfs_repair when running in
this mode. xfs_repair has had it's scalability problems solved, too,
so this is a safe thing to do.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
common.rc | 18 ++++++++++--------
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git a/common.rc b/common.rc
index d221be1..ee4f522 100644
--- a/common.rc
+++ b/common.rc
@@ -1273,19 +1273,17 @@ _check_xfs_filesystem()
extra_mount_options=""
device=$1
if [ "$2" != "none" ]; then
- extra_log_options="-l$2"
+ extra_log_options="-l$2"
extra_mount_options="-ologdev=$2"
fi
if [ "$3" != "none" ]; then
- extra_rt_options="-r$3"
+ extra_rt_options="-r$3"
extra_mount_options=$extra_mount_options" -ortdev=$3"
fi
extra_mount_options=$extra_mount_options" $MOUNT_OPTIONS"
[ "$FSTYP" != xfs ] && return 0
- testoption=""
- [ "$LARGE_SCRATCH_DEV" = yes ] && testoption=-t
type=`_fs_type $device`
ok=1
@@ -1310,8 +1308,13 @@ _check_xfs_filesystem()
ok=0
fi
- $XFS_CHECK_PROG $testoption $extra_log_options $device 2>&1 |\
- _fix_malloc >$tmp.fs_check
+ # xfs_check runs out of memory on large files, so even providing the test
+ # option (-t) to avoid indexing the free space trees doesn't make it pass on
+ # large filesystems. Avoid it.
+ if [ "$LARGE_SCRATCH_DEV" != yes ]; then
+ $XFS_CHECK_PROG $extra_log_options $device 2>&1 |\
+ _fix_malloc >$tmp.fs_check
+ fi
if [ -s $tmp.fs_check ]
then
echo "_check_xfs_filesystem: filesystem on $device is inconsistent (c) (see $seq.full)"
@@ -1323,8 +1326,7 @@ _check_xfs_filesystem()
ok=0
fi
- # repair doesn't scale massively at this stage, optionally skip it for now
- [ "$LARGE_SCRATCH_DEV" = yes ] || \
+
$XFS_REPAIR_PROG -n $extra_log_options $extra_rt_options $device >$tmp.repair 2>&1
if [ $? -ne 0 ]
then
--
1.7.10.4
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH 08/13] xfstests: always use test option when checking large scratch device
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
` (6 preceding siblings ...)
2013-03-15 11:53 ` [PATCH 07/13] xfstest: enable xfs_repair for large filesystem testing Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
2013-03-15 11:53 ` [PATCH 09/13] xfstests: enable large fs testing on ext4 Dave Chinner
` (5 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
To: xfs
From: Dave Chinner <dchinner@redhat.com>
Some tests call _check_scratch_device directly and when using a
large filesystem this needs to run with a -t option to avoid
consuming large amounts of memory. Make this happen in all cases
that the scratch device is checked.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
017 | 7 ++-----
common.rc | 2 ++
2 files changed, 4 insertions(+), 5 deletions(-)
diff --git a/017 b/017
index 7db4667..fb92375 100755
--- a/017
+++ b/017
@@ -48,9 +48,6 @@ _supported_os Linux
_require_scratch
-checkopts=""
-[ "$USE_BIG_LOOPFS" = yes ] && checkopts=-t
-
echo "*** init FS"
rm -f $seq.full
@@ -82,8 +79,8 @@ do
echo "" >>$seq.full
echo "*** XFS_CHECK ***" >>$seq.full
echo "" >>$seq.full
- _scratch_xfs_check $checkopts >>$seq.full 2>&1 \
- || _fail "xfs_check $checkopts failed"
+ _scratch_xfs_check >>$seq.full 2>&1 \
+ || _fail "xfs_check failed"
_scratch_mount -o remount,rw \
|| _fail "remount rw failed"
done
diff --git a/common.rc b/common.rc
index ee4f522..1026572 100644
--- a/common.rc
+++ b/common.rc
@@ -475,6 +475,8 @@ _scratch_xfs_check()
SCRATCH_OPTIONS=""
[ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ] && \
SCRATCH_OPTIONS="-l $SCRATCH_LOGDEV"
+ [ "$LARGE_SCRATCH_DEV" = yes ] && \
+ SCRATCH_OPTIONS=$SCRATCH_OPTIONS" -t"
$XFS_CHECK_PROG $SCRATCH_OPTIONS $* $SCRATCH_DEV
}
--
1.7.10.4
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH 09/13] xfstests: enable large fs testing on ext4
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
` (7 preceding siblings ...)
2013-03-15 11:53 ` [PATCH 08/13] xfstests: always use test option when checking large scratch device Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
2013-03-15 11:53 ` [PATCH 10/13] xfstests: disable tests that typically fail on large filesystems Dave Chinner
` (4 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
To: xfs
From: Dave Chinner <dchinner@redhat.com>
Now that setting up large filesystem testing on sparse loopback
devices uses a generic method for filling the filesystem, extent
support to ext4 filesystems.
ext4 is slightly more complex to fill as it does not support files
larger than 16TB. Hence a slightly more complex method of using
multiple smaller files to fill the space is necessary.
WARNING: be prepeared for ext4 to take forever to prepare large
filesystems as allocation of large amounts of space (especially as
it approaches ENOSPC) is can take minutes rather than milliseconds
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
common.rc | 90 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 90 insertions(+)
diff --git a/common.rc b/common.rc
index 1026572..690c6a3 100644
--- a/common.rc
+++ b/common.rc
@@ -366,6 +366,93 @@ _scratch_mkfs_xfs()
return $mkfs_status
}
+_setup_large_ext4_fs()
+{
+ fs_size=$1
+ local tmp_dir=/tmp/
+
+ [ "$LARGE_SCRATCH_DEV" != yes ] && return 0
+ [ -z "$SCRATCH_DEV_EMPTY_SPACE" ] && SCRATCH_DEV_EMPTY_SPACE=0
+ [ $SCRATCH_DEV_EMPTY_SPACE -ge $fs_size ] && return 0
+
+ # Default free space in the FS is 50GB, but you can specify more via
+ # SCRATCH_DEV_EMPTY_SPACE
+ space_to_consume=$(($fs_size - 50*1024*1024*1024 - $SCRATCH_DEV_EMPTY_SPACE))
+
+ # mount the filesystem and create 16TB - 4KB files until we consume
+ # all the necessary space.
+ _scratch_mount 2>&1 >$tmp_dir/mnt.err
+ local status=$?
+ if [ $status -ne 0 ]; then
+ echo "mount failed"
+ cat $tmp_dir/mnt.err >&2
+ rm -f $tmp_dir/mnt.err
+ return $status
+ fi
+ rm -f $tmp_dir/mnt.err
+
+ file_size=$((16*1024*1024*1024*1024 - 4096))
+ nfiles=0
+ while [ $space_to_consume -gt $file_size ]; do
+
+ xfs_io -F -f \
+ -c "truncate $file_size" \
+ -c "falloc -k 0 $file_size" \
+ $SCRATCH_MNT/.use_space.$nfiles 2>&1
+ status=$?
+ if [ $status -ne 0 ]; then
+ break;
+ fi
+
+ space_to_consume=$(( $space_to_consume - $file_size ))
+ nfiles=$(($nfiles + 1))
+ done
+
+ # consume the remaining space.
+ if [ $space_to_consume -gt 0 ]; then
+ xfs_io -F -f \
+ -c "truncate $space_to_consume" \
+ -c "falloc -k 0 $space_to_consume" \
+ $SCRATCH_MNT/.use_space.$nfiles 2>&1
+ status=$?
+ fi
+
+ umount $SCRATCH_MNT
+ if [ $status -ne 0 ]; then
+ echo "large file prealloc failed"
+ cat $tmp_dir/mnt.err >&2
+ return $status
+ fi
+ return 0
+}
+_scratch_mkfs_ext4()
+{
+ local tmp_dir=/tmp/
+
+ /sbin/mkfs -t $FSTYP -- $MKFS_OPTIONS $* $SCRATCH_DEV \
+ 2>$tmp_dir.mkfserr 1>$tmp_dir.mkfsstd
+ local mkfs_status=$?
+
+ if [ $mkfs_status -eq 0 -a "$LARGE_SCRATCH_DEV" = yes ]; then
+ # manually parse the mkfs output to get the fs size in bytes
+ fs_size=`cat $tmp_dir.mkfsstd | awk ' \
+ /^Block size/ { split($2, a, "="); bs = a[2] ; } \
+ / inodes, / { blks = $3 } \
+ /reserved for the super user/ { resv = $1 } \
+ END { fssize = bs * blks - resv; print fssize }'`
+
+ _setup_large_ext4_fs $fs_size
+ mkfs_status=$?
+ fi
+
+ # output stored mkfs output
+ cat $tmp_dir.mkfserr >&2
+ cat $tmp_dir.mkfsstd
+ rm -f $tmp_dir.mkfserr $tmp_dir.mkfsstd
+
+ return $mkfs_status
+}
+
_scratch_mkfs()
{
case $FSTYP in
@@ -381,6 +468,9 @@ _scratch_mkfs()
btrfs)
$MKFS_BTRFS_PROG $MKFS_OPTIONS $* $SCRATCH_DEV > /dev/null
;;
+ ext4)
+ _scratch_mkfs_ext4 $*
+ ;;
*)
yes | $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS $* $SCRATCH_DEV
;;
--
1.7.10.4
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH 10/13] xfstests: disable tests that typically fail on large filesystems
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
` (8 preceding siblings ...)
2013-03-15 11:53 ` [PATCH 09/13] xfstests: enable large fs testing on ext4 Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
2013-03-15 11:53 ` [PATCH 11/13] xfstests: exclude largefs fill files from dump tests Dave Chinner
` (3 subsequent siblings)
13 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
To: xfs
From: Dave Chinner <dchinner@redhat.com>
Test 017 typically fails due to the OOM killer kicking in and
killing everything but xfs_db so panics the machine.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
---
017 | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/017 b/017
index fb92375..65ddce1 100755
--- a/017
+++ b/017
@@ -48,6 +48,10 @@ _supported_os Linux
_require_scratch
+# xfs_db will OOM kill the machine if you don't have huge amounts of RAM, so
+# don't run this on large filesystems.
+_require_no_large_scratch_dev
+
echo "*** init FS"
rm -f $seq.full
--
1.7.10.4
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH 11/13] xfstests: exclude largefs fill files from dump tests
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
` (9 preceding siblings ...)
2013-03-15 11:53 ` [PATCH 10/13] xfstests: disable tests that typically fail on large filesystems Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
2013-03-19 19:49 ` Rich Johnston
2013-03-15 11:53 ` [PATCH 12/13] xfstests: filter files in 019 for --large-fs Dave Chinner
` (2 subsequent siblings)
13 siblings, 1 reply; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
To: xfs
From: Dave Chinner <dchinner@redhat.com>
The xfsdump/restore tests will see the space filling files and may
try to backthem up and restore them, consuming huge amounts of time
to do (espcially when diffing the results). Exclude the space
filling files by setting the no dump attributes on them and ensure
that xfsdump runs with the -e flag to exclude such files.
This also needs a dump filter addition to remove the output that
files were skipped, and to decrement the count of files processed by
xfsrestore because the inventory still includes excluded files.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
common.config | 3 ++-
common.dump | 34 ++++++++++++++++++++++++++--------
common.filter | 4 +++-
common.rc | 1 +
4 files changed, 32 insertions(+), 10 deletions(-)
diff --git a/common.config b/common.config
index 5ac58bf..ab19772 100644
--- a/common.config
+++ b/common.config
@@ -160,7 +160,8 @@ export XFS_GROWFS_PROG=`set_prog_path xfs_growfs`
export XFS_IO_PROG="`set_prog_path xfs_io`"
export XFS_PARALLEL_REPAIR_PROG="`set_prog_path xfs_prepair`"
export XFS_PARALLEL_REPAIR64_PROG="`set_prog_path xfs_prepair64`"
-export XFSDUMP_PROG="`set_prog_path xfsdump`"
+export __XFSDUMP_PROG="`set_prog_path xfsdump`"
+export XFSDUMP_PROG="$__XFSDUMP_PROG -e"
export XFSRESTORE_PROG="`set_prog_path xfsrestore`"
export XFSINVUTIL_PROG="`set_prog_path xfsinvutil`"
export DBENCH_PROG="`set_prog_path dbench`"
diff --git a/common.dump b/common.dump
index e89dedc..be55249 100644
--- a/common.dump
+++ b/common.dump
@@ -29,7 +29,8 @@ if [ -n "$DEBUGDUMP" ]; then
# Use dump/restore in qa directory (copy them here) for debugging
export PATH="$here:$PATH"
- export XFSDUMP_PROG="`set_prog_path xfsdump`"
+ export __XFSDUMP_PROG="`set_prog_path xfsdump`"
+ export XFSDUMP_PROG="$__XFSDUMP_PROG -e"
export XFSRESTORE_PROG="`set_prog_path xfsrestore`"
export XFSINVUTIL_PROG="`set_prog_path xfsinvutil`"
[ -x $here/xfsdump ] && echo "Using xfstests' xfsdump for debug"
@@ -829,7 +830,7 @@ _dump_filter_main()
{
_filter_devchar |\
sed \
- -e "s#$XFSDUMP_PROG#xfsdump#" \
+ -e "s#$__XFSDUMP_PROG#xfsdump#" \
-e "s#$XFSRESTORE_PROG#xfsrestore#" \
-e "s#$XFSINVUTIL_PROG#xfsinvutil#" \
-e "s/`hostname`/HOSTNAME/" \
@@ -872,13 +873,29 @@ _dump_filter_main()
-e '/xfsdump: ino map phase [5]/,1d' \
-e 's/id:[[:space:]]*[0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}/ID: ID/' \
-e 's/\[y\/n\][- ]----------------------*/\[y\/n\]/' \
+ -e '/skip attribute set/d' \
| perl -ne '
- if ($_ =~ /(?:Dump|Restore) Summary/) {
- $skip = 1;
- } elsif ($_ =~ /(?:Dump|Restore) Status/) {
- $skip = 0;
- }
- print if (! $skip);'
+ # filter out all the output between the lines "Dump Summary:"
+ # and "Dump Status:"
+ if ($_ =~ /(?:Dump|Restore) Summary/) {
+ $skip = 1;
+ } elsif ($_ =~ /(?:Dump|Restore) Status/) {
+ $skip = 0;
+ }
+ print if (! $skip);' \
+ | perl -ne '
+ # correct the file count if large scratch devices are being used
+ $skip = 0;
+ if ($_ =~ /(\S+) directories and (\S+) entries/) {
+ $foo = $2;
+ if ($ENV{'LARGE_SCRATCH_DEV'} && $foo > 0) {
+ $foo -= 1;
+ }
+ printf("xfsrestore: %u directories and %u entries processed\n",
+ $1, $foo);
+ $skip = 1;
+ }
+ print if (! $skip);'
}
_dump_filter()
@@ -916,6 +933,7 @@ _dir_filter()
-e "s#$dump_sdir#DUMP_SUBDIR#g" \
-e "s#$restore_sdir#RESTORE_SUBDIR#g" \
-e "s#$$#PID#g" \
+ -e "/Only in SCRATCH_MNT: .use_space/d" \
}
diff --git a/common.filter b/common.filter
index c34809f..da5675f 100644
--- a/common.filter
+++ b/common.filter
@@ -223,7 +223,9 @@ _filter_test_dir()
_filter_scratch()
{
- sed -e "s,$SCRATCH_DEV,SCRATCH_DEV,g" -e "s,$SCRATCH_MNT,SCRATCH_MNT,g"
+ sed -e "s,$SCRATCH_DEV,SCRATCH_DEV,g" \
+ -e "s,$SCRATCH_MNT,SCRATCH_MNT,g" \
+ -e "/.use_space/d"
}
# Turn any device in the scratch pool into SCRATCH_DEV
diff --git a/common.rc b/common.rc
index 690c6a3..96d5077 100644
--- a/common.rc
+++ b/common.rc
@@ -306,6 +306,7 @@ _setup_large_xfs_fs()
xfs_io -F -f \
-c "truncate $file_size" \
-c "falloc -k 0 $file_size" \
+ -c "chattr +d" \
$SCRATCH_MNT/.use_space 2>&1 > /dev/null
status=$?
umount $SCRATCH_MNT
--
1.7.10.4
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH 12/13] xfstests: filter files in 019 for --large-fs
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
` (10 preceding siblings ...)
2013-03-15 11:53 ` [PATCH 11/13] xfstests: exclude largefs fill files from dump tests Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
2013-03-19 19:52 ` Rich Johnston
2013-03-15 11:53 ` [PATCH 13/13] xfstests: correct quota file count " Dave Chinner
2013-03-19 20:20 ` [PATCH 00/13] xfstests: generic --large-fs support Rich Johnston
13 siblings, 1 reply; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
To: xfs
From: Dave Chinner <dchinner@redhat.com>
Make sure that the .use_space files don't appear in the files dumped
to the output files.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
019 | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/019 b/019
index 42e3c22..88be464 100755
--- a/019
+++ b/019
@@ -128,15 +128,16 @@ _verify_fs()
echo "*** verify FS"
(cd $SCRATCH_MNT ; find . | LC_COLLATE=POSIX sort \
+ | grep -v ".use_space" \
| xargs $here/src/lstat64 | _filter_stat)
diff -q $SCRATCH_MNT/bigfile $tempfile.2 \
|| _fail "bigfile corrupted"
-
+
echo "*** unmount FS"
_full "umount"
umount $SCRATCH_DEV >>$seqfull 2>&1 \
|| _fail "umount failed"
-
+
echo "*** check FS"
_check_scratch_fs
}
--
1.7.10.4
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH 13/13] xfstests: correct quota file count for --large-fs
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
` (11 preceding siblings ...)
2013-03-15 11:53 ` [PATCH 12/13] xfstests: filter files in 019 for --large-fs Dave Chinner
@ 2013-03-15 11:53 ` Dave Chinner
2013-03-19 19:52 ` Rich Johnston
2013-03-19 20:20 ` [PATCH 00/13] xfstests: generic --large-fs support Rich Johnston
13 siblings, 1 reply; 23+ messages in thread
From: Dave Chinner @ 2013-03-15 11:53 UTC (permalink / raw)
To: xfs
From: Dave Chinner <dchinner@redhat.com>
The files used to fill space when using --large-fs contribute
towards the quota of uid 0. Modify the quota output filter in test
050 to "hide" these files from the test output.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
050 | 8 +++++++-
common.rc | 2 ++
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/050 b/050
index cf70b5a..acca654 100755
--- a/050
+++ b/050
@@ -69,7 +69,13 @@ _filter_report()
perl -npe '
s/^\#'$id' /[NAME] /g;
s/^\#0 \d+ /[ROOT] 0 /g;
- s/6 days/7 days/g'
+ s/6 days/7 days/g' |
+ perl -npe '
+ $val = 0;
+ if ($ENV{'LARGE_SCRATCH_DEV'}) {
+ $val = $ENV{'NUM_SPACE_FILES'};
+ }
+ s/(^\[ROOT\] \S+ \S+ \S+ \S+ \[--------\] )(\S+)/$1@{[$2 - $val]}/g'
}
# The actual point at which limit enforcement takes place for the
diff --git a/common.rc b/common.rc
index 96d5077..319680d 100644
--- a/common.rc
+++ b/common.rc
@@ -308,6 +308,7 @@ _setup_large_xfs_fs()
-c "falloc -k 0 $file_size" \
-c "chattr +d" \
$SCRATCH_MNT/.use_space 2>&1 > /dev/null
+ export NUM_SPACE_FILES=1
status=$?
umount $SCRATCH_MNT
if [ $status -ne 0 ]; then
@@ -417,6 +418,7 @@ _setup_large_ext4_fs()
$SCRATCH_MNT/.use_space.$nfiles 2>&1
status=$?
fi
+ export NUM_SPACE_FILES=$nfiles
umount $SCRATCH_MNT
if [ $status -ne 0 ]; then
--
1.7.10.4
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [PATCH 00/13] xfstests: generic --large-fs support
2013-03-15 11:53 [PATCH 00/13] xfstests: generic --large-fs support Dave Chinner
` (12 preceding siblings ...)
2013-03-15 11:53 ` [PATCH 13/13] xfstests: correct quota file count " Dave Chinner
@ 2013-03-19 20:20 ` Rich Johnston
13 siblings, 0 replies; 23+ messages in thread
From: Rich Johnston @ 2013-03-19 20:20 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
This patchset has been committed, thanks for submitting it Dave.
Thanks
--Rich
On 03/15/2013 06:53 AM, Dave Chinner wrote:
> Hi folks,
>
> This is a repost of my current version of the large filesystem
> testing support for xfstests. I've been using it for more than a
> year, and I think the ext4 guys would like to have it available to
> them.
>
> Last posted here:
>
> http://oss.sgi.com/archives/xfs/2012-07/msg00361.html
>
> There are changes from last time, but it's months since i've made
> them, so there's no fine grained changelog. :/
>
> Cheers,
>
> Dave.
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread