[PATCH 0/5] xfstests: various fixes

[PATCH 0/5] xfstests: various fixes

[Dave Chinner]

Hi folks,

These are various fixes for xfstests.

The first patch is an infrastructure regression fix, resulting from
the patchset that split up the tests into subdirectories. It also
moves all the check.* output files to the result directory rather
than leaving them in the root directory of xfstests.

The second is a bunch of changes to generic/310 that should have
been done in the review cycle. It now works reliably for me.

The last 3 patches fix bugs in tests that result in files being left
in the root directory of xfstests.

Cheers,

Dave.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 1/5] xfstests: fix last test runtime output

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

Prior to the test directory split-up, xfstests used to output the
time it previously took to run a specific test in it's output. This
was broken during the split up, as test identifiers changed and the
result output changed.

To fix this, the search for previous test results needs to look at
the sequence number rather than the absolute sequence for the test.
The new output looks the same as the old functionality:

generic/001 4s ... 5s
generic/002 1s ... 0s
generic/005 1s ... 1s
generic/006 1s ... 1s
generic/007 2s ... 1s

Where the first column is the time of the previous test run, and the
second column is the time that this run took.

Further, the check files used to generate this information are not
being output properly in the result directory, and so various log
files are not getting written to the correct location or file names.
For example, the calls to _check_test_fs would output failures to
".full", while other messages would be output to check.full, but
none would output to the corect location of RESULT_BASE/check.full.

Fix all this mess up so that all the check files for a specific run
end up in RESULT_BASE, and ensure the timing code checks the right
file and dumps output.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 check |   48 +++++++++++++++++++++++++++---------------------
 1 file changed, 27 insertions(+), 21 deletions(-)

diff --git a/check b/check
index 966fbe5..a79747e 100755
--- a/check
+++ b/check
@@ -293,14 +293,17 @@ fi
 
 _wrapup()
 {
+    seq="check"
+    check="$RESULT_BASE/check"
+
     if $showme
     then
 	:
     elif $needwrap
     then
-	if [ -f check.time -a -f $tmp.time ]
+	if [ -f $check.time -a -f $tmp.time ]
 	then
-	    cat check.time $tmp.time \
+	    cat $check.time $tmp.time \
 	    | $AWK_PROG '
 	{ t[$1] = $2 }
 END	{ if (NR > 0) {
@@ -308,13 +311,13 @@ END	{ if (NR > 0) {
 	  }
 	}' \
 	    | sort -n >$tmp.out
-	    mv $tmp.out check.time
+	    mv $tmp.out $check.time
 	fi
 
-	echo "" >>check.log
-	date >>check.log
-	echo $list | fmt | sed -e 's/^/    /' -e "s;$SRC_DIR/;;g" >>check.log
-	$interrupt && echo "Interrupted!" >>check.log
+	echo "" >>$check.log
+	date >>$check.log
+	echo $list | fmt | sed -e 's/^/    /' -e "s;$SRC_DIR/;;g" >>$check.log
+	$interrupt && echo "Interrupted!" >>$check.log
         
         if [ ! -z "$n_try" -a $n_try != 0 ]
 	then
@@ -324,18 +327,18 @@ END	{ if (NR > 0) {
 	if [ ! -z "$notrun" ]
 	then
 	    echo "Not run:$notrun"
-	    echo "Not run:$notrun" >>check.log
+	    echo "Not run:$notrun" >>$check.log
 	fi
 
         if [ ! -z "$n_bad" -a $n_bad != 0 ]
 	then
 	    echo "Failures:$bad"
 	    echo "Failed $n_bad of $n_try tests"
-	    echo "Failures:$bad" | fmt >>check.log
-	    echo "Failed $n_bad of $n_try tests" >>check.log
+	    echo "Failures:$bad" | fmt >>$check.log
+	    echo "Failed $n_bad of $n_try tests" >>$check.log
 	else
 	    echo "Passed all $n_try tests"
-	    echo "Passed all $n_try tests" >>check.log
+	    echo "Passed all $n_try tests" >>$check.log
 	fi
 	needwrap=false
     fi
@@ -346,10 +349,19 @@ END	{ if (NR > 0) {
 
 trap "_wrapup; exit \$status" 0 1 2 3 15
 
+mkdir -p $RESULT_BASE
+if [ ! -d $RESULT_BASE ]; then
+	echo "failed to create results directory $RESULTS_BASE"
+	exit 1;
+fi
+
+seq="check"
+check="$RESULT_BASE/check"
+
 # don't leave old full output behind on a clean run
-rm -f check.full
+rm -f $check.full
 
-[ -f check.time ] || touch check.time
+[ -f $check.time ] || touch $check.time
 
 # print out our test configuration
 echo "FSTYP         -- `_full_fstyp_details`"
@@ -385,13 +397,7 @@ if [ ! -z "$SCRATCH_DEV" ]; then
   fi
 fi
 
-mkdir -p $RESULT_BASE
-if [ ! -d $RESULT_BASE ]; then
-	echo "failed to create results directory $RESULTS_BASE"
-	exit 1;
-fi
-
-seq="check"
+seqres="$check"
 _check_test_fs
 
 for seq in $list
@@ -432,7 +438,7 @@ do
 	fi
 
 	# slashes now in names, sed barfs on them so use grep
-	lasttime=`grep -w ^$seq check.time | awk '// {print $2}'`
+	lasttime=`grep -w ^$seqnum $check.time | awk '// {print $2}'`
 	if [ "X$lasttime" != X ]; then
 		echo -n " ${lasttime}s ..."
 	else
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 2/5] xfstests: 310 fails with existing directory error

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

Test 310 fails with:

mkdir: cannot create directory `/mnt/test/tmp': File exists

$TEST_DIR is persistent, so test directories need to be created with
"mkdir -p" so they don't fail if the directory already exists.

Many other things need fixing, too.
	- Tests should define directories they use on $TEST_DIR by
	  their sequence number, not generic names.

	- Use a variable for the directory the test runs in
	  ($SEQ_DIR, in this case) to avoid having to manually code
	  it everywhere.

	- New binaries need to be added to .gitignore.

	- Return status for shell functions is 0 for success,
	  non-zero for failure.

	- Setting status=0 if there is no failure in the first test
	  means that even if the second test fails, the test will
	  still pass. Change the test to use "_fatal" when a kernel
	  bug is detected, and only set status=0 when the entire
	  test has finished.

	- reduce the default runtime by to roughly a minute and
	  scale it with the stress load factor variables. In most
	  cases, this test is never going to hit problems (as
	  they've already been fixed) so running it for ~4 minutes
	  is mostly a waste of time...

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 .gitignore        |    2 ++
 tests/generic/310 |   35 +++++++++++++++++------------------
 2 files changed, 19 insertions(+), 18 deletions(-)

diff --git a/.gitignore b/.gitignore
index 7a10feb..0bd48c3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -67,6 +67,8 @@
 /src/preallo_rw_pattern_writer
 /src/pwrite_mmap_blocked
 /src/randholes
+/src/t_readdir_1
+/src/t_readdir_2
 /src/rename
 /src/resvtest
 /src/runas
diff --git a/tests/generic/310 b/tests/generic/310
index b5316cd..26d2d4a 100755
--- a/tests/generic/310
+++ b/tests/generic/310
@@ -75,12 +75,11 @@ check_kernel_bug()
 	new_warning=`dmesg | grep -c "^WARNING"`
 	new_lockdep=`dmesg | grep -c "possible recursive locking detected"`
 
-
 	# no kernel bug is detected
 	if [ $new_bug -eq $nr_bug -a $new_error -eq $nr_error -a \
 	     $new_null -eq $nr_null -a $new_warning -eq $nr_warning -a \
 	     $new_lockdep -eq $nr_lockdep ]; then
-		return 1
+		return 0
 	fi
 
 	nr_bug=$new_bug
@@ -88,37 +87,36 @@ check_kernel_bug()
 	nr_null=$new_null
 	nr_warning=$new_warning
 	nr_lockdep=$new_lockdep
+	return 1
 }
-mkdir $TEST_DIR/tmp
+
+RUN_TIME=$((30 * $TIME_FACTOR))
+
+SEQ_DIR=$TEST_DIR/$seq
+mkdir -p $SEQ_DIR
 for n in {1..4096}; do
-	touch $TEST_DIR/tmp/$n
+	touch $SEQ_DIR/$n
 done
 
 _test_read()
 {
-	src/t_readdir_1 $TEST_DIR/tmp &
-	sleep 100
+	src/t_readdir_1 $SEQ_DIR &
+	sleep $RUN_TIME
 	killall src/t_readdir_1
 	check_kernel_bug
-	if [ $? -eq 1 ]; then
-		status=0
-	else
-		echo "error: kernel bug was found, you can see the
-			dmesg for more infomation."
+	if [ $? -ne 0 ]; then
+		_fatal "kernel bug detected, check dmesg for more infomation."
 	fi
 }
 
 _test_lseek()
 {
-	src/t_readdir_2 $TEST_DIR/tmp &
-	sleep 100
+	src/t_readdir_2 $SEQ_DIR &
+	sleep $RUN_TIME
 	killall src/t_readdir_2
 	check_kernel_bug
-	if [ $? -eq 1 ]; then
-		status=0
-	else
-		echo "error: kernel bug was found, you can see the
-			dmesg for more infomation."
+	if [ $? -ne 0 ]; then
+		_fatal "kernel bug detected, check dmesg for more infomation."
 	fi
 }
 
@@ -127,4 +125,5 @@ _test_lseek
 
 # success, all done
 echo "*** done"
+status=0
 exit
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 3/5] xfstests: filter EA paths used by dump

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

Test 063 fails because the diff output now has entire paths to the
files in the results directory in it rather than just the file name.
Add the results directory to the directory filter used by the dump
tests to remove the path from the diff output.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 common/dump |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/common/dump b/common/dump
index 73d0304..7fe50f6 100644
--- a/common/dump
+++ b/common/dump
@@ -927,6 +927,7 @@ _dir_filter()
     -e "s#$restore_sdir#RESTORE_SUBDIR#g" \
     -e "s#$$#PID#g" \
     -e "/Only in SCRATCH_MNT: .use_space/d" \
+    -e "s#$RESULT_DIR/##g" \
 
 }
 
@@ -1358,7 +1359,7 @@ _diff_compare_eas()
     | sed -e "s#$restore_sdir\/##" \
     | tee $seqres.ea2 \
     | _dir_filter
-    diff -s $seqres.ea1 $seqres.ea2
+    diff -s $seqres.ea1 $seqres.ea2 | _dir_filter
 }
 
 
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 4/5] xfstests: fix incorrect redirect in generic/232

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

generic/232 attempts to direct output to tee, but instead of using a
pipe it uses an append operator. Hence it leaves a file named "tee"
in the root directory of the xfstests execution path. Just direct
the output to the $seqres.full file rather than trying to tee it
into the test output as well.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 tests/generic/232 |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/generic/232 b/tests/generic/232
index 2402c45..8ada8aa 100755
--- a/tests/generic/232
+++ b/tests/generic/232
@@ -55,7 +55,7 @@ _fsstress()
 	count=2000
 	args=`_scale_fsstress_args -d $out -n $count -p 7`
 
-	echo "fsstress $args" >> tee -a $seqres.full
+	echo "fsstress $args" >> $seqres.full
 	if ! $FSSTRESS_PROG $args | tee -a $seqres.full | _filter_num
 	then
 		echo "    fsstress $args returned $?"
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 5/5] xfstests: fix broken redirects in generic/131

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

generic/131 attempts to kill processes that may no longer exist when
the test finishes and has a broken redirect for the error messages
and they end up in a file nemaed "1" in the xfstests root instead of
/dev/null.

Not only that, the attempts to redirect stderr to stdout in the
middle of the test use incorrect redirect syntax, so they create an
empty file named "1" in the xfstests root...

IOWs, all the redirects in the test are broken. Fix them and clean
up the failure case to use the exit trap to trigger the cleanup
function....

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 tests/generic/131 |   18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)

diff --git a/tests/generic/131 b/tests/generic/131
index 95eb612..6c3131c 100755
--- a/tests/generic/131
+++ b/tests/generic/131
@@ -33,8 +33,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _cleanup()
 {
-    kill $locktest_pid2 2&>1 /dev/null
-    kill $locktest_pid1 2&>1 /dev/null
+    kill $locktest_pid2 > /dev/null 2>&1
+    kill $locktest_pid1 > /dev/null 2>&1
     _cleanup_testdir
 }
 
@@ -60,23 +60,21 @@ elif [ $$ -gt 32000 ]; then
 fi
 
 # Start the server
-src/locktest -p $PORT $TESTFILE > $testdir/server.out 2>&1 &
+src/locktest -p $PORT $TESTFILE 2>&1 > $testdir/server.out &
 locktest_pid1=$!
 
 sleep 1
 
 # Start the client
-src/locktest -p $PORT -h localhost $TESTFILE > $testdir/client.out 2>&1
+src/locktest -p $PORT -h localhost $TESTFILE 2>&1 > $testdir/client.out
 locktest_pid2=$!
 result=$?
 if [ $result -eq 0 ]; then
-    echo success!
+	echo success!
+	status=0
 else
-    echo "Client reported failure ($result)"
-    cat $testdir/*.out
-    _cleanup
-    exit $status
+	echo "Client reported failure ($result)"
+	cat $testdir/*.out
 fi
 
-status=0
 exit
-- 
1.7.10.4

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 0/5] xfstests: various fixes

[Dave Chinner]

On Wed, May 01, 2013 at 06:31:58PM +1000, Dave Chinner wrote:
> Hi folks,
> 
> These are various fixes for xfstests.
> 
> The first patch is an infrastructure regression fix, resulting from
> the patchset that split up the tests into subdirectories. It also
> moves all the check.* output files to the result directory rather
> than leaving them in the root directory of xfstests.
> 
> The second is a bunch of changes to generic/310 that should have
> been done in the review cycle. It now works reliably for me.
> 
> The last 3 patches fix bugs in tests that result in files being left
> in the root directory of xfstests.

FWIW, with the V2 xfsprogs kernel sync tarball I've posted and these
fixes to xfstests, I'm down to only 3 failures in xfstests for 4k
block size filesystem tests:

Failures: generic/233 shared/298 xfs/296

generic/233 appears to be a small quota accounting mismatch,
probably related to speculative preallocation. Not a major issue.

shared/298 is failing due to the ENOSPC problem Eric posted patches
to fix. Test bug.

xfs/296 is failing due to an unresolved xfsdump issue w.r.t
restoring security attributes. Unresolved.

So, I'm much happier now about the state of xfsprogs and xfstests
than I was this morning.  I'll be happier still when I fix the fsx
failures on 512 byte block size filesystems, because that will bring
my 512 byte block size filesystem tests to close to the same level.
That's for tomorrow, though.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 1/5] xfstests: fix last test runtime output

[Rich Johnston]

Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 2/5] xfstests: 310 fails with existing directory error

[Rich Johnston]

Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 3/5] xfstests: filter EA paths used by dump

[Rich Johnston]

Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 4/5] xfstests: fix incorrect redirect in generic/232

[Rich Johnston]

Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 5/5] xfstests: fix broken redirects in generic/131

[Rich Johnston]

Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 0/5] xfstests: various fixes

[Rich Johnston]

Thanks Dave, this patchset has been committed.

--Rich

commit 87f5e8e2dbf02b3c4d804a797ed5a7a643f4b64e
Author: Dave Chinner <dchinner@redhat.com>
Date:   Wed May 1 08:31:59 2013 +0000

     xfstests: fix last test runtime output

commit 97540f14f71e6785838532a982c443571380cfcc
Author: Dave Chinner <dchinner@redhat.com>
Date:   Wed May 1 08:32:00 2013 +0000

     xfstests: 310 fails with existing directory error

commit 3ac75605de1fbbaae9505799b46c44fbe053f04e
Author: Dave Chinner <dchinner@redhat.com>
Date:   Wed May 1 08:32:01 2013 +0000

     xfstests: filter EA paths used by dump

commit e6c6abd6dcacafa778d610451a023ec93c1b352e
Author: Dave Chinner <dchinner@redhat.com>
Date:   Wed May 1 08:32:02 2013 +0000

     xfstests: fix incorrect redirect in generic/232

commit 627427bb1bb4007c661a178bbeb4ea0bd0167c6f
Author: Dave Chinner <dchinner@redhat.com>
Date:   Wed May 1 08:32:03 2013 +0000

     xfstests: fix broken redirects in generic/131

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs