[PATCH 0/5] more testsuite improvements

[PATCH 0/5] more testsuite improvements

[Luc Van Oostenryck]

This serie is a continuation of a similar previous sent in
November. It has three parts:
- patch 1: give a bit more flexibility when testing the test output,
  from test-linearize for example.
- patch 2: somehow correct a message from the testsuite
- patch 3-5: make the testsuite quieter regarding the 'known-to-fail'
  tests.


This serie can also be found as:
  git://github.com/lucvoo/sparse.git sent/testsuite2


Luc Van Oostenryck (5):
  testsuite: check the nbr of times a pattern should be present
  testsuite: use 'error' instead of 'info' for successful tests known to fail
  testsuite: get 'check-known-to-fail' earlier
  testsuite: allow quieter error reporting
  testsuite: quieter error reporting for 'known-to-fail'

 Documentation/test-suite |  4 ++++
 validation/test-suite    | 42 ++++++++++++++++++++++++++++++++++++------
 2 files changed, 40 insertions(+), 6 deletions(-)

[PATCH 1/5] testsuite: check the nbr of times a pattern should be present

[Luc Van Oostenryck]

Complement the 'check-output-contains/excludes' tags to also
be able to specify the number of times a given pattern should
occurs in the output.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 Documentation/test-suite |  4 ++++
 validation/test-suite    | 28 ++++++++++++++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/Documentation/test-suite b/Documentation/test-suite
index a0f205f4..2e786bbf 100644
--- a/Documentation/test-suite
+++ b/Documentation/test-suite
@@ -49,6 +49,10 @@ check-output-excludes: <pattern> (optional)
 	Several such tags can be given, in which case the output
 	must contains none of the patterns.
 
+check-output-pattern-<nbr>-times: <pattern> (optional)
+	Similar than the contains/excludes her above, but with full control
+	of the number of times the pattern should occurs in the output.
+
 	Using test-suite
 	~~~~~~~~~~~~~~~~
 
diff --git a/validation/test-suite b/validation/test-suite
index e5317109..c14a4c5a 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -93,6 +93,27 @@ has_none_patterns()
 	has_patterns "$1" "$2" "$3" -eq
 }
 
+##
+# nbr_patterns(ifile tag ofile) - does ofile contains the
+#                        the patterns given by ifile's tags
+#                        the right number of time?
+nbr_patterns()
+{
+	ifile="$1"
+	patt="$2"
+	ofile="$3"
+	grep "$patt-[0-9][0-9]*-times:" "$ifile" | \
+	sed -e "s/^.*$patt-\([0-9][0-9]*\)-times: *\(.*\)/\1 \2/" | \
+	while read nbr pat; do
+		n=$(grep -s "$pat" "$ofile" | wc -l)
+		if [ "$n" -ne "$nbr" ]; then
+			return 1
+		fi
+	done
+
+	return $?
+}
+
 ##
 # verbose(string) - prints string if we are in verbose mode
 verbose()
@@ -221,6 +242,13 @@ do_test()
 		test_failed=1
 	fi
 
+	# verify the 'check-output-pattern-X-times' tags
+	nbr_patterns "$file" 'check-output-pattern' $file.output.got
+	if [ "$?" -ne "0" ]; then
+		error "Actual output doesn't contain the pattern the expected number."
+		test_failed=1
+	fi
+
 	get_tag "check-known-to-fail" $file
 	must_fail=`expr "$?" = 0`
 	known_ko_tests=`expr $known_ko_tests + $must_fail`
-- 
2.11.0

[PATCH 2/5] testsuite: use 'error' instead of 'info' for successful tests known to fail

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/validation/test-suite b/validation/test-suite
index c14a4c5a..852904a4 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -259,7 +259,7 @@ do_test()
 		if [ "$test_failed" -eq "1" ]; then
 			echo "info: test '$file' is known to fail"
 		else
-			echo "info: test '$file' is known to fail but succeed!"
+			echo "error: test '$file' is known to fail but succeed!"
 			test_failed=1
 		fi
 	fi
-- 
2.11.0

[PATCH 3/5] testsuite: get 'check-known-to-fail' earlier

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 852904a4..fdb7cbc2 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -212,6 +212,10 @@ do_test()
 	$cmd 1> $file.output.got 2> $file.error.got
 	actual_exit_value=$?
 
+	get_tag "check-known-to-fail" $file
+	must_fail=`expr "$?" = 0`
+	known_ko_tests=`expr $known_ko_tests + $must_fail`
+
 	for stream in output error; do
 		grep -s -q "check-$stream-ignore" $file && continue
 
@@ -249,10 +253,6 @@ do_test()
 		test_failed=1
 	fi
 
-	get_tag "check-known-to-fail" $file
-	must_fail=`expr "$?" = 0`
-	known_ko_tests=`expr $known_ko_tests + $must_fail`

[PATCH 4/5] testsuite: allow quieter error reporting

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index fdb7cbc2..47015c61 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -126,7 +126,7 @@ verbose()
 # error(string[, die]) - prints an error and exits with value die if given
 error()
 {
-	echo "error: $1"
+	[ "$quiet" -ne 1 ] && echo "error: $1"
 	[ -n "$2" ] && exit $2
 	return 0
 }
@@ -223,7 +223,7 @@ do_test()
 		if [ "$?" -ne "0" ]; then
 			error "actual $stream text does not match expected $stream text."
 			error  "see $file.$stream.* for further investigation."
-			cat "$file".$stream.diff
+			[ $quiet -ne 1 ] && cat "$file".$stream.diff
 			test_failed=1
 		fi
 	done
-- 
2.11.0

[PATCH 5/5] testsuite: quieter error reporting for 'known-to-fail'

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/validation/test-suite b/validation/test-suite
index 47015c61..a5f5e25e 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -214,6 +214,8 @@ do_test()
 
 	get_tag "check-known-to-fail" $file
 	must_fail=`expr "$?" = 0`
+	quiet=0
+	[ $must_fail -eq 1 ] && [ $V -eq 0 ] && quiet=1
 	known_ko_tests=`expr $known_ko_tests + $must_fail`
 
 	for stream in output error; do
-- 
2.11.0