[PATCH 00/28] testsuite improvements

[PATCH 00/28] testsuite improvements

[Luc Van Oostenryck]

This series contains diverse small improvements to
the testsuite.


The series is also available for review in the git repository at:
  git://github.com/lucvoo/sparse.git testsuite-clean

----------------------------------------------------------------
Luc Van Oostenryck (28):
      testsuite: 'echo -n' may not be interpreted as '-n'
      testsuite: allow to test a few cases at once
      testsuite: move verbose() & error()
      testsuite: better message for pattern nbr checking
      testsuite: better message for pattern absence/presence
      testsuite: saner checking of pattern occurences
      testsuite: clearer result summary
      testsuite: check error messages first
      testsuite: saner handling of 'must_fail'
      testsuite: allow to parse several options
      testsuite: add support for -q|--quiet
      testsuite: add support for -a|--abort
      testsuite: get options from env too
      testsuite: allow --format & --single
      testsuite: remove useless selftest
      testsuite: remove useless test-be.c
      testsuite: extract disable()
      testsuite: simplify documentation
      testsuite: allow arch-specific tests
      testsuite: save screen real estate
      testsuite: add a blank line before format
      testsuite: 'quiet' must be initialized earlier
      testsuite: move up arg_file()
      testsuite: make do_format() more self-contained
      testsuite: format: saner defaults handling
      testsuite: format: strip .c from default name
      testsuite: add support for 'format -f'
      testsuite: add support for 'format -l'

 Documentation/test-suite          |  35 +++--
 validation/test-be.c              |  46 ------
 validation/test-suite             | 286 ++++++++++++++++++++++++++++++--------
 validation/testsuite-selfcheck1.c |  10 --
 validation/testsuite-selfcheck2.c |  10 --
 validation/testsuite-selfcheck3.c |  10 --
 6 files changed, 249 insertions(+), 148 deletions(-)
 delete mode 100644 validation/test-be.c
 delete mode 100644 validation/testsuite-selfcheck1.c
 delete mode 100644 validation/testsuite-selfcheck2.c
 delete mode 100644 validation/testsuite-selfcheck3.c

[PATCH 01/28] testsuite: 'echo -n' may not be interpreted as '-n'

[Luc Van Oostenryck]

POSIX tell that the interpretation of the '-n' is implementation
independent. So avoid it.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index cf151a361..53cd30190 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -303,8 +303,8 @@ do_test_suite()
 
 	# prints some numbers
 	tests_nr=$(($ok_tests + $ko_tests))
-	echo -n "Out of $tests_nr tests, $ok_tests passed, $ko_tests failed"
-	echo " ($known_ko_tests of them are known to fail)"
+	echo "Out of $tests_nr tests, $ok_tests passed, $ko_tests failed" \
+		" ($known_ko_tests of them are known to fail)"
 	if [ "$unhandled_tests" -ne "0" ]; then
 		echo "$unhandled_tests tests could not be handled by $prog_name"
 	fi
-- 
2.14.0

[PATCH 02/28] testsuite: allow to test a few cases at once

[Luc Van Oostenryck]

Sometimes we want to only launch a few tests, not the whole
testsuite. the 'single' option allow to test a single case
but it's too restrictive.

Changes this by allowing arbirary arguments as files to be tested.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/validation/test-suite b/validation/test-suite
index 53cd30190..9e30ed937 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -6,7 +6,7 @@ cd $(dirname "$0")
 
 default_path=".."
 default_cmd="sparse \$file"
-tests_list=`find . -name '*.c' | sed -e 's#^\./\(.*\)#\1#' | sort`
+tests_list=""
 prog_name=`basename $0`
 
 if [ ! -x "$default_path/sparse-llvm" ]; then
@@ -156,6 +156,7 @@ echo "Usage: $prog_name [command] [command arguments]"
 echo
 echo "commands:"
 echo "    none                       runs the whole test suite"
+echo "    file ...                   runs the test suite on the given file(s)"
 echo "    single file                runs the test in 'file'"
 echo "    format file [name [cmd]]   helps writing a new test case using cmd"
 echo
@@ -370,8 +371,14 @@ arg_file()
 
 case "$1" in
 	'')
+		tests_list=`find . -name '*.c' | sed -e 's#^\./\(.*\)#\1#' | sort`
 		do_test_suite
 		;;
+	*.c)
+		tests_list="$@"
+		do_test_suite
+		;;
+
 	single)
 		arg_file "$2"
 		do_test "$2"
-- 
2.14.0

[PATCH 03/28] testsuite: move verbose() & error()

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 9e30ed937..5f63a23bb 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -71,6 +71,23 @@ get_tag_value()
 EOT
 }
 
+##
+# verbose(string) - prints string if we are in verbose mode
+verbose()
+{
+	[ "$V" -eq "1" ] && echo "        $1"
+	return 0
+}
+
+##
+# error(string[, die]) - prints an error and exits with value die if given
+error()
+{
+	[ "$quiet" -ne 1 ] && echo "error: $1"
+	[ -n "$2" ] && exit $2
+	return 0
+}
+
 ##
 # helper for has_(each|none)_patterns()
 has_patterns()
@@ -132,23 +149,6 @@ nbr_patterns()
 	return $?
 }
 
-##
-# verbose(string) - prints string if we are in verbose mode
-verbose()
-{
-	[ "$V" -eq "1" ] && echo "        $1"
-	return 0
-}
-
-##
-# error(string[, die]) - prints an error and exits with value die if given
-error()
-{
-	[ "$quiet" -ne 1 ] && echo "error: $1"
-	[ -n "$2" ] && exit $2
-	return 0
-}

[PATCH 04/28] testsuite: better message for pattern nbr checking

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/validation/test-suite b/validation/test-suite
index 5f63a23bb..953b7637e 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -142,6 +142,8 @@ nbr_patterns()
 	while read nbr pat; do
 		n=$(grep -s "$pat" "$ofile" | wc -l)
 		if [ "$n" -ne "$nbr" ]; then
+			error "test '$ifile' failed"
+			error "	Pattern '$pat' expected $nbr times but got $n times"
 			return 1
 		fi
 	done
@@ -271,7 +273,6 @@ do_test()
 		# verify the 'check-output-pattern-X-times' tags
 		nbr_patterns "$file" 'check-output-pattern' $file.output.got
 		if [ "$?" -ne "0" ]; then
-			error "Actual output doesn't contain the pattern the expected number."
 			test_failed=1
 		fi
 	fi
-- 
2.14.0

[PATCH 05/28] testsuite: better message for pattern absence/presence

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 953b7637e..bdc1424f4 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -96,11 +96,14 @@ has_patterns()
 	patt="$2"
 	ofile="$3"
 	cmp="$4"
+	msg="$5"
 	grep "$patt:" "$ifile" | \
 	sed -e "s/^.*$patt: *\(.*\)$/\1/" | \
 	while read val; do
 		grep -s -q "$val" "$ofile"
 		if [ "$?" $cmp 0 ]; then
+			error "test '$ifile' failed"
+			error "	Pattern '$val' unexpectedly $msg"
 			return 1
 		fi
 	done
@@ -115,7 +118,7 @@ has_patterns()
 # returns 0 if all present, 1 otherwise
 has_each_patterns()
 {
-	has_patterns "$1" "$2" "$3" -ne
+	has_patterns "$1" "$2" "$4" -ne "$3"
 }
 
 ##
@@ -125,7 +128,7 @@ has_each_patterns()
 # returns 1 if any present, 0 otherwise
 has_none_patterns()
 {
-	has_patterns "$1" "$2" "$3" -eq
+	has_patterns "$1" "$2" "$4" -eq "$3"
 }
 
 ##
@@ -256,16 +259,14 @@ do_test()
 
 	# verify the 'check-output-contains/excludes' tags
 	if [ $check_output_contains -eq 1 ]; then
-		has_each_patterns "$file" 'check-output-contains' $file.output.got
+		has_each_patterns "$file" 'check-output-contains' absent $file.output.got
 		if [ "$?" -ne "0" ]; then
-			error "Actual output doesn't contain some of the expected patterns."
 			test_failed=1
 		fi
 	fi
 	if [ $check_output_excludes -eq 1 ]; then
-		has_none_patterns "$file" 'check-output-excludes' $file.output.got
+		has_none_patterns "$file" 'check-output-excludes' present $file.output.got
 		if [ "$?" -ne "0" ]; then
-			error "Actual output contains some patterns which are not expected."
 			test_failed=1
 		fi
 	fi
-- 
2.14.0

[PATCH 06/28] testsuite: saner checking of pattern occurences

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 Documentation/test-suite |  6 ++++++
 validation/test-suite    | 49 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+)

diff --git a/Documentation/test-suite b/Documentation/test-suite
index a288c81a5..e2dda61d7 100644
--- a/Documentation/test-suite
+++ b/Documentation/test-suite
@@ -57,6 +57,12 @@ check-output-pattern-<nbr>-times: <pattern> (optional)
 	Similar than the contains/excludes her above, but with full control
 	of the number of times the pattern should occurs in the output.
 
+check-output-pattern(eq,<nbr>): <pattern> (optional)
+check-output-pattern(<min>,<max>): <pattern> (optional)
+	Similar than the contains/excludes her above, but with full control
+	of the number of times the pattern should occurs in the output.
+	If <min> or <max> is '-' the corresponding check is ignored.
+
 	Using test-suite
 	~~~~~~~~~~~~~~~~
 
diff --git a/validation/test-suite b/validation/test-suite
index bdc1424f4..8b2e0c85a 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -46,6 +46,7 @@ get_tag_value()
 	check_output_contains=0
 	check_output_excludes=0
 	check_output_pattern=0
+	check_output_pattern_minmax=0
 
 	lines=$(grep 'check-[a-z-]*' $1 | \
 		sed -e 's/^.*\(check-[a-z-]*:*\) *\(.*\)$/\1 \2/')
@@ -65,6 +66,7 @@ get_tag_value()
 		check-output-contains:)	check_output_contains=1 ;;
 		check-output-excludes:)	check_output_excludes=1 ;;
 		check-output-pattern-)	check_output_pattern=1 ;;
+		check-output-pattern)	check_output_pattern_minmax=1 ;;
 		esac
 	done << EOT
 	$lines
@@ -154,6 +156,46 @@ nbr_patterns()
 	return $?
 }
 
+##
+# minmax_patterns(ifile tag ofile) - does ofile contains the
+#                        the patterns given by ifile's tags
+#                        the right number of time?
+minmax_patterns()
+{
+	ifile="$1"
+	patt="$2"
+	ofile="$3"
+	grep "$patt([0-9a-z-]*, *[0-9-]*):" "$ifile" | \
+	sed -e "s/^.*$patt(\([0-9a-z-]*\), *\([0-9-]*\)): *\(.*\)/\1 \2 \3/" | \
+	while read min max pat; do
+		n=$(grep -s "$pat" "$ofile" | wc -l)
+		if [ "$min" = 'eq' ]; then
+		    if [ "$n" -ne "$max" ]; then
+			error "test '$ifile' failed"
+			error "	Pattern '$pat' expected $max times but got $n times"
+			return 1
+		    fi
+		    continue
+		fi
+		if [ "$min" != '-' ]; then
+		    if [ "$n" -lt "$min" ]; then
+			error "test '$ifile' failed"
+			error "	Pattern '$pat' expected min $nbr times but got $n times"
+			return 1
+		    fi
+		fi
+		if [ "$max" != '-' ]; then
+		    if [ "$n" -gt "$max" ]; then
+			error "test '$ifile' failed"
+			error "	Pattern '$pat' expected max $nbr times but got $n times"
+			return 1
+		    fi
+		fi
+	done
+
+	return $?
+}
+
 do_usage()
 {
 echo "$prog_name - a tiny automatic testing script"
@@ -277,6 +319,13 @@ do_test()
 			test_failed=1
 		fi
 	fi
+	if [ $check_output_pattern_minmax -eq 1 ]; then
+		# verify the 'check-output-pattern(...)' tags
+		minmax_patterns "$file" 'check-output-pattern' $file.output.got
+		if [ "$?" -ne "0" ]; then
+			test_failed=1
+		fi
+	fi
 
 	[ "$test_failed" -eq "$must_fail" ] || failed=1
 
-- 
2.14.0

[PATCH 07/28] testsuite: clearer result summary

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 8b2e0c85a..3d2192178 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -353,15 +353,20 @@ do_test_suite()
 		do_test "$i"
 	done
 
+	OK=OK
+	[ $failed -eq 0 ] || OK=KO
+
 	# prints some numbers
 	tests_nr=$(($ok_tests + $ko_tests))
-	echo "Out of $tests_nr tests, $ok_tests passed, $ko_tests failed" \
-		" ($known_ko_tests of them are known to fail)"
+	echo "$OK: out of $tests_nr tests, $ok_tests passed, $ko_tests failed"
+	if [ "$known_ko_tests" -ne 0 ]; then
+		echo "	$known_ko_tests of them are known to fail"
+	fi
 	if [ "$unhandled_tests" -ne "0" ]; then
-		echo "$unhandled_tests tests could not be handled by $prog_name"
+		echo "	$unhandled_tests tests could not be handled by $prog_name"
 	fi
 	if [ "$disabled_tests" -ne "0" ]; then
-		echo "$disabled_tests tests were disabled"
+		echo "	$disabled_tests tests were disabled"
 	fi
 }
 
-- 
2.14.0

[PATCH 08/28] testsuite: check error messages first

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/validation/test-suite b/validation/test-suite
index 3d2192178..256d31550 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -276,7 +276,7 @@ do_test()
 	[ $must_fail -eq 1 ] && [ $V -eq 0 ] && quiet=1
 	known_ko_tests=$(($known_ko_tests + $must_fail))
 
-	for stream in output error; do
+	for stream in error output; do
 		eval ignore=\$check_${stream}_ignore
 		[ $ignore -eq 1 ] && continue
 
-- 
2.14.0

[PATCH 09/28] testsuite: saner handling of 'must_fail'

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 256d31550..5dfe9e7f3 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -327,17 +327,19 @@ do_test()
 		fi
 	fi
 
-	[ "$test_failed" -eq "$must_fail" ] || failed=1

[PATCH 10/28] testsuite: allow to parse several options

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 5dfe9e7f3..7a491dfb4 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -427,7 +427,8 @@ arg_file()
 	return 0
 }
 
-case "$1" in
+while true; do
+	case "$1" in
 	'')
 		tests_list=`find . -name '*.c' | sed -e 's#^\./\(.*\)#\1#' | sort`
 		do_test_suite
@@ -454,7 +455,9 @@ case "$1" in
 		do_usage
 		exit 1
 		;;
-esac
+	esac
+	break
+done
 
 exit $failed
 
-- 
2.14.0

[PATCH 11/28] testsuite: add support for -q|--quiet

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 7a491dfb4..9371130d9 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -31,6 +31,7 @@ known_ko_tests=0
 
 # defaults to not verbose
 [ -z "$V" ] && V=0
+vquiet=""
 
 ##
 # get_tag_value(file) - get the 'check-<...>' tags & values
@@ -199,7 +200,10 @@ minmax_patterns()
 do_usage()
 {
 echo "$prog_name - a tiny automatic testing script"
-echo "Usage: $prog_name [command] [command arguments]"
+echo "Usage: $prog_name [option(s)] [command] [arguments]"
+echo
+echo "options:"
+echo "    -q|--quiet                 be extra quiet while running the tests"
 echo
 echo "commands:"
 echo "    none                       runs the whole test suite"
@@ -254,7 +258,9 @@ do_test()
 
 	cmd=`eval echo $default_path/$check_command`
 
-	echo "     TEST    $test_name ($file)"
+	if [ -z "$vquiet" ]; then
+		echo "     TEST    $test_name ($file)"
+	fi
 
 	verbose "Using command       : $cmd"
 
@@ -329,6 +335,7 @@ do_test()
 
 	if [ "$must_fail" -eq "1" ]; then
 		if [ "$test_failed" -eq "1" ]; then
+			[ -z "$vquiet" ] && \
 			echo "info: test '$file' is known to fail"
 		else
 			echo "error: test '$file' is known to fail but succeed!"
@@ -429,6 +436,12 @@ arg_file()
 
 while true; do
 	case "$1" in
+	-q|--quiet)
+		vquiet=1
+		shift
+		continue
+		;;
+
 	'')
 		tests_list=`find . -name '*.c' | sed -e 's#^\./\(.*\)#\1#' | sort`
 		do_test_suite
-- 
2.14.0

[PATCH 12/28] testsuite: add support for -a|--abort

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/validation/test-suite b/validation/test-suite
index 9371130d9..49b3e0c24 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -32,6 +32,7 @@ known_ko_tests=0
 # defaults to not verbose
 [ -z "$V" ] && V=0
 vquiet=""
+abort=0
 
 ##
 # get_tag_value(file) - get the 'check-<...>' tags & values
@@ -203,6 +204,7 @@ echo "$prog_name - a tiny automatic testing script"
 echo "Usage: $prog_name [option(s)] [command] [arguments]"
 echo
 echo "options:"
+echo "    -a|--abort                 abort the tests as soon as one fails"
 echo "    -q|--quiet                 be extra quiet while running the tests"
 echo
 echo "commands:"
@@ -343,6 +345,7 @@ do_test()
 	fi
 
 	if [ "$test_failed" -ne "$must_fail" ]; then
+		[ $abort -eq 1 ] && exit 1
 		test_failed=1
 		failed=1
 	fi
@@ -436,6 +439,11 @@ arg_file()
 
 while true; do
 	case "$1" in
+	-a|--abort)
+		abort=1
+		shift
+		continue
+		;;
 	-q|--quiet)
 		vquiet=1
 		shift
-- 
2.14.0

[PATCH 13/28] testsuite: get options from env too

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/validation/test-suite b/validation/test-suite
index 49b3e0c24..c03348028 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -2,6 +2,13 @@
 
 #set -x
 
+## allow flags from environment
+flags="$SPARSE_TEST_FLAGS"
+if [ ! -z "$flags" ]; then
+	unset SPARSE_TEST_FLAGS
+	exec "$0" $flags "$@"
+fi
+
 cd $(dirname "$0")
 
 default_path=".."
-- 
2.14.0

[PATCH 14/28] testsuite: allow --format & --single

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index c03348028..9364b9d7f 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -466,7 +466,7 @@ while true; do
 		do_test_suite
 		;;
 
-	single)
+	single|--single)
 		arg_file "$2"
 		do_test "$2"
 		case "$?" in
@@ -475,7 +475,7 @@ while true; do
 			2) echo "$2 can't be handled by $prog_name";;
 		esac
 		;;
-	format)
+	format|--format)
 		arg_file "$2"
 		do_format "$2" "$3" "$4"
 		;;
-- 
2.14.0

[PATCH 15/28] testsuite: remove useless selftest

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/testsuite-selfcheck1.c | 10 ----------
 validation/testsuite-selfcheck2.c | 10 ----------
 validation/testsuite-selfcheck3.c | 10 ----------
 3 files changed, 30 deletions(-)
 delete mode 100644 validation/testsuite-selfcheck1.c
 delete mode 100644 validation/testsuite-selfcheck2.c
 delete mode 100644 validation/testsuite-selfcheck3.c

diff --git a/validation/testsuite-selfcheck1.c b/validation/testsuite-selfcheck1.c
deleted file mode 100644
index d927f9961..000000000
--- a/validation/testsuite-selfcheck1.c
+++ /dev/null
@@ -1,10 +0,0 @@
-good
-
-/*
- * check-name: selfcheck1
- * check-command: sparse -E $file
- * check-output-ignore
- *
- * check-output-contains: good
- * check-output-excludes: evil
- */
diff --git a/validation/testsuite-selfcheck2.c b/validation/testsuite-selfcheck2.c
deleted file mode 100644
index 5309e32f3..000000000
--- a/validation/testsuite-selfcheck2.c
+++ /dev/null
@@ -1,10 +0,0 @@
-evil
-
-/*
- * check-name: selfcheck2
- * check-command: sparse -E $file
- * check-output-ignore
- * check-known-to-fail
- *
- * check-output-contains: good
- */
diff --git a/validation/testsuite-selfcheck3.c b/validation/testsuite-selfcheck3.c
deleted file mode 100644
index 6d834e68d..000000000
--- a/validation/testsuite-selfcheck3.c
+++ /dev/null
@@ -1,10 +0,0 @@
-evil

[PATCH 16/28] testsuite: remove useless test-be.c

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-be.c | 46 ----------------------------------------------
 1 file changed, 46 deletions(-)
 delete mode 100644 validation/test-be.c

diff --git a/validation/test-be.c b/validation/test-be.c
deleted file mode 100644
index deda3cc14..000000000
--- a/validation/test-be.c
+++ /dev/null
@@ -1,46 +0,0 @@
-int printf(char *c, ...);
-void exit(int c);
-
-#undef PRINT_OUTPUTS
-
-static void test_func_args(int x, int y)
-{
-	if (x == y)
-		exit(1);
-}
-
-static int binop_s32(int x, int y)
-{
-	int a;
-
-	a = a + x;
-	a = a / y;
-	a = a * x;
-	a = a - y;
-
-	return a;
-}
-
-static void test_binops(void)
-{
-	int tmp_s32 = binop_s32(987123, 234);
-
-#ifdef PRINT_OUTPUTS
-	printf("binop_s32(987123, 234) == %d\n", tmp_s32);
-#else
-	if (tmp_s32 != -1470599007)
-		exit(2);
-#endif
-}
-
-int main (int argc, char *argv[])
-{
-	test_func_args(1, 2);
-	test_binops();
-
-	return 0;
-}

[PATCH 17/28] testsuite: extract disable()

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 9364b9d7f..84e359fb8 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -223,6 +223,14 @@ echo
 echo "    help                       prints usage"
 }
 
+disable()
+{
+	disabled_tests=$(($disabled_tests + 1))
+	if [ -z "$vquiet" ]; then
+		echo "     DISABLE $1 ($2)"
+	fi
+}
+
 ##
 # do_test(file) - tries to validate a test case
 #
@@ -259,8 +267,7 @@ do_test()
 	base_cmd=$1
 	for i in $disabled_cmds; do
 		if [ "$i" = "$base_cmd" ] ; then
-			disabled_tests=$(($disabled_tests + 1))
-			echo "     DISABLE $test_name ($file)"
+			disable "$test_name" "$file"
 			return 3
 		fi
 	done
-- 
2.14.0

[PATCH 18/28] testsuite: simplify documentation

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 Documentation/test-suite | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/Documentation/test-suite b/Documentation/test-suite
index e2dda61d7..a745da850 100644
--- a/Documentation/test-suite
+++ b/Documentation/test-suite
@@ -7,13 +7,13 @@ Sparse has a number of test cases in its validation directory. The test-suite
 script aims at making automated checking of these tests possible. It works by
 embedding tags in C comments in the test cases.
 
-check-name: (mandatory)
-	Name of the test.
+check-name: <name>
+	Name of the test. This is the only mandatory tag.
 
-check-description: (optional)
+check-description: <description ...>
 	A description of what the test checks.
 
-check-command: (optional)
+check-command: <command arg ...>
 	There are different kinds of tests. Some can validate the sparse
 	preprocessor, while others will use sparse, cgcc, or even other backends
 	of the library. check-command allows you to give a custom command to
@@ -22,43 +22,43 @@ check-command: (optional)
 	run time.
 	It defaults to "sparse $file".
 
-check-exit-value: (optional)
+check-exit-value: <value>
 	The expected exit value of check-command. It defaults to 0.
 
-check-timeout: (optional)
+check-timeout: <timeout>
 	The maximum expected duration of check-command, in seconds.
 	It defaults to 1.
 
-check-output-start / check-output-end (optional)
+check-output-start / check-output-end
 	The expected output (stdout and stderr) of check-command lies between
 	those two tags. It defaults to no output.
 
-check-output-ignore / check-error-ignore (optional)
+check-output-ignore / check-error-ignore
 	Don't check the expected output (stdout or stderr) of check-command
 	(usefull when this output is not comparable or if you're only interested
 	in the exit value).
 	By default this check is done.
 
-check-known-to-fail (optional)
+check-known-to-fail
 	Mark the test as being known to fail.
 
-check-output-contains: <pattern> (optional)
+check-output-contains: <pattern>
 	Check that the output (stdout) contains the given pattern.
 	Several such tags can be given, in which case the output
 	must contains all the patterns.
 
-check-output-excludes: <pattern> (optional)
+check-output-excludes: <pattern>
 	Similar than the above one, but with opposite logic.
 	Check that the output (stdout) doesn't contain the given pattern.
 	Several such tags can be given, in which case the output
 	must contains none of the patterns.
 
-check-output-pattern-<nbr>-times: <pattern> (optional)
+check-output-pattern-<nbr>-times: <pattern>
 	Similar than the contains/excludes her above, but with full control
 	of the number of times the pattern should occurs in the output.
 
-check-output-pattern(eq,<nbr>): <pattern> (optional)
-check-output-pattern(<min>,<max>): <pattern> (optional)
+check-output-pattern(eq,<nbr>): <pattern>
+check-output-pattern(<min>,<max>): <pattern>
 	Similar than the contains/excludes her above, but with full control
 	of the number of times the pattern should occurs in the output.
 	If <min> or <max> is '-' the corresponding check is ignored.
-- 
2.14.0

[PATCH 19/28] testsuite: allow arch-specific tests

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 Documentation/test-suite |  5 +++++
 validation/test-suite    | 18 ++++++++++++++++++
 2 files changed, 23 insertions(+)

diff --git a/Documentation/test-suite b/Documentation/test-suite
index a745da850..3c07100e7 100644
--- a/Documentation/test-suite
+++ b/Documentation/test-suite
@@ -22,6 +22,11 @@ check-command: <command arg ...>
 	run time.
 	It defaults to "sparse $file".
 
+check-arch-ignore: <arch[|...]>
+check-arch-only: <arch[|...]>
+	Ignore the test if the current architecture (as returned by 'uname -m')
+	match or not one of the archs given in the pattern.
+
 check-exit-value: <value>
 	The expected exit value of check-command. It defaults to 0.
 
diff --git a/validation/test-suite b/validation/test-suite
index 84e359fb8..746c0fb3a 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -56,6 +56,8 @@ get_tag_value()
 	check_output_excludes=0
 	check_output_pattern=0
 	check_output_pattern_minmax=0
+	check_arch_ignore=""
+	check_arch_only=""
 
 	lines=$(grep 'check-[a-z-]*' $1 | \
 		sed -e 's/^.*\(check-[a-z-]*:*\) *\(.*\)$/\1 \2/')
@@ -76,6 +78,10 @@ get_tag_value()
 		check-output-excludes:)	check_output_excludes=1 ;;
 		check-output-pattern-)	check_output_pattern=1 ;;
 		check-output-pattern)	check_output_pattern_minmax=1 ;;
+		check-arch-ignore:)	arch=$(uname -m)
+					check_arch_ignore="$val" ;;
+		check-arch-only:)	arch=$(uname -m)
+					check_arch_only="$val" ;;
 		esac
 	done << EOT
 	$lines
@@ -271,6 +277,18 @@ do_test()
 			return 3
 		fi
 	done
+	if [ "$check_arch_ignore" != "" ]; then
+		if echo $arch | egrep -q -w "$check_arch_ignore"; then
+			disable "$test_name" "$file"
+			return 3
+		fi
+	fi
+	if [ "$check_arch_only" != "" ]; then
+		if ! (echo $arch | egrep -q -w "$check_arch_only"); then
+			disable "$test_name" "$file"
+			return 3
+		fi
+	fi
 
 	cmd=`eval echo $default_path/$check_command`
 
-- 
2.14.0

[PATCH 20/28] testsuite: save screen real estate

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 746c0fb3a..f2e1c3242 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -233,7 +233,7 @@ disable()
 {
 	disabled_tests=$(($disabled_tests + 1))
 	if [ -z "$vquiet" ]; then
-		echo "     DISABLE $1 ($2)"
+		echo "  SKIP	$1 ($2)"
 	fi
 }
 
@@ -293,7 +293,7 @@ do_test()
 	cmd=`eval echo $default_path/$check_command`
 
 	if [ -z "$vquiet" ]; then
-		echo "     TEST    $test_name ($file)"
+		echo "  TEST	$test_name ($file)"
 	fi
 
 	verbose "Using command       : $cmd"
-- 
2.14.0

[PATCH 21/28] testsuite: add a blank line before format

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 1 +
 1 file changed, 1 insertion(+)

diff --git a/validation/test-suite b/validation/test-suite
index f2e1c3242..96eb90581 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -433,6 +433,7 @@ do_format()
 	$cmd 1> $file.output.got 2> $file.error.got
 	fexit_value=$?
 	cat <<_EOF
+
 /*
  * check-name: $fname
 _EOF
-- 
2.14.0

[PATCH 22/28] testsuite: 'quiet' must be initialized earlier

[Luc Van Oostenryck]

as it can, for example, be needed via arg_file() -> error()

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 1 +
 1 file changed, 1 insertion(+)

diff --git a/validation/test-suite b/validation/test-suite
index 96eb90581..1ef18f288 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -39,6 +39,7 @@ known_ko_tests=0
 # defaults to not verbose
 [ -z "$V" ] && V=0
 vquiet=""
+quiet=0
 abort=0
 
 ##
-- 
2.14.0

[PATCH 23/28] testsuite: move up arg_file()

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 32 +++++++++++++++++---------------
 1 file changed, 17 insertions(+), 15 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 1ef18f288..35138cb37 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -212,6 +212,23 @@ minmax_patterns()
 	return $?
 }
 
+##
+# arg_file(filename) - checks if filename exists
+arg_file()
+{
+	[ -z "$1" ] && {
+		do_usage
+		exit 1
+	}
+	[ -e "$1" ] || {
+		error "Can't open file $1"
+		exit 1
+	}
+	return 0
+}
+
+
+##
 do_usage()
 {
 echo "$prog_name - a tiny automatic testing script"
@@ -456,21 +473,6 @@ _EOF
 	return 0
 }
 
-##
-# arg_file(filename) - checks if filename exists
-arg_file()
-{
-	[ -z "$1" ] && {
-		do_usage
-		exit 1
-	}
-	[ -e "$1" ] || {
-		error "Can't open file $1"
-		exit 1
-	}
-	return 0
-}

[PATCH 24/28] testsuite: make do_format() more self-contained

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 30 +++++++++++++++++++++++++++---
 1 file changed, 27 insertions(+), 3 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 35138cb37..b4ba39f3f 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -244,7 +244,7 @@ echo "    file ...                   runs the test suite on the given file(s)"
 echo "    single file                runs the test in 'file'"
 echo "    format file [name [cmd]]   helps writing a new test case using cmd"
 echo
-echo "    help                       prints usage"
+echo "    [command] help             prints usage"
 }
 
 disable()
@@ -432,10 +432,34 @@ do_test_suite()
 	fi
 }
 
+##
+do_format_help() {
+echo "Usage: $prog_name [--]format file [name [cmd]]"
+echo
+echo "argument(s):"
+echo "    file                       file containing the test case(s)"
+echo "    name                       name for the test case (defaults to file)"
+echo "    cmd                        command to be used (defaults to 'sparse \$file')"
+}
+
 ##
 # do_format(file[, name[, cmd]]) - helps a test writer to format test-suite tags
 do_format()
 {
+	while [ $# -gt 1 ] ; do
+		case "$1" in
+		help|-*)
+			do_format_help
+			return 0
+			;;
+		*)	break ;;
+		esac
+		shift
+		continue
+	done
+
+	arg_file "$1" || return 1
+
 	if [ -z "$2" ]; then
 		fname="$1"
 		fcmd=$default_cmd
@@ -505,8 +529,8 @@ while true; do
 		esac
 		;;
 	format|--format)
-		arg_file "$2"
-		do_format "$2" "$3" "$4"
+		shift
+		do_format "$@"
 		;;
 	help | *)
 		do_usage
-- 
2.14.0

[PATCH 25/28] testsuite: format: saner defaults handling

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index b4ba39f3f..599f5e7e6 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -460,17 +460,12 @@ do_format()
 
 	arg_file "$1" || return 1
 
-	if [ -z "$2" ]; then
-		fname="$1"
-		fcmd=$default_cmd
-	elif [ -z "$3" ]; then
-		fname="$2"
-		fcmd=$default_cmd
-	else
-		fname="$2"
-		fcmd="$3"
-	fi
 	file="$1"
+	fname="$2"
+	[ -z "$fname" ] && fname="$1"
+	fcmd="$3"
+	[ -z "$fcmd" ] && fcmd=$default_cmd
+
 	cmd=`eval echo $default_path/$fcmd`
 	$cmd 1> $file.output.got 2> $file.error.got
 	fexit_value=$?
-- 
2.14.0

[PATCH 26/28] testsuite: format: strip .c from default name

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/validation/test-suite b/validation/test-suite
index 599f5e7e6..5821f4428 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -462,7 +462,7 @@ do_format()
 
 	file="$1"
 	fname="$2"
-	[ -z "$fname" ] && fname="$1"
+	[ -z "$fname" ] && fname="$(basename "$1" .c)"
 	fcmd="$3"
 	[ -z "$fcmd" ] && fcmd=$default_cmd
 
-- 
2.14.0

[PATCH 27/28] testsuite: add support for 'format -f'

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 5821f4428..b5bd937d5 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -434,7 +434,10 @@ do_test_suite()
 
 ##
 do_format_help() {
-echo "Usage: $prog_name [--]format file [name [cmd]]"
+echo "Usage: $prog_name [option(s)] [--]format file [name [cmd]]"
+echo
+echo "options:"
+echo "    -f                         write a test known to fail"
 echo
 echo "argument(s):"
 echo "    file                       file containing the test case(s)"
@@ -443,11 +446,15 @@ echo "    cmd                        command to be used (defaults to 'sparse \$f
 }
 
 ##
-# do_format(file[, name[, cmd]]) - helps a test writer to format test-suite tags
+# do_format([options,] file[, name[, cmd]]) - helps a test writer to format test-suite tags
 do_format()
 {
+	fail=0
+
 	while [ $# -gt 1 ] ; do
 		case "$1" in
+		-f)
+			fail=1 ;;
 		help|-*)
 			do_format_help
 			return 0
@@ -480,6 +487,9 @@ _EOF
 	if [ "$fexit_value" -ne "0" ]; then
 		echo " * check-exit-value: $fexit_value"
 	fi
+	if [ $fail != 0 ]; then
+		echo " * check-known-to-fail"
+	fi
 	for stream in output error; do
 		if [ -s "$file.$stream.got" ]; then
 			echo " *"
-- 
2.14.0

[PATCH 28/28] testsuite: add support for 'format -l'

[Luc Van Oostenryck]

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/validation/test-suite b/validation/test-suite
index b5bd937d5..8cbfb94e9 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -438,6 +438,7 @@ echo "Usage: $prog_name [option(s)] [--]format file [name [cmd]]"
 echo
 echo "options:"
 echo "    -f                         write a test known to fail"
+echo "    -l                         write a test for linearized code"
 echo
 echo "argument(s):"
 echo "    file                       file containing the test case(s)"
@@ -449,12 +450,17 @@ echo "    cmd                        command to be used (defaults to 'sparse \$f
 # do_format([options,] file[, name[, cmd]]) - helps a test writer to format test-suite tags
 do_format()
 {
+	def_cmd="$default_cmd"
+	linear=0
 	fail=0
 
 	while [ $# -gt 1 ] ; do
 		case "$1" in
 		-f)
 			fail=1 ;;
+		-l)
+			def_cmd='test-linearize -Wno-decl $file'
+			linear=1 ;;
 		help|-*)
 			do_format_help
 			return 0
@@ -471,7 +477,7 @@ do_format()
 	fname="$2"
 	[ -z "$fname" ] && fname="$(basename "$1" .c)"
 	fcmd="$3"
-	[ -z "$fcmd" ] && fcmd=$default_cmd
+	[ -z "$fcmd" ] && fcmd="$def_cmd"
 
 	cmd=`eval echo $default_path/$fcmd`
 	$cmd 1> $file.output.got 2> $file.error.got
@@ -490,6 +496,12 @@ _EOF
 	if [ $fail != 0 ]; then
 		echo " * check-known-to-fail"
 	fi
+	if [ $linear != 0 ]; then
+		echo " *"
+		echo " * check-output-ignore"
+		echo " * check-output-contains: xyz\\\\."
+		echo " * check-output-excludes: \\\\."
+	fi
 	for stream in output error; do
 		if [ -s "$file.$stream.got" ]; then
 			echo " *"
-- 
2.14.0

Re: [PATCH 00/28] testsuite improvements

[Christopher Li]

On Fri, Sep 15, 2017 at 3:13 AM, Luc Van Oostenryck
<luc.vanoostenryck@gmail.com> wrote:
> This series contains diverse small improvements to
> the testsuite.
>

Thanks for the patch. It will keep me busy for a little while.

I am might not able to finish reading it today.

BTW, patchworks seems not updating. I haven't able to fetch
yours and some other  patch on the sparse mailing list.

https://patchwork.kernel.org/project/linux-sparse/list/

Not a big deal. I can use your git tree branch.

Chris

Re: [PATCH 00/28] testsuite improvements

[Luc Van Oostenryck]

On Fri, Sep 15, 2017 at 7:03 PM, Christopher Li <sparse@chrisli.org> wrote:
> On Fri, Sep 15, 2017 at 3:13 AM, Luc Van Oostenryck
> <luc.vanoostenryck@gmail.com> wrote:
>> This series contains diverse small improvements to
>> the testsuite.
>>
>
> Thanks for the patch. It will keep me busy for a little while.
>
> I am might not able to finish reading it today.
>
> BTW, patchworks seems not updating. I haven't able to fetch
> yours and some other  patch on the sparse mailing list.
>
> https://patchwork.kernel.org/project/linux-sparse/list/

Yes, I saw that too.

-- Luc

Re: [PATCH 00/28] testsuite improvements

[Christopher Li]

On Fri, Sep 15, 2017 at 3:13 AM, Luc Van Oostenryck
<luc.vanoostenryck@gmail.com> wrote:
> This series contains diverse small improvements to
> the testsuite.
>
>
> The series is also available for review in the git repository at:
>   git://github.com/lucvoo/sparse.git testsuite-clean

I take a brief look at this series. It seems fine to me.

I will wait for the git pull for it then.

Chris