[PATCH 00/16] testsuite improvements

[PATCH 00/16] testsuite improvements

[Luc Van Oostenryck]

This series contains a few fixes & improvements to the
testsuite; mainly:
- allow to run the testsuite on all the tests of a subdir
- teach 'format' to directly append to the testcase
- validate the 'check-...' tags

Luc Van Oostenryck (16):
  testsuite: make the '%.t' rule depends on PROGRAMS too
  testsuite: fix a few more incorrect check-commands
  testsuite: convert to the new pattern syntax
  testsuite: remove old ugly pattern syntax
  testsuite: move verbose/error() before get_tag_value()
  testsuite: add & use warning()
  testsuite: reset 'quiet' at the start of each testcase
  testsuite: fix invalid 'check-...' tags
  testsuite: validate the 'check-...' tags
  testsuite: early return in getopt loop
  testsuite: move do_test_suite out of the getopt loop
  testsuite: move no-arg out of the getopt loop
  testsuite: change do_usage text
  testsuite: allow to test only a subdir
  testsuite: default to shift in the getopt loop
  testsuite: add support for 'format -a'

 Makefile                          |   2 +-
 validation/cond-address.c         |   2 +-
 validation/cond-err-expand.c      |   2 +-
 validation/fp-ops.c               |   2 +-
 validation/fp2i-cast.c            |   2 +-
 validation/nested-declarator.c    |   4 +-
 validation/nested-declarator2.c   |   4 +-
 validation/optim/canonical-cmp.c  |   2 +-
 validation/optim/canonical-fcmp.c |   2 +-
 validation/option-parsing-00.c    |   2 +-
 validation/option-parsing-01.c    |   2 +-
 validation/reserved.c             |   4 +-
 validation/test-suite             | 149 +++++++++++++++++++-------------------
 validation/typedef_shadow.c       |   4 +-
 14 files changed, 92 insertions(+), 91 deletions(-)

-- 
2.15.0

[PATCH 01/16] testsuite: make the '%.t' rule depends on PROGRAMS too

[Luc Van Oostenryck]

The testsuite can be run on a specific testcase directly
via the Makefile via a 'validation/%.t' pattern rule but
this rule had no dependency on the programs being tested
and thus could be run on a previous version.

Fix this by adding the needed dependencies.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 1ae0a33f0..dab38aacd 100644
--- a/Makefile
+++ b/Makefile
@@ -214,7 +214,7 @@ version.h: FORCE
 
 check: all
 	$(Q)cd validation && ./test-suite
-validation/%.t: FORCE
+validation/%.t: $(PROGRAMS)
 	@validation/test-suite single $*.c
 
 
-- 
2.15.0

[PATCH 02/16] testsuite: fix a few more incorrect check-commands

[Luc Van Oostenryck]

The exact syntax for commands is:
	'check-command: ' <command> <args>...
and the command itself must *not* be prefixed with './'.

Fix the last three that had it wrong.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/fp-ops.c            | 2 +-
 validation/option-parsing-00.c | 2 +-
 validation/option-parsing-01.c | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/validation/fp-ops.c b/validation/fp-ops.c
index 7f58a72fc..71cb9848b 100644
--- a/validation/fp-ops.c
+++ b/validation/fp-ops.c
@@ -7,7 +7,7 @@ _Bool  ftst(double x)           { return !x; }
 
 /*
  * check-name: floating-point ops
- * check-command: ./test-linearize -Wno-decl $file
+ * check-command: test-linearize -Wno-decl $file
 
  * check-output-start
 fadd:
diff --git a/validation/option-parsing-00.c b/validation/option-parsing-00.c
index 9b85943c1..9dceab7fa 100644
--- a/validation/option-parsing-00.c
+++ b/validation/option-parsing-00.c
@@ -1,5 +1,5 @@
 
 /*
  * check-name: option parsing 00
- * check-command sparse -foptimize-xyz $file
+ * check-command: sparse -foptimize-xyz $file
  */
diff --git a/validation/option-parsing-01.c b/validation/option-parsing-01.c
index e33a2ef0b..a2875bd15 100644
--- a/validation/option-parsing-01.c
+++ b/validation/option-parsing-01.c
@@ -1,5 +1,5 @@
 
 /*
  * check-name: option parsing 01
- * check-command sparse -fno-optimize-xyz $file
+ * check-command: sparse -fno-optimize-xyz $file
  */
-- 
2.15.0

[PATCH 03/16] testsuite: convert to the new pattern syntax

[Luc Van Oostenryck]

The old one is too ugly and has to die.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/fp2i-cast.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/validation/fp2i-cast.c b/validation/fp2i-cast.c
index 08f8c9252..98d7453e5 100644
--- a/validation/fp2i-cast.c
+++ b/validation/fp2i-cast.c
@@ -26,5 +26,5 @@ ul d2ul(double a) { return a; }
  * check-command: test-linearize -Wno-decl $file
  *
  * check-output-ignore
- * check-output-pattern-8-times: cast\\.
+ * check-output-pattern(8): cast\\.
  */
-- 
2.15.0

[PATCH 04/16] testsuite: remove old ugly pattern syntax

[Luc Van Oostenryck]

It was too ugly (and a bit longish).

Remove it.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 32 --------------------------------
 1 file changed, 32 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 903a2b908..07d9e7fc9 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -55,7 +55,6 @@ get_tag_value()
 	check_output_ignore=0
 	check_output_contains=0
 	check_output_excludes=0
-	check_output_pattern_obsolete=0
 	check_output_pattern=0
 	check_arch_ignore=""
 	check_arch_only=""
@@ -77,7 +76,6 @@ get_tag_value()
 		check-output-ignore)	check_output_ignore=1 ;;
 		check-output-contains:)	check_output_contains=1 ;;
 		check-output-excludes:)	check_output_excludes=1 ;;
-		check-output-pattern-)	check_output_pattern_obsolete=1 ;;
 		check-output-pattern)	check_output_pattern=1 ;;
 		check-arch-ignore:)	arch=$(uname -m)
 					check_arch_ignore="$val" ;;
@@ -149,29 +147,6 @@ has_none_patterns()
 	has_patterns "$1" "$2" "$4" -eq "$3"
 }
 
-##
-# nbr_patterns(ifile tag ofile) - does ofile contains the
-#                        the patterns given by ifile's tags
-#                        the right number of time?
-nbr_patterns()
-{
-	ifile="$1"
-	patt="$2"
-	ofile="$3"
-	grep "$patt-[0-9][0-9]*-times:" "$ifile" | \
-	sed -e "s/^.*$patt-\([0-9][0-9]*\)-times: *\(.*\)/\1 \2/" | \
-	while read nbr pat; do
-		n=$(grep -s "$pat" "$ofile" | wc -l)
-		if [ "$n" -ne "$nbr" ]; then
-			error "test '$ifile' failed"
-			error "	Pattern '$pat' expected $nbr times but got $n times"
-			return 1
-		fi
-	done
-
-	return $?
-}

[PATCH 05/16] testsuite: move verbose/error() before get_tag_value()

[Luc Van Oostenryck]

So, we can use them inside get_tag_value().

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 36 +++++++++++++++++++-----------------
 1 file changed, 19 insertions(+), 17 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 07d9e7fc9..f04d81779 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -42,6 +42,25 @@ vquiet=""
 quiet=0
 abort=0
 
+
+##
+# verbose(string) - prints string if we are in verbose mode
+verbose()
+{
+	[ "$V" -eq "1" ] && echo "        $1"
+	return 0
+}
+
+##
+# error(string[, die]) - prints an error and exits with value die if given
+error()
+{
+	[ "$quiet" -ne 1 ] && echo "error: $1"
+	[ -n "$2" ] && exit $2
+	return 0
+}
+
+
 ##
 # get_tag_value(file) - get the 'check-<...>' tags & values
 get_tag_value()
@@ -87,23 +106,6 @@ get_tag_value()
 EOT
 }
 
-##
-# verbose(string) - prints string if we are in verbose mode
-verbose()
-{
-	[ "$V" -eq "1" ] && echo "        $1"
-	return 0
-}
-
-##
-# error(string[, die]) - prints an error and exits with value die if given
-error()
-{
-	[ "$quiet" -ne 1 ] && echo "error: $1"
-	[ -n "$2" ] && exit $2
-	return 0
-}

[PATCH 06/16] testsuite: add & use warning()

[Luc Van Oostenryck]

Allow this new helper to indicate wich file trigger the
warning and replace the existing call to 'echo "warning: ...'.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/validation/test-suite b/validation/test-suite
index f04d81779..420214c65 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -51,6 +51,14 @@ verbose()
 	return 0
 }
 
+##
+# warning(string) - prints a warning
+warning()
+{
+	[ "$quiet" -ne 1 ] && echo "warning: $1"
+	return 0
+}
+
 ##
 # error(string[, die]) - prints an error and exits with value die if given
 error()
@@ -253,7 +261,7 @@ do_test()
 	# can this test be handled by test-suite ?
 	# (it has to have a check-name key in it)
 	if [ "$check_name" = "" ]; then
-		echo "warning: test '$file' unhandled"
+		warning "$file: test unhandled"
 		unhandled_tests=$(($unhandled_tests + 1))
 		return 2
 	fi
-- 
2.15.0

[PATCH 07/16] testsuite: reset 'quiet' at the start of each testcase

[Luc Van Oostenryck]

The flag 'quiet' is used to quiets unwanted error messages,
for example for testcases known to fail, but this flag is reset
too late so that the beginning of the next testcases will run
with the value for the previous case.

Fix this by reseting the flag at the begining of each testcase.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/validation/test-suite b/validation/test-suite
index 420214c65..6a939846f 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -255,6 +255,7 @@ do_test()
 {
 	test_failed=0
 	file="$1"
+	quiet=0
 
 	get_tag_value $file
 
@@ -316,7 +317,6 @@ do_test()
 	actual_exit_value=$?
 
 	must_fail=$check_known_to_fail
-	quiet=0
 	[ $must_fail -eq 1 ] && [ $V -eq 0 ] && quiet=1
 	known_ko_tests=$(($known_ko_tests + $must_fail))
 
-- 
2.15.0

[PATCH 08/16] testsuite: fix invalid 'check-...' tags

[Luc Van Oostenryck]

A few testcases had typos in their 'check-...' tags or
the tag was plainly invalid.

Fix them in accordance to the doc.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/cond-address.c         | 2 +-
 validation/cond-err-expand.c      | 2 +-
 validation/nested-declarator.c    | 4 ++--
 validation/nested-declarator2.c   | 4 ++--
 validation/optim/canonical-cmp.c  | 2 +-
 validation/optim/canonical-fcmp.c | 2 +-
 validation/reserved.c             | 4 ++--
 validation/typedef_shadow.c       | 4 ++--
 8 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/validation/cond-address.c b/validation/cond-address.c
index 2a69f4b92..c9253ad04 100644
--- a/validation/cond-address.c
+++ b/validation/cond-address.c
@@ -10,5 +10,5 @@ int qux(void) { if (f && a) return 1; return 0; }
  * check-command: test-linearize -Wno-decl $file
  * check-output-ignore
  *
- * check-excludes: VOID
+ * check-output-excludes: VOID
  */
diff --git a/validation/cond-err-expand.c b/validation/cond-err-expand.c
index 93bbac153..7936a2d00 100644
--- a/validation/cond-err-expand.c
+++ b/validation/cond-err-expand.c
@@ -23,5 +23,5 @@ cond-err-expand.c:13:11: error: incompatible types in conditional expression (di
  * check-error-end
  *
  * check-output-ignore
- * check-excludes: call.* __builtin_constant_p
+ * check-output-excludes: call.* __builtin_constant_p
  */
diff --git a/validation/nested-declarator.c b/validation/nested-declarator.c
index 1efe20ce4..f258d457b 100644
--- a/validation/nested-declarator.c
+++ b/validation/nested-declarator.c
@@ -15,7 +15,7 @@ int i(void (void)(*f));
 int j(int [2](*));
 /*
  * check-name: nested declarator vs. parameters
- * check-error-start:
+ * check-error-start
 nested-declarator.c:11:23: warning: missing identifier in declaration
 nested-declarator.c:11:23: error: Expected ; at the end of type declaration
 nested-declarator.c:11:23: error: got (
@@ -25,5 +25,5 @@ nested-declarator.c:14:18: error: Expected ) in function declarator
 nested-declarator.c:14:18: error: got (
 nested-declarator.c:15:14: error: Expected ) in function declarator
 nested-declarator.c:15:14: error: got (
- * check-error-end:
+ * check-error-end
  */
diff --git a/validation/nested-declarator2.c b/validation/nested-declarator2.c
index 345a04b06..983b7ef5c 100644
--- a/validation/nested-declarator2.c
+++ b/validation/nested-declarator2.c
@@ -27,7 +27,7 @@ static int (-bad2);
 static void [2](*bad3);
 /*
  * check-name: more on handling of ( in direct-declarator
- * check-error-start:
+ * check-error-start
 nested-declarator2.c:17:1: warning: non-ANSI definition of function 'w1'
 nested-declarator2.c:21:21: warning: non-ANSI function declaration of function '<noident>'
 nested-declarator2.c:22:16: warning: variadic functions must have one named argument
@@ -37,5 +37,5 @@ nested-declarator2.c:26:13: error: Expected ) in nested declarator
 nested-declarator2.c:26:13: error: got -
 nested-declarator2.c:27:16: error: Expected ; at the end of type declaration
 nested-declarator2.c:27:16: error: got (
- * check-error-end:
+ * check-error-end
  */
diff --git a/validation/optim/canonical-cmp.c b/validation/optim/canonical-cmp.c
index 19b416310..6f3b5180e 100644
--- a/validation/optim/canonical-cmp.c
+++ b/validation/optim/canonical-cmp.c
@@ -21,7 +21,7 @@ uint uat(uint p, uint a) { return (123 <  p) ? a : 0; }
  * check-name: canonical-cmp
  * check-command: test-linearize -Wno-decl $file
  *
- * check-output-exclude: \$123,
+ * check-output-excludes: \$123,
  *
  * check-output-start
 seq:
diff --git a/validation/optim/canonical-fcmp.c b/validation/optim/canonical-fcmp.c
index e3e758a9b..d098da64f 100644
--- a/validation/optim/canonical-fcmp.c
+++ b/validation/optim/canonical-fcmp.c
@@ -20,7 +20,7 @@ int nfcmp_ge(double a) { return !(g <  a); }
  * check-name: canonical-cmp
  * check-command: test-linearize -Wno-decl $file
  *
- * check-output-exclude: \$123,
+ * check-output-excludes: \$123,
  *
  * check-output-start
 fcmp_eq:
diff --git a/validation/reserved.c b/validation/reserved.c
index 29554560b..6a2163e9a 100644
--- a/validation/reserved.c
+++ b/validation/reserved.c
@@ -81,7 +81,7 @@ static int (__builtin_va_list);
 
 /*
  * check-name: const et.al. are reserved identifiers
- * check-error-start:
+ * check-error-start
 reserved.c:1:12: error: Trying to use reserved word 'auto' as identifier
 reserved.c:2:12: error: Trying to use reserved word 'break' as identifier
 reserved.c:3:12: error: Trying to use reserved word 'case' as identifier
@@ -154,5 +154,5 @@ reserved.c:77:12: error: Trying to use reserved word '__builtin_ms_va_list' as i
 reserved.c:78:12: error: Trying to use reserved word '__builtin_offsetof' as identifier
 reserved.c:79:12: error: Trying to use reserved word '__builtin_types_compatible_p' as identifier
 reserved.c:80:12: error: Trying to use reserved word '__builtin_va_list' as identifier
- * check-error-end:
+ * check-error-end
  */
diff --git a/validation/typedef_shadow.c b/validation/typedef_shadow.c
index e52de80f2..e8e4c229b 100644
--- a/validation/typedef_shadow.c
+++ b/validation/typedef_shadow.c
@@ -5,9 +5,9 @@ static void f(int T)
 }
 /*
  * check-name: typedef shadowing
- * check-error-start:
+ * check-error-start
 typedef_shadow.c:4:16: warning: 'T' has implicit type
 typedef_shadow.c:4:18: error: Expected ; at end of declaration
 typedef_shadow.c:4:18: error: got a
- * check-error-end:
+ * check-error-end
  */
-- 
2.15.0

[PATCH 09/16] testsuite: validate the 'check-...' tags

[Luc Van Oostenryck]

Making a typo in one of the 'check-...' tags can make
a testcase useless and thus incapable of detecting a regression.

Add some validation to these tags in order to detect wrong
tags.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/validation/test-suite b/validation/test-suite
index 6a939846f..d50a3bfa2 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -108,6 +108,18 @@ get_tag_value()
 					check_arch_ignore="$val" ;;
 		check-arch-only:)	arch=$(uname -m)
 					check_arch_only="$val" ;;
+
+		check-description:)	;;	# ignore
+		check-note:)		;;	# ignore
+		check-warning:)		;;	# ignore
+		check-error-start)	;;	# ignore
+		check-error-end)	;;	# ignore
+		check-output-start)	;;	# ignore
+		check-output-end)	;;	# ignore
+		check-should-pass)	;;	# ignore, unused annotation
+		check-should-fail)	;;	# ignore, unused annotation
+		check-should-warn)	;;	# ignore, unused annotation
+		check-*)		error "$1: unknown tag '$tag'" 1 ;;
 		esac
 	done << EOT
 	$lines
-- 
2.15.0

[PATCH 10/16] testsuite: early return in getopt loop

[Luc Van Oostenryck]

This is a preparatory step to allow to run only a part
of the testsuite (a subdir).

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/validation/test-suite b/validation/test-suite
index d50a3bfa2..c16c7c843 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -521,10 +521,12 @@ while true; do
 	'')
 		tests_list=`find . -name '*.c' | sed -e 's#^\./\(.*\)#\1#' | sort`
 		do_test_suite
+		break
 		;;
 	*.c)
 		tests_list="$@"
 		do_test_suite
+		break
 		;;
 
 	single|--single)
@@ -535,10 +537,12 @@ while true; do
 			1) echo "$2 failed !";;
 			2) echo "$2 can't be handled by $prog_name";;
 		esac
+		exit $failed
 		;;
 	format|--format)
 		shift
 		do_format "$@"
+		exit 0
 		;;
 	help | *)
 		do_usage
-- 
2.15.0

[PATCH 11/16] testsuite: move do_test_suite out of the getopt loop

[Luc Van Oostenryck]

This is a preparatory step to allow to run only a part
of the testsuite (a subdir).

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index c16c7c843..72bee9dce 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -520,12 +520,10 @@ while true; do
 
 	'')
 		tests_list=`find . -name '*.c' | sed -e 's#^\./\(.*\)#\1#' | sort`
-		do_test_suite
 		break
 		;;
 	*.c)
 		tests_list="$@"
-		do_test_suite
 		break
 		;;
 
@@ -552,5 +550,6 @@ while true; do
 	break
 done
 
+do_test_suite
 exit $failed
 
-- 
2.15.0

[PATCH 12/16] testsuite: move no-arg out of the getopt loop

[Luc Van Oostenryck]

This is a preparatory step to allow to run only a part
of the testsuite (a subdir).

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 72bee9dce..410190bd3 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -505,7 +505,7 @@ _EOF
 	return 0
 }
 
-while true; do
+while [ "$#" -gt "0" ]; do
 	case "$1" in
 	-a|--abort)
 		abort=1
@@ -517,11 +517,6 @@ while true; do
 		shift
 		continue
 		;;

[PATCH 13/16] testsuite: change do_usage text

[Luc Van Oostenryck]

The text for the testsuite usage used 'none' as if it was an
option/keyword while it only meant the absence of arguments.

Make the text clearer by removing the 'none' and being explicit
about the absence of arguments.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 410190bd3..854ad8add 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -233,16 +233,16 @@ echo "$prog_name - a tiny automatic testing script"
 echo "Usage: $prog_name [option(s)] [command] [arguments]"
 echo
 echo "options:"
-echo "    -a|--abort                 abort the tests as soon as one fails"
-echo "    -q|--quiet                 be extra quiet while running the tests"
+echo "    -a|--abort                 Abort the tests as soon as one fails."
+echo "    -q|--quiet                 Be extra quiet while running the tests."
 echo
 echo "commands:"
-echo "    none                       runs the whole test suite"
-echo "    file ...                   runs the test suite on the given file(s)"
-echo "    single file                runs the test in 'file'"
-echo "    format file [name [cmd]]   helps writing a new test case using cmd"
+echo "    [file ...]                 Runs the test suite on the given file(s)."
+echo "                               If no file is given, run the whole testsuite."
+echo "    single file                Run the test in 'file'."
+echo "    format file [name [cmd]]   Help writing a new test case using cmd."
 echo
-echo "    [command] help             prints usage"
+echo "    [command] help             Print usage."
 }
 
 disable()
-- 
2.15.0

[PATCH 14/16] testsuite: allow to test only a subdir

[Luc Van Oostenryck]

During development, it is very useful to be able to run only
some of the tests, maybe a whole class.

Help this by allowing to run the testsuite on only a subdir
of the 'validation/' directory.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 854ad8add..098cc7869 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -238,6 +238,7 @@ echo "    -q|--quiet                 Be extra quiet while running the tests."
 echo
 echo "commands:"
 echo "    [file ...]                 Runs the test suite on the given file(s)."
+echo "                               If a directory is given, run only those files."
 echo "                               If no file is given, run the whole testsuite."
 echo "    single file                Run the test in 'file'."
 echo "    format file [name [cmd]]   Help writing a new test case using cmd."
@@ -517,10 +518,6 @@ while [ "$#" -gt "0" ]; do
 		shift
 		continue
 		;;
-	*.c)
-		tests_list="$@"
-		break
-		;;
 
 	single|--single)
 		arg_file "$2"
@@ -537,10 +534,25 @@ while [ "$#" -gt "0" ]; do
 		do_format "$@"
 		exit 0
 		;;
-	help | *)
+	help)
 		do_usage
 		exit 1
 		;;
+
+	*.c)
+		tests_list="$tests_list $1"
+		shift
+		continue
+		;;
+	*)
+		if [ ! -d "$1" ]; then
+			do_usage
+			exit 1
+		fi
+		tests_list="$tests_list $(find "$1" -name '*.c' | sort)"
+		shift
+		continue
+		;;
 	esac
 	break
 done
-- 
2.15.0

[PATCH 15/16] testsuite: default to shift in the getopt loop

[Luc Van Oostenryck]

The getopt loop used to bear by default and only some
options had to explicitly call 'shift' and 'continue'
to process further elements.

Change this to a 'normal' loop, shifting the next arg by default
and breaking of the loop when needed.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 098cc7869..8c0453590 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -510,13 +510,9 @@ while [ "$#" -gt "0" ]; do
 	case "$1" in
 	-a|--abort)
 		abort=1
-		shift
-		continue
 		;;
 	-q|--quiet)
 		vquiet=1
-		shift
-		continue
 		;;
 
 	single|--single)
@@ -541,8 +537,6 @@ while [ "$#" -gt "0" ]; do
 
 	*.c)
 		tests_list="$tests_list $1"
-		shift
-		continue
 		;;
 	*)
 		if [ ! -d "$1" ]; then
@@ -550,11 +544,9 @@ while [ "$#" -gt "0" ]; do
 			exit 1
 		fi
 		tests_list="$tests_list $(find "$1" -name '*.c' | sort)"
-		shift
-		continue
 		;;
 	esac
-	break
+	shift
 done
 
 if [ -z "$tests_list" ]; then
-- 
2.15.0

[PATCH 16/16] testsuite: add support for 'format -a'

[Luc Van Oostenryck]

The 'format' command create the information needed for the testcase
from the input file and output this on stdout. The developper must
then add this to the input file.

Let's do this automatically by adding an option '-a' to the 'format'
command to directly append the infos to the input file.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/validation/test-suite b/validation/test-suite
index 8c0453590..192fba300 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -429,6 +429,7 @@ do_format_help() {
 echo "Usage: $prog_name [option(s)] [--]format file [name [cmd]]"
 echo
 echo "options:"
+echo "    -a                         append the created test to the input file"
 echo "    -f                         write a test known to fail"
 echo "    -l                         write a test for linearized code"
 echo
@@ -443,11 +444,14 @@ echo "    cmd                        command to be used (defaults to 'sparse \$f
 do_format()
 {
 	def_cmd="$default_cmd"
+	append=0
 	linear=0
 	fail=0
 
 	while [ $# -gt 1 ] ; do
 		case "$1" in
+		-a)
+			append=1 ;;
 		-f)
 			fail=1 ;;
 		-l)
@@ -474,6 +478,7 @@ do_format()
 	cmd=`eval echo $default_path/$fcmd`
 	$cmd 1> $file.output.got 2> $file.error.got
 	fexit_value=$?
+	[ "append" != 0 ] && exec >> $file
 	cat <<_EOF
 
 /*
-- 
2.15.0

Re: [PATCH 00/16] testsuite improvements

[Christopher Li]

On Fri, Dec 8, 2017 at 9:14 PM, Luc Van Oostenryck
<luc.vanoostenryck@gmail.com> wrote:
> This series contains a few fixes & improvements to the
> testsuite; mainly:
> - allow to run the testsuite on all the tests of a subdir
> - teach 'format' to directly append to the testcase
> - validate the 'check-...' tags
>
> Luc Van Oostenryck (16):
>   testsuite: make the '%.t' rule depends on PROGRAMS too
>   testsuite: fix a few more incorrect check-commands
>   testsuite: convert to the new pattern syntax
>   testsuite: remove old ugly pattern syntax
>   testsuite: move verbose/error() before get_tag_value()
>   testsuite: add & use warning()
>   testsuite: reset 'quiet' at the start of each testcase
>   testsuite: fix invalid 'check-...' tags
>   testsuite: validate the 'check-...' tags
>   testsuite: early return in getopt loop
>   testsuite: move do_test_suite out of the getopt loop
>   testsuite: move no-arg out of the getopt loop
>   testsuite: change do_usage text
>   testsuite: allow to test only a subdir
>   testsuite: default to shift in the getopt loop
>   testsuite: add support for 'format -a'

I haven't have a chance to do the full review as I did before
applying them. I just briefly go over the patches. From the
distance I look, the series looks fine for me.

Usually as long as the make check was not broken,
I am not too worry about the testsuite changes.

Chris

Re: [PATCH 00/16] testsuite improvements

[Luc Van Oostenryck]

On Sat, Dec 09, 2017 at 10:09:05AM +0800, Christopher Li wrote:
> 
> I haven't have a chance to do the full review as I did before
> applying them. I just briefly go over the patches. From the
> distance I look, the series looks fine for me.
> 
> Usually as long as the make check was not broken,
> I am not too worry about the testsuite changes.
> 
> Chris

Thanks,

This has now been pushed to:
	git://github.com/lucvoo/sparse.git


-- Luc Van Oostenryck