Re: [PATCH 6/6] Add a simple test script, embed expected results into test files

["Damien Lespiau" <damien.lespiau@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1386 bytes --]

A small update,

> Yes, but I still prefer the delimited format for readability.

Ok: expected output is now between check-output-start / check-output-end
tags.
I've added a clean-check target in the Makefile, updated the provided
test-cases and documentation.

> You can't redirect two things independently to the same file; that will
> open the file twice, and the writes will conflict, giving exactly the
> result you saw.  > o 2>&1 should work; it has exactly the same effect as
> &> .

For some reason I was expecting that:

[pid  8877] open("o", O_WRONLY|O_CREAT|O_TRUNC| \
        O_LARGEFILE, 0666) = 3
[pid  8877] dup2(3, 1)                  = 1
[pid  8877] close(3)                    = 0
[pid  8877] open("o", O_WRONLY|O_CREAT|O_TRUNC| \
        O_LARGEFILE, 0666) = 3
[pid  8877] dup2(3, 2)                  = 2
[pid  8877] close(3)                    = 0

and

[pid 10777] open("o", O_WRONLY|O_CREAT|O_TRUNC| \
        O_LARGEFILE, 0666) = 3
[pid 10777] dup2(3, 1)                  = 1
[pid 10777] close(3)                    = 0
[pid 10777] dup2(1, 2)                  = 2

followed by:

[pid  xxxxx] write(2, "validation/preprocessor19.c:4:9:"..., 73) = 73
[pid  xxxxx] write(2, "validation/preprocessor19.c:3:9:"..., 66) = 66
[pid  xxxxx] write(1, "\ny\n", 3)        = o

to give the same result and that the first one would not eat characters.
I was wrong.

-- 
Damien

[-- Attachment #2: 0001-test-suite-a-tiny-test-automation-script.patch --]
[-- Type: text/x-patch, Size: 6857 bytes --]

From 4977f1c73fcb53d260facbee0be60c78bdb7af76 Mon Sep 17 00:00:00 2001
From: Damien Lespiau <damien.lespiau@gmail.com>
Date: Mon, 2 Jul 2007 06:41:34 +0200
Subject: [PATCH] test-suite: a tiny test automation script

This patch introduces test-suite, a simple script that makes test cases
verification easier. Test cases in the validation directory are annotated
and this script parses them to check if the actual result is the one
expected by the test writer.

Signed-off-by: Damien Lespiau <damien.lespiau@gmail.com>
---
 .gitignore            |    5 +
 Makefile              |    7 ++
 validation/test-suite |  252 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 264 insertions(+), 0 deletions(-)
 create mode 100755 validation/test-suite

diff --git a/.gitignore b/.gitignore
index e22a8c6..faa046b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -32,3 +32,8 @@ patches-*
 patches
 series
 
+# test-suite
+*.diff
+*.got
+*.expected
+
diff --git a/Makefile b/Makefile
index 039fe38..6bfb2ea 100644
--- a/Makefile
+++ b/Makefile
@@ -168,3 +168,10 @@ dist:
 		exit 1 ; \
 	fi
 	git archive --format=tar --prefix=sparse-$(VERSION)/ HEAD^{tree} | gzip -9 > sparse-$(VERSION).tar.gz
+
+check: all
+	$(Q)cd validation && ./test-suite
+
+clean-check:
+	find validation/ -name "*.c.[egd]*" -exec rm {} \;
+
diff --git a/validation/test-suite b/validation/test-suite
new file mode 100755
index 0000000..6b9d1b7
--- /dev/null
+++ b/validation/test-suite
@@ -0,0 +1,252 @@
+#!/bin/sh
+
+#set -x
+
+default_path=".."
+default_cmd="sparse \$file"
+tests_list=`find . -name '*.c' | sed -e 's#^\./\(.*\)#\1#' | sort`
+prog_name=`basename $0`
+
+# counts:
+#	- tests that have not been converted to test-suite format
+#	- tests that passed
+#	- tests that failed
+#	- tests that failed but are known to fail
+unhandled_tests=0
+ok_tests=0
+ko_tests=0
+known_ko_tests=0
+
+# defaults to not verbose
+[ -z "$V" ] && V=0
+
+##
+# get_value(key, file) - gets the value of a (key, value) pair in file.
+#
+# returns 0 on success, 1 if the file does not have the key
+get_value()
+{
+	last_result=`grep $1: $2 | sed -e "s/^.*$1:\(.*\)$/\1/"`
+	[ -z "$last_result" ] && return 1
+	return 0
+}
+
+##
+# get_tag(key, file) - does file has the tag key in it ?
+#
+# returns 0 if present, 1 otherwise
+get_tag()
+{
+	last_result=`grep $1 $2`
+	return $?
+}
+
+##
+# verbose(string) - prints string if we are in verbose mode
+verbose()
+{
+	[ "$V" -eq "1" ] && echo "        $1"
+	return 0
+}
+
+##
+# error(string[, die]) - prints an error and exits with value die if given
+error()
+{
+	echo "error: $1"
+	[ -n "$2" ] && exit $2
+	return 0
+}
+
+do_usage()
+{
+echo "$prog_name - a tiny automatic testing script"
+echo "Usage: $prog_name [command] [command arguments]"
+echo
+echo "commands:"
+echo "    none                       runs the whole test suite"
+echo "    single file                runs the test in 'file'"
+echo "    format file [name [cmd]]   helps writing a new test case using cmd"
+echo
+echo "    help                       prints usage"
+}
+
+##
+# do_test(file) - tries to validate a test case
+#
+# it "parses" file, looking for check-* tags and tries to validate
+# the test against an expected result
+# returns:
+#	- 0 if the test passed,
+#	- 1 if it failed,
+#	- 2 if it is not a "test-suite" test.
+do_test()
+{
+	test_failed=0
+	file="$1"
+
+	# can this test be handled by test-suite ?
+	# (it has to have a check-name key in it)
+	get_value "check-name" $file
+	if [ "$?" -eq 1 ]; then
+		unhandled_tests=`expr $unhandled_tests + 1`
+		return 2
+	fi
+	test_name=$last_result
+
+	echo "     TEST    $test_name ($file)"
+
+	# does the test provide a specific command ?
+	cmd=`eval echo $default_path/$default_cmd`
+	get_value "check-command" $file
+	if [ "$?" -eq "0" ]; then
+		last_result=`echo $last_result | sed -e 's/^ *//'`
+		cmd=`eval echo $default_path/$last_result`
+	fi
+	verbose "Using command       : $cmd"
+
+	# grab the expected exit value
+	get_value "check-exit-value" $file
+	if [ "$?" -eq "0" ]; then
+		expected_exit_value=`echo $last_result | tr -d ' '`
+	else
+		expected_exit_value=0
+	fi
+	verbose "Expecting exit value: $expected_exit_value"
+
+	# grab the expected output
+	sed -n '/check-output-start/,/check-output-end/p' $file \
+		| grep -v check-output > "$file".expected
+
+	# grab the actual output & exit value
+	$cmd 1> $file.got 2>&1
+	actual_exit_value=$?
+
+	diff -u "$file".expected "$file".got > "$file".diff
+	if [ "$?" -ne "0" ]; then
+		error "actual output does not match the expected one."
+		error  "see $file.* for further investigation."
+		test_failed=1
+	fi
+
+	if [ "$actual_exit_value" -ne "$expected_exit_value" ]; then
+		error "Actual exit value does not match the expected one."
+		error "expected $expected_exit_value, got $actual_exit_value."
+		test_failed=1
+	fi
+
+	if [ "$test_failed" -eq "1" ]; then
+		ko_tests=`expr $ko_tests + 1`
+		get_tag "check-known-to-fail" $file
+		[ "$?" -eq "0" ] && known_ko_tests=`expr $known_ko_tests + 1`
+		return 1
+	else
+		ok_tests=`expr $ok_tests + 1`
+		return 0
+	fi
+}
+
+do_test_suite()
+{
+	for i in $tests_list; do
+		do_test "$i"
+	done
+
+	# prints some numbers
+	tests_nr=`expr $ok_tests + $ko_tests`
+	echo -n "Out of $tests_nr tests, $ok_tests passed, $ko_tests failed"
+	echo " ($known_ko_tests of them are known to fail)"
+	if [ "$unhandled_tests" -ne "0" ]; then
+		echo "$unhandled_tests tests could not be handled by $prog_name"
+	fi
+}
+
+##
+# do_format(file[, name[, cmd]]) - helps a test writer to format test-suite tags
+do_format()
+{
+	if [ -z "$2" ]; then
+		fname="$1"
+		fcmd=$default_cmd
+	elif [ -z "$3" ]; then
+		fname="$2"
+		fcmd=$default_cmd
+	else
+		fname="$2"
+		fcmd="$3"
+	fi
+	file="$1"
+	cmd=`eval echo $default_path/$fcmd`
+	$cmd 1> $file.got 2>&1
+	fexit_value=$?
+	#foutput=`sed -e "s/^\(.*\)/ * check-output:\1/" $file.got`
+	foutput=`cat $file.got`
+	if [ -z "$foutput" ]; then
+	format=`cat <<_EOF
+/*
+ * check-name: $fname
+ *
+ * check-command: $fcmd
+ * check-exit-value: $fexit_value
+ */
+_EOF
+`
+	else
+	format=`cat <<_EOF
+/*
+ * check-name: $fname
+ *
+ * check-command: $fcmd
+ * check-exit-value: $fexit_value
+ *
+ * check-output-start
+$foutput
+ * check-output-end
+ */
+_EOF
+`
+	fi
+	echo "$format"
+	return 0
+}
+
+##
+# arg_file(filename) - checks if filename exists
+arg_file()
+{
+	[ -z "$1" ] && {
+		do_usage
+		exit 1
+	}
+	[ -e "$1" ] || {
+		error "Can't open file $1"
+		exit 1
+	}
+	return 0
+}
+
+case "$1" in
+	'')
+		do_test_suite
+		;;
+	single)
+		arg_file "$2"
+		do_test "$2"
+		case "$?" in
+			0) echo "$2 passed !";;
+			1) echo "$2 failed !";;
+			2) echo "$2 can't be handled by $prog_name";;
+		esac
+		;;
+	format)
+		arg_file "$2"
+		do_format "$2" "$3" "$4"
+		;;
+	help | *)
+		do_usage
+		exit 1
+		;;
+esac
+
+exit 0
+
-- 
1.5.2.1.280.g38570


[-- Attachment #3: 0002-Sample-test-suite-test-cases.patch --]
[-- Type: text/x-patch, Size: 3613 bytes --]

From 922bed97f061491ff2e7ca11e22161726fd6d7e5 Mon Sep 17 00:00:00 2001
From: Damien Lespiau <damien.lespiau@gmail.com>
Date: Mon, 2 Jul 2007 06:41:39 +0200
Subject: [PATCH] Sample test-suite test cases

A few examples meant to show the use of test-suite

Signed-off-by: Damien Lespiau <damien.lespiau@gmail.com>
---
 validation/bad-assignment.c              |   13 +++++++++++++
 validation/preprocessor/preprocessor1.c  |   11 +++++++++++
 validation/preprocessor/preprocessor19.c |   13 +++++++++++++
 validation/preprocessor/preprocessor2.c  |   11 +++++++++++
 validation/preprocessor/preprocessor3.c  |   14 ++++++++++++++
 validation/struct-as.c                   |    6 ++++++
 6 files changed, 68 insertions(+), 0 deletions(-)

diff --git a/validation/bad-assignment.c b/validation/bad-assignment.c
index 3b66a11..28e8c8d 100644
--- a/validation/bad-assignment.c
+++ b/validation/bad-assignment.c
@@ -4,3 +4,16 @@ static int foo(int a)
 
 	return a;
 }
+/*
+ * check-name: bad assignement
+ *
+ * check-command: sparse $file
+ * check-exit-value: 1
+ *
+ * check-output-start
+bad-assignment.c:3:6: error: Expected ; at end of statement
+bad-assignment.c:3:6: error: got \
+ * check-output-end
+ *
+ * check-known-to-fail
+ */
diff --git a/validation/preprocessor/preprocessor1.c b/validation/preprocessor/preprocessor1.c
index 5ae20aa..a02ccf6 100644
--- a/validation/preprocessor/preprocessor1.c
+++ b/validation/preprocessor/preprocessor1.c
@@ -12,3 +12,14 @@
 #define bar func(
 #define foo bar foo
 foo )
+/*
+ * check-name: Preprocessor #1
+ *
+ * check-command: sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output-start
+
+foo
+ * check-output-end
+ */
diff --git a/validation/preprocessor/preprocessor19.c b/validation/preprocessor/preprocessor19.c
index 0f7da47..c036c2e 100644
--- a/validation/preprocessor/preprocessor19.c
+++ b/validation/preprocessor/preprocessor19.c
@@ -3,3 +3,16 @@
 #define A x
 #define A y
 A
+/*
+ * check-name: Preprocessor #19
+ *
+ * check-command: sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output-start
+preprocessor/preprocessor19.c:4:9: warning: preprocessor token A redefined
+preprocessor/preprocessor19.c:3:9: this was the original definition
+
+y
+ * check-output-end
+ */
diff --git a/validation/preprocessor/preprocessor2.c b/validation/preprocessor/preprocessor2.c
index 340938e..c95641d 100644
--- a/validation/preprocessor/preprocessor2.c
+++ b/validation/preprocessor/preprocessor2.c
@@ -13,3 +13,14 @@
 #define BINARY(x, y) x + y
 
 UNARY(TWO)
+/*
+ * check-name: Preprocessor #2
+ *
+ * check-command: sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output-start
+
+a + b
+ * check-output-end
+ */
diff --git a/validation/preprocessor/preprocessor3.c b/validation/preprocessor/preprocessor3.c
index 71b9acd..0735667 100644
--- a/validation/preprocessor/preprocessor3.c
+++ b/validation/preprocessor/preprocessor3.c
@@ -35,3 +35,17 @@ A()                     // B ( )
 SCAN( A() )             // A ( )
 SCAN(SCAN( A() ))       // B ( )
 SCAN(SCAN(SCAN( A() ))) // A ( )
+/*
+ * check-name: Preprocessor #3
+ *
+ * check-command: sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output-start
+
+B ( )
+A ( )
+B ( )
+A ( )
+ * check-output-end
+ */
diff --git a/validation/struct-as.c b/validation/struct-as.c
index 86b90d3..31bdc88 100644
--- a/validation/struct-as.c
+++ b/validation/struct-as.c
@@ -14,3 +14,9 @@ static int broken(struct hello __user *sp)
 {
 	test(&sp->a);
 }
+/*
+ * check-name: Address space of a struct member
+ *
+ * check-command: sparse $file
+ * check-exit-value: 0
+ */
-- 
1.5.2.1.280.g38570


[-- Attachment #4: 0003-test-suite-documentation.patch --]
[-- Type: text/x-patch, Size: 3640 bytes --]

From 3881b34f3e103aecdc4581bfcef1925d7a8636b6 Mon Sep 17 00:00:00 2001
From: Damien Lespiau <damien.lespiau@gmail.com>
Date: Mon, 2 Jul 2007 06:49:04 +0200
Subject: [PATCH] test-suite documentation

A quick description of test-suite usage.

Signed-off-by: Damien Lespiau <damien.lespiau@gmail.com>
---
 Documentation/test-suite |  112 ++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 112 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/test-suite

diff --git a/Documentation/test-suite b/Documentation/test-suite
new file mode 100644
index 0000000..1755156
--- /dev/null
+++ b/Documentation/test-suite
@@ -0,0 +1,112 @@
+
+
+	Sparse test suite
+	~~~~~~~~~~~~~~~~~
+
+Sparse has a number of test cases in its validation directory. The test-suite
+script aims at making automated checking of these tests possible. It works by
+embedding tags in C comments in the test cases.
+
+check-name: (mandatory)
+	Name of the test.
+
+check-description: (optional)
+	A description of what checks the test.
+
+check-command: (optional)
+	There are different kinds of tests. Some can validate sparse
+	preprocessor while others will use sparse, cgcc of even other backends
+	of the library. check-command allows you to give a custom command to
+	run the test-case.
+	The '$file' string is special. It will be expanded to the file name at
+	run time.
+	It defaults to "sparse $file".
+
+check-exit-value: (optional)
+	The expected exit value of check-command. It defaults to 0.
+
+check-output-start / check-output-end (optional)
+	The expected output (stdout and stderr) of check-command lies between
+	those two tags. It defaults to no output.
+
+check-known-to-fail (optional)
+	Mark the test as being known to fail.
+
+
+	Using test-suite
+	~~~~~~~~~~~~~~~~
+
+The test-suite script is called trough the check target of the Makefile. It
+will try to check every test case it finds (find validation -name '*.c').
+
+It can be called to check a single test with:
+$ cd validation
+$ ./test-suite single preprocessor/preprocessor1.c
+     TEST     Preprocessor #1 (preprocessor/preprocessor1.c)
+preprocessor/preprocessor1.c passed !
+
+
+	Writing a test
+	~~~~~~~~~~~~~~
+
+test-suite comes with a format command to make a test easier to write:
+
+	test-suite format file [name [cmd]]
+
+name:
+	check-name value. If no name is provided, it defaults to the file name.
+cmd:
+	check-command value. It no cmd if provided, it defaults to
+	"sparse $file".
+
+The output of the test-suite format command can be redirected into the
+test case to create a test-suite formated file.
+
+$ ./test-suite format bad-assignment.c Assignement >> bad-assignment.c
+$ cat !$
+cat bad-assignment.c
+/*
+ * check-name: bad assignement
+ *
+ * check-command: sparse $file
+ * check-exit-value: 1
+ *
+ * check-output-start
+bad-assignment.c:3:6: error: Expected ; at end of statement
+bad-assignment.c:3:6: error: got \
+ * check-output-end
+ */
+
+You can define the check-command you want to use for the test. $file will be
+extended to the file name at run time.
+
+$ ./test-suite format validation/preprocessor2.c "Preprocessor #2" \
+		"sparse -E \$file" >> validation/preprocessor2.c
+$ cat !$
+cat validation/preprocessor2.c
+/*
+ * This one we happen to get right.
+ *
+ * It should result in a simple
+ *
+ *	a + b
+ *
+ * for a proper preprocessor.
+ */
+#define TWO a, b
+
+#define UNARY(x) BINARY(x)
+#define BINARY(x, y) x + y
+
+UNARY(TWO)
+/*
+ * check-name: Preprocessor #2
+ *
+ * check-command: sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output-start
+
+a + b
+ * check-output-end
+ */
-- 
1.5.2.1.280.g38570