* [PATCH 01/28] testsuite: 'echo -n' may not be interpreted as '-n'
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 02/28] testsuite: allow to test a few cases at once Luc Van Oostenryck
` (28 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
POSIX tell that the interpretation of the '-n' is implementation
independent. So avoid it.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/validation/test-suite b/validation/test-suite
index cf151a361..53cd30190 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -303,8 +303,8 @@ do_test_suite()
# prints some numbers
tests_nr=$(($ok_tests + $ko_tests))
- echo -n "Out of $tests_nr tests, $ok_tests passed, $ko_tests failed"
- echo " ($known_ko_tests of them are known to fail)"
+ echo "Out of $tests_nr tests, $ok_tests passed, $ko_tests failed" \
+ " ($known_ko_tests of them are known to fail)"
if [ "$unhandled_tests" -ne "0" ]; then
echo "$unhandled_tests tests could not be handled by $prog_name"
fi
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 02/28] testsuite: allow to test a few cases at once
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 01/28] testsuite: 'echo -n' may not be interpreted as '-n' Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 03/28] testsuite: move verbose() & error() Luc Van Oostenryck
` (27 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Sometimes we want to only launch a few tests, not the whole
testsuite. the 'single' option allow to test a single case
but it's too restrictive.
Changes this by allowing arbirary arguments as files to be tested.
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/validation/test-suite b/validation/test-suite
index 53cd30190..9e30ed937 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -6,7 +6,7 @@ cd $(dirname "$0")
default_path=".."
default_cmd="sparse \$file"
-tests_list=`find . -name '*.c' | sed -e 's#^\./\(.*\)#\1#' | sort`
+tests_list=""
prog_name=`basename $0`
if [ ! -x "$default_path/sparse-llvm" ]; then
@@ -156,6 +156,7 @@ echo "Usage: $prog_name [command] [command arguments]"
echo
echo "commands:"
echo " none runs the whole test suite"
+echo " file ... runs the test suite on the given file(s)"
echo " single file runs the test in 'file'"
echo " format file [name [cmd]] helps writing a new test case using cmd"
echo
@@ -370,8 +371,14 @@ arg_file()
case "$1" in
'')
+ tests_list=`find . -name '*.c' | sed -e 's#^\./\(.*\)#\1#' | sort`
do_test_suite
;;
+ *.c)
+ tests_list="$@"
+ do_test_suite
+ ;;
+
single)
arg_file "$2"
do_test "$2"
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 03/28] testsuite: move verbose() & error()
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 01/28] testsuite: 'echo -n' may not be interpreted as '-n' Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 02/28] testsuite: allow to test a few cases at once Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 04/28] testsuite: better message for pattern nbr checking Luc Van Oostenryck
` (26 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 34 +++++++++++++++++-----------------
1 file changed, 17 insertions(+), 17 deletions(-)
diff --git a/validation/test-suite b/validation/test-suite
index 9e30ed937..5f63a23bb 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -71,6 +71,23 @@ get_tag_value()
EOT
}
+##
+# verbose(string) - prints string if we are in verbose mode
+verbose()
+{
+ [ "$V" -eq "1" ] && echo " $1"
+ return 0
+}
+
+##
+# error(string[, die]) - prints an error and exits with value die if given
+error()
+{
+ [ "$quiet" -ne 1 ] && echo "error: $1"
+ [ -n "$2" ] && exit $2
+ return 0
+}
+
##
# helper for has_(each|none)_patterns()
has_patterns()
@@ -132,23 +149,6 @@ nbr_patterns()
return $?
}
-##
-# verbose(string) - prints string if we are in verbose mode
-verbose()
-{
- [ "$V" -eq "1" ] && echo " $1"
- return 0
-}
-
-##
-# error(string[, die]) - prints an error and exits with value die if given
-error()
-{
- [ "$quiet" -ne 1 ] && echo "error: $1"
- [ -n "$2" ] && exit $2
- return 0
-}
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 04/28] testsuite: better message for pattern nbr checking
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (2 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 03/28] testsuite: move verbose() & error() Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 05/28] testsuite: better message for pattern absence/presence Luc Van Oostenryck
` (25 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/validation/test-suite b/validation/test-suite
index 5f63a23bb..953b7637e 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -142,6 +142,8 @@ nbr_patterns()
while read nbr pat; do
n=$(grep -s "$pat" "$ofile" | wc -l)
if [ "$n" -ne "$nbr" ]; then
+ error "test '$ifile' failed"
+ error " Pattern '$pat' expected $nbr times but got $n times"
return 1
fi
done
@@ -271,7 +273,6 @@ do_test()
# verify the 'check-output-pattern-X-times' tags
nbr_patterns "$file" 'check-output-pattern' $file.output.got
if [ "$?" -ne "0" ]; then
- error "Actual output doesn't contain the pattern the expected number."
test_failed=1
fi
fi
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 05/28] testsuite: better message for pattern absence/presence
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (3 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 04/28] testsuite: better message for pattern nbr checking Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 06/28] testsuite: saner checking of pattern occurences Luc Van Oostenryck
` (24 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/validation/test-suite b/validation/test-suite
index 953b7637e..bdc1424f4 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -96,11 +96,14 @@ has_patterns()
patt="$2"
ofile="$3"
cmp="$4"
+ msg="$5"
grep "$patt:" "$ifile" | \
sed -e "s/^.*$patt: *\(.*\)$/\1/" | \
while read val; do
grep -s -q "$val" "$ofile"
if [ "$?" $cmp 0 ]; then
+ error "test '$ifile' failed"
+ error " Pattern '$val' unexpectedly $msg"
return 1
fi
done
@@ -115,7 +118,7 @@ has_patterns()
# returns 0 if all present, 1 otherwise
has_each_patterns()
{
- has_patterns "$1" "$2" "$3" -ne
+ has_patterns "$1" "$2" "$4" -ne "$3"
}
##
@@ -125,7 +128,7 @@ has_each_patterns()
# returns 1 if any present, 0 otherwise
has_none_patterns()
{
- has_patterns "$1" "$2" "$3" -eq
+ has_patterns "$1" "$2" "$4" -eq "$3"
}
##
@@ -256,16 +259,14 @@ do_test()
# verify the 'check-output-contains/excludes' tags
if [ $check_output_contains -eq 1 ]; then
- has_each_patterns "$file" 'check-output-contains' $file.output.got
+ has_each_patterns "$file" 'check-output-contains' absent $file.output.got
if [ "$?" -ne "0" ]; then
- error "Actual output doesn't contain some of the expected patterns."
test_failed=1
fi
fi
if [ $check_output_excludes -eq 1 ]; then
- has_none_patterns "$file" 'check-output-excludes' $file.output.got
+ has_none_patterns "$file" 'check-output-excludes' present $file.output.got
if [ "$?" -ne "0" ]; then
- error "Actual output contains some patterns which are not expected."
test_failed=1
fi
fi
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 06/28] testsuite: saner checking of pattern occurences
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (4 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 05/28] testsuite: better message for pattern absence/presence Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 07/28] testsuite: clearer result summary Luc Van Oostenryck
` (23 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
Documentation/test-suite | 6 ++++++
validation/test-suite | 49 ++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 55 insertions(+)
diff --git a/Documentation/test-suite b/Documentation/test-suite
index a288c81a5..e2dda61d7 100644
--- a/Documentation/test-suite
+++ b/Documentation/test-suite
@@ -57,6 +57,12 @@ check-output-pattern-<nbr>-times: <pattern> (optional)
Similar than the contains/excludes her above, but with full control
of the number of times the pattern should occurs in the output.
+check-output-pattern(eq,<nbr>): <pattern> (optional)
+check-output-pattern(<min>,<max>): <pattern> (optional)
+ Similar than the contains/excludes her above, but with full control
+ of the number of times the pattern should occurs in the output.
+ If <min> or <max> is '-' the corresponding check is ignored.
+
Using test-suite
~~~~~~~~~~~~~~~~
diff --git a/validation/test-suite b/validation/test-suite
index bdc1424f4..8b2e0c85a 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -46,6 +46,7 @@ get_tag_value()
check_output_contains=0
check_output_excludes=0
check_output_pattern=0
+ check_output_pattern_minmax=0
lines=$(grep 'check-[a-z-]*' $1 | \
sed -e 's/^.*\(check-[a-z-]*:*\) *\(.*\)$/\1 \2/')
@@ -65,6 +66,7 @@ get_tag_value()
check-output-contains:) check_output_contains=1 ;;
check-output-excludes:) check_output_excludes=1 ;;
check-output-pattern-) check_output_pattern=1 ;;
+ check-output-pattern) check_output_pattern_minmax=1 ;;
esac
done << EOT
$lines
@@ -154,6 +156,46 @@ nbr_patterns()
return $?
}
+##
+# minmax_patterns(ifile tag ofile) - does ofile contains the
+# the patterns given by ifile's tags
+# the right number of time?
+minmax_patterns()
+{
+ ifile="$1"
+ patt="$2"
+ ofile="$3"
+ grep "$patt([0-9a-z-]*, *[0-9-]*):" "$ifile" | \
+ sed -e "s/^.*$patt(\([0-9a-z-]*\), *\([0-9-]*\)): *\(.*\)/\1 \2 \3/" | \
+ while read min max pat; do
+ n=$(grep -s "$pat" "$ofile" | wc -l)
+ if [ "$min" = 'eq' ]; then
+ if [ "$n" -ne "$max" ]; then
+ error "test '$ifile' failed"
+ error " Pattern '$pat' expected $max times but got $n times"
+ return 1
+ fi
+ continue
+ fi
+ if [ "$min" != '-' ]; then
+ if [ "$n" -lt "$min" ]; then
+ error "test '$ifile' failed"
+ error " Pattern '$pat' expected min $nbr times but got $n times"
+ return 1
+ fi
+ fi
+ if [ "$max" != '-' ]; then
+ if [ "$n" -gt "$max" ]; then
+ error "test '$ifile' failed"
+ error " Pattern '$pat' expected max $nbr times but got $n times"
+ return 1
+ fi
+ fi
+ done
+
+ return $?
+}
+
do_usage()
{
echo "$prog_name - a tiny automatic testing script"
@@ -277,6 +319,13 @@ do_test()
test_failed=1
fi
fi
+ if [ $check_output_pattern_minmax -eq 1 ]; then
+ # verify the 'check-output-pattern(...)' tags
+ minmax_patterns "$file" 'check-output-pattern' $file.output.got
+ if [ "$?" -ne "0" ]; then
+ test_failed=1
+ fi
+ fi
[ "$test_failed" -eq "$must_fail" ] || failed=1
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 07/28] testsuite: clearer result summary
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (5 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 06/28] testsuite: saner checking of pattern occurences Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 08/28] testsuite: check error messages first Luc Van Oostenryck
` (22 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/validation/test-suite b/validation/test-suite
index 8b2e0c85a..3d2192178 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -353,15 +353,20 @@ do_test_suite()
do_test "$i"
done
+ OK=OK
+ [ $failed -eq 0 ] || OK=KO
+
# prints some numbers
tests_nr=$(($ok_tests + $ko_tests))
- echo "Out of $tests_nr tests, $ok_tests passed, $ko_tests failed" \
- " ($known_ko_tests of them are known to fail)"
+ echo "$OK: out of $tests_nr tests, $ok_tests passed, $ko_tests failed"
+ if [ "$known_ko_tests" -ne 0 ]; then
+ echo " $known_ko_tests of them are known to fail"
+ fi
if [ "$unhandled_tests" -ne "0" ]; then
- echo "$unhandled_tests tests could not be handled by $prog_name"
+ echo " $unhandled_tests tests could not be handled by $prog_name"
fi
if [ "$disabled_tests" -ne "0" ]; then
- echo "$disabled_tests tests were disabled"
+ echo " $disabled_tests tests were disabled"
fi
}
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 08/28] testsuite: check error messages first
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (6 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 07/28] testsuite: clearer result summary Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 09/28] testsuite: saner handling of 'must_fail' Luc Van Oostenryck
` (21 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/validation/test-suite b/validation/test-suite
index 3d2192178..256d31550 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -276,7 +276,7 @@ do_test()
[ $must_fail -eq 1 ] && [ $V -eq 0 ] && quiet=1
known_ko_tests=$(($known_ko_tests + $must_fail))
- for stream in output error; do
+ for stream in error output; do
eval ignore=\$check_${stream}_ignore
[ $ignore -eq 1 ] && continue
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 09/28] testsuite: saner handling of 'must_fail'
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (7 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 08/28] testsuite: check error messages first Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 10/28] testsuite: allow to parse several options Luc Van Oostenryck
` (20 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/validation/test-suite b/validation/test-suite
index 256d31550..5dfe9e7f3 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -327,17 +327,19 @@ do_test()
fi
fi
- [ "$test_failed" -eq "$must_fail" ] || failed=1
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 10/28] testsuite: allow to parse several options
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (8 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 09/28] testsuite: saner handling of 'must_fail' Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 11/28] testsuite: add support for -q|--quiet Luc Van Oostenryck
` (19 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/validation/test-suite b/validation/test-suite
index 5dfe9e7f3..7a491dfb4 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -427,7 +427,8 @@ arg_file()
return 0
}
-case "$1" in
+while true; do
+ case "$1" in
'')
tests_list=`find . -name '*.c' | sed -e 's#^\./\(.*\)#\1#' | sort`
do_test_suite
@@ -454,7 +455,9 @@ case "$1" in
do_usage
exit 1
;;
-esac
+ esac
+ break
+done
exit $failed
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 11/28] testsuite: add support for -q|--quiet
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (9 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 10/28] testsuite: allow to parse several options Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 12/28] testsuite: add support for -a|--abort Luc Van Oostenryck
` (18 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 17 +++++++++++++++--
1 file changed, 15 insertions(+), 2 deletions(-)
diff --git a/validation/test-suite b/validation/test-suite
index 7a491dfb4..9371130d9 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -31,6 +31,7 @@ known_ko_tests=0
# defaults to not verbose
[ -z "$V" ] && V=0
+vquiet=""
##
# get_tag_value(file) - get the 'check-<...>' tags & values
@@ -199,7 +200,10 @@ minmax_patterns()
do_usage()
{
echo "$prog_name - a tiny automatic testing script"
-echo "Usage: $prog_name [command] [command arguments]"
+echo "Usage: $prog_name [option(s)] [command] [arguments]"
+echo
+echo "options:"
+echo " -q|--quiet be extra quiet while running the tests"
echo
echo "commands:"
echo " none runs the whole test suite"
@@ -254,7 +258,9 @@ do_test()
cmd=`eval echo $default_path/$check_command`
- echo " TEST $test_name ($file)"
+ if [ -z "$vquiet" ]; then
+ echo " TEST $test_name ($file)"
+ fi
verbose "Using command : $cmd"
@@ -329,6 +335,7 @@ do_test()
if [ "$must_fail" -eq "1" ]; then
if [ "$test_failed" -eq "1" ]; then
+ [ -z "$vquiet" ] && \
echo "info: test '$file' is known to fail"
else
echo "error: test '$file' is known to fail but succeed!"
@@ -429,6 +436,12 @@ arg_file()
while true; do
case "$1" in
+ -q|--quiet)
+ vquiet=1
+ shift
+ continue
+ ;;
+
'')
tests_list=`find . -name '*.c' | sed -e 's#^\./\(.*\)#\1#' | sort`
do_test_suite
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 12/28] testsuite: add support for -a|--abort
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (10 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 11/28] testsuite: add support for -q|--quiet Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 13/28] testsuite: get options from env too Luc Van Oostenryck
` (17 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/validation/test-suite b/validation/test-suite
index 9371130d9..49b3e0c24 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -32,6 +32,7 @@ known_ko_tests=0
# defaults to not verbose
[ -z "$V" ] && V=0
vquiet=""
+abort=0
##
# get_tag_value(file) - get the 'check-<...>' tags & values
@@ -203,6 +204,7 @@ echo "$prog_name - a tiny automatic testing script"
echo "Usage: $prog_name [option(s)] [command] [arguments]"
echo
echo "options:"
+echo " -a|--abort abort the tests as soon as one fails"
echo " -q|--quiet be extra quiet while running the tests"
echo
echo "commands:"
@@ -343,6 +345,7 @@ do_test()
fi
if [ "$test_failed" -ne "$must_fail" ]; then
+ [ $abort -eq 1 ] && exit 1
test_failed=1
failed=1
fi
@@ -436,6 +439,11 @@ arg_file()
while true; do
case "$1" in
+ -a|--abort)
+ abort=1
+ shift
+ continue
+ ;;
-q|--quiet)
vquiet=1
shift
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 13/28] testsuite: get options from env too
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (11 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 12/28] testsuite: add support for -a|--abort Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 14/28] testsuite: allow --format & --single Luc Van Oostenryck
` (16 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/validation/test-suite b/validation/test-suite
index 49b3e0c24..c03348028 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -2,6 +2,13 @@
#set -x
+## allow flags from environment
+flags="$SPARSE_TEST_FLAGS"
+if [ ! -z "$flags" ]; then
+ unset SPARSE_TEST_FLAGS
+ exec "$0" $flags "$@"
+fi
+
cd $(dirname "$0")
default_path=".."
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 14/28] testsuite: allow --format & --single
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (12 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 13/28] testsuite: get options from env too Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 15/28] testsuite: remove useless selftest Luc Van Oostenryck
` (15 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/validation/test-suite b/validation/test-suite
index c03348028..9364b9d7f 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -466,7 +466,7 @@ while true; do
do_test_suite
;;
- single)
+ single|--single)
arg_file "$2"
do_test "$2"
case "$?" in
@@ -475,7 +475,7 @@ while true; do
2) echo "$2 can't be handled by $prog_name";;
esac
;;
- format)
+ format|--format)
arg_file "$2"
do_format "$2" "$3" "$4"
;;
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 15/28] testsuite: remove useless selftest
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (13 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 14/28] testsuite: allow --format & --single Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 16/28] testsuite: remove useless test-be.c Luc Van Oostenryck
` (14 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/testsuite-selfcheck1.c | 10 ----------
validation/testsuite-selfcheck2.c | 10 ----------
validation/testsuite-selfcheck3.c | 10 ----------
3 files changed, 30 deletions(-)
delete mode 100644 validation/testsuite-selfcheck1.c
delete mode 100644 validation/testsuite-selfcheck2.c
delete mode 100644 validation/testsuite-selfcheck3.c
diff --git a/validation/testsuite-selfcheck1.c b/validation/testsuite-selfcheck1.c
deleted file mode 100644
index d927f9961..000000000
--- a/validation/testsuite-selfcheck1.c
+++ /dev/null
@@ -1,10 +0,0 @@
-good
-
-/*
- * check-name: selfcheck1
- * check-command: sparse -E $file
- * check-output-ignore
- *
- * check-output-contains: good
- * check-output-excludes: evil
- */
diff --git a/validation/testsuite-selfcheck2.c b/validation/testsuite-selfcheck2.c
deleted file mode 100644
index 5309e32f3..000000000
--- a/validation/testsuite-selfcheck2.c
+++ /dev/null
@@ -1,10 +0,0 @@
-evil
-
-/*
- * check-name: selfcheck2
- * check-command: sparse -E $file
- * check-output-ignore
- * check-known-to-fail
- *
- * check-output-contains: good
- */
diff --git a/validation/testsuite-selfcheck3.c b/validation/testsuite-selfcheck3.c
deleted file mode 100644
index 6d834e68d..000000000
--- a/validation/testsuite-selfcheck3.c
+++ /dev/null
@@ -1,10 +0,0 @@
-evil
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 16/28] testsuite: remove useless test-be.c
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (14 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 15/28] testsuite: remove useless selftest Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 17/28] testsuite: extract disable() Luc Van Oostenryck
` (13 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-be.c | 46 ----------------------------------------------
1 file changed, 46 deletions(-)
delete mode 100644 validation/test-be.c
diff --git a/validation/test-be.c b/validation/test-be.c
deleted file mode 100644
index deda3cc14..000000000
--- a/validation/test-be.c
+++ /dev/null
@@ -1,46 +0,0 @@
-int printf(char *c, ...);
-void exit(int c);
-
-#undef PRINT_OUTPUTS
-
-static void test_func_args(int x, int y)
-{
- if (x == y)
- exit(1);
-}
-
-static int binop_s32(int x, int y)
-{
- int a;
-
- a = a + x;
- a = a / y;
- a = a * x;
- a = a - y;
-
- return a;
-}
-
-static void test_binops(void)
-{
- int tmp_s32 = binop_s32(987123, 234);
-
-#ifdef PRINT_OUTPUTS
- printf("binop_s32(987123, 234) == %d\n", tmp_s32);
-#else
- if (tmp_s32 != -1470599007)
- exit(2);
-#endif
-}
-
-int main (int argc, char *argv[])
-{
- test_func_args(1, 2);
- test_binops();
-
- return 0;
-}
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 17/28] testsuite: extract disable()
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (15 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 16/28] testsuite: remove useless test-be.c Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 18/28] testsuite: simplify documentation Luc Van Oostenryck
` (12 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/validation/test-suite b/validation/test-suite
index 9364b9d7f..84e359fb8 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -223,6 +223,14 @@ echo
echo " help prints usage"
}
+disable()
+{
+ disabled_tests=$(($disabled_tests + 1))
+ if [ -z "$vquiet" ]; then
+ echo " DISABLE $1 ($2)"
+ fi
+}
+
##
# do_test(file) - tries to validate a test case
#
@@ -259,8 +267,7 @@ do_test()
base_cmd=$1
for i in $disabled_cmds; do
if [ "$i" = "$base_cmd" ] ; then
- disabled_tests=$(($disabled_tests + 1))
- echo " DISABLE $test_name ($file)"
+ disable "$test_name" "$file"
return 3
fi
done
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 18/28] testsuite: simplify documentation
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (16 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 17/28] testsuite: extract disable() Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 19/28] testsuite: allow arch-specific tests Luc Van Oostenryck
` (11 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
Documentation/test-suite | 28 ++++++++++++++--------------
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/Documentation/test-suite b/Documentation/test-suite
index e2dda61d7..a745da850 100644
--- a/Documentation/test-suite
+++ b/Documentation/test-suite
@@ -7,13 +7,13 @@ Sparse has a number of test cases in its validation directory. The test-suite
script aims at making automated checking of these tests possible. It works by
embedding tags in C comments in the test cases.
-check-name: (mandatory)
- Name of the test.
+check-name: <name>
+ Name of the test. This is the only mandatory tag.
-check-description: (optional)
+check-description: <description ...>
A description of what the test checks.
-check-command: (optional)
+check-command: <command arg ...>
There are different kinds of tests. Some can validate the sparse
preprocessor, while others will use sparse, cgcc, or even other backends
of the library. check-command allows you to give a custom command to
@@ -22,43 +22,43 @@ check-command: (optional)
run time.
It defaults to "sparse $file".
-check-exit-value: (optional)
+check-exit-value: <value>
The expected exit value of check-command. It defaults to 0.
-check-timeout: (optional)
+check-timeout: <timeout>
The maximum expected duration of check-command, in seconds.
It defaults to 1.
-check-output-start / check-output-end (optional)
+check-output-start / check-output-end
The expected output (stdout and stderr) of check-command lies between
those two tags. It defaults to no output.
-check-output-ignore / check-error-ignore (optional)
+check-output-ignore / check-error-ignore
Don't check the expected output (stdout or stderr) of check-command
(usefull when this output is not comparable or if you're only interested
in the exit value).
By default this check is done.
-check-known-to-fail (optional)
+check-known-to-fail
Mark the test as being known to fail.
-check-output-contains: <pattern> (optional)
+check-output-contains: <pattern>
Check that the output (stdout) contains the given pattern.
Several such tags can be given, in which case the output
must contains all the patterns.
-check-output-excludes: <pattern> (optional)
+check-output-excludes: <pattern>
Similar than the above one, but with opposite logic.
Check that the output (stdout) doesn't contain the given pattern.
Several such tags can be given, in which case the output
must contains none of the patterns.
-check-output-pattern-<nbr>-times: <pattern> (optional)
+check-output-pattern-<nbr>-times: <pattern>
Similar than the contains/excludes her above, but with full control
of the number of times the pattern should occurs in the output.
-check-output-pattern(eq,<nbr>): <pattern> (optional)
-check-output-pattern(<min>,<max>): <pattern> (optional)
+check-output-pattern(eq,<nbr>): <pattern>
+check-output-pattern(<min>,<max>): <pattern>
Similar than the contains/excludes her above, but with full control
of the number of times the pattern should occurs in the output.
If <min> or <max> is '-' the corresponding check is ignored.
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 19/28] testsuite: allow arch-specific tests
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (17 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 18/28] testsuite: simplify documentation Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 20/28] testsuite: save screen real estate Luc Van Oostenryck
` (10 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
Documentation/test-suite | 5 +++++
validation/test-suite | 18 ++++++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/Documentation/test-suite b/Documentation/test-suite
index a745da850..3c07100e7 100644
--- a/Documentation/test-suite
+++ b/Documentation/test-suite
@@ -22,6 +22,11 @@ check-command: <command arg ...>
run time.
It defaults to "sparse $file".
+check-arch-ignore: <arch[|...]>
+check-arch-only: <arch[|...]>
+ Ignore the test if the current architecture (as returned by 'uname -m')
+ match or not one of the archs given in the pattern.
+
check-exit-value: <value>
The expected exit value of check-command. It defaults to 0.
diff --git a/validation/test-suite b/validation/test-suite
index 84e359fb8..746c0fb3a 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -56,6 +56,8 @@ get_tag_value()
check_output_excludes=0
check_output_pattern=0
check_output_pattern_minmax=0
+ check_arch_ignore=""
+ check_arch_only=""
lines=$(grep 'check-[a-z-]*' $1 | \
sed -e 's/^.*\(check-[a-z-]*:*\) *\(.*\)$/\1 \2/')
@@ -76,6 +78,10 @@ get_tag_value()
check-output-excludes:) check_output_excludes=1 ;;
check-output-pattern-) check_output_pattern=1 ;;
check-output-pattern) check_output_pattern_minmax=1 ;;
+ check-arch-ignore:) arch=$(uname -m)
+ check_arch_ignore="$val" ;;
+ check-arch-only:) arch=$(uname -m)
+ check_arch_only="$val" ;;
esac
done << EOT
$lines
@@ -271,6 +277,18 @@ do_test()
return 3
fi
done
+ if [ "$check_arch_ignore" != "" ]; then
+ if echo $arch | egrep -q -w "$check_arch_ignore"; then
+ disable "$test_name" "$file"
+ return 3
+ fi
+ fi
+ if [ "$check_arch_only" != "" ]; then
+ if ! (echo $arch | egrep -q -w "$check_arch_only"); then
+ disable "$test_name" "$file"
+ return 3
+ fi
+ fi
cmd=`eval echo $default_path/$check_command`
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 20/28] testsuite: save screen real estate
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (18 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 19/28] testsuite: allow arch-specific tests Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 21/28] testsuite: add a blank line before format Luc Van Oostenryck
` (9 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/validation/test-suite b/validation/test-suite
index 746c0fb3a..f2e1c3242 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -233,7 +233,7 @@ disable()
{
disabled_tests=$(($disabled_tests + 1))
if [ -z "$vquiet" ]; then
- echo " DISABLE $1 ($2)"
+ echo " SKIP $1 ($2)"
fi
}
@@ -293,7 +293,7 @@ do_test()
cmd=`eval echo $default_path/$check_command`
if [ -z "$vquiet" ]; then
- echo " TEST $test_name ($file)"
+ echo " TEST $test_name ($file)"
fi
verbose "Using command : $cmd"
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 21/28] testsuite: add a blank line before format
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (19 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 20/28] testsuite: save screen real estate Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 22/28] testsuite: 'quiet' must be initialized earlier Luc Van Oostenryck
` (8 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 1 +
1 file changed, 1 insertion(+)
diff --git a/validation/test-suite b/validation/test-suite
index f2e1c3242..96eb90581 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -433,6 +433,7 @@ do_format()
$cmd 1> $file.output.got 2> $file.error.got
fexit_value=$?
cat <<_EOF
+
/*
* check-name: $fname
_EOF
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 22/28] testsuite: 'quiet' must be initialized earlier
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (20 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 21/28] testsuite: add a blank line before format Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 23/28] testsuite: move up arg_file() Luc Van Oostenryck
` (7 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
as it can, for example, be needed via arg_file() -> error()
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 1 +
1 file changed, 1 insertion(+)
diff --git a/validation/test-suite b/validation/test-suite
index 96eb90581..1ef18f288 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -39,6 +39,7 @@ known_ko_tests=0
# defaults to not verbose
[ -z "$V" ] && V=0
vquiet=""
+quiet=0
abort=0
##
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 23/28] testsuite: move up arg_file()
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (21 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 22/28] testsuite: 'quiet' must be initialized earlier Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 24/28] testsuite: make do_format() more self-contained Luc Van Oostenryck
` (6 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 32 +++++++++++++++++---------------
1 file changed, 17 insertions(+), 15 deletions(-)
diff --git a/validation/test-suite b/validation/test-suite
index 1ef18f288..35138cb37 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -212,6 +212,23 @@ minmax_patterns()
return $?
}
+##
+# arg_file(filename) - checks if filename exists
+arg_file()
+{
+ [ -z "$1" ] && {
+ do_usage
+ exit 1
+ }
+ [ -e "$1" ] || {
+ error "Can't open file $1"
+ exit 1
+ }
+ return 0
+}
+
+
+##
do_usage()
{
echo "$prog_name - a tiny automatic testing script"
@@ -456,21 +473,6 @@ _EOF
return 0
}
-##
-# arg_file(filename) - checks if filename exists
-arg_file()
-{
- [ -z "$1" ] && {
- do_usage
- exit 1
- }
- [ -e "$1" ] || {
- error "Can't open file $1"
- exit 1
- }
- return 0
-}
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 24/28] testsuite: make do_format() more self-contained
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (22 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 23/28] testsuite: move up arg_file() Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 25/28] testsuite: format: saner defaults handling Luc Van Oostenryck
` (5 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 30 +++++++++++++++++++++++++++---
1 file changed, 27 insertions(+), 3 deletions(-)
diff --git a/validation/test-suite b/validation/test-suite
index 35138cb37..b4ba39f3f 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -244,7 +244,7 @@ echo " file ... runs the test suite on the given file(s)"
echo " single file runs the test in 'file'"
echo " format file [name [cmd]] helps writing a new test case using cmd"
echo
-echo " help prints usage"
+echo " [command] help prints usage"
}
disable()
@@ -432,10 +432,34 @@ do_test_suite()
fi
}
+##
+do_format_help() {
+echo "Usage: $prog_name [--]format file [name [cmd]]"
+echo
+echo "argument(s):"
+echo " file file containing the test case(s)"
+echo " name name for the test case (defaults to file)"
+echo " cmd command to be used (defaults to 'sparse \$file')"
+}
+
##
# do_format(file[, name[, cmd]]) - helps a test writer to format test-suite tags
do_format()
{
+ while [ $# -gt 1 ] ; do
+ case "$1" in
+ help|-*)
+ do_format_help
+ return 0
+ ;;
+ *) break ;;
+ esac
+ shift
+ continue
+ done
+
+ arg_file "$1" || return 1
+
if [ -z "$2" ]; then
fname="$1"
fcmd=$default_cmd
@@ -505,8 +529,8 @@ while true; do
esac
;;
format|--format)
- arg_file "$2"
- do_format "$2" "$3" "$4"
+ shift
+ do_format "$@"
;;
help | *)
do_usage
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 25/28] testsuite: format: saner defaults handling
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (23 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 24/28] testsuite: make do_format() more self-contained Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 26/28] testsuite: format: strip .c from default name Luc Van Oostenryck
` (4 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 15 +++++----------
1 file changed, 5 insertions(+), 10 deletions(-)
diff --git a/validation/test-suite b/validation/test-suite
index b4ba39f3f..599f5e7e6 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -460,17 +460,12 @@ do_format()
arg_file "$1" || return 1
- if [ -z "$2" ]; then
- fname="$1"
- fcmd=$default_cmd
- elif [ -z "$3" ]; then
- fname="$2"
- fcmd=$default_cmd
- else
- fname="$2"
- fcmd="$3"
- fi
file="$1"
+ fname="$2"
+ [ -z "$fname" ] && fname="$1"
+ fcmd="$3"
+ [ -z "$fcmd" ] && fcmd=$default_cmd
+
cmd=`eval echo $default_path/$fcmd`
$cmd 1> $file.output.got 2> $file.error.got
fexit_value=$?
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 26/28] testsuite: format: strip .c from default name
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (24 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 25/28] testsuite: format: saner defaults handling Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 27/28] testsuite: add support for 'format -f' Luc Van Oostenryck
` (3 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/validation/test-suite b/validation/test-suite
index 599f5e7e6..5821f4428 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -462,7 +462,7 @@ do_format()
file="$1"
fname="$2"
- [ -z "$fname" ] && fname="$1"
+ [ -z "$fname" ] && fname="$(basename "$1" .c)"
fcmd="$3"
[ -z "$fcmd" ] && fcmd=$default_cmd
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 27/28] testsuite: add support for 'format -f'
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (25 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 26/28] testsuite: format: strip .c from default name Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 7:13 ` [PATCH 28/28] testsuite: add support for 'format -l' Luc Van Oostenryck
` (2 subsequent siblings)
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/validation/test-suite b/validation/test-suite
index 5821f4428..b5bd937d5 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -434,7 +434,10 @@ do_test_suite()
##
do_format_help() {
-echo "Usage: $prog_name [--]format file [name [cmd]]"
+echo "Usage: $prog_name [option(s)] [--]format file [name [cmd]]"
+echo
+echo "options:"
+echo " -f write a test known to fail"
echo
echo "argument(s):"
echo " file file containing the test case(s)"
@@ -443,11 +446,15 @@ echo " cmd command to be used (defaults to 'sparse \$f
}
##
-# do_format(file[, name[, cmd]]) - helps a test writer to format test-suite tags
+# do_format([options,] file[, name[, cmd]]) - helps a test writer to format test-suite tags
do_format()
{
+ fail=0
+
while [ $# -gt 1 ] ; do
case "$1" in
+ -f)
+ fail=1 ;;
help|-*)
do_format_help
return 0
@@ -480,6 +487,9 @@ _EOF
if [ "$fexit_value" -ne "0" ]; then
echo " * check-exit-value: $fexit_value"
fi
+ if [ $fail != 0 ]; then
+ echo " * check-known-to-fail"
+ fi
for stream in output error; do
if [ -s "$file.$stream.got" ]; then
echo " *"
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* [PATCH 28/28] testsuite: add support for 'format -l'
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (26 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 27/28] testsuite: add support for 'format -f' Luc Van Oostenryck
@ 2017-09-15 7:13 ` Luc Van Oostenryck
2017-09-15 17:03 ` [PATCH 00/28] testsuite improvements Christopher Li
2017-09-20 5:53 ` Christopher Li
29 siblings, 0 replies; 32+ messages in thread
From: Luc Van Oostenryck @ 2017-09-15 7:13 UTC (permalink / raw)
To: linux-sparse; +Cc: Christopher Li, Luc Van Oostenryck
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
validation/test-suite | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/validation/test-suite b/validation/test-suite
index b5bd937d5..8cbfb94e9 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -438,6 +438,7 @@ echo "Usage: $prog_name [option(s)] [--]format file [name [cmd]]"
echo
echo "options:"
echo " -f write a test known to fail"
+echo " -l write a test for linearized code"
echo
echo "argument(s):"
echo " file file containing the test case(s)"
@@ -449,12 +450,17 @@ echo " cmd command to be used (defaults to 'sparse \$f
# do_format([options,] file[, name[, cmd]]) - helps a test writer to format test-suite tags
do_format()
{
+ def_cmd="$default_cmd"
+ linear=0
fail=0
while [ $# -gt 1 ] ; do
case "$1" in
-f)
fail=1 ;;
+ -l)
+ def_cmd='test-linearize -Wno-decl $file'
+ linear=1 ;;
help|-*)
do_format_help
return 0
@@ -471,7 +477,7 @@ do_format()
fname="$2"
[ -z "$fname" ] && fname="$(basename "$1" .c)"
fcmd="$3"
- [ -z "$fcmd" ] && fcmd=$default_cmd
+ [ -z "$fcmd" ] && fcmd="$def_cmd"
cmd=`eval echo $default_path/$fcmd`
$cmd 1> $file.output.got 2> $file.error.got
@@ -490,6 +496,12 @@ _EOF
if [ $fail != 0 ]; then
echo " * check-known-to-fail"
fi
+ if [ $linear != 0 ]; then
+ echo " *"
+ echo " * check-output-ignore"
+ echo " * check-output-contains: xyz\\\\."
+ echo " * check-output-excludes: \\\\."
+ fi
for stream in output error; do
if [ -s "$file.$stream.got" ]; then
echo " *"
--
2.14.0
^ permalink raw reply related [flat|nested] 32+ messages in thread* Re: [PATCH 00/28] testsuite improvements
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (27 preceding siblings ...)
2017-09-15 7:13 ` [PATCH 28/28] testsuite: add support for 'format -l' Luc Van Oostenryck
@ 2017-09-15 17:03 ` Christopher Li
2017-09-15 17:35 ` Luc Van Oostenryck
2017-09-20 5:53 ` Christopher Li
29 siblings, 1 reply; 32+ messages in thread
From: Christopher Li @ 2017-09-15 17:03 UTC (permalink / raw)
To: Luc Van Oostenryck; +Cc: Linux-Sparse
On Fri, Sep 15, 2017 at 3:13 AM, Luc Van Oostenryck
<luc.vanoostenryck@gmail.com> wrote:
> This series contains diverse small improvements to
> the testsuite.
>
Thanks for the patch. It will keep me busy for a little while.
I am might not able to finish reading it today.
BTW, patchworks seems not updating. I haven't able to fetch
yours and some other patch on the sparse mailing list.
https://patchwork.kernel.org/project/linux-sparse/list/
Not a big deal. I can use your git tree branch.
Chris
^ permalink raw reply [flat|nested] 32+ messages in thread* Re: [PATCH 00/28] testsuite improvements
2017-09-15 7:13 [PATCH 00/28] testsuite improvements Luc Van Oostenryck
` (28 preceding siblings ...)
2017-09-15 17:03 ` [PATCH 00/28] testsuite improvements Christopher Li
@ 2017-09-20 5:53 ` Christopher Li
29 siblings, 0 replies; 32+ messages in thread
From: Christopher Li @ 2017-09-20 5:53 UTC (permalink / raw)
To: Luc Van Oostenryck; +Cc: Linux-Sparse
On Fri, Sep 15, 2017 at 3:13 AM, Luc Van Oostenryck
<luc.vanoostenryck@gmail.com> wrote:
> This series contains diverse small improvements to
> the testsuite.
>
>
> The series is also available for review in the git repository at:
> git://github.com/lucvoo/sparse.git testsuite-clean
I take a brief look at this series. It seems fine to me.
I will wait for the git pull for it then.
Chris
^ permalink raw reply [flat|nested] 32+ messages in thread