[PATCH 0/6] testsuite speedup

[PATCH 0/6] testsuite speedup

[Luc Van Oostenryck]

The goal of this series is to speedup the testsuite,
mainly by avoiding to unneedlessly spawn child processes.

The resulting speedup here is over 2.5, which is quite appreciable.

Luc Van Oostenryck (6):
  testsuite: get all tags in once
  testsuite: grep the expected output only when needed
  testsuite: grep the output patterns only when needed
  testsuite: use shell arithmetic instead of fork-execing expr
  testsuite: remove unneeded './' before commands
  testsuite: avoid fork+execing basename

 validation/backend/arithmetic-ops.c   |   2 +-
 validation/backend/array.c            |   2 +-
 validation/backend/bitwise-ops.c      |   2 +-
 validation/backend/bool-test.c        |   2 +-
 validation/backend/cast.c             |   2 +-
 validation/backend/cmp-ops.c          |   2 +-
 validation/backend/extern.c           |   2 +-
 validation/backend/function-ptr.c     |   2 +-
 validation/backend/hello.c            |   2 +-
 validation/backend/int-cond.c         |   2 +-
 validation/backend/load-type.c        |   2 +-
 validation/backend/logical-ops.c      |   2 +-
 validation/backend/loop.c             |   2 +-
 validation/backend/loop2.c            |   2 +-
 validation/backend/ptrcast.c          |   2 +-
 validation/backend/store-type.c       |   2 +-
 validation/backend/struct-access.c    |   2 +-
 validation/backend/struct.c           |   2 +-
 validation/backend/sum.c              |   2 +-
 validation/backend/union.c            |   2 +-
 validation/backend/void-return-type.c |   2 +-
 validation/prototype.c                |   2 +-
 validation/test-suite                 | 137 +++++++++++++++++++---------------
 23 files changed, 97 insertions(+), 84 deletions(-)

-- 
2.13.0

[PATCH 1/6] testsuite: get all tags in once

[Luc Van Oostenryck]

The test cases contain annotations with the following:
	check-<tagname>[: <value>]
These are extracted with grep & sed but this is done
separately for each tags which means that we need fork+exec
two processes for each possible tags.

Change this by trying to get all the tag+value in once by storing
the result in a variables instead of doing the grep & sed thing
at each time we need to check the tag.

This speedup the testsuite by around 30% for me.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 77 ++++++++++++++++++++++++++++-----------------------
 1 file changed, 43 insertions(+), 34 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index 904a2dbbd..fa4cd36cf 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -33,24 +33,39 @@ known_ko_tests=0
 [ -z "$V" ] && V=0
 
 ##
-# get_value(key, file) - gets the value of a (key, value) pair in file.
-#
-# returns 0 on success, 1 if the file does not have the key
-get_value()
-{
-	last_result=`grep $1: $2 | sed -e "s/^.*$1:\(.*\)$/\1/"`
-	[ -z "$last_result" ] && return 1
-	return 0
-}
-
-##
-# get_tag(key, file) - does file has the tag key in it ?
-#
-# returns 0 if present, 1 otherwise
-get_tag()
+# get_tag_value(file) - get the 'check-<...>' tags & values
+get_tag_value()
 {
-	last_result=`grep $1 $2`
-	return $?
+	check_name=""
+	check_command="$default_cmd"
+	check_exit_value=0
+	check_known_to_fail=0
+	check_error_ignore=0
+	check_output_ignore=0
+	check_output_contains=0
+	check_output_excludes=0
+	check_output_pattern=0
+
+	lines=$(grep 'check-[a-z-]*' $1 | \
+		sed -e 's/^.*\(check-[a-z-]*:*\) *\(.*\)$/\1 \2/')
+
+	while read tag val; do
+		#echo "-> tag: '$tag'"
+		#echo "-> val: '$val'"
+		case $tag in
+		check-name:)		check_name="$val" ;;
+		check-command:)		check_command="$val" ;;
+		check-exit-value:)	check_exit_value="$val" ;;
+		check-known-to-fail)	check_known_to_fail=1 ;;
+		check-error-ignore)	check_error_ignore=1 ;;
+		check-output-ignore)	check_output_ignore=1 ;;
+		check-output-contains:)	check_output_contains=1 ;;
+		check-output-excludes:)	check_output_excludes=1 ;;
+		check-output-pattern-)	check_output_pattern=1 ;;
+		esac
+	done << EOT
+	$lines
+EOT
 }
 
 ##
@@ -159,23 +174,22 @@ do_test()
 	test_failed=0
 	file="$1"
 
+	get_tag_value $file
+
 	# can this test be handled by test-suite ?
 	# (it has to have a check-name key in it)
-	get_value "check-name" $file
-	if [ "$?" -eq 1 ]; then
+	if [ "$check_name" = "" ]; then
 		echo "warning: test '$file' unhandled"
 		unhandled_tests=`expr $unhandled_tests + 1`
 		return 2
 	fi
-	test_name=$last_result
+	test_name="$check_name"
 
 	# does the test provide a specific command ?
-	cmd=`eval echo $default_path/$default_cmd`
-	get_value "check-command" $file
-	if [ "$?" -eq "0" ]; then
-		last_result=`echo $last_result | sed -e 's/^ *//'`
-		cmd=`eval echo $default_path/$last_result`
+	if [ "$check_command" = "" ]; then
+		check_command="$defaut_command"
 	fi
+	cmd=`eval echo $default_path/$check_command`
 
 	# check for disabled commands
 	set -- $cmd
@@ -199,12 +213,7 @@ do_test()
 		| grep -v check-error > "$file".error.expected
 
 	# grab the expected exit value
-	get_value "check-exit-value" $file
-	if [ "$?" -eq "0" ]; then
-		expected_exit_value=`echo $last_result | tr -d ' '`
-	else
-		expected_exit_value=0
-	fi
+	expected_exit_value=$check_exit_value
 	verbose "Expecting exit value: $expected_exit_value"
 
 
@@ -212,14 +221,14 @@ do_test()
 	$cmd 1> $file.output.got 2> $file.error.got
 	actual_exit_value=$?
 
-	get_tag "check-known-to-fail" $file
-	must_fail=`expr "$?" = 0`
+	must_fail=$check_known_to_fail
 	quiet=0
 	[ $must_fail -eq 1 ] && [ $V -eq 0 ] && quiet=1
 	known_ko_tests=`expr $known_ko_tests + $must_fail`
 
 	for stream in output error; do
-		grep -s -q "check-$stream-ignore" $file && continue
+		eval ignore=\$check_${stream}_ignore
+		[ $ignore -eq 1 ] && continue
 
 		diff -u "$file".$stream.expected "$file".$stream.got > "$file".$stream.diff
 		if [ "$?" -ne "0" ]; then
-- 
2.13.0

[PATCH 2/6] testsuite: grep the expected output only when needed

[Luc Van Oostenryck]

Currently the testsuite always try to extract the expected
output from the test case description.
This is done with grep & sed.

But in some case we're not interested to compare the actual
output with the expected one. And in those cases the grep & sed
are just wasted CPU cycles.

Fix this by extracting the expected output only when we know
that it won't be ignored.

This speedup the testsuite by a modest 1%.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index fa4cd36cf..ce2ca454c 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -206,12 +206,6 @@ do_test()
 
 	verbose "Using command       : $cmd"
 
-	# grab the expected output
-	sed -n '/check-output-start/,/check-output-end/p' $file \
-		| grep -v check-output > "$file".output.expected
-	sed -n '/check-error-start/,/check-error-end/p' $file \
-		| grep -v check-error > "$file".error.expected

[PATCH 3/6] testsuite: grep the output patterns only when needed

[Luc Van Oostenryck]

For some testcase, it is checked if the output contains or not
some given pattern, or we're checking the number of time a
pattern is present in the output.
This is done with grep and some glue.

But for most testcases there is nothing to check and this
grepping is just wasted CPU cycles.

Fix this by using the already known tags to see if we need to
do this grepping or not.

This speedup the testsuite by another 15%.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 33 +++++++++++++++++++--------------
 1 file changed, 19 insertions(+), 14 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index ce2ca454c..e33d97804 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -244,22 +244,27 @@ do_test()
 	fi
 
 	# verify the 'check-output-contains/excludes' tags
-	has_each_patterns "$file" 'check-output-contains' $file.output.got
-	if [ "$?" -ne "0" ]; then
-		error "Actual output doesn't contain some of the expected patterns."
-		test_failed=1
+	if [ $check_output_contains -eq 1 ]; then
+		has_each_patterns "$file" 'check-output-contains' $file.output.got
+		if [ "$?" -ne "0" ]; then
+			error "Actual output doesn't contain some of the expected patterns."
+			test_failed=1
+		fi
 	fi
-	has_none_patterns "$file" 'check-output-excludes' $file.output.got
-	if [ "$?" -ne "0" ]; then
-		error "Actual output contains some patterns which are not expected."
-		test_failed=1
+	if [ $check_output_excludes -eq 1 ]; then
+		has_none_patterns "$file" 'check-output-excludes' $file.output.got
+		if [ "$?" -ne "0" ]; then
+			error "Actual output contains some patterns which are not expected."
+			test_failed=1
+		fi
 	fi

[PATCH 4/6] testsuite: use shell arithmetic instead of fork-execing expr

[Luc Van Oostenryck]

The tessuite use a few counters to keep track of things.
These counters are incremented via the 'expr' command.

But this command is not a shell builtin and need thus
to be fork+execed which need more CPU ressources than
using some builtin operation.

Change this by doing all the arithmetic via the portable
'$(( ... ))' shell builtin.

This speedup the testsuite by another 10%.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index e33d97804..da9d88159 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -180,7 +180,7 @@ do_test()
 	# (it has to have a check-name key in it)
 	if [ "$check_name" = "" ]; then
 		echo "warning: test '$file' unhandled"
-		unhandled_tests=`expr $unhandled_tests + 1`
+		unhandled_tests=$(($unhandled_tests + 1))
 		return 2
 	fi
 	test_name="$check_name"
@@ -196,7 +196,7 @@ do_test()
 	base_cmd=`basename $1`
 	for i in $disabled_cmds; do
 		if [ "$i" = "$base_cmd" ] ; then
-			disabled_tests=`expr $disabled_tests + 1`
+			disabled_tests=$(($disabled_tests + 1))
 			echo "     DISABLE $test_name ($file)"
 			return 3
 		fi
@@ -218,7 +218,7 @@ do_test()
 	must_fail=$check_known_to_fail
 	quiet=0
 	[ $must_fail -eq 1 ] && [ $V -eq 0 ] && quiet=1
-	known_ko_tests=`expr $known_ko_tests + $must_fail`
+	known_ko_tests=$(($known_ko_tests + $must_fail))
 
 	for stream in output error; do
 		eval ignore=\$check_${stream}_ignore
@@ -279,9 +279,9 @@ do_test()
 	fi
 
 	if [ "$test_failed" -eq "1" ]; then
-		ko_tests=`expr $ko_tests + 1`
+		ko_tests=$(($ko_tests + 1))
 	else
-		ok_tests=`expr $ok_tests + 1`
+		ok_tests=$(($ok_tests + 1))
 		rm -f $file.{error,output}.{expected,got,diff}
 	fi
 	return $test_failed
@@ -294,7 +294,7 @@ do_test_suite()
 	done
 
 	# prints some numbers
-	tests_nr=`expr $ok_tests + $ko_tests`
+	tests_nr=$(($ok_tests + $ko_tests))
 	echo -n "Out of $tests_nr tests, $ok_tests passed, $ko_tests failed"
 	echo " ($known_ko_tests of them are known to fail)"
 	if [ "$unhandled_tests" -ne "0" ]; then
-- 
2.13.0

[PATCH 5/6] testsuite: remove unneeded './' before commands

[Luc Van Oostenryck]

Some testcase have their command specified as './<command name>'
but the './' part is unneeded as all commands are first prefixed
with '../' before being run.

Furthermore, the possible presence of these './' force the
use of 'basename' when filtering out the disabled commands.

Change this by stripping the './' where is was used.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/backend/arithmetic-ops.c   | 2 +-
 validation/backend/array.c            | 2 +-
 validation/backend/bitwise-ops.c      | 2 +-
 validation/backend/bool-test.c        | 2 +-
 validation/backend/cast.c             | 2 +-
 validation/backend/cmp-ops.c          | 2 +-
 validation/backend/extern.c           | 2 +-
 validation/backend/function-ptr.c     | 2 +-
 validation/backend/hello.c            | 2 +-
 validation/backend/int-cond.c         | 2 +-
 validation/backend/load-type.c        | 2 +-
 validation/backend/logical-ops.c      | 2 +-
 validation/backend/loop.c             | 2 +-
 validation/backend/loop2.c            | 2 +-
 validation/backend/ptrcast.c          | 2 +-
 validation/backend/store-type.c       | 2 +-
 validation/backend/struct-access.c    | 2 +-
 validation/backend/struct.c           | 2 +-
 validation/backend/sum.c              | 2 +-
 validation/backend/union.c            | 2 +-
 validation/backend/void-return-type.c | 2 +-
 validation/prototype.c                | 2 +-
 22 files changed, 22 insertions(+), 22 deletions(-)

diff --git a/validation/backend/arithmetic-ops.c b/validation/backend/arithmetic-ops.c
index 7c299d033..55996d9c3 100644
--- a/validation/backend/arithmetic-ops.c
+++ b/validation/backend/arithmetic-ops.c
@@ -90,5 +90,5 @@ static unsigned int umod(unsigned int x, unsigned int y)
 
 /*
  * check-name: Arithmetic operator code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/array.c b/validation/backend/array.c
index bd3ec5960..4e87b79ee 100644
--- a/validation/backend/array.c
+++ b/validation/backend/array.c
@@ -2,5 +2,5 @@ static char array[128];
 
 /*
  * check-name: Array code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/bitwise-ops.c b/validation/backend/bitwise-ops.c
index 659c76398..9f61fca14 100644
--- a/validation/backend/bitwise-ops.c
+++ b/validation/backend/bitwise-ops.c
@@ -60,5 +60,5 @@ static unsigned int unot(unsigned int x)
 
 /*
  * check-name: Bitwise operator code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/bool-test.c b/validation/backend/bool-test.c
index a6f33a1ac..cd20a4bd9 100644
--- a/validation/backend/bool-test.c
+++ b/validation/backend/bool-test.c
@@ -5,5 +5,5 @@ static _Bool return_false(void)
 
 /*
  * check-name: Boolean type code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/cast.c b/validation/backend/cast.c
index 598b16aab..4ca531b53 100644
--- a/validation/backend/cast.c
+++ b/validation/backend/cast.c
@@ -46,5 +46,5 @@ DEFINE_CASTS(double)
 
 /*
  * check-name: Cast code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/cmp-ops.c b/validation/backend/cmp-ops.c
index a5f736d7a..c7918da1a 100644
--- a/validation/backend/cmp-ops.c
+++ b/validation/backend/cmp-ops.c
@@ -80,5 +80,5 @@ static int setfge(float x, float y)
 
 /*
  * check-name: Comparison operator code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/extern.c b/validation/backend/extern.c
index 24cbae557..b5c14d842 100644
--- a/validation/backend/extern.c
+++ b/validation/backend/extern.c
@@ -7,5 +7,5 @@ static unsigned long bar(void)
 
 /*
  * check-name: Extern symbol code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/function-ptr.c b/validation/backend/function-ptr.c
index fc022b3cd..35fb678af 100644
--- a/validation/backend/function-ptr.c
+++ b/validation/backend/function-ptr.c
@@ -7,5 +7,5 @@ static int run(fn_t fn, int x, int y)
 
 /*
  * check-name: Function pointer code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/hello.c b/validation/backend/hello.c
index 79905004e..b0e514be4 100644
--- a/validation/backend/hello.c
+++ b/validation/backend/hello.c
@@ -9,5 +9,5 @@ int main(int argc, char *argv[])
 
 /*
  * check-name: 'hello, world' code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/int-cond.c b/validation/backend/int-cond.c
index 48b25a776..2a5a43482 100644
--- a/validation/backend/int-cond.c
+++ b/validation/backend/int-cond.c
@@ -26,5 +26,5 @@ static long bar_bool(_Bool a, long b, long c)
 
 /*
  * check-name: Non-bool condition values in branch/select
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/load-type.c b/validation/backend/load-type.c
index 80416cad9..825567694 100644
--- a/validation/backend/load-type.c
+++ b/validation/backend/load-type.c
@@ -8,5 +8,5 @@ static void test(void) {
 
 /*
  * check-name: Type of loaded objects
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/logical-ops.c b/validation/backend/logical-ops.c
index 8b2a6a85b..ec5b7c6dc 100644
--- a/validation/backend/logical-ops.c
+++ b/validation/backend/logical-ops.c
@@ -20,5 +20,5 @@ static unsigned int uor_bool(unsigned int x, unsigned int y)
 
 /*
  * check-name: Logical operator code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/loop.c b/validation/backend/loop.c
index 31054f52f..181a00689 100644
--- a/validation/backend/loop.c
+++ b/validation/backend/loop.c
@@ -17,5 +17,5 @@ int foo (int x)
 
 /*
  * check-name: Loops
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/loop2.c b/validation/backend/loop2.c
index 279af2146..d827b5d32 100644
--- a/validation/backend/loop2.c
+++ b/validation/backend/loop2.c
@@ -10,5 +10,5 @@ static void test(void)
 
 /*
  * check-name: Loops with unused counter
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/ptrcast.c b/validation/backend/ptrcast.c
index 46f8add82..8e4df0700 100644
--- a/validation/backend/ptrcast.c
+++ b/validation/backend/ptrcast.c
@@ -5,5 +5,5 @@ static char *ptrcast(unsigned long *x)
 
 /*
  * check-name: Pointer cast code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/store-type.c b/validation/backend/store-type.c
index 9e2ce73fa..2246a3b3b 100644
--- a/validation/backend/store-type.c
+++ b/validation/backend/store-type.c
@@ -8,5 +8,5 @@ static void set(struct foo *f)
 
 /*
  * check-name: Type of stored objects
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/struct-access.c b/validation/backend/struct-access.c
index 884b4706d..3d193e331 100644
--- a/validation/backend/struct-access.c
+++ b/validation/backend/struct-access.c
@@ -24,5 +24,5 @@ static void store_d(struct st *st, int *d)
 
 /*
  * check-name: struct access code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/struct.c b/validation/backend/struct.c
index 905339afb..083b05c3b 100644
--- a/validation/backend/struct.c
+++ b/validation/backend/struct.c
@@ -21,5 +21,5 @@ static struct unnamed un;
 
 /*
  * check-name: Struct code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/sum.c b/validation/backend/sum.c
index c9451d4bc..06042999f 100644
--- a/validation/backend/sum.c
+++ b/validation/backend/sum.c
@@ -19,7 +19,7 @@ int main(int argc, char **argv)
 
 /*
  * check-name: sum from 1 to n
- * check-command: ./sparsei $file
+ * check-command: sparsei $file
  *
  * check-output-start
 15
diff --git a/validation/backend/union.c b/validation/backend/union.c
index e155f6ad1..7ec35c7a8 100644
--- a/validation/backend/union.c
+++ b/validation/backend/union.c
@@ -8,5 +8,5 @@ static union foo foo;
 
 /*
  * check-name: Union code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/backend/void-return-type.c b/validation/backend/void-return-type.c
index b282fdee0..e540a9d41 100644
--- a/validation/backend/void-return-type.c
+++ b/validation/backend/void-return-type.c
@@ -9,5 +9,5 @@ static void *bar(void *p)
 
 /*
  * check-name: void return type code generation
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
diff --git a/validation/prototype.c b/validation/prototype.c
index 481cc93e0..c338e1619 100644
--- a/validation/prototype.c
+++ b/validation/prototype.c
@@ -2,5 +2,5 @@ static int prototype(void);
 
 /*
  * check-name: Compile skip function prototype
- * check-command: ./sparsec -c $file -o tmp.o
+ * check-command: sparsec -c $file -o tmp.o
  */
-- 
2.13.0

[PATCH 6/6] testsuite: avoid fork+execing basename

[Luc Van Oostenryck]

Some testcase (the ones related to sparse-llvm) are disabled if
the needed support is not present. This is done by checking
the name of the command used by the testcase.

The previous possible presence of './' before the command meant
that the command was checked via the 'basename' command, which
need to ne fork+execed.
Since the './' have now been stripped from all command names
this is no more needed.

Change this by comparing directly the command name.

This speedup the testsuite by another 6%.

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 validation/test-suite | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/validation/test-suite b/validation/test-suite
index da9d88159..3056fce90 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -189,11 +189,10 @@ do_test()
 	if [ "$check_command" = "" ]; then
 		check_command="$defaut_command"
 	fi
-	cmd=`eval echo $default_path/$check_command`
 
 	# check for disabled commands
-	set -- $cmd
-	base_cmd=`basename $1`
+	set -- $check_command
+	base_cmd=$1
 	for i in $disabled_cmds; do
 		if [ "$i" = "$base_cmd" ] ; then
 			disabled_tests=$(($disabled_tests + 1))
@@ -202,6 +201,8 @@ do_test()
 		fi
 	done
 
+	cmd=`eval echo $default_path/$check_command`
+
 	echo "     TEST    $test_name ($file)"
 
 	verbose "Using command       : $cmd"
-- 
2.13.0

Re: [PATCH 0/6] testsuite speedup

[Ramsay Jones]

On 28/05/17 20:29, Luc Van Oostenryck wrote:
> The goal of this series is to speedup the testsuite,
> mainly by avoiding to unneedlessly spawn child processes.
> 
> The resulting speedup here is over 2.5, which is quite appreciable.

I would have liked to try these out, particularly on cygwin, but I
could not get them to apply. (patch #4 has a line: 'rm -f $file.{error,output}.{expected,got,diff}' in the context which I don't have).

I was attempting to apply on top of 'sparse-next-20170516'.
Should I apply to a different branch?
Were they built on a branch I don't have?

ATB,
Ramsay Jones

Re: [PATCH 0/6] testsuite speedup

[Luc Van Oostenryck]

On Mon, May 29, 2017 at 12:04 AM, Ramsay Jones
<ramsay@ramsayjones.plus.com> wrote:
>
>
> On 28/05/17 20:29, Luc Van Oostenryck wrote:
>> The goal of this series is to speedup the testsuite,
>> mainly by avoiding to unneedlessly spawn child processes.
>>
>> The resulting speedup here is over 2.5, which is quite appreciable.
>
> I would have liked to try these out, particularly on cygwin, but I
> could not get them to apply. (patch #4 has a line: 'rm -f $file.{error,output}.{expected,got,diff}' in the context which I don't have).
>
> I was attempting to apply on top of 'sparse-next-20170516'.
> Should I apply to a different branch?
> Were they built on a branch I don't have?

Ah yes, sorry for that.
It should be applied on top of the next sparse-next but Chris hasn't yet
pulled it.

If you wish you can directly use the branch on my github tree:
        git://github.com/lucvoo/sparse.git testsuite-faster

-- Luc

Re: [PATCH 0/6] testsuite speedup

[Ramsay Jones]

On 28/05/17 23:39, Luc Van Oostenryck wrote:
> On Mon, May 29, 2017 at 12:04 AM, Ramsay Jones
> <ramsay@ramsayjones.plus.com> wrote:
>>
>>
>> On 28/05/17 20:29, Luc Van Oostenryck wrote:
>>> The goal of this series is to speedup the testsuite,
>>> mainly by avoiding to unneedlessly spawn child processes.
>>>
>>> The resulting speedup here is over 2.5, which is quite appreciable.
>>
>> I would have liked to try these out, particularly on cygwin, but I
>> could not get them to apply. (patch #4 has a line: 'rm -f $file.{error,output}.{expected,got,diff}' in the context which I don't have).
>>
>> I was attempting to apply on top of 'sparse-next-20170516'.
>> Should I apply to a different branch?
>> Were they built on a branch I don't have?
> 
> Ah yes, sorry for that.
> It should be applied on top of the next sparse-next but Chris hasn't yet
> pulled it.
> 
> If you wish you can directly use the branch on my github tree:
>         git://github.com/lucvoo/sparse.git testsuite-faster

Thanks. I very quickly tried it on cygwin and the time to run the
testsuite fell from 2m14.607s down to 1m0.298s. So, a nice time
saver. ;-)

ATB,
Ramsay Jones

Re: [PATCH 0/6] testsuite speedup

[Christopher Li]

On Sun, May 28, 2017 at 12:29 PM, Luc Van Oostenryck
<luc.vanoostenryck@gmail.com> wrote:
> The goal of this series is to speedup the testsuite,
> mainly by avoiding to unneedlessly spawn child processes.
>
> The resulting speedup here is over 2.5, which is quite appreciable.

This series looks fine to me.

BTW, another way to speed up the test suit is using makefile to invoke
more than one job of test suite. Currently all test suit are execute
one by one. But there is no inter dependency between the test.
So in theory we can change the Makefile a bit to run more than one
job on the test suit using pattern rules.

Chris

Re: [PATCH 0/6] testsuite speedup

[Luc Van Oostenryck]

On Thu, Jun 1, 2017 at 11:16 AM, Christopher Li <sparse@chrisli.org> wrote:
> BTW, another way to speed up the test suit is using makefile to invoke
> more than one job of test suite. Currently all test suit are execute
> one by one. But there is no inter dependency between the test.
> So in theory we can change the Makefile a bit to run more than one
> job on the test suit using pattern rules.

Yes, but there is a few issues with this:
- 'test-suite single file.c' doesn't exit with the right return code yet
- parallelizing tests with make won't return the summary of the tests
  (which may or may not be needed)
- the error messages could be mixed in
- make would stop at the first error (which may be OK or not)
- would be better used with something like 'test-suite --very-quiet file.c'

-- Luc

Re: [PATCH 0/6] testsuite speedup

[Christopher Li]

On Thu, Jun 1, 2017 at 7:05 AM, Luc Van Oostenryck
<luc.vanoostenryck@gmail.com> wrote:
> Yes, but there is a few issues with this:
> - 'test-suite single file.c' doesn't exit with the right return code yet
> - parallelizing tests with make won't return the summary of the tests
>   (which may or may not be needed)
> - the error messages could be mixed in
> - make would stop at the first error (which may be OK or not)
> - would be better used with something like 'test-suite --very-quiet file.c'

How about the parallel running just run individual test and
save the result/error in per test file. Have the a report target that process
all the individual run result. That will get rid of the result
ordering issue, no
error mix in, no early stop etc.

test-suite will need to change a bit for the separate report phrase.
I am not suggesting make that into this series. This series is fine
as it is. I am only making suggestion that parallelizing can speed
up the check run by a few times.

Chris

Re: [PATCH 0/6] testsuite speedup

[Luc Van Oostenryck]

On Fri, Jun 2, 2017 at 9:29 AM, Christopher Li <sparse@chrisli.org> wrote:
> On Thu, Jun 1, 2017 at 7:05 AM, Luc Van Oostenryck
> <luc.vanoostenryck@gmail.com> wrote:
>> Yes, but there is a few issues with this:
>> - 'test-suite single file.c' doesn't exit with the right return code yet
>> - parallelizing tests with make won't return the summary of the tests
>>   (which may or may not be needed)
>> - the error messages could be mixed in
>> - make would stop at the first error (which may be OK or not)
>> - would be better used with something like 'test-suite --very-quiet file.c'
>
> How about the parallel running just run individual test and
> save the result/error in per test file. Have the a report target that process
> all the individual run result. That will get rid of the result
> ordering issue, no
> error mix in, no early stop etc.

Yes, it's a possibility.

> test-suite will need to change a bit for the separate report phrase.
> I am not suggesting make that into this series. This series is fine
> as it is. I am only making suggestion that parallelizing can speed
> up the check run by a few times.

Yes, it's how I understood it.
I'll see about it next time I make some changes at the testsuite.

-- Luc