[Qemu-devel] [kvm-unit-tests PATCH v6 0/3] run_tests: support concurrent test execution

[Qemu-devel] [kvm-unit-tests PATCH v6 0/3] run_tests: support concurrent test execution

[Peter Xu]

v6:
- some tunes on how to rm/mv logs/logs.old [Drew]
- fix errno to 2 when param parse fail [Drew]
- add one more patch to fix *) case errno [Drew]
  (please either take/squash/... this one)
- comment fix again [Drew]

v5:
- add "/" at start/end of line where proper [Drew]
- remove useless newline in Makefile [Drew]
- don't check "mv" since it won't fail [Drew]
- avoid using '"s in (( )) [Drew]
- comment fix [Drew]

v4:
- add .gitignore for logs/ [Drew]
- instead of create globals.bash, renaming function.bash into
  common.bash, put globals inside [Drew]
- instead of removing logs/ directly when start run_tests, move it
  into logs.old so we at least have the last time result cached [Drew]
- s/ut_/unittest_/ through the whole series [Drew]
- remove unittest_log_summary var [Drew]
- remove radim's s-b in patch 2 since it does not suite [Drew]
- tiny fix on the usage lines [Drew]
- use bash arithmetic where proper [Drew]
- remove ut_in_parallel since not used [Drew]

v3:
- better handling for ctrl-c during run_tests.sh [Radim]

v2:
- patch 1: do per-test logging in all cases
- patch 2: throw away task.bash, instead, take Radim's suggestion to
  use jobs

run_tests.sh is getting slower. Maybe it's time to let it run faster.
An obvious issue is that, we were running the tests sequentially in
the past.

This series provides another new "-j" parameter. "-j 8" means we run
the tests on 8 task queues. That'll fasten the script a lot. A very
quick test of mine shows 3x speed boost with 8 task queues.

Please review, thanks.

Peter Xu (3):
  run_tests: fix errno for param parsing
  run_tests: put logs into per-test file
  run_tests: allow run tests in parallel

 .gitignore                              |  3 ++-
 Makefile                                |  5 ++---
 run_tests.sh                            | 33 +++++++++++++++++++++++----------
 scripts/{functions.bash => common.bash} | 27 +++++++++++++++++++++++++--
 scripts/mkstandalone.sh                 |  2 +-
 5 files changed, 53 insertions(+), 17 deletions(-)
 rename scripts/{functions.bash => common.bash} (63%)

-- 
2.7.4

[Qemu-devel] [kvm-unit-tests PATCH v6 1/3] run_tests: fix errno for param parsing

[Peter Xu]

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 run_tests.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/run_tests.sh b/run_tests.sh
index 254129d..2cfa365 100755
--- a/run_tests.sh
+++ b/run_tests.sh
@@ -41,7 +41,7 @@ while getopts "g:hv" opt; do
             verbose="yes"
             ;;
         *)
-            exit 1
+            exit 2
             ;;
     esac
 done
-- 
2.7.4

Re: [Qemu-devel] [kvm-unit-tests PATCH v6 1/3] run_tests: fix errno for param parsing

[Andrew Jones]

On Thu, Jan 12, 2017 at 11:36:20AM +0800, Peter Xu wrote:
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  run_tests.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/run_tests.sh b/run_tests.sh
> index 254129d..2cfa365 100755
> --- a/run_tests.sh
> +++ b/run_tests.sh
> @@ -41,7 +41,7 @@ while getopts "g:hv" opt; do
>              verbose="yes"
>              ;;
>          *)
> -            exit 1
> +            exit 2
>              ;;
>      esac
>  done
> -- 
> 2.7.4
> 
>

Reviewed-by: Andrew Jones <drjones@redhat.com> 

[Qemu-devel] [kvm-unit-tests PATCH v6 2/3] run_tests: put logs into per-test file

[Peter Xu]

We were using test.log before to keep all the test logs. This patch
creates one log file per test case under logs/ directory with name
"TESTNAME.log". Meanwhile, we will keep the last time log into
logs.old/.

Renaming scripts/functions.bash into scripts/common.bash to store some
more global variables.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 .gitignore                              |  3 ++-
 Makefile                                |  5 ++---
 run_tests.sh                            | 19 ++++++++++++-------
 scripts/{functions.bash => common.bash} | 13 +++++++++++--
 scripts/mkstandalone.sh                 |  2 +-
 5 files changed, 28 insertions(+), 14 deletions(-)
 rename scripts/{functions.bash => common.bash} (75%)

diff --git a/.gitignore b/.gitignore
index 3155418..2213b9b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -12,7 +12,8 @@ cscope.*
 /lib/asm
 /config.mak
 /*-run
-/test.log
 /msr.out
 /tests
 /build-head
+/logs/
+/logs.old/
diff --git a/Makefile b/Makefile
index a32333b..844bacc 100644
--- a/Makefile
+++ b/Makefile
@@ -94,9 +94,8 @@ libfdt_clean:
 	$(LIBFDT_objdir)/.*.d
 
 distclean: clean libfdt_clean
-	$(RM) lib/asm config.mak $(TEST_DIR)-run test.log msr.out cscope.* \
-	      build-head
-	$(RM) -r tests
+	$(RM) lib/asm config.mak $(TEST_DIR)-run msr.out cscope.* build-head
+	$(RM) -r tests logs logs.old
 
 cscope: cscope_dirs = lib lib/libfdt lib/linux $(TEST_DIR) $(ARCH_LIBDIRS) lib/asm-generic
 cscope:
diff --git a/run_tests.sh b/run_tests.sh
index 2cfa365..afd3d95 100755
--- a/run_tests.sh
+++ b/run_tests.sh
@@ -7,7 +7,7 @@ if [ ! -f config.mak ]; then
     exit 1
 fi
 source config.mak
-source scripts/functions.bash
+source scripts/common.bash
 
 function usage()
 {
@@ -46,17 +46,22 @@ while getopts "g:hv" opt; do
     esac
 done
 
-RUNTIME_log_stderr () { cat >> test.log; }
+# RUNTIME_log_file will be configured later
+RUNTIME_log_stderr () { cat >> $RUNTIME_log_file; }
 RUNTIME_log_stdout () {
     if [ "$PRETTY_PRINT_STACKS" = "yes" ]; then
-        ./scripts/pretty_print_stacks.py $1 >> test.log
+        ./scripts/pretty_print_stacks.py $1 >> $RUNTIME_log_file
     else
-        cat >> test.log
+        cat >> $RUNTIME_log_file
     fi
 }
 
-
 config=$TEST_DIR/unittests.cfg
-rm -f test.log
-printf "BUILD_HEAD=$(cat build-head)\n\n" > test.log
+
+rm -rf $unittest_log_dir.old
+[ -d $unittest_log_dir ] && mv $unittest_log_dir $unittest_log_dir.old
+mkdir $unittest_log_dir || exit 2
+
+echo "BUILD_HEAD=$(cat build-head)" > $unittest_log_dir/SUMMARY
+
 for_each_unittest $config run
diff --git a/scripts/functions.bash b/scripts/common.bash
similarity index 75%
rename from scripts/functions.bash
rename to scripts/common.bash
index ee9143c..2dd7360 100644
--- a/scripts/functions.bash
+++ b/scripts/common.bash
@@ -1,3 +1,12 @@
+: ${unittest_log_dir:=logs}
+
+function run_task()
+{
+	local testname="$2"
+
+	RUNTIME_log_file="${unittest_log_dir}/${testname}.log"
+	"$@"
+}
 
 function for_each_unittest()
 {
@@ -17,7 +26,7 @@ function for_each_unittest()
 
 	while read -u $fd line; do
 		if [[ "$line" =~ ^\[(.*)\]$ ]]; then
-			"$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
+			run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
 			testname=${BASH_REMATCH[1]}
 			smp=1
 			kernel=""
@@ -45,6 +54,6 @@ function for_each_unittest()
 			timeout=${BASH_REMATCH[1]}
 		fi
 	done
-	"$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
+	run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
 	exec {fd}<&-
 }
diff --git a/scripts/mkstandalone.sh b/scripts/mkstandalone.sh
index d2bae19..3c1938e 100755
--- a/scripts/mkstandalone.sh
+++ b/scripts/mkstandalone.sh
@@ -5,7 +5,7 @@ if [ ! -f config.mak ]; then
 	exit 1
 fi
 source config.mak
-source scripts/functions.bash
+source scripts/common.bash
 
 escape ()
 {
-- 
2.7.4

Re: [Qemu-devel] [kvm-unit-tests PATCH v6 2/3] run_tests: put logs into per-test file

[Andrew Jones]

On Thu, Jan 12, 2017 at 11:36:21AM +0800, Peter Xu wrote:
> We were using test.log before to keep all the test logs. This patch
> creates one log file per test case under logs/ directory with name
> "TESTNAME.log". Meanwhile, we will keep the last time log into
> logs.old/.
> 
> Renaming scripts/functions.bash into scripts/common.bash to store some
> more global variables.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  .gitignore                              |  3 ++-
>  Makefile                                |  5 ++---
>  run_tests.sh                            | 19 ++++++++++++-------
>  scripts/{functions.bash => common.bash} | 13 +++++++++++--
>  scripts/mkstandalone.sh                 |  2 +-
>  5 files changed, 28 insertions(+), 14 deletions(-)
>  rename scripts/{functions.bash => common.bash} (75%)
> 
> diff --git a/.gitignore b/.gitignore
> index 3155418..2213b9b 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -12,7 +12,8 @@ cscope.*
>  /lib/asm
>  /config.mak
>  /*-run
> -/test.log
>  /msr.out
>  /tests
>  /build-head
> +/logs/
> +/logs.old/
> diff --git a/Makefile b/Makefile
> index a32333b..844bacc 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -94,9 +94,8 @@ libfdt_clean:
>  	$(LIBFDT_objdir)/.*.d
>  
>  distclean: clean libfdt_clean
> -	$(RM) lib/asm config.mak $(TEST_DIR)-run test.log msr.out cscope.* \
> -	      build-head
> -	$(RM) -r tests
> +	$(RM) lib/asm config.mak $(TEST_DIR)-run msr.out cscope.* build-head
> +	$(RM) -r tests logs logs.old
>  
>  cscope: cscope_dirs = lib lib/libfdt lib/linux $(TEST_DIR) $(ARCH_LIBDIRS) lib/asm-generic
>  cscope:
> diff --git a/run_tests.sh b/run_tests.sh
> index 2cfa365..afd3d95 100755
> --- a/run_tests.sh
> +++ b/run_tests.sh
> @@ -7,7 +7,7 @@ if [ ! -f config.mak ]; then
>      exit 1
>  fi
>  source config.mak
> -source scripts/functions.bash
> +source scripts/common.bash
>  
>  function usage()
>  {
> @@ -46,17 +46,22 @@ while getopts "g:hv" opt; do
>      esac
>  done
>  
> -RUNTIME_log_stderr () { cat >> test.log; }
> +# RUNTIME_log_file will be configured later
> +RUNTIME_log_stderr () { cat >> $RUNTIME_log_file; }
>  RUNTIME_log_stdout () {
>      if [ "$PRETTY_PRINT_STACKS" = "yes" ]; then
> -        ./scripts/pretty_print_stacks.py $1 >> test.log
> +        ./scripts/pretty_print_stacks.py $1 >> $RUNTIME_log_file
>      else
> -        cat >> test.log
> +        cat >> $RUNTIME_log_file
>      fi
>  }
>  
> -
>  config=$TEST_DIR/unittests.cfg
> -rm -f test.log
> -printf "BUILD_HEAD=$(cat build-head)\n\n" > test.log
> +
> +rm -rf $unittest_log_dir.old
> +[ -d $unittest_log_dir ] && mv $unittest_log_dir $unittest_log_dir.old
> +mkdir $unittest_log_dir || exit 2
> +
> +echo "BUILD_HEAD=$(cat build-head)" > $unittest_log_dir/SUMMARY
> +

nit: to be 100% correct all the references to the log dir should have
"'s around them in order to handle spaces, or other shell ambiguous
characters, in the name. For example, mkdir $log_dir, where $log_dir
is "my tests" would create two directories, "my" and "tests" without
the quotes. Anyway, I'm not too worried about someone choosing a
weird log dir name, nor even changing it from the default.

Isn't bash fun :-)

>  for_each_unittest $config run
> diff --git a/scripts/functions.bash b/scripts/common.bash
> similarity index 75%
> rename from scripts/functions.bash
> rename to scripts/common.bash
> index ee9143c..2dd7360 100644
> --- a/scripts/functions.bash
> +++ b/scripts/common.bash
> @@ -1,3 +1,12 @@
> +: ${unittest_log_dir:=logs}
> +
> +function run_task()
> +{
> +	local testname="$2"
> +
> +	RUNTIME_log_file="${unittest_log_dir}/${testname}.log"
> +	"$@"
> +}
>  
>  function for_each_unittest()
>  {
> @@ -17,7 +26,7 @@ function for_each_unittest()
>  
>  	while read -u $fd line; do
>  		if [[ "$line" =~ ^\[(.*)\]$ ]]; then
> -			"$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
> +			run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
>  			testname=${BASH_REMATCH[1]}
>  			smp=1
>  			kernel=""
> @@ -45,6 +54,6 @@ function for_each_unittest()
>  			timeout=${BASH_REMATCH[1]}
>  		fi
>  	done
> -	"$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
> +	run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
>  	exec {fd}<&-
>  }
> diff --git a/scripts/mkstandalone.sh b/scripts/mkstandalone.sh
> index d2bae19..3c1938e 100755
> --- a/scripts/mkstandalone.sh
> +++ b/scripts/mkstandalone.sh
> @@ -5,7 +5,7 @@ if [ ! -f config.mak ]; then
>  	exit 1
>  fi
>  source config.mak
> -source scripts/functions.bash
> +source scripts/common.bash
>  
>  escape ()
>  {
> -- 
> 2.7.4
> 
>

Reviewed-by: Andrew Jones <drjones@redhat.com>

Re: [Qemu-devel] [kvm-unit-tests PATCH v6 2/3] run_tests: put logs into per-test file

[Peter Xu]

On Thu, Jan 12, 2017 at 11:04:05AM +0100, Andrew Jones wrote:

[...]

> >  config=$TEST_DIR/unittests.cfg
> > -rm -f test.log
> > -printf "BUILD_HEAD=$(cat build-head)\n\n" > test.log
> > +
> > +rm -rf $unittest_log_dir.old
> > +[ -d $unittest_log_dir ] && mv $unittest_log_dir $unittest_log_dir.old
> > +mkdir $unittest_log_dir || exit 2
> > +
> > +echo "BUILD_HEAD=$(cat build-head)" > $unittest_log_dir/SUMMARY
> > +
> 
> nit: to be 100% correct all the references to the log dir should have
> "'s around them in order to handle spaces, or other shell ambiguous
> characters, in the name. For example, mkdir $log_dir, where $log_dir
> is "my tests" would create two directories, "my" and "tests" without
> the quotes. Anyway, I'm not too worried about someone choosing a
> weird log dir name, nor even changing it from the default.
> 
> Isn't bash fun :-)

It is, as long as it won't break my system... (Once I accidentally did
a "rm -rf /lib" in Bash when I was still using FreeBSD... Bash became
less funny since then ;-).

I'll add them if I'm going to have another spin.

[...]

> Reviewed-by: Andrew Jones <drjones@redhat.com>

Thanks for reviewing!

-- peterx

Re: [Qemu-devel] [kvm-unit-tests PATCH v6 2/3] run_tests: put logs into per-test file

[Paolo Bonzini]

On 12/01/2017 04:36, Peter Xu wrote:
> We were using test.log before to keep all the test logs. This patch
> creates one log file per test case under logs/ directory with name
> "TESTNAME.log". Meanwhile, we will keep the last time log into
> logs.old/.
> 
> Renaming scripts/functions.bash into scripts/common.bash to store some
> more global variables.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  .gitignore                              |  3 ++-
>  Makefile                                |  5 ++---
>  run_tests.sh                            | 19 ++++++++++++-------
>  scripts/{functions.bash => common.bash} | 13 +++++++++++--
>  scripts/mkstandalone.sh                 |  2 +-
>  5 files changed, 28 insertions(+), 14 deletions(-)
>  rename scripts/{functions.bash => common.bash} (75%)
> 
> diff --git a/.gitignore b/.gitignore
> index 3155418..2213b9b 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -12,7 +12,8 @@ cscope.*
>  /lib/asm
>  /config.mak
>  /*-run
> -/test.log
>  /msr.out
>  /tests
>  /build-head
> +/logs/
> +/logs.old/
> diff --git a/Makefile b/Makefile
> index a32333b..844bacc 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -94,9 +94,8 @@ libfdt_clean:
>  	$(LIBFDT_objdir)/.*.d
>  
>  distclean: clean libfdt_clean
> -	$(RM) lib/asm config.mak $(TEST_DIR)-run test.log msr.out cscope.* \
> -	      build-head
> -	$(RM) -r tests
> +	$(RM) lib/asm config.mak $(TEST_DIR)-run msr.out cscope.* build-head
> +	$(RM) -r tests logs logs.old
>  
>  cscope: cscope_dirs = lib lib/libfdt lib/linux $(TEST_DIR) $(ARCH_LIBDIRS) lib/asm-generic
>  cscope:
> diff --git a/run_tests.sh b/run_tests.sh
> index 2cfa365..afd3d95 100755
> --- a/run_tests.sh
> +++ b/run_tests.sh
> @@ -7,7 +7,7 @@ if [ ! -f config.mak ]; then
>      exit 1
>  fi
>  source config.mak
> -source scripts/functions.bash
> +source scripts/common.bash
>  
>  function usage()
>  {
> @@ -46,17 +46,22 @@ while getopts "g:hv" opt; do
>      esac
>  done
>  
> -RUNTIME_log_stderr () { cat >> test.log; }
> +# RUNTIME_log_file will be configured later
> +RUNTIME_log_stderr () { cat >> $RUNTIME_log_file; }
>  RUNTIME_log_stdout () {
>      if [ "$PRETTY_PRINT_STACKS" = "yes" ]; then
> -        ./scripts/pretty_print_stacks.py $1 >> test.log
> +        ./scripts/pretty_print_stacks.py $1 >> $RUNTIME_log_file
>      else
> -        cat >> test.log
> +        cat >> $RUNTIME_log_file
>      fi
>  }
>  
> -
>  config=$TEST_DIR/unittests.cfg
> -rm -f test.log
> -printf "BUILD_HEAD=$(cat build-head)\n\n" > test.log
> +
> +rm -rf $unittest_log_dir.old
> +[ -d $unittest_log_dir ] && mv $unittest_log_dir $unittest_log_dir.old
> +mkdir $unittest_log_dir || exit 2
> +
> +echo "BUILD_HEAD=$(cat build-head)" > $unittest_log_dir/SUMMARY
> +
>  for_each_unittest $config run
> diff --git a/scripts/functions.bash b/scripts/common.bash
> similarity index 75%
> rename from scripts/functions.bash
> rename to scripts/common.bash
> index ee9143c..2dd7360 100644
> --- a/scripts/functions.bash
> +++ b/scripts/common.bash
> @@ -1,3 +1,12 @@
> +: ${unittest_log_dir:=logs}
> +
> +function run_task()
> +{
> +	local testname="$2"
> +
> +	RUNTIME_log_file="${unittest_log_dir}/${testname}.log"
> +	"$@"
> +}

This should go in run_tests.sh since that's the only place that uses
unittest_log_dir.  (The last line then becomes 'run "$@"' and the
for_each_unittest call must be adjusted).  Done and queued.

Paolo

>  function for_each_unittest()
>  {
> @@ -17,7 +26,7 @@ function for_each_unittest()
>  
>  	while read -u $fd line; do
>  		if [[ "$line" =~ ^\[(.*)\]$ ]]; then
> -			"$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
> +			run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
>  			testname=${BASH_REMATCH[1]}
>  			smp=1
>  			kernel=""
> @@ -45,6 +54,6 @@ function for_each_unittest()
>  			timeout=${BASH_REMATCH[1]}
>  		fi
>  	done
> -	"$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
> +	run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
>  	exec {fd}<&-
>  }
> diff --git a/scripts/mkstandalone.sh b/scripts/mkstandalone.sh
> index d2bae19..3c1938e 100755
> --- a/scripts/mkstandalone.sh
> +++ b/scripts/mkstandalone.sh
> @@ -5,7 +5,7 @@ if [ ! -f config.mak ]; then
>  	exit 1
>  fi
>  source config.mak
> -source scripts/functions.bash
> +source scripts/common.bash
>  
>  escape ()
>  {
> 

Re: [Qemu-devel] [kvm-unit-tests PATCH v6 2/3] run_tests: put logs into per-test file

[Peter Xu]

On Thu, Jan 12, 2017 at 01:55:48PM +0100, Paolo Bonzini wrote:

[...]

> > diff --git a/scripts/functions.bash b/scripts/common.bash
> > similarity index 75%
> > rename from scripts/functions.bash
> > rename to scripts/common.bash
> > index ee9143c..2dd7360 100644
> > --- a/scripts/functions.bash
> > +++ b/scripts/common.bash
> > @@ -1,3 +1,12 @@
> > +: ${unittest_log_dir:=logs}
> > +
> > +function run_task()
> > +{
> > +	local testname="$2"
> > +
> > +	RUNTIME_log_file="${unittest_log_dir}/${testname}.log"
> > +	"$@"
> > +}
> 
> This should go in run_tests.sh since that's the only place that uses
> unittest_log_dir.  (The last line then becomes 'run "$@"' and the
> for_each_unittest call must be adjusted).  Done and queued.

Yes, this should be cleaner. Thanks,

-- peterx

[Qemu-devel] [kvm-unit-tests PATCH v6 3/3] run_tests: allow run tests in parallel

[Peter Xu]

run_task.sh is getting slow. This patch is trying to make it faster by
running the tests concurrently.

We provide a new parameter "-j" for the run_tests.sh, which can be used
to specify how many run queues we want for the tests. Default queue
length is 1, which is the old behavior.

Quick test on my laptop (4 cores, 2 threads each) shows 3x speed boost:

   |-----------------+-----------|
   | command         | time used |
   |-----------------+-----------|
   | run_test.sh     | 75s       |
   | run_test.sh -j8 | 27s       |
   |-----------------+-----------|

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 run_tests.sh        | 12 ++++++++++--
 scripts/common.bash | 16 +++++++++++++++-
 2 files changed, 25 insertions(+), 3 deletions(-)

diff --git a/run_tests.sh b/run_tests.sh
index afd3d95..4d57ff9 100755
--- a/run_tests.sh
+++ b/run_tests.sh
@@ -13,10 +13,11 @@ function usage()
 {
 cat <<EOF
 
-Usage: $0 [-g group] [-h] [-v]
+Usage: $0 [-g group] [-h] [-v] [-j num_run_queues]
 
     -g: Only execute tests in the given group
     -h: Output this help text
+    -j: Execute tests in parallel
     -v: Enables verbose mode
 
 Set the environment variable QEMU=/path/to/qemu-system-ARCH to
@@ -28,7 +29,7 @@ EOF
 RUNTIME_arch_run="./$TEST_DIR/run"
 source scripts/runtime.bash
 
-while getopts "g:hv" opt; do
+while getopts "g:hj:v" opt; do
     case $opt in
         g)
             only_group=$OPTARG
@@ -37,6 +38,13 @@ while getopts "g:hv" opt; do
             usage
             exit
             ;;
+        j)
+            unittest_run_queues=$OPTARG
+            if (( $unittest_run_queues <= 0 )); then
+                echo "Invalid -j option: $unittest_run_queues"
+                exit 2
+            fi
+            ;;
         v)
             verbose="yes"
             ;;
diff --git a/scripts/common.bash b/scripts/common.bash
index 2dd7360..9bd560f 100644
--- a/scripts/common.bash
+++ b/scripts/common.bash
@@ -1,11 +1,19 @@
 : ${unittest_log_dir:=logs}
+: ${unittest_run_queues:=1}
 
 function run_task()
 {
 	local testname="$2"
 
+	while (( $(jobs | wc -l) == $unittest_run_queues )); do
+		# wait for any background test to finish
+		wait -n
+	done
+
 	RUNTIME_log_file="${unittest_log_dir}/${testname}.log"
-	"$@"
+
+	# start the testcase in the background
+	"$@" &
 }
 
 function for_each_unittest()
@@ -22,6 +30,8 @@ function for_each_unittest()
 	local accel
 	local timeout
 
+	trap "wait; exit 130" SIGINT
+
 	exec {fd}<"$unittests"
 
 	while read -u $fd line; do
@@ -55,5 +65,9 @@ function for_each_unittest()
 		fi
 	done
 	run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
+
+	# wait until all tasks finish
+	wait
+
 	exec {fd}<&-
 }
-- 
2.7.4

Re: [Qemu-devel] [kvm-unit-tests PATCH v6 3/3] run_tests: allow run tests in parallel

[Andrew Jones]

On Thu, Jan 12, 2017 at 11:36:22AM +0800, Peter Xu wrote:
> run_task.sh is getting slow. This patch is trying to make it faster by
> running the tests concurrently.
> 
> We provide a new parameter "-j" for the run_tests.sh, which can be used
> to specify how many run queues we want for the tests. Default queue
> length is 1, which is the old behavior.
> 
> Quick test on my laptop (4 cores, 2 threads each) shows 3x speed boost:
> 
>    |-----------------+-----------|
>    | command         | time used |
>    |-----------------+-----------|
>    | run_test.sh     | 75s       |
>    | run_test.sh -j8 | 27s       |
>    |-----------------+-----------|
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  run_tests.sh        | 12 ++++++++++--
>  scripts/common.bash | 16 +++++++++++++++-
>  2 files changed, 25 insertions(+), 3 deletions(-)
> 
> diff --git a/run_tests.sh b/run_tests.sh
> index afd3d95..4d57ff9 100755
> --- a/run_tests.sh
> +++ b/run_tests.sh
> @@ -13,10 +13,11 @@ function usage()
>  {
>  cat <<EOF
>  
> -Usage: $0 [-g group] [-h] [-v]
> +Usage: $0 [-g group] [-h] [-v] [-j num_run_queues]
>  
>      -g: Only execute tests in the given group
>      -h: Output this help text
> +    -j: Execute tests in parallel
>      -v: Enables verbose mode
>  
>  Set the environment variable QEMU=/path/to/qemu-system-ARCH to
> @@ -28,7 +29,7 @@ EOF
>  RUNTIME_arch_run="./$TEST_DIR/run"
>  source scripts/runtime.bash
>  
> -while getopts "g:hv" opt; do
> +while getopts "g:hj:v" opt; do
>      case $opt in
>          g)
>              only_group=$OPTARG
> @@ -37,6 +38,13 @@ while getopts "g:hv" opt; do
>              usage
>              exit
>              ;;
> +        j)
> +            unittest_run_queues=$OPTARG
> +            if (( $unittest_run_queues <= 0 )); then
> +                echo "Invalid -j option: $unittest_run_queues"
> +                exit 2
> +            fi
> +            ;;
>          v)
>              verbose="yes"
>              ;;
> diff --git a/scripts/common.bash b/scripts/common.bash
> index 2dd7360..9bd560f 100644
> --- a/scripts/common.bash
> +++ b/scripts/common.bash
> @@ -1,11 +1,19 @@
>  : ${unittest_log_dir:=logs}
> +: ${unittest_run_queues:=1}
>  
>  function run_task()
>  {
>  	local testname="$2"
>  
> +	while (( $(jobs | wc -l) == $unittest_run_queues )); do
> +		# wait for any background test to finish
> +		wait -n
> +	done
> +
>  	RUNTIME_log_file="${unittest_log_dir}/${testname}.log"
> -	"$@"
> +
> +	# start the testcase in the background
> +	"$@" &
>  }
>  
>  function for_each_unittest()
> @@ -22,6 +30,8 @@ function for_each_unittest()
>  	local accel
>  	local timeout
>  
> +	trap "wait; exit 130" SIGINT
> +
>  	exec {fd}<"$unittests"
>  
>  	while read -u $fd line; do
> @@ -55,5 +65,9 @@ function for_each_unittest()
>  		fi
>  	done
>  	run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
> +
> +	# wait until all tasks finish
> +	wait
> +
>  	exec {fd}<&-
>  }
> -- 
> 2.7.4
> 
>

Reviewed-by: Andrew Jones <drjones@redhat.com> 

Re: [Qemu-devel] [kvm-unit-tests PATCH v6 3/3] run_tests: allow run tests in parallel

[Paolo Bonzini]

On 12/01/2017 12:42, Andrew Jones wrote:
> On Thu, Jan 12, 2017 at 11:36:22AM +0800, Peter Xu wrote:
>> run_task.sh is getting slow. This patch is trying to make it faster by
>> running the tests concurrently.
>>
>> We provide a new parameter "-j" for the run_tests.sh, which can be used
>> to specify how many run queues we want for the tests. Default queue
>> length is 1, which is the old behavior.
>>
>> Quick test on my laptop (4 cores, 2 threads each) shows 3x speed boost:
>>
>>    |-----------------+-----------|
>>    | command         | time used |
>>    |-----------------+-----------|
>>    | run_test.sh     | 75s       |
>>    | run_test.sh -j8 | 27s       |
>>    |-----------------+-----------|

wait -n is very new, so I've added "2> /dev/null" (the result will be a
busy wait).  To avoid a busy wait in the common -j1 case, I also added:

+	if [ $unittest_run_queues = 1 ]; then
+		run "$@"
+	else
+		run "$@" &
+	fi

(I tried using a named pipe too, but it's messy to open both ends
without blocking).

Paolo

>> Signed-off-by: Peter Xu <peterx@redhat.com>
>> ---
>>  run_tests.sh        | 12 ++++++++++--
>>  scripts/common.bash | 16 +++++++++++++++-
>>  2 files changed, 25 insertions(+), 3 deletions(-)
>>
>> diff --git a/run_tests.sh b/run_tests.sh
>> index afd3d95..4d57ff9 100755
>> --- a/run_tests.sh
>> +++ b/run_tests.sh
>> @@ -13,10 +13,11 @@ function usage()
>>  {
>>  cat <<EOF
>>  
>> -Usage: $0 [-g group] [-h] [-v]
>> +Usage: $0 [-g group] [-h] [-v] [-j num_run_queues]
>>  
>>      -g: Only execute tests in the given group
>>      -h: Output this help text
>> +    -j: Execute tests in parallel
>>      -v: Enables verbose mode
>>  
>>  Set the environment variable QEMU=/path/to/qemu-system-ARCH to
>> @@ -28,7 +29,7 @@ EOF
>>  RUNTIME_arch_run="./$TEST_DIR/run"
>>  source scripts/runtime.bash
>>  
>> -while getopts "g:hv" opt; do
>> +while getopts "g:hj:v" opt; do
>>      case $opt in
>>          g)
>>              only_group=$OPTARG
>> @@ -37,6 +38,13 @@ while getopts "g:hv" opt; do
>>              usage
>>              exit
>>              ;;
>> +        j)
>> +            unittest_run_queues=$OPTARG
>> +            if (( $unittest_run_queues <= 0 )); then
>> +                echo "Invalid -j option: $unittest_run_queues"
>> +                exit 2
>> +            fi
>> +            ;;
>>          v)
>>              verbose="yes"
>>              ;;
>> diff --git a/scripts/common.bash b/scripts/common.bash
>> index 2dd7360..9bd560f 100644
>> --- a/scripts/common.bash
>> +++ b/scripts/common.bash
>> @@ -1,11 +1,19 @@
>>  : ${unittest_log_dir:=logs}
>> +: ${unittest_run_queues:=1}
>>  
>>  function run_task()
>>  {
>>  	local testname="$2"
>>  
>> +	while (( $(jobs | wc -l) == $unittest_run_queues )); do
>> +		# wait for any background test to finish
>> +		wait -n
>> +	done
>> +
>>  	RUNTIME_log_file="${unittest_log_dir}/${testname}.log"
>> -	"$@"
>> +
>> +	# start the testcase in the background
>> +	"$@" &
>>  }
>>  
>>  function for_each_unittest()
>> @@ -22,6 +30,8 @@ function for_each_unittest()
>>  	local accel
>>  	local timeout
>>  
>> +	trap "wait; exit 130" SIGINT
>> +
>>  	exec {fd}<"$unittests"
>>  
>>  	while read -u $fd line; do
>> @@ -55,5 +65,9 @@ function for_each_unittest()
>>  		fi
>>  	done
>>  	run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
>> +
>> +	# wait until all tasks finish
>> +	wait
>> +
>>  	exec {fd}<&-
>>  }
>> -- 
>> 2.7.4
>>
>>
> 
> Reviewed-by: Andrew Jones <drjones@redhat.com> 
>