From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37384) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cRBSK-00027i-8r for qemu-devel@nongnu.org; Wed, 11 Jan 2017 00:28:17 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cRBSF-0003Zc-JM for qemu-devel@nongnu.org; Wed, 11 Jan 2017 00:28:16 -0500 Received: from mx1.redhat.com ([209.132.183.28]:34346) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cRBSF-0003Z3-BD for qemu-devel@nongnu.org; Wed, 11 Jan 2017 00:28:11 -0500 Date: Wed, 11 Jan 2017 13:28:03 +0800 From: Peter Xu Message-ID: <20170111052803.GB4450@pxdev.xzpeter.org> References: <1483934694-32425-1-git-send-email-peterx@redhat.com> <1483934694-32425-3-git-send-email-peterx@redhat.com> <20170110173959.ckfhipadyv3bsijs@kamzik.brq.redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20170110173959.ckfhipadyv3bsijs@kamzik.brq.redhat.com> Subject: Re: [Qemu-devel] [kvm-unit-tests PATCH v4 2/2] run_tests: allow run tests in parallel List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Andrew Jones Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, Paolo Bonzini , Radim =?utf-8?B?S3LEjW3DocWZ?= On Tue, Jan 10, 2017 at 06:39:59PM +0100, Andrew Jones wrote: > On Mon, Jan 09, 2017 at 12:04:54PM +0800, Peter Xu wrote: > > run_task.sh is getting slow. This patch is trying to make it faster by > > running the tests concurrently. > > > > We provide a new parameter "-j" for the run_tests.sh, which can be used > > to specify how many run queues we want for the tests. Default queue > > length is 1, which is the old behavior. > > > > Quick test on my laptop (4 cores, 2 threads each) shows 3x speed boost: > > > > |-----------------+-----------| > > | command | time used | > > |-----------------+-----------| > > | run_test.sh | 75s | > > | run_test.sh -j8 | 27s | > > |-----------------+-----------| > > > > Signed-off-by: Peter Xu > > --- > > run_tests.sh | 12 ++++++++++-- > > scripts/common.bash | 16 +++++++++++++++- > > 2 files changed, 25 insertions(+), 3 deletions(-) > > > > diff --git a/run_tests.sh b/run_tests.sh > > index 1e36d66..795cf73 100755 > > --- a/run_tests.sh > > +++ b/run_tests.sh > > @@ -13,10 +13,11 @@ function usage() > > { > > cat < > > > -Usage: $0 [-g group] [-h] [-v] > > +Usage: $0 [-g group] [-h] [-v] [-j num_run_queues] > > > > -g: Only execute tests in the given group > > -h: Output this help text > > + -j: Execute tests in parallel > > -v: Enables verbose mode > > > > Set the environment variable QEMU=/path/to/qemu-system-ARCH to > > @@ -28,7 +29,7 @@ EOF > > RUNTIME_arch_run="./$TEST_DIR/run" > > source scripts/runtime.bash > > > > -while getopts "g:hv" opt; do > > +while getopts "g:hj:v" opt; do > > case $opt in > > g) > > only_group=$OPTARG > > @@ -37,6 +38,13 @@ while getopts "g:hv" opt; do > > usage > > exit > > ;; > > + j) > > + unittest_run_queues=$OPTARG > > + if ! (( "$unittest_run_queues" > 0 )); then > > The equivalent expression without '!' is (( $var <= 0 )), and > no need for the "'s around the var. > > > + echo "Invalid -j option: $unittest_run_queues" > > + exit 1 > > + fi > > + ;; > > v) > > verbose="yes" > > ;; > > diff --git a/scripts/common.bash b/scripts/common.bash > > index 2dd7360..83aebf8 100644 > > --- a/scripts/common.bash > > +++ b/scripts/common.bash > > @@ -1,11 +1,19 @@ > > : ${unittest_log_dir:=logs} > > +: ${unittest_run_queues:=1} > > > > function run_task() > > { > > local testname="$2" > > > > + while (( "$(jobs | wc -l)" == $unittest_run_queues )); do > > Why the "'s? The result must resolve to an int, otherwise we don't > want ((...)) > > > + # wait for any background test to finish > > + wait -n > > + done > > + > > RUNTIME_log_file="${unittest_log_dir}/${testname}.log" > > - "$@" > > + > > + # start the testcase in the background > > + "$@" & > > } > > > > function for_each_unittest() > > @@ -22,6 +30,8 @@ function for_each_unittest() > > local accel > > local timeout > > > > + trap "wait; exit 130" SIGINT > > + > > exec {fd}<"$unittests" > > > > while read -u $fd line; do > > @@ -55,5 +65,9 @@ function for_each_unittest() > > fi > > done > > run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout" > > + > > + # wait all task finish > > wait until all tasks finish Fixing all three places, thanks! -- peterx