All of lore.kernel.org
 help / color / mirror / Atom feed
From: mcgrof@kernel.org (Luis R. Rodriguez)
To: cocci@systeme.lip6.fr
Subject: [Cocci] [PATCH v2 1/5] coccicheck: enable parmap support
Date: Tue, 14 Jun 2016 13:05:48 -0700	[thread overview]
Message-ID: <1465934752-13792-2-git-send-email-mcgrof@kernel.org> (raw)
In-Reply-To: <1465934752-13792-1-git-send-email-mcgrof@kernel.org>

Coccinelle has had parmap support since 1.0.2, this means
it supports --jobs, enabling built-in multithreaded functionality,
instead of needing one to script it out. Just look for --jobs
in the help output to determine if this is supported.

Also enable the load balancing to be dynamic, so that if a
thread finishes early we keep feeding it.

Note: now that we have all things handled for us, redirect stderr to
stdout as well to capture any possible errors or warnings issued by
coccinelle.

If --jobs is not supported we fallback to the old mechanism.

As a small example, prior to this change, on an 8-core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report
...

real    29m14.912s
user    103m1.796s
sys     0m4.464s

After:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

v2:

o redirect coccinelle stderr to ${DIR}/coccicheck.$$.err
o fix typo of paramap/parmap

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index aa5e78fba270..02602381be59 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then
     exit 1
 fi
 
-trap kill_running SIGTERM SIGINT
-declare -a SPATCH_PID
+USE_JOBS="no"
+$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
@@ -82,7 +82,21 @@ if [ "$ONLINE" = "0" ] ; then
     echo ''
 fi
 
-run_cmd() {
+if [ "$USE_JOBS" = "no" ]; then
+	trap kill_running SIGTERM SIGINT
+	declare -a SPATCH_PID
+else
+	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
+fi
+
+run_cmd_parmap() {
+	if [ $VERBOSE -ne 0 ] ; then
+		echo "Running ($NPROC in parallel): $@"
+	fi
+	$@ 2> ${DIR}/coccicheck.$$.err
+}
+
+run_cmd_old() {
 	local i
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
@@ -97,6 +111,14 @@ run_cmd() {
 	wait
 }
 
+run_cmd() {
+	if [ "$USE_JOBS" = "yes" ]; then
+		run_cmd_parmap $@
+	else
+		run_cmd_old $@
+	fi
+}
+
 kill_running() {
 	for i in $(seq 0 $(( NPROC - 1 )) ); do
 		if [ $VERBOSE -eq 2 ] ; then
-- 
2.8.2

WARNING: multiple messages have this Message-ID (diff)
From: "Luis R. Rodriguez" <mcgrof@kernel.org>
To: Julia.Lawall@lip6.fr, Gilles.Muller@lip6.fr,
	nicolas.palix@imag.fr, mmarek@suse.com
Cc: linux-kernel@vger.kernel.org, cocci@systeme.lip6.fr,
	"Luis R. Rodriguez" <mcgrof@kernel.org>
Subject: [PATCH v2 1/5] coccicheck: enable parmap support
Date: Tue, 14 Jun 2016 13:05:48 -0700	[thread overview]
Message-ID: <1465934752-13792-2-git-send-email-mcgrof@kernel.org> (raw)
In-Reply-To: <1465934752-13792-1-git-send-email-mcgrof@kernel.org>

Coccinelle has had parmap support since 1.0.2, this means
it supports --jobs, enabling built-in multithreaded functionality,
instead of needing one to script it out. Just look for --jobs
in the help output to determine if this is supported.

Also enable the load balancing to be dynamic, so that if a
thread finishes early we keep feeding it.

Note: now that we have all things handled for us, redirect stderr to
stdout as well to capture any possible errors or warnings issued by
coccinelle.

If --jobs is not supported we fallback to the old mechanism.

As a small example, prior to this change, on an 8-core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report
...

real    29m14.912s
user    103m1.796s
sys     0m4.464s

After:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

v2:

o redirect coccinelle stderr to ${DIR}/coccicheck.$$.err
o fix typo of paramap/parmap

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index aa5e78fba270..02602381be59 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then
     exit 1
 fi
 
-trap kill_running SIGTERM SIGINT
-declare -a SPATCH_PID
+USE_JOBS="no"
+$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
@@ -82,7 +82,21 @@ if [ "$ONLINE" = "0" ] ; then
     echo ''
 fi
 
-run_cmd() {
+if [ "$USE_JOBS" = "no" ]; then
+	trap kill_running SIGTERM SIGINT
+	declare -a SPATCH_PID
+else
+	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
+fi
+
+run_cmd_parmap() {
+	if [ $VERBOSE -ne 0 ] ; then
+		echo "Running ($NPROC in parallel): $@"
+	fi
+	$@ 2> ${DIR}/coccicheck.$$.err
+}
+
+run_cmd_old() {
 	local i
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
@@ -97,6 +111,14 @@ run_cmd() {
 	wait
 }
 
+run_cmd() {
+	if [ "$USE_JOBS" = "yes" ]; then
+		run_cmd_parmap $@
+	else
+		run_cmd_old $@
+	fi
+}
+
 kill_running() {
 	for i in $(seq 0 $(( NPROC - 1 )) ); do
 		if [ $VERBOSE -eq 2 ] ; then
-- 
2.8.2

  reply	other threads:[~2016-06-14 20:05 UTC|newest]

Thread overview: 44+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-06-14 20:05 [Cocci] [PATCH v2 0/5] coccicheck: extend with parmap and indexing heuristics Luis R. Rodriguez
2016-06-14 20:05 ` Luis R. Rodriguez
2016-06-14 20:05 ` Luis R. Rodriguez [this message]
2016-06-14 20:05   ` [PATCH v2 1/5] coccicheck: enable parmap support Luis R. Rodriguez
2016-06-14 20:05 ` [Cocci] [PATCH v2 2/5] scripts: add glimpse.sh for indexing the kernel Luis R. Rodriguez
2016-06-14 20:05   ` Luis R. Rodriguez
2016-06-14 20:05 ` [Cocci] [PATCH v2 3/5] coccicheck: add indexing enhancement options Luis R. Rodriguez
2016-06-14 20:05   ` Luis R. Rodriguez
2016-06-14 20:05 ` [Cocci] [PATCH v2 4/5] coccicheck: document how to pass extra options to coccinelle Luis R. Rodriguez
2016-06-14 20:05   ` Luis R. Rodriguez
2016-06-14 20:05 ` [Cocci] [PATCH v2 5/5] coccicheck: refer to coccicheck bottest wiki for documentation Luis R. Rodriguez
2016-06-14 20:05   ` Luis R. Rodriguez
2016-06-14 20:19 ` [Cocci] [PATCH v2 0/5] coccicheck: extend with parmap and indexing heuristics Luis R. Rodriguez
2016-06-14 20:19   ` Luis R. Rodriguez
2016-06-14 20:23   ` [Cocci] [PATCH v3 0/6] " Luis R. Rodriguez
2016-06-14 20:23     ` Luis R. Rodriguez
2016-06-14 20:23     ` [Cocci] [PATCH v3 1/6] coccicheck: move spatch binary check up Luis R. Rodriguez
2016-06-14 20:23       ` Luis R. Rodriguez
2016-06-14 20:23     ` [Cocci] [PATCH v3 2/6] coccicheck: enable parmap support Luis R. Rodriguez
2016-06-14 20:23       ` Luis R. Rodriguez
2016-06-14 20:23     ` [Cocci] [PATCH v3 3/6] scripts: add glimpse.sh for indexing the kernel Luis R. Rodriguez
2016-06-14 20:23       ` Luis R. Rodriguez
2016-06-15 10:26       ` [Cocci] " SF Markus Elfring
2016-06-15 10:26         ` Aw: " SF Markus Elfring
2016-06-15 18:00         ` Luis R. Rodriguez
2016-06-15 18:00           ` Luis R. Rodriguez
2016-06-15 18:24           ` [Cocci] " SF Markus Elfring
2016-06-15 18:24             ` SF Markus Elfring
2016-06-15 18:43             ` [Cocci] " Luis R. Rodriguez
2016-06-15 18:43               ` Luis R. Rodriguez
2016-06-17 16:05               ` [Cocci] " SF Markus Elfring
2016-06-17 16:05                 ` SF Markus Elfring
2016-06-17 17:23                 ` [Cocci] " Luis R. Rodriguez
2016-06-17 17:23                   ` Luis R. Rodriguez
2016-06-14 20:23     ` [Cocci] [PATCH v3 4/6] coccicheck: add indexing enhancement options Luis R. Rodriguez
2016-06-14 20:23       ` Luis R. Rodriguez
2016-06-15 11:20       ` [Cocci] " SF Markus Elfring
2016-06-15 11:20         ` Aw: " SF Markus Elfring
2016-06-15 20:00         ` Luis R. Rodriguez
2016-06-15 20:00           ` Luis R. Rodriguez
2016-06-14 20:23     ` [Cocci] [PATCH v3 5/6] coccicheck: document how to pass extra options to coccinelle Luis R. Rodriguez
2016-06-14 20:23       ` Luis R. Rodriguez
2016-06-14 20:23     ` [Cocci] [PATCH v3 6/6] coccicheck: refer to coccicheck bottest wiki for documentation Luis R. Rodriguez
2016-06-14 20:23       ` Luis R. Rodriguez

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1465934752-13792-2-git-send-email-mcgrof@kernel.org \
    --to=mcgrof@kernel.org \
    --cc=cocci@systeme.lip6.fr \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.