From: mcgrof@kernel.org (Luis R. Rodriguez)
To: cocci@systeme.lip6.fr
Subject: [Cocci] [PATCH v2 1/5] coccicheck: enable parmap support
Date: Tue, 14 Jun 2016 13:05:48 -0700 [thread overview]
Message-ID: <1465934752-13792-2-git-send-email-mcgrof@kernel.org> (raw)
In-Reply-To: <1465934752-13792-1-git-send-email-mcgrof@kernel.org>
Coccinelle has had parmap support since 1.0.2, this means
it supports --jobs, enabling built-in multithreaded functionality,
instead of needing one to script it out. Just look for --jobs
in the help output to determine if this is supported.
Also enable the load balancing to be dynamic, so that if a
thread finishes early we keep feeding it.
Note: now that we have all things handled for us, redirect stderr to
stdout as well to capture any possible errors or warnings issued by
coccinelle.
If --jobs is not supported we fallback to the old mechanism.
As a small example, prior to this change, on an 8-core system:
Before:
$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report
...
real 29m14.912s
user 103m1.796s
sys 0m4.464s
After:
real 16m22.435s
user 128m30.060s
sys 0m2.712s
v2:
o redirect coccinelle stderr to ${DIR}/coccicheck.$$.err
o fix typo of paramap/parmap
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
scripts/coccicheck | 28 +++++++++++++++++++++++++---
1 file changed, 25 insertions(+), 3 deletions(-)
diff --git a/scripts/coccicheck b/scripts/coccicheck
index aa5e78fba270..02602381be59 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then
exit 1
fi
-trap kill_running SIGTERM SIGINT
-declare -a SPATCH_PID
+USE_JOBS="no"
+$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
# The verbosity may be set by the environmental parameter V=
# as for example with 'make V=1 coccicheck'
@@ -82,7 +82,21 @@ if [ "$ONLINE" = "0" ] ; then
echo ''
fi
-run_cmd() {
+if [ "$USE_JOBS" = "no" ]; then
+ trap kill_running SIGTERM SIGINT
+ declare -a SPATCH_PID
+else
+ OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
+fi
+
+run_cmd_parmap() {
+ if [ $VERBOSE -ne 0 ] ; then
+ echo "Running ($NPROC in parallel): $@"
+ fi
+ $@ 2> ${DIR}/coccicheck.$$.err
+}
+
+run_cmd_old() {
local i
if [ $VERBOSE -ne 0 ] ; then
echo "Running ($NPROC in parallel): $@"
@@ -97,6 +111,14 @@ run_cmd() {
wait
}
+run_cmd() {
+ if [ "$USE_JOBS" = "yes" ]; then
+ run_cmd_parmap $@
+ else
+ run_cmd_old $@
+ fi
+}
+
kill_running() {
for i in $(seq 0 $(( NPROC - 1 )) ); do
if [ $VERBOSE -eq 2 ] ; then
--
2.8.2
WARNING: multiple messages have this Message-ID (diff)
From: "Luis R. Rodriguez" <mcgrof@kernel.org>
To: Julia.Lawall@lip6.fr, Gilles.Muller@lip6.fr,
nicolas.palix@imag.fr, mmarek@suse.com
Cc: linux-kernel@vger.kernel.org, cocci@systeme.lip6.fr,
"Luis R. Rodriguez" <mcgrof@kernel.org>
Subject: [PATCH v2 1/5] coccicheck: enable parmap support
Date: Tue, 14 Jun 2016 13:05:48 -0700 [thread overview]
Message-ID: <1465934752-13792-2-git-send-email-mcgrof@kernel.org> (raw)
In-Reply-To: <1465934752-13792-1-git-send-email-mcgrof@kernel.org>
Coccinelle has had parmap support since 1.0.2, this means
it supports --jobs, enabling built-in multithreaded functionality,
instead of needing one to script it out. Just look for --jobs
in the help output to determine if this is supported.
Also enable the load balancing to be dynamic, so that if a
thread finishes early we keep feeding it.
Note: now that we have all things handled for us, redirect stderr to
stdout as well to capture any possible errors or warnings issued by
coccinelle.
If --jobs is not supported we fallback to the old mechanism.
As a small example, prior to this change, on an 8-core system:
Before:
$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report
...
real 29m14.912s
user 103m1.796s
sys 0m4.464s
After:
real 16m22.435s
user 128m30.060s
sys 0m2.712s
v2:
o redirect coccinelle stderr to ${DIR}/coccicheck.$$.err
o fix typo of paramap/parmap
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
scripts/coccicheck | 28 +++++++++++++++++++++++++---
1 file changed, 25 insertions(+), 3 deletions(-)
diff --git a/scripts/coccicheck b/scripts/coccicheck
index aa5e78fba270..02602381be59 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then
exit 1
fi
-trap kill_running SIGTERM SIGINT
-declare -a SPATCH_PID
+USE_JOBS="no"
+$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
# The verbosity may be set by the environmental parameter V=
# as for example with 'make V=1 coccicheck'
@@ -82,7 +82,21 @@ if [ "$ONLINE" = "0" ] ; then
echo ''
fi
-run_cmd() {
+if [ "$USE_JOBS" = "no" ]; then
+ trap kill_running SIGTERM SIGINT
+ declare -a SPATCH_PID
+else
+ OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
+fi
+
+run_cmd_parmap() {
+ if [ $VERBOSE -ne 0 ] ; then
+ echo "Running ($NPROC in parallel): $@"
+ fi
+ $@ 2> ${DIR}/coccicheck.$$.err
+}
+
+run_cmd_old() {
local i
if [ $VERBOSE -ne 0 ] ; then
echo "Running ($NPROC in parallel): $@"
@@ -97,6 +111,14 @@ run_cmd() {
wait
}
+run_cmd() {
+ if [ "$USE_JOBS" = "yes" ]; then
+ run_cmd_parmap $@
+ else
+ run_cmd_old $@
+ fi
+}
+
kill_running() {
for i in $(seq 0 $(( NPROC - 1 )) ); do
if [ $VERBOSE -eq 2 ] ; then
--
2.8.2
next prev parent reply other threads:[~2016-06-14 20:05 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-06-14 20:05 [Cocci] [PATCH v2 0/5] coccicheck: extend with parmap and indexing heuristics Luis R. Rodriguez
2016-06-14 20:05 ` Luis R. Rodriguez
2016-06-14 20:05 ` Luis R. Rodriguez [this message]
2016-06-14 20:05 ` [PATCH v2 1/5] coccicheck: enable parmap support Luis R. Rodriguez
2016-06-14 20:05 ` [Cocci] [PATCH v2 2/5] scripts: add glimpse.sh for indexing the kernel Luis R. Rodriguez
2016-06-14 20:05 ` Luis R. Rodriguez
2016-06-14 20:05 ` [Cocci] [PATCH v2 3/5] coccicheck: add indexing enhancement options Luis R. Rodriguez
2016-06-14 20:05 ` Luis R. Rodriguez
2016-06-14 20:05 ` [Cocci] [PATCH v2 4/5] coccicheck: document how to pass extra options to coccinelle Luis R. Rodriguez
2016-06-14 20:05 ` Luis R. Rodriguez
2016-06-14 20:05 ` [Cocci] [PATCH v2 5/5] coccicheck: refer to coccicheck bottest wiki for documentation Luis R. Rodriguez
2016-06-14 20:05 ` Luis R. Rodriguez
2016-06-14 20:19 ` [Cocci] [PATCH v2 0/5] coccicheck: extend with parmap and indexing heuristics Luis R. Rodriguez
2016-06-14 20:19 ` Luis R. Rodriguez
2016-06-14 20:23 ` [Cocci] [PATCH v3 0/6] " Luis R. Rodriguez
2016-06-14 20:23 ` Luis R. Rodriguez
2016-06-14 20:23 ` [Cocci] [PATCH v3 1/6] coccicheck: move spatch binary check up Luis R. Rodriguez
2016-06-14 20:23 ` Luis R. Rodriguez
2016-06-14 20:23 ` [Cocci] [PATCH v3 2/6] coccicheck: enable parmap support Luis R. Rodriguez
2016-06-14 20:23 ` Luis R. Rodriguez
2016-06-14 20:23 ` [Cocci] [PATCH v3 3/6] scripts: add glimpse.sh for indexing the kernel Luis R. Rodriguez
2016-06-14 20:23 ` Luis R. Rodriguez
2016-06-15 10:26 ` [Cocci] " SF Markus Elfring
2016-06-15 10:26 ` Aw: " SF Markus Elfring
2016-06-15 18:00 ` Luis R. Rodriguez
2016-06-15 18:00 ` Luis R. Rodriguez
2016-06-15 18:24 ` [Cocci] " SF Markus Elfring
2016-06-15 18:24 ` SF Markus Elfring
2016-06-15 18:43 ` [Cocci] " Luis R. Rodriguez
2016-06-15 18:43 ` Luis R. Rodriguez
2016-06-17 16:05 ` [Cocci] " SF Markus Elfring
2016-06-17 16:05 ` SF Markus Elfring
2016-06-17 17:23 ` [Cocci] " Luis R. Rodriguez
2016-06-17 17:23 ` Luis R. Rodriguez
2016-06-14 20:23 ` [Cocci] [PATCH v3 4/6] coccicheck: add indexing enhancement options Luis R. Rodriguez
2016-06-14 20:23 ` Luis R. Rodriguez
2016-06-15 11:20 ` [Cocci] " SF Markus Elfring
2016-06-15 11:20 ` Aw: " SF Markus Elfring
2016-06-15 20:00 ` Luis R. Rodriguez
2016-06-15 20:00 ` Luis R. Rodriguez
2016-06-14 20:23 ` [Cocci] [PATCH v3 5/6] coccicheck: document how to pass extra options to coccinelle Luis R. Rodriguez
2016-06-14 20:23 ` Luis R. Rodriguez
2016-06-14 20:23 ` [Cocci] [PATCH v3 6/6] coccicheck: refer to coccicheck bottest wiki for documentation Luis R. Rodriguez
2016-06-14 20:23 ` Luis R. Rodriguez
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1465934752-13792-2-git-send-email-mcgrof@kernel.org \
--to=mcgrof@kernel.org \
--cc=cocci@systeme.lip6.fr \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.