[RESEND 0/3] coccinelle: add parallel execution

[RESEND 0/3] coccinelle: add parallel execution

[Gregory.Dietsche]

From: Greg Dietsche <Gregory.Dietsche@cuw.edu>

This is a re-post of the patches found here: https://lkml.org/lkml/2011/7/12/397

Could I get the appropriate aks/naks to get these into the kernel? Also, I've added Michal Marek to the CC list since it is my understanding that he's the one that will eventually commit these. Thanks!!

---

This patch series adds a new flag PARALLEL= to the coccicheck build target and documents the new feature.

Performance is quite a bit better than single threaded. On my 6 core box, the checks generally run around 3x faster.


Greg Dietsche (3):
  coccinelle.txt: add overview section
  coccinelle.txt: add documentation of PARALLEL= flag
  coccicheck: add parallel execution

 Documentation/coccinelle.txt |   35 +++++++++++++++++++++++++++++++++++
 scripts/coccicheck           |   27 ++++++++++++++++++++++++++-
 2 files changed, 61 insertions(+), 1 deletions(-)

-- 
1.7.6.4

[RESEND 1/3] coccinelle.txt: add overview section

[Gregory.Dietsche]

From: Greg Dietsche <Gregory.Dietsche@cuw.edu>

Add an overview section to the coccinelle documentation.

Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>
Signed-off-by: Nicolas Palix <npalix.work@gmail.com>
---
 Documentation/coccinelle.txt |   31 +++++++++++++++++++++++++++++++
 1 files changed, 31 insertions(+), 0 deletions(-)

diff --git a/Documentation/coccinelle.txt b/Documentation/coccinelle.txt
index 96b6903..9495a4b 100644
--- a/Documentation/coccinelle.txt
+++ b/Documentation/coccinelle.txt
@@ -41,6 +41,37 @@ The semantic patches in the kernel will work best with Coccinelle version
 semantic patch code, but any results that are obtained should still be
 correct.
 
+
+ Overview / Quick Start
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are two ways to use Coccinelle with the Linux kernel.
+
+1) Coccinelle can be used like sparse (see Documentation/sparse.txt):
+	make C=1 CHECK="scripts/coccicheck"
+	make C=2 CHECK="scripts/coccicheck"
+
+2) Coccinelle can be used via a build target in the kernel's Makefile:
+	make coccicheck
+
+There are a number of optional parameters that can be used with the build target.
+
+	make coccicheck MODE={patch,report,context,org} COCCI=? M=?
+
+MODE:
+	Determines what mode cocci operates in. If no mode is specified
+	then cocci will default to 'chain' mode which will run for each
+	available mode (patch, report, context, org).
+
+COCCI:
+	Process a specific .cocci SmPL script. If this is not set then
+	process all scripts under scripts/coccinelle/
+
+M:
+	Limit cocci to a subset of directories. This is very similar to the
+	way the build system works when building modules.
+
+
  Using Coccinelle on the Linux kernel
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-- 
1.7.6.4

[RESEND 2/3] coccinelle.txt: add documentation of PARALLEL= flag

[Gregory.Dietsche]

From: Greg Dietsche <Gregory.Dietsche@cuw.edu>

Document the new PARALLEL= option of Coccinelle which
allows running SmPL scripts in parallel.

Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>
---
 Documentation/coccinelle.txt |    6 +++++-
 1 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/Documentation/coccinelle.txt b/Documentation/coccinelle.txt
index 9495a4b..3bccb50 100644
--- a/Documentation/coccinelle.txt
+++ b/Documentation/coccinelle.txt
@@ -56,7 +56,7 @@ There are two ways to use Coccinelle with the Linux kernel.
 
 There are a number of optional parameters that can be used with the build target.
 
-	make coccicheck MODE={patch,report,context,org} COCCI=? M=?
+	make coccicheck MODE={patch,report,context,org} COCCI=? M=? PARALLEL=?
 
 MODE:
 	Determines what mode cocci operates in. If no mode is specified
@@ -71,6 +71,10 @@ M:
 	Limit cocci to a subset of directories. This is very similar to the
 	way the build system works when building modules.
 
+PARALLEL:
+	Number of *.cocci SmPL scripts found under scripts/coccinelle/ to run
+	at the same time.
+
 
  Using Coccinelle on the Linux kernel
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- 
1.7.6.4

[RESEND 3/3] coccicheck: add parallel execution

[Gregory.Dietsche]

From: Greg Dietsche <Gregory.Dietsche@cuw.edu>

For example to process 6 SmPL patches at the same time:
	make coccicheck PARALLEL=6

Results are held in /tmp until the scripts finish. By doing this the
script is able to collate the results from each SmPL patch instead of
interleaving them in the output.

Version 2: change pkill -P $$ to pkill -s 0

Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>
Acked-by: Nicolas Palix <npalix.work@gmail.com>
---
 scripts/coccicheck |   27 ++++++++++++++++++++++++++-
 1 files changed, 26 insertions(+), 1 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 1bb1a1b..c2a04c2 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -97,10 +97,35 @@ coccinelle () {
 
 }
 
+parallel_cleanup () {
+	pkill -s 0 > /dev/null
+	rm /tmp/cocci_parallel_$$_* 2>/dev/null
+	exit
+}
+
+trap parallel_cleanup SIGINT
+PARALLEL_ID=0
+
 if [ "$COCCI" = "" ] ; then
     for f in `find $srctree/scripts/coccinelle/ -name '*.cocci' -type f | sort`; do
-	coccinelle $f
+	if [ "$PARALLEL" -eq "$PARALLEL" 2>/dev/null ] ; then
+		if [ "$ONLINE" = "0" ] ; then
+			echo Processing: `basename $f`
+		fi
+		coccinelle $f>/tmp/cocci_parallel_$$_$PARALLEL_ID 2>&1 &
+		PARALLEL_ID=$(($PARALLEL_ID + 1))
+		while [ "`jobs -p | wc -l`" -ge "$PARALLEL" ] ; do
+			sleep 3
+		done
+	else
+		coccinelle $f
+	fi
     done
+    wait
+    if [ "$PARALLEL_ID" -ge "0" ] ; then
+	cat /tmp/cocci_parallel_$$_*
+	parallel_cleanup
+    fi
 else
     coccinelle $COCCI
 fi
-- 
1.7.6.4

Re: [RESEND 1/3] coccinelle.txt: add overview section

[Julia Lawall]

On Sat, 5 Nov 2011, Gregory.Dietsche@cuw.edu wrote:

> From: Greg Dietsche <Gregory.Dietsche@cuw.edu>
>
> Add an overview section to the coccinelle documentation.
>
> Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>
> Signed-off-by: Nicolas Palix <npalix.work@gmail.com>

Acked-by: Julia Lawall <julia.lawall@lip6.fr>

> ---
> Documentation/coccinelle.txt |   31 +++++++++++++++++++++++++++++++
> 1 files changed, 31 insertions(+), 0 deletions(-)
>
> diff --git a/Documentation/coccinelle.txt b/Documentation/coccinelle.txt
> index 96b6903..9495a4b 100644
> --- a/Documentation/coccinelle.txt
> +++ b/Documentation/coccinelle.txt
> @@ -41,6 +41,37 @@ The semantic patches in the kernel will work best with Coccinelle version
> semantic patch code, but any results that are obtained should still be
> correct.
>
> +
> + Overview / Quick Start
> +~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +There are two ways to use Coccinelle with the Linux kernel.
> +
> +1) Coccinelle can be used like sparse (see Documentation/sparse.txt):
> +	make C=1 CHECK="scripts/coccicheck"
> +	make C=2 CHECK="scripts/coccicheck"
> +
> +2) Coccinelle can be used via a build target in the kernel's Makefile:
> +	make coccicheck
> +
> +There are a number of optional parameters that can be used with the build target.
> +
> +	make coccicheck MODE={patch,report,context,org} COCCI=? M=?
> +
> +MODE:
> +	Determines what mode cocci operates in. If no mode is specified
> +	then cocci will default to 'chain' mode which will run for each
> +	available mode (patch, report, context, org).
> +
> +COCCI:
> +	Process a specific .cocci SmPL script. If this is not set then
> +	process all scripts under scripts/coccinelle/
> +
> +M:
> +	Limit cocci to a subset of directories. This is very similar to the
> +	way the build system works when building modules.
> +
> +
>  Using Coccinelle on the Linux kernel
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> -- 
> 1.7.6.4
>
>

Re: [RESEND 2/3] coccinelle.txt: add documentation of PARALLEL= flag

[Julia Lawall]

On Sat, 5 Nov 2011, Gregory.Dietsche@cuw.edu wrote:

> From: Greg Dietsche <Gregory.Dietsche@cuw.edu>
>
> Document the new PARALLEL= option of Coccinelle which
> allows running SmPL scripts in parallel.
>
> Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>

Acked-by: Julia Lawall <julia.lawall@lip6.fr>

> ---
> Documentation/coccinelle.txt |    6 +++++-
> 1 files changed, 5 insertions(+), 1 deletions(-)
>
> diff --git a/Documentation/coccinelle.txt b/Documentation/coccinelle.txt
> index 9495a4b..3bccb50 100644
> --- a/Documentation/coccinelle.txt
> +++ b/Documentation/coccinelle.txt
> @@ -56,7 +56,7 @@ There are two ways to use Coccinelle with the Linux kernel.
>
> There are a number of optional parameters that can be used with the build target.
>
> -	make coccicheck MODE={patch,report,context,org} COCCI=? M=?
> +	make coccicheck MODE={patch,report,context,org} COCCI=? M=? PARALLEL=?
>
> MODE:
> 	Determines what mode cocci operates in. If no mode is specified
> @@ -71,6 +71,10 @@ M:
> 	Limit cocci to a subset of directories. This is very similar to the
> 	way the build system works when building modules.
>
> +PARALLEL:
> +	Number of *.cocci SmPL scripts found under scripts/coccinelle/ to run
> +	at the same time.
> +
>
>  Using Coccinelle on the Linux kernel
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> -- 
> 1.7.6.4
>
>

Re: [RESEND 3/3] coccicheck: add parallel execution

[Julia Lawall]

On Sat, 5 Nov 2011, Gregory.Dietsche@cuw.edu wrote:

> From: Greg Dietsche <Gregory.Dietsche@cuw.edu>
>
> For example to process 6 SmPL patches at the same time:
> 	make coccicheck PARALLEL=6
>
> Results are held in /tmp until the scripts finish. By doing this the
> script is able to collate the results from each SmPL patch instead of
> interleaving them in the output.
>
> Version 2: change pkill -P $$ to pkill -s 0
>
> Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>
> Acked-by: Nicolas Palix <npalix.work@gmail.com>

Acked-by: Julia Lawall <julia.lawall@lip6.fr>

> ---
> scripts/coccicheck |   27 ++++++++++++++++++++++++++-
> 1 files changed, 26 insertions(+), 1 deletions(-)
>
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index 1bb1a1b..c2a04c2 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -97,10 +97,35 @@ coccinelle () {
>
> }
>
> +parallel_cleanup () {
> +	pkill -s 0 > /dev/null
> +	rm /tmp/cocci_parallel_$$_* 2>/dev/null
> +	exit
> +}
> +
> +trap parallel_cleanup SIGINT
> +PARALLEL_ID=0
> +
> if [ "$COCCI" = "" ] ; then
>     for f in `find $srctree/scripts/coccinelle/ -name '*.cocci' -type f | sort`; do
> -	coccinelle $f
> +	if [ "$PARALLEL" -eq "$PARALLEL" 2>/dev/null ] ; then
> +		if [ "$ONLINE" = "0" ] ; then
> +			echo Processing: `basename $f`
> +		fi
> +		coccinelle $f>/tmp/cocci_parallel_$$_$PARALLEL_ID 2>&1 &
> +		PARALLEL_ID=$(($PARALLEL_ID + 1))
> +		while [ "`jobs -p | wc -l`" -ge "$PARALLEL" ] ; do
> +			sleep 3
> +		done
> +	else
> +		coccinelle $f
> +	fi
>     done
> +    wait
> +    if [ "$PARALLEL_ID" -ge "0" ] ; then
> +	cat /tmp/cocci_parallel_$$_*
> +	parallel_cleanup
> +    fi
> else
>     coccinelle $COCCI
> fi
> -- 
> 1.7.6.4
>
>

Re: [RESEND 3/3] coccicheck: add parallel execution

[Michal Marek]

On 6.11.2011 03:14, Gregory.Dietsche@cuw.edu wrote:
> +		coccinelle $f>/tmp/cocci_parallel_$$_$PARALLEL_ID 2>&1 &

This opens a symlink race.

And this whole parallel execution should be done by make. Just have a
generic rule for getting result from a single check, make the default
target depend on the list of the wanted results and concatenate the
results there. make -j will then take of parallelizing it.

Michal

Re: [RESEND 3/3] coccicheck: add parallel execution

[Greg Dietsche]

On 12-11-11 02:48pm, Michal Marek wrote:
> On 6.11.2011 03:14, Gregory.Dietsche@cuw.edu wrote:
>> +		coccinelle $f>/tmp/cocci_parallel_$$_$PARALLEL_ID 2>&1 &
>
> This opens a symlink race.
>
> And this whole parallel execution should be done by make. Just have a
> generic rule for getting result from a single check, make the default
> target depend on the list of the wanted results and concatenate the
> results there. make -j will then take of parallelizing it.

Michal,
Sorry for the slow response - my mail filters weren't behaving well at 
all...

For now, lets just drop this patch. I don't have the time to fix it up.
A few months ago I did take a look at what it would mean to use make's 
-j option
and it seemed to me that doing so would require completely changing how 
the coccinelle
script works.

I would like to see my M= patch set merged though:
https://lkml.org/lkml/2011/11/5/94

Thanks,
Greg