[PATCH 0/3] coccinelle: add parallel execution

[PATCH 0/3] coccinelle: add parallel execution

[Greg Dietsche]

This patch series adds a new flag PARALLEL= to the coccicheck build target and documents the new feature.
It works a lot like make's -j flag. I looked for a way to have make pass on the value of -j, but
didn't find one - if anyone knows how to do that, I'd love to hear about it! 

I suppose an alternate approach would be to accomplish this via make itself, but the approach in this patch series seemed simpler to implement... 

Performance is quite a bit better than single threaded. On my 6 core box, the checks generally run around 3x faster.

Feedback welcome!! :)

Greg

Greg Dietsche (3):
  coccinelle.txt: add overview section
  coccinelle.txt: add documentation of PARALLEL= flag
  coccicheck: add parallel execution

 Documentation/coccinelle.txt |   35 +++++++++++++++++++++++++++++++++++
 scripts/coccicheck           |   27 ++++++++++++++++++++++++++-
 2 files changed, 61 insertions(+), 1 deletions(-)

-- 
1.7.2.5

[PATCH 1/3] coccinelle.txt: add overview section

[Greg Dietsche]

Add an overview section to the Coccinelle documentation.

Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>
---
 Documentation/coccinelle.txt |   31 +++++++++++++++++++++++++++++++
 1 files changed, 31 insertions(+), 0 deletions(-)

diff --git a/Documentation/coccinelle.txt b/Documentation/coccinelle.txt
index 96b6903..9495a4b 100644
--- a/Documentation/coccinelle.txt
+++ b/Documentation/coccinelle.txt
@@ -41,6 +41,37 @@ The semantic patches in the kernel will work best with Coccinelle version
 semantic patch code, but any results that are obtained should still be
 correct.
 
+
+ Overview / Quick Start
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are two ways to use Coccinelle with the Linux kernel.
+
+1) Coccinelle can be used like sparse (see Documentation/sparse.txt):
+	make C=1 CHECK="scripts/coccicheck"
+	make C=2 CHECK="scripts/coccicheck"
+
+2) Coccinelle can be used via a build target in the kernel's Makefile:
+	make coccicheck
+
+There are a number of optional parameters that can be used with the build target.
+
+	make coccicheck MODE={patch,report,context,org} COCCI=? M=?
+
+MODE:
+	Determines what mode cocci operates in. If no mode is specified
+	then cocci will default to 'chain' mode which will run for each
+	available mode (patch, report, context, org).
+
+COCCI:
+	Process a specific .cocci SmPL script. If this is not set then
+	process all scripts under scripts/coccinelle/
+
+M:
+	Limit cocci to a subset of directories. This is very similar to the
+	way the build system works when building modules.
+
+
  Using Coccinelle on the Linux kernel
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-- 
1.7.2.5

Re: [PATCH 1/3] coccinelle.txt: add overview section

[Nicolas Palix]

On Wed, Jul 13, 2011 at 3:26 AM, Greg Dietsche <Gregory.Dietsche@cuw.edu> wrote:
> Add an overview section to the Coccinelle documentation.
>
> Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>

Signed-off-by: Nicolas Palix <npalix.work@gmail.com>

> ---
>  Documentation/coccinelle.txt |   31 +++++++++++++++++++++++++++++++
>  1 files changed, 31 insertions(+), 0 deletions(-)
>
> diff --git a/Documentation/coccinelle.txt b/Documentation/coccinelle.txt
> index 96b6903..9495a4b 100644
> --- a/Documentation/coccinelle.txt
> +++ b/Documentation/coccinelle.txt
> @@ -41,6 +41,37 @@ The semantic patches in the kernel will work best with Coccinelle version
>  semantic patch code, but any results that are obtained should still be
>  correct.
>
> +
> + Overview / Quick Start
> +~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +There are two ways to use Coccinelle with the Linux kernel.
> +
> +1) Coccinelle can be used like sparse (see Documentation/sparse.txt):
> +       make C=1 CHECK="scripts/coccicheck"
> +       make C=2 CHECK="scripts/coccicheck"
> +
> +2) Coccinelle can be used via a build target in the kernel's Makefile:
> +       make coccicheck
> +
> +There are a number of optional parameters that can be used with the build target.
> +
> +       make coccicheck MODE={patch,report,context,org} COCCI=? M=?
> +
> +MODE:
> +       Determines what mode cocci operates in. If no mode is specified
> +       then cocci will default to 'chain' mode which will run for each
> +       available mode (patch, report, context, org).
> +
> +COCCI:
> +       Process a specific .cocci SmPL script. If this is not set then
> +       process all scripts under scripts/coccinelle/
> +
> +M:
> +       Limit cocci to a subset of directories. This is very similar to the
> +       way the build system works when building modules.
> +
> +
>  Using Coccinelle on the Linux kernel
>  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> --
> 1.7.2.5
>
>



-- 
Nicolas Palix
http://sardes.inrialpes.fr/~npalix/

[PATCH 2/3] coccinelle.txt: add documentation of PARALLEL= flag

[Greg Dietsche]

Document the new PARALLEL= option of Coccinelle which
allows running SmPL scripts in parallel.

Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>
---
 Documentation/coccinelle.txt |    6 +++++-
 1 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/Documentation/coccinelle.txt b/Documentation/coccinelle.txt
index 9495a4b..3bccb50 100644
--- a/Documentation/coccinelle.txt
+++ b/Documentation/coccinelle.txt
@@ -56,7 +56,7 @@ There are two ways to use Coccinelle with the Linux kernel.
 
 There are a number of optional parameters that can be used with the build target.
 
-	make coccicheck MODE={patch,report,context,org} COCCI=? M=?
+	make coccicheck MODE={patch,report,context,org} COCCI=? M=? PARALLEL=?
 
 MODE:
 	Determines what mode cocci operates in. If no mode is specified
@@ -71,6 +71,10 @@ M:
 	Limit cocci to a subset of directories. This is very similar to the
 	way the build system works when building modules.
 
+PARALLEL:
+	Number of *.cocci SmPL scripts found under scripts/coccinelle/ to run
+	at the same time.
+
 
  Using Coccinelle on the Linux kernel
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- 
1.7.2.5

Re: [PATCH 2/3] coccinelle.txt: add documentation of PARALLEL= flag

[Nicolas Palix]

On Wed, Jul 13, 2011 at 3:26 AM, Greg Dietsche <Gregory.Dietsche@cuw.edu> wrote:
> Document the new PARALLEL= option of Coccinelle which
> allows running SmPL scripts in parallel.
>
> Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>
Acked-by: Nicolas Palix <npalix.work@gmail.com>
> ---
>  Documentation/coccinelle.txt |    6 +++++-
>  1 files changed, 5 insertions(+), 1 deletions(-)
>
> diff --git a/Documentation/coccinelle.txt b/Documentation/coccinelle.txt
> index 9495a4b..3bccb50 100644
> --- a/Documentation/coccinelle.txt
> +++ b/Documentation/coccinelle.txt
> @@ -56,7 +56,7 @@ There are two ways to use Coccinelle with the Linux kernel.
>
>  There are a number of optional parameters that can be used with the build target.
>
> -       make coccicheck MODE={patch,report,context,org} COCCI=? M=?
> +       make coccicheck MODE={patch,report,context,org} COCCI=? M=? PARALLEL=?
>
>  MODE:
>        Determines what mode cocci operates in. If no mode is specified
> @@ -71,6 +71,10 @@ M:
>        Limit cocci to a subset of directories. This is very similar to the
>        way the build system works when building modules.
>
> +PARALLEL:
> +       Number of *.cocci SmPL scripts found under scripts/coccinelle/ to run
> +       at the same time.
> +
>
>  Using Coccinelle on the Linux kernel
>  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> --
> 1.7.2.5
>
>



-- 
Nicolas Palix
http://sardes.inrialpes.fr/~npalix/

[PATCH 3/3] coccicheck: add parallel execution

[Greg Dietsche]

For example to process 6 SmPL patches at the same time:
	make coccicheck PARALLEL=6

Results are held in /tmp until the scripts finish. By doing this the
script is able to collate the results from each SmPL patch instead of
interleaving them in the output.

Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>
---
 scripts/coccicheck |   27 ++++++++++++++++++++++++++-
 1 files changed, 26 insertions(+), 1 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 1bb1a1b..27e8ff4 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -97,10 +97,35 @@ coccinelle () {
 
 }
 
+parallel_cleanup () {
+	pkill -P $$ > /dev/null
+	rm /tmp/cocci_parallel_$$_* 2>/dev/null
+	exit
+}
+
+trap parallel_cleanup SIGINT
+PARALLEL_ID=0
+
 if [ "$COCCI" = "" ] ; then
     for f in `find $srctree/scripts/coccinelle/ -name '*.cocci' -type f | sort`; do
-	coccinelle $f
+	if [ "$PARALLEL" -eq "$PARALLEL" 2>/dev/null ] ; then
+		if [ "$ONLINE" = "0" ] ; then
+			echo Processing: `basename $f`
+		fi
+		coccinelle $f>/tmp/cocci_parallel_$$_$PARALLEL_ID 2>&1 &
+		PARALLEL_ID=$(($PARALLEL_ID + 1))
+		while [ "`jobs -p | wc -l`" -ge "$PARALLEL" ] ; do
+			sleep 3
+		done
+	else
+		coccinelle $f
+	fi
     done
+    wait
+    if [ "$PARALLEL_ID" -ge "0" ] ; then
+	cat /tmp/cocci_parallel_$$_*
+	parallel_cleanup
+    fi
 else
     coccinelle $COCCI
 fi
-- 
1.7.2.5

Re: [PATCH 3/3] coccicheck: add parallel execution

[Joe Perches]

On Tue, 2011-07-12 at 20:26 -0500, Greg Dietsche wrote:
> For example to process 6 SmPL patches at the same time:
> 	make coccicheck PARALLEL=6
> Results are held in /tmp until the scripts finish. By doing this the
> script is able to collate the results from each SmPL patch instead of
> interleaving them in the output.

You might try adding support for gnu parallel instead.
http://www.gnu.org/software/parallel/

Re: [PATCH 3/3] coccicheck: add parallel execution

[Greg Dietsche]

On 07/12/2011 09:42 PM, Joe Perches wrote:
> On Tue, 2011-07-12 at 20:26 -0500, Greg Dietsche wrote:
>    
>> For example to process 6 SmPL patches at the same time:
>> 	make coccicheck PARALLEL=6
>> Results are held in /tmp until the scripts finish. By doing this the
>> script is able to collate the results from each SmPL patch instead of
>> interleaving them in the output.
>>      
> You might try adding support for gnu parallel instead.
> http://www.gnu.org/software/parallel/
>
>    
That is a very interesting and useful tool :) I'd never heard of it 
before. The man page scares me a little bit - lots of options are marked 
as "alpha testing," but the ones i tried seemed to work fine. Anyway, 
without any modifications to Coccinelle, it is possible to do this:

ls scripts/coccinelle/*/*.cocci | parallel -j+0 make coccicheck COCCI={} 
MODE=patch

it seems to behave mostly like my patch, but doesn't cleanup quite as 
well if you decide to kill it with ctrl-c (it leaves instances of spatch 
running). On the upside, it is up to 3 seconds faster than my patch 
(because my patch waits 3 seconds before checking on the number of jobs 
that are running).

Greg

Re: [PATCH 3/3] coccicheck: add parallel execution

[Nicolas Palix]

Hi,

On Thu, Jul 14, 2011 at 4:14 AM, Greg Dietsche <gregory.dietsche@cuw.edu> wrote:
> On 07/12/2011 09:42 PM, Joe Perches wrote:
>>
>> On Tue, 2011-07-12 at 20:26 -0500, Greg Dietsche wrote:
>>
>>>
>>> For example to process 6 SmPL patches at the same time:
>>>        make coccicheck PARALLEL=6
>>> Results are held in /tmp until the scripts finish. By doing this the
>>> script is able to collate the results from each SmPL patch instead of
>>> interleaving them in the output.

On my system, when killing the make task, the cleanup function only
kills the first level of
children processes (shell processes) thus living spatch running, both
the frontend script and the actual code.

Could you resubmit a patch with "pkill -P $$" replaced by  "pkill -s 0"
to kill all related processes ?

That is the only problem I notice. I will thus acked your patch.
Keep Michal in CC for your resubmission as he will commit
your patch.

>>>
>>
>> You might try adding support for gnu parallel instead.
>> http://www.gnu.org/software/parallel/
>>
>>
>
> That is a very interesting and useful tool :) I'd never heard of it before.
> The man page scares me a little bit - lots of options are marked as "alpha
> testing," but the ones i tried seemed to work fine. Anyway, without any
> modifications to Coccinelle, it is possible to do this:
>
> ls scripts/coccinelle/*/*.cocci | parallel -j+0 make coccicheck COCCI={}
> MODE=patch

Maybe that alternative could be added to the documentation ?

>
> it seems to behave mostly like my patch, but doesn't cleanup quite as well
> if you decide to kill it with ctrl-c (it leaves instances of spatch
> running). On the upside, it is up to 3 seconds faster than my patch (because
> my patch waits 3 seconds before checking on the number of jobs that are
> running).
>
> Greg
>



-- 
Nicolas Palix
http://sardes.inrialpes.fr/~npalix/

Re: [PATCH 3/3] coccicheck: add parallel execution

[Michal Marek]

On 26.7.2011 15:33, Nicolas Palix wrote:
> Hi,
>
> On Thu, Jul 14, 2011 at 4:14 AM, Greg Dietsche<gregory.dietsche@cuw.edu>  wrote:
>> On 07/12/2011 09:42 PM, Joe Perches wrote:
>>>
>>> On Tue, 2011-07-12 at 20:26 -0500, Greg Dietsche wrote:
>>>
>>>>
>>>> For example to process 6 SmPL patches at the same time:
>>>>         make coccicheck PARALLEL=6
>>>> Results are held in /tmp until the scripts finish. By doing this the
>>>> script is able to collate the results from each SmPL patch instead of
>>>> interleaving them in the output.

Why not move the loop into some Makefile and let make take care of the 
parallel execution? Something among the lines of

results := check1.out check2.out ... (generated)
coccicheck: $(results)
	cat $^

%.out: FORCE
	... run given test and store its result ...

then the user can simply type make -j6 coccicheck.

Note: I haven't seen the patch, I only got Nicolas' mail now.

Michal

Re: [PATCH 3/3] coccicheck: add parallel execution

[Greg Dietsche]

On 7/26/2011 8:53 AM, Michal Marek wrote:
> Why not move the loop into some Makefile and let make take care of the 
> parallel execution? Something among the lines of
>
> results := check1.out check2.out ... (generated)
> coccicheck: $(results)
>     cat $^
>
> %.out: FORCE
>     ... run given test and store its result ...
>
> then the user can simply type make -j6 coccicheck.
>
> Note: I haven't seen the patch, I only got Nicolas' mail now.
>
> Michal
>

yes, I agree, that would be better - especially from a user perspective.
I'm not much of a makefile expert at all. I did try that approach 
initially, but it seemed like it was going to turn into a rewrite of the 
coccinelle script. If someone can do it via make, I'd say go for it :)  
I won't have much free time for the foreseeable future to tackle it. 
using "make -j" is much more comfortable and intuitive than typing 
PARALLEL= ...

Greg

[PATCH v2] coccicheck: add parallel execution

[Greg Dietsche]

For example to process 6 SmPL patches at the same time:
	make coccicheck PARALLEL=6

Results are held in /tmp until the scripts finish. By doing this the
script is able to collate the results from each SmPL patch instead of
interleaving them in the output.

Version 2: change pkill -P $$ to pkill -s 0

Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>
---
 scripts/coccicheck |   27 ++++++++++++++++++++++++++-
 1 files changed, 26 insertions(+), 1 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 1bb1a1b..c2a04c2 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -97,10 +97,35 @@ coccinelle () {
 
 }
 
+parallel_cleanup () {
+	pkill -s 0 > /dev/null
+	rm /tmp/cocci_parallel_$$_* 2>/dev/null
+	exit
+}
+
+trap parallel_cleanup SIGINT
+PARALLEL_ID=0
+
 if [ "$COCCI" = "" ] ; then
     for f in `find $srctree/scripts/coccinelle/ -name '*.cocci' -type f | sort`; do
-	coccinelle $f
+	if [ "$PARALLEL" -eq "$PARALLEL" 2>/dev/null ] ; then
+		if [ "$ONLINE" = "0" ] ; then
+			echo Processing: `basename $f`
+		fi
+		coccinelle $f>/tmp/cocci_parallel_$$_$PARALLEL_ID 2>&1 &
+		PARALLEL_ID=$(($PARALLEL_ID + 1))
+		while [ "`jobs -p | wc -l`" -ge "$PARALLEL" ] ; do
+			sleep 3
+		done
+	else
+		coccinelle $f
+	fi
     done
+    wait
+    if [ "$PARALLEL_ID" -ge "0" ] ; then
+	cat /tmp/cocci_parallel_$$_*
+	parallel_cleanup
+    fi
 else
     coccinelle $COCCI
 fi
-- 
1.7.2.5

Re: [PATCH v2] coccicheck: add parallel execution

[Nicolas Palix]

On Wed, Jul 27, 2011 at 12:36 AM, Greg Dietsche
<Gregory.Dietsche@cuw.edu> wrote:
> For example to process 6 SmPL patches at the same time:
>        make coccicheck PARALLEL=6
>
> Results are held in /tmp until the scripts finish. By doing this the
> script is able to collate the results from each SmPL patch instead of
> interleaving them in the output.
>
> Version 2: change pkill -P $$ to pkill -s 0
>
> Signed-off-by: Greg Dietsche <Gregory.Dietsche@cuw.edu>
Acked-by: Nicolas Palix <npalix.work@gmail.com>
> ---
>  scripts/coccicheck |   27 ++++++++++++++++++++++++++-
>  1 files changed, 26 insertions(+), 1 deletions(-)
>
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index 1bb1a1b..c2a04c2 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -97,10 +97,35 @@ coccinelle () {
>
>  }
>
> +parallel_cleanup () {
> +       pkill -s 0 > /dev/null
> +       rm /tmp/cocci_parallel_$$_* 2>/dev/null
> +       exit
> +}
> +
> +trap parallel_cleanup SIGINT
> +PARALLEL_ID=0
> +
>  if [ "$COCCI" = "" ] ; then
>     for f in `find $srctree/scripts/coccinelle/ -name '*.cocci' -type f | sort`; do
> -       coccinelle $f
> +       if [ "$PARALLEL" -eq "$PARALLEL" 2>/dev/null ] ; then
> +               if [ "$ONLINE" = "0" ] ; then
> +                       echo Processing: `basename $f`
> +               fi
> +               coccinelle $f>/tmp/cocci_parallel_$$_$PARALLEL_ID 2>&1 &
> +               PARALLEL_ID=$(($PARALLEL_ID + 1))
> +               while [ "`jobs -p | wc -l`" -ge "$PARALLEL" ] ; do
> +                       sleep 3
> +               done
> +       else
> +               coccinelle $f
> +       fi
>     done
> +    wait
> +    if [ "$PARALLEL_ID" -ge "0" ] ; then
> +       cat /tmp/cocci_parallel_$$_*
> +       parallel_cleanup
> +    fi
>  else
>     coccinelle $COCCI
>  fi
> --
> 1.7.2.5
>
>



-- 
Nicolas Palix
http://sardes.inrialpes.fr/~npalix/

Re: [PATCH 0/3] coccinelle: add parallel execution

[Ian Campbell]

On Tue, 2011-07-12 at 20:26 -0500, Greg Dietsche wrote:
> This patch series adds a new flag PARALLEL= to the coccicheck build target and documents the new feature.
> It works a lot like make's -j flag. I looked for a way to have make pass on the value of -j, but
> didn't find one - if anyone knows how to do that, I'd love to hear about it! 

I suppose in principal you could parse $MAKEFLAGS (or $MFLAGS) for
--jobserver-fds and implement compatibility with make's jobserver
protocol ( http://mad-scientist.net/make/jobserver.html ). I don't know
stable or outward facing the make guys consider that interface, plus it
all sounds a bit fiddly to me...

Ian.

-- 
Ian Campbell

QOTD:
	"I thought I saw a unicorn on the way over, but it was just a
	horse with one of the horns broken off."