[PATCH] pycocci: sync pycocci with Coccinelle upstream

[PATCH] pycocci: sync pycocci with Coccinelle upstream

[Luis R. Rodriguez]

pycocci is now part of Coccinelle upstream. Coccinelle has a plan
to actually extend multithreaded support to enable us to not have
to script this out. Until then pyccoci is used and I'll sync both
solutions up as there is a bit of latency between a coccinelle release
and what we may need from pycocci.

Please send patches or report issues about pycocci to the coccinelle
mailing list.

This commit syncs pycocci with the upstream pycocci with just one
minor change introduced:

commit cf8ef1aee7d40cddfee0f8707b2f481eb8258b4d
Author: Luis R. Rodriguez <mcgrof@suse.com>
Date:   Fri May 29 15:15:57 2015 -0700

    pycocci: update Python script to print errors clearly

    When you run into errors with the pycocci script errors
    seem cryptic, fix this. While at it also spit out what the
    script will run, as typically when debugging you just want
    to runt he same commands on your own.

    For instance, when run on a 32 core machine you'll now see:

    mcgrof@drvbp1 ~/linux-next (git::master)$ time pycocci rename-dma.cocci ./
    spatch --sp-file rename-dma.cocci --in-place --recursive-includes --relax-include-path --use-coccigrep --timeout 120 --dir ./ -max 32 -index 0
    ...
    spatch --sp-file rename-dma.cocci --in-place --recursive-includes --relax-include-path --use-coccigrep --timeout 120 --dir ./ -max 32 -index 27

    real    0m50.074s
    user    14m50.168s
    sys     2m8.260s

Signed-off-by: Luis R. Rodriguez <mcgrof@do-not-panic.com>
---
 devel/pycocci | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/devel/pycocci b/devel/pycocci
index b6bb78e..4aa1d36 100755
--- a/devel/pycocci
+++ b/devel/pycocci
@@ -45,22 +45,22 @@ class CoccinelleError(Exception):
 class ExecutionErrorThread(CoccinelleError):
     def __init__(self, errcode, fn, cocci_file, threads, t, logwrite, print_name):
         self.error_code = errcode
-        logwrite("Failed to apply changes from %s" % print_name)
+        logwrite("Failed to apply changes from %s\n" % print_name)
 
-        logwrite("Specific log output from change that failed using %s" % print_name)
+        logwrite("Specific log output from change that failed using %s\n" % print_name)
         tf = open(fn, 'r')
         for line in tf.read():
-            logwrite('> %s' % line)
+            logwrite(line)
         tf.close()
 
-        logwrite("Full log using %s" % print_name)
+        logwrite("Full log using %s\n" % print_name)
         for num in range(threads):
             fn = os.path.join(t, '.tmp_spatch_worker.' + str(num))
             if (not os.path.isfile(fn)):
                 continue
             tf = open(fn, 'r')
             for line in tf.read():
-                logwrite('> %s' % line)
+                logwrite(line)
             tf.close()
             os.unlink(fn)
 
@@ -82,6 +82,7 @@ def spatch(cocci_file, outdir,
 
     fn = os.path.join(temp_dir, '.tmp_spatch_worker.' + str(thread_id))
     outfile = open(fn, 'w')
+    logwrite("%s\n" % " ".join(cmd))
 
     sprocess = subprocess.Popen(cmd,
                                stdout=outfile, stderr=subprocess.STDOUT,
-- 
2.1.0

[Cocci] [PATCH] pycocci: sync pycocci with Coccinelle upstream

[Luis R. Rodriguez]

pycocci is now part of Coccinelle upstream. Coccinelle has a plan
to actually extend multithreaded support to enable us to not have
to script this out. Until then pyccoci is used and I'll sync both
solutions up as there is a bit of latency between a coccinelle release
and what we may need from pycocci.

Please send patches or report issues about pycocci to the coccinelle
mailing list.

This commit syncs pycocci with the upstream pycocci with just one
minor change introduced:

commit cf8ef1aee7d40cddfee0f8707b2f481eb8258b4d
Author: Luis R. Rodriguez <mcgrof@suse.com>
Date:   Fri May 29 15:15:57 2015 -0700

    pycocci: update Python script to print errors clearly

    When you run into errors with the pycocci script errors
    seem cryptic, fix this. While at it also spit out what the
    script will run, as typically when debugging you just want
    to runt he same commands on your own.

    For instance, when run on a 32 core machine you'll now see:

    mcgrof at drvbp1 ~/linux-next (git::master)$ time pycocci rename-dma.cocci ./
    spatch --sp-file rename-dma.cocci --in-place --recursive-includes --relax-include-path --use-coccigrep --timeout 120 --dir ./ -max 32 -index 0
    ...
    spatch --sp-file rename-dma.cocci --in-place --recursive-includes --relax-include-path --use-coccigrep --timeout 120 --dir ./ -max 32 -index 27

    real    0m50.074s
    user    14m50.168s
    sys     2m8.260s

Signed-off-by: Luis R. Rodriguez <mcgrof@do-not-panic.com>
---
 devel/pycocci | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/devel/pycocci b/devel/pycocci
index b6bb78e..4aa1d36 100755
--- a/devel/pycocci
+++ b/devel/pycocci
@@ -45,22 +45,22 @@ class CoccinelleError(Exception):
 class ExecutionErrorThread(CoccinelleError):
     def __init__(self, errcode, fn, cocci_file, threads, t, logwrite, print_name):
         self.error_code = errcode
-        logwrite("Failed to apply changes from %s" % print_name)
+        logwrite("Failed to apply changes from %s\n" % print_name)
 
-        logwrite("Specific log output from change that failed using %s" % print_name)
+        logwrite("Specific log output from change that failed using %s\n" % print_name)
         tf = open(fn, 'r')
         for line in tf.read():
-            logwrite('> %s' % line)
+            logwrite(line)
         tf.close()
 
-        logwrite("Full log using %s" % print_name)
+        logwrite("Full log using %s\n" % print_name)
         for num in range(threads):
             fn = os.path.join(t, '.tmp_spatch_worker.' + str(num))
             if (not os.path.isfile(fn)):
                 continue
             tf = open(fn, 'r')
             for line in tf.read():
-                logwrite('> %s' % line)
+                logwrite(line)
             tf.close()
             os.unlink(fn)
 
@@ -82,6 +82,7 @@ def spatch(cocci_file, outdir,
 
     fn = os.path.join(temp_dir, '.tmp_spatch_worker.' + str(thread_id))
     outfile = open(fn, 'w')
+    logwrite("%s\n" % " ".join(cmd))
 
     sprocess = subprocess.Popen(cmd,
                                stdout=outfile, stderr=subprocess.STDOUT,
-- 
2.1.0

Re: [PATCH] pycocci: sync pycocci with Coccinelle upstream

[Johannes Berg]

On Mon, 2015-06-08 at 13:01 -0700, Luis R. Rodriguez wrote:
> pycocci is now part of Coccinelle upstream. Coccinelle has a plan
> to actually extend multithreaded support to enable us to not have
> to script this out. Until then pyccoci is used and I'll sync both
> solutions up as there is a bit of latency between a coccinelle release
> and what we may need from pycocci.

If it's upstream, do they ship it? If so it wouldn't be needed that we
have a copy, it seems?

johannes

[Cocci] [PATCH] pycocci: sync pycocci with Coccinelle upstream

[Johannes Berg]

On Mon, 2015-06-08 at 13:01 -0700, Luis R. Rodriguez wrote:
> pycocci is now part of Coccinelle upstream. Coccinelle has a plan
> to actually extend multithreaded support to enable us to not have
> to script this out. Until then pyccoci is used and I'll sync both
> solutions up as there is a bit of latency between a coccinelle release
> and what we may need from pycocci.

If it's upstream, do they ship it? If so it wouldn't be needed that we
have a copy, it seems?

johannes

Re: [PATCH] pycocci: sync pycocci with Coccinelle upstream

[Luis R. Rodriguez]

On Mon, Jun 8, 2015 at 1:31 PM, Johannes Berg <johannes@sipsolutions.net> wrote:
> On Mon, 2015-06-08 at 13:01 -0700, Luis R. Rodriguez wrote:
>> pycocci is now part of Coccinelle upstream. Coccinelle has a plan
>> to actually extend multithreaded support to enable us to not have
>> to script this out. Until then pyccoci is used and I'll sync both
>> solutions up as there is a bit of latency between a coccinelle release
>> and what we may need from pycocci.
>
> If it's upstream, do they ship it? If so it wouldn't be needed that we
> have a copy, it seems?

Indeed, Coccinelle 1.0 has pycocci except for this last patch I just
mentioned. I think it is important enough for our sake that the errors
are reported well, I think its fair to drop this in backports once
Coccinelle 1.1 is released and we can reasonably expect users to have
it installed.

 Luis

[Cocci] [PATCH] pycocci: sync pycocci with Coccinelle upstream

[Luis R. Rodriguez]

On Mon, Jun 8, 2015 at 1:31 PM, Johannes Berg <johannes@sipsolutions.net> wrote:
> On Mon, 2015-06-08 at 13:01 -0700, Luis R. Rodriguez wrote:
>> pycocci is now part of Coccinelle upstream. Coccinelle has a plan
>> to actually extend multithreaded support to enable us to not have
>> to script this out. Until then pyccoci is used and I'll sync both
>> solutions up as there is a bit of latency between a coccinelle release
>> and what we may need from pycocci.
>
> If it's upstream, do they ship it? If so it wouldn't be needed that we
> have a copy, it seems?

Indeed, Coccinelle 1.0 has pycocci except for this last patch I just
mentioned. I think it is important enough for our sake that the errors
are reported well, I think its fair to drop this in backports once
Coccinelle 1.1 is released and we can reasonably expect users to have
it installed.

 Luis

Re: [PATCH] pycocci: sync pycocci with Coccinelle upstream

[Johannes Berg]

On Mon, 2015-06-08 at 13:43 -0700, Luis R. Rodriguez wrote:
> On Mon, Jun 8, 2015 at 1:31 PM, Johannes Berg <johannes@sipsolutions.net> wrote:
> > On Mon, 2015-06-08 at 13:01 -0700, Luis R. Rodriguez wrote:
> >> pycocci is now part of Coccinelle upstream. Coccinelle has a plan
> >> to actually extend multithreaded support to enable us to not have
> >> to script this out. Until then pyccoci is used and I'll sync both
> >> solutions up as there is a bit of latency between a coccinelle release
> >> and what we may need from pycocci.
> >
> > If it's upstream, do they ship it? If so it wouldn't be needed that we
> > have a copy, it seems?
> 
> Indeed, Coccinelle 1.0 has pycocci except for this last patch I just
> mentioned. I think it is important enough for our sake that the errors
> are reported well, I think its fair to drop this in backports once
> Coccinelle 1.1 is released and we can reasonably expect users to have
> it installed.

Not sure I see the point in applying it now - clearly we haven't had any
issues here that required this change, so why bother?

johannes

[Cocci] [PATCH] pycocci: sync pycocci with Coccinelle upstream

[Johannes Berg]

On Mon, 2015-06-08 at 13:43 -0700, Luis R. Rodriguez wrote:
> On Mon, Jun 8, 2015 at 1:31 PM, Johannes Berg <johannes@sipsolutions.net> wrote:
> > On Mon, 2015-06-08 at 13:01 -0700, Luis R. Rodriguez wrote:
> >> pycocci is now part of Coccinelle upstream. Coccinelle has a plan
> >> to actually extend multithreaded support to enable us to not have
> >> to script this out. Until then pyccoci is used and I'll sync both
> >> solutions up as there is a bit of latency between a coccinelle release
> >> and what we may need from pycocci.
> >
> > If it's upstream, do they ship it? If so it wouldn't be needed that we
> > have a copy, it seems?
> 
> Indeed, Coccinelle 1.0 has pycocci except for this last patch I just
> mentioned. I think it is important enough for our sake that the errors
> are reported well, I think its fair to drop this in backports once
> Coccinelle 1.1 is released and we can reasonably expect users to have
> it installed.

Not sure I see the point in applying it now - clearly we haven't had any
issues here that required this change, so why bother?

johannes

Re: [PATCH] pycocci: sync pycocci with Coccinelle upstream

[Luis R. Rodriguez]

On Mon, Jun 8, 2015 at 11:29 PM, Johannes Berg
<johannes@sipsolutions.net> wrote:
> On Mon, 2015-06-08 at 13:43 -0700, Luis R. Rodriguez wrote:
>> On Mon, Jun 8, 2015 at 1:31 PM, Johannes Berg <johannes@sipsolutions.net> wrote:
>> > On Mon, 2015-06-08 at 13:01 -0700, Luis R. Rodriguez wrote:
>> >> pycocci is now part of Coccinelle upstream. Coccinelle has a plan
>> >> to actually extend multithreaded support to enable us to not have
>> >> to script this out. Until then pyccoci is used and I'll sync both
>> >> solutions up as there is a bit of latency between a coccinelle release
>> >> and what we may need from pycocci.
>> >
>> > If it's upstream, do they ship it? If so it wouldn't be needed that we
>> > have a copy, it seems?
>>
>> Indeed, Coccinelle 1.0 has pycocci except for this last patch I just
>> mentioned. I think it is important enough for our sake that the errors
>> are reported well, I think its fair to drop this in backports once
>> Coccinelle 1.1 is released and we can reasonably expect users to have
>> it installed.
>
> Not sure I see the point in applying it now - clearly we haven't had any
> issues here that required this change, so why bother?

Well, I had issues with it :P its why I fixed it.

 Luis

[Cocci] [PATCH] pycocci: sync pycocci with Coccinelle upstream

[Luis R. Rodriguez]

On Mon, Jun 8, 2015 at 11:29 PM, Johannes Berg
<johannes@sipsolutions.net> wrote:
> On Mon, 2015-06-08 at 13:43 -0700, Luis R. Rodriguez wrote:
>> On Mon, Jun 8, 2015 at 1:31 PM, Johannes Berg <johannes@sipsolutions.net> wrote:
>> > On Mon, 2015-06-08 at 13:01 -0700, Luis R. Rodriguez wrote:
>> >> pycocci is now part of Coccinelle upstream. Coccinelle has a plan
>> >> to actually extend multithreaded support to enable us to not have
>> >> to script this out. Until then pyccoci is used and I'll sync both
>> >> solutions up as there is a bit of latency between a coccinelle release
>> >> and what we may need from pycocci.
>> >
>> > If it's upstream, do they ship it? If so it wouldn't be needed that we
>> > have a copy, it seems?
>>
>> Indeed, Coccinelle 1.0 has pycocci except for this last patch I just
>> mentioned. I think it is important enough for our sake that the errors
>> are reported well, I think its fair to drop this in backports once
>> Coccinelle 1.1 is released and we can reasonably expect users to have
>> it installed.
>
> Not sure I see the point in applying it now - clearly we haven't had any
> issues here that required this change, so why bother?

Well, I had issues with it :P its why I fixed it.

 Luis

Re: [PATCH] pycocci: sync pycocci with Coccinelle upstream

[Luis R. Rodriguez]

On Tue, Jun 9, 2015 at 3:20 PM, Luis R. Rodriguez
<mcgrof@do-not-panic.com> wrote:
> On Mon, Jun 8, 2015 at 11:29 PM, Johannes Berg
> <johannes@sipsolutions.net> wrote:
>> On Mon, 2015-06-08 at 13:43 -0700, Luis R. Rodriguez wrote:
>>> On Mon, Jun 8, 2015 at 1:31 PM, Johannes Berg <johannes@sipsolutions.net> wrote:
>>> > On Mon, 2015-06-08 at 13:01 -0700, Luis R. Rodriguez wrote:
>>> >> pycocci is now part of Coccinelle upstream. Coccinelle has a plan
>>> >> to actually extend multithreaded support to enable us to not have
>>> >> to script this out. Until then pyccoci is used and I'll sync both
>>> >> solutions up as there is a bit of latency between a coccinelle release
>>> >> and what we may need from pycocci.
>>> >
>>> > If it's upstream, do they ship it? If so it wouldn't be needed that we
>>> > have a copy, it seems?
>>>
>>> Indeed, Coccinelle 1.0 has pycocci except for this last patch I just
>>> mentioned. I think it is important enough for our sake that the errors
>>> are reported well, I think its fair to drop this in backports once
>>> Coccinelle 1.1 is released and we can reasonably expect users to have
>>> it installed.
>>
>> Not sure I see the point in applying it now - clearly we haven't had any
>> issues here that required this change, so why bother?
>
> Well, I had issues with it :P its why I fixed it.

Coccinelle now supports -j option so after proper testing we should
just remove pycocci now.

 Luis

[Cocci] [PATCH] pycocci: sync pycocci with Coccinelle upstream

[Luis R. Rodriguez]

On Tue, Jun 9, 2015 at 3:20 PM, Luis R. Rodriguez
<mcgrof@do-not-panic.com> wrote:
> On Mon, Jun 8, 2015 at 11:29 PM, Johannes Berg
> <johannes@sipsolutions.net> wrote:
>> On Mon, 2015-06-08 at 13:43 -0700, Luis R. Rodriguez wrote:
>>> On Mon, Jun 8, 2015 at 1:31 PM, Johannes Berg <johannes@sipsolutions.net> wrote:
>>> > On Mon, 2015-06-08 at 13:01 -0700, Luis R. Rodriguez wrote:
>>> >> pycocci is now part of Coccinelle upstream. Coccinelle has a plan
>>> >> to actually extend multithreaded support to enable us to not have
>>> >> to script this out. Until then pyccoci is used and I'll sync both
>>> >> solutions up as there is a bit of latency between a coccinelle release
>>> >> and what we may need from pycocci.
>>> >
>>> > If it's upstream, do they ship it? If so it wouldn't be needed that we
>>> > have a copy, it seems?
>>>
>>> Indeed, Coccinelle 1.0 has pycocci except for this last patch I just
>>> mentioned. I think it is important enough for our sake that the errors
>>> are reported well, I think its fair to drop this in backports once
>>> Coccinelle 1.1 is released and we can reasonably expect users to have
>>> it installed.
>>
>> Not sure I see the point in applying it now - clearly we haven't had any
>> issues here that required this change, so why bother?
>
> Well, I had issues with it :P its why I fixed it.

Coccinelle now supports -j option so after proper testing we should
just remove pycocci now.

 Luis