[Cocci] Using Coccinelle to add #include lines

[Cocci] Using Coccinelle to add #include lines

[Peter Maydell]

Hi. I'm trying to use Coccinelle to automate the process of imposing
this rule:
 * every .c file should #include "qemu/osdep.h" as the first thing it
   #includes

but I'm having some trouble figuring out how to get Coccinelle to do this.
I'm hoping somebody can suggest to me where I'm going wrong and/or
the right way to do this. Below I'll explain the things I've tried
already:

My first attempt was:

===begin===
// Add new include before first "..." include
@ add1 @
@@

+ #include "qemu/osdep.h"
 #include "..."

===endit===

This almost works, but if the first include happens to be "#include <...>"
rather than "..." then it will put the include in the wrong place. (It
also doesn't remove any existing-but-not-at-the-top include of osdep.h,
but I'll get to that later). So I tried:

===begin===
@ add2 @
@@

+ #include "qemu/osdep.h"
(
 #include "..."
|
 #include <...>
)

===endit===

This causes Coccinelle to complain:
init_defs_builtins: /usr/share/coccinelle/standard.h
Fatal error: exception Failure("6: no available token to attach to")

So then I tried:
===begin===
@ add3 @
@@

+ #include "qemu/osdep.h"
 #include
(
 "..."
|
 <...>
)

===endit===

which produces a different cryptic error:
init_defs_builtins: /usr/share/coccinelle/standard.h
File "/home/petmay01/linaro/coccinelle-patches/osdep-include-2.spatch",
line 7, column 1,  charpos = 87
    around = '#', whole content =  #include
Fatal error: exception Lexer_cocci.Lexical("unrecognised symbol, in
token rule: #")


OK, plan B: have two rules like 'add1', one for each kind of include. This
might result in two includes per file, but we need to remove any
extra existing includes anyway, and that will delete the extras.
However my removal rule:

===begin===
@ delete @
@@

 #include "qemu/osdep.h"
 ...
- #include "qemu/osdep.h"

===endit===

doesn't result in coccinelle thinking it needs to change the file.
How do I say "delete the 2nd, 3rd, etc instances of this line" ?

(I'm using "spatch version 1.0.0-rc19 with Python support and with
PCRE support", which is the version in Ubuntu Trusty.)

thanks in advance
-- PMM

[Cocci] Using Coccinelle to add #include lines

[SF Markus Elfring]

> (I'm using "spatch version 1.0.0-rc19 with Python support and with
> PCRE support", which is the version in Ubuntu Trusty.)

How are the chances for you to retry your adjustments of include statements
with the current software "Coccinelle 1.0.4"?

Regards,
Markus

[Cocci] Using Coccinelle to add #include lines

[Julia Lawall]

On Fri, 4 Dec 2015, Peter Maydell wrote:

> Hi. I'm trying to use Coccinelle to automate the process of imposing
> this rule:
>  * every .c file should #include "qemu/osdep.h" as the first thing it
>    #includes
>
> but I'm having some trouble figuring out how to get Coccinelle to do this.
> I'm hoping somebody can suggest to me where I'm going wrong and/or
> the right way to do this. Below I'll explain the things I've tried
> already:
>
> My first attempt was:
>
> ===begin===
> // Add new include before first "..." include
> @ add1 @
> @@
>
> + #include "qemu/osdep.h"
>  #include "..."
>
> ===endit===i
>
> This almost works, but if the first include happens to be "#include <...>"
> rather than "..." then it will put the include in the wrong place. (It
> also doesn't remove any existing-but-not-at-the-top include of osdep.h,
> but I'll get to that later). So I tried:
>
> ===begin===
> @ add2 @
> @@
>
> + #include "qemu/osdep.h"
> (
>  #include "..."
> |
>  #include <...>
> )

Propagate the modification into the ( | ), ie duplicating the
modification.

> ===endit===
>
> This causes Coccinelle to complain:
> init_defs_builtins: /usr/share/coccinelle/standard.h
> Fatal error: exception Failure("6: no available token to attach to")
>
> So then I tried:
> ===begin===
> @ add3 @
> @@
>
> + #include "qemu/osdep.h"
>  #include
> (
>  "..."
> |
>  <...>
> )
>
> ===endit===
>
> which produces a different cryptic error:
> init_defs_builtins: /usr/share/coccinelle/standard.h
> File "/home/petmay01/linaro/coccinelle-patches/osdep-include-2.spatch",
> line 7, column 1,  charpos = 87
>     around = '#', whole content =  #include
> Fatal error: exception Lexer_cocci.Lexical("unrecognised symbol, in
> token rule: #")

You can't do this one.  A disjunction is not supported here.

>
>
> OK, plan B: have two rules like 'add1', one for each kind of include. This
> might result in two includes per file, but we need to remove any
> extra existing includes anyway, and that will delete the extras.
> However my removal rule:
>
> ===begin===
> @ delete @
> @@
>
>  #include "qemu/osdep.h"
>  ...
> - #include "qemu/osdep.h"

Coccinelle processes one top-level thing at a time.  The includes are in
different top level units.

The solution would be to take the position of each one, and then save only
the position of the earliest one.  This would require using scripts and
hash tables, and is kind of clunky.

julia

> ===endit===
>
> doesn't result in coccinelle thinking it needs to change the file.
> How do I say "delete the 2nd, 3rd, etc instances of this line" ?
>
> (I'm using "spatch version 1.0.0-rc19 with Python support and with
> PCRE support", which is the version in Ubuntu Trusty.)
>
> thanks in advance
> -- PMM
> _______________________________________________
> Cocci mailing list
> Cocci at systeme.lip6.fr
> https://systeme.lip6.fr/mailman/listinfo/cocci
>

[Cocci] Using Coccinelle to add #include lines

[Peter Maydell]

On 4 December 2015 at 16:05, Julia Lawall <julia.lawall@lip6.fr> wrote:
> On Fri, 4 Dec 2015, Peter Maydell wrote:
>> So I tried:
>>
>> ===begin===
>> @ add2 @
>> @@
>>
>> + #include "qemu/osdep.h"
>> (
>>  #include "..."
>> |
>>  #include <...>
>> )
>
> Propagate the modification into the ( | ), ie duplicating the
> modification.

Do you mean like this:

===begin===
@ add4 @
@@

(
+ #include "qemu/osdep.h"
 #include "..."
|
+ #include "qemu/osdep.h"
 #include <...>
)

===endit===

?

That runs but has the same effect as having two separate
rules: a new #include line is added both before the first
#include "..." and before the first #include <...>, rather
than just once before the first incidence of any #include.

>> OK, plan B: have two rules like 'add1', one for each kind of include. This
>> might result in two includes per file, but we need to remove any
>> extra existing includes anyway, and that will delete the extras.
>> However my removal rule:
>>
>> ===begin===
>> @ delete @
>> @@
>>
>>  #include "qemu/osdep.h"
>>  ...
>> - #include "qemu/osdep.h"
>
> Coccinelle processes one top-level thing at a time.  The includes are in
> different top level units.
>
> The solution would be to take the position of each one, and then save only
> the position of the earliest one.  This would require using scripts and
> hash tables, and is kind of clunky.

Hmm. Maybe I'm better off just using sed or perl for this job?
#include lines are really pretty much line-based after all.

thanks
-- PMM

[Cocci] Using Coccinelle to add #include lines

[Julia Lawall]

> Hmm. Maybe I'm better off just using sed or perl for this job?
> #include lines are really pretty much line-based after all.

If you have a good solution for getting the first include with sed or
perl, I think it would be better to do that.  Coccinelle is designed to be
independent of the position of top-level entities in the file, so it is
not very convenient to make it sensitive to that.

julia

[Cocci] Using Coccinelle to add #include lines

[SF Markus Elfring]

>> The solution would be to take the position of each one, and then save only
>> the position of the earliest one.  This would require using scripts and
>> hash tables, and is kind of clunky.
> 
> Maybe I'm better off just using sed or perl for this job?

Would you dare to try the semantic patch language out a bit more together
with capabilities from programming languages like "Python" or "OCaml"?


> #include lines are really pretty much line-based after all.

How safe and robust should the desired adjustment of include statements
become for your source files?

Regards,
Markus

[Cocci] Using Coccinelle to add #include lines

[Peter Maydell]

On 4 December 2015 at 16:16, Julia Lawall <julia.lawall@lip6.fr> wrote:
>> Hmm. Maybe I'm better off just using sed or perl for this job?
>> #include lines are really pretty much line-based after all.
>
> If you have a good solution for getting the first include with sed or
> perl, I think it would be better to do that.  Coccinelle is designed to be
> independent of the position of top-level entities in the file, so it is
> not very convenient to make it sensitive to that.

OK. I think I'll use coccinelle to add the new #includes and then
perl to remove the duplicates. The removal is a perl one-liner:
perl -n -i -e 'print if !/#include "qemu\/osdep.h"/ || !$n++;' foo.c

Thanks for your help.

-- PMM