[Cocci] Parsing errors in several headers

[Cocci] Parsing errors in several headers

[Jean-Sébastien Pédron]

Hi!

I'm very new to Coccinelle and I don't understand why spatch(1) returns
parsing errors on some headers and what I can do about that.

== First example ==

Header to parse:
https://github.com/freebsd/freebsd/blob/master/sys/ofed/include/linux/bitops.h

Full spatch command line and output:
http://pastebin.com/DbNhBAF7

spatch(1) fails to parse the #define at the end of the header. This
#define starts a for() loop and uses functions declared ealier in the
same file.

== Second example ==

Header to parse:
https://github.com/freebsd/freebsd/blob/master/sys/ofed/include/linux/spinlock.h

Full spatch command line and output:
http://pastebin.com/C0h5ZN7r

Here, it doesn't like the last #define again. This #define calls other
macros defined in other headers.

FYI, I'm using Coccinelle 1.0.0rc19 on FreeBSD.

Can somebody help me understand what's wrong here?

Thanks!

-- 
Jean-S?bastien P?dron

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: OpenPGP digital signature
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20141028/a7c088ee/attachment.asc>

[Cocci] Parsing errors in several headers

[Julia Lawall]

On Tue, 28 Oct 2014, Jean-S?bastien P?dron wrote:

> Hi!
>
> I'm very new to Coccinelle and I don't understand why spatch(1) returns
> parsing errors on some headers and what I can do about that.
>
> == First example ==
>
> Header to parse:
> https://github.com/freebsd/freebsd/blob/master/sys/ofed/include/linux/bitops.h
>
> Full spatch command line and output:
> http://pastebin.com/DbNhBAF7
>
> spatch(1) fails to parse the #define at the end of the header. This
> #define starts a for() loop and uses functions declared ealier in the
> same file.

Do you need to match or transform some code that is related to this?  If
not, it doesn't matter.

julia

> == Second example ==
>
> Header to parse:
> https://github.com/freebsd/freebsd/blob/master/sys/ofed/include/linux/spinlock.h
>
> Full spatch command line and output:
> http://pastebin.com/C0h5ZN7r
>
> Here, it doesn't like the last #define again. This #define calls other
> macros defined in other headers.
>
> FYI, I'm using Coccinelle 1.0.0rc19 on FreeBSD.
>
> Can somebody help me understand what's wrong here?
>
> Thanks!
>
> --
> Jean-S?bastien P?dron
>
>

[Cocci] Parsing errors in several headers

[Jean-Sébastien Pédron]

On 28.10.2014 19:33, Julia Lawall wrote:
> Do you need to match or transform some code that is related to this?  If
> not, it doesn't matter.

Ok, then I must have a problem with my semantic patch, because it
doesn't change anything :) I thought it was because of the parsing error.

Thank you for the fast reply!

-- 
Jean-S?bastien P?dron

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: OpenPGP digital signature
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20141028/f0d061c8/attachment.asc>

[Cocci] Parsing errors in several headers

[SF Markus Elfring]

> Ok, then I must have a problem with my semantic patch, because it
> doesn't change anything :) I thought it was because of the parsing error.

Would you like to show your SmPL approach here so that we could help you more
eventually?

Regards,
Markus

[Cocci] Parsing errors in several headers

[Jean-Sébastien Pédron]

On 28.10.2014 20:10, SF Markus Elfring wrote:
>> Ok, then I must have a problem with my semantic patch, because it
>> doesn't change anything :) I thought it was because of the parsing error.
> 
> Would you like to show your SmPL approach here so that we could help you more
> eventually?

Sure! I wanted to add a #define just above a function prototype or the
function itself.

Example #1:
  int foo();

should become:
  #define foo BAR(foo)
  int foo();

Example #2:
  int foo() { /* ... */ }

should become:
  #define foo BAR(foo)
  int foo() { /* ... */ }

Here's the script I used, which worked with example #1:

---8<---
@rule1@
identifier func;
@@

-#define        func    BAR(func)

@rule2@
identifier func;
type ret;
@@

+#define        func    BAR(func)
ret func(...);
---8<---

I can handle example #2 with this one:

---8<---
@rule1@
identifier func;
@@

-#define        func    BAR(func)

@rule2@
identifier func;
type ret;
@@

+#define        func    BAR(func)
ret func(...) { ... }
---8<---

However, I fail to assemble those two patches in a single concise one.

Thank you for offering your help :)

-- 
Jean-S?bastien P?dron

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: OpenPGP digital signature
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20141028/09605fdc/attachment.asc>

[Cocci] Parsing errors in several headers

[Julia Lawall]

On Tue, 28 Oct 2014, Jean-S?bastien P?dron wrote:

> On 28.10.2014 20:10, SF Markus Elfring wrote:
> >> Ok, then I must have a problem with my semantic patch, because it
> >> doesn't change anything :) I thought it was because of the parsing error.
> > 
> > Would you like to show your SmPL approach here so that we could help you more
> > eventually?
> 
> Sure! I wanted to add a #define just above a function prototype or the
> function itself.
> 
> Example #1:
>   int foo();
> 
> should become:
>   #define foo BAR(foo)
>   int foo();
> 
> Example #2:
>   int foo() { /* ... */ }
> 
> should become:
>   #define foo BAR(foo)
>   int foo() { /* ... */ }
> 
> Here's the script I used, which worked with example #1:
> 
> ---8<---
> @rule1@
> identifier func;
> @@
> 
> -#define        func    BAR(func)
> 
> @rule2@
> identifier func;
> type ret;
> @@
> 
> +#define        func    BAR(func)
> ret func(...);
> ---8<---
> 
> I can handle example #2 with this one:
> 
> ---8<---
> @rule1@
> identifier func;
> @@
> 
> -#define        func    BAR(func)
> 
> @rule2@
> identifier func;
> type ret;
> @@
> 
> +#define        func    BAR(func)
> ret func(...) { ... }
> ---8<---
> 
> However, I fail to assemble those two patches in a single concise one.

It is not possible.  You need two rules.

julia

> 
> Thank you for offering your help :)
> 
> -- 
> Jean-S?bastien P?dron
> 
> 

[Cocci] Parsing errors in several headers

[SF Markus Elfring]

> I wanted to add a #define just above a function prototype or the
> function itself.
[...]
> @rule1@
> identifier func;
> @@
> 
> -#define        func    BAR(func)

I find this SmPL approach strange according to your description.
- Would you really like to delete the definition for a preprocessor symbol here?
- Should this semantic patch express the change to move the position of such a
statement to other source code places?

Regards,
Markus

[Cocci] Parsing errors in several headers

[Jean-Sébastien Pédron]

On 29.10.2014 09:29, SF Markus Elfring wrote:
>> I wanted to add a #define just above a function prototype or the
>> function itself.
> [...]
>> @rule1@
>> identifier func;
>> @@
>>
>> -#define        func    BAR(func)
> 
> I find this SmPL approach strange according to your description.
> - Would you really like to delete the definition for a preprocessor symbol here?
> - Should this semantic patch express the change to move the position of such a
> statement to other source code places?

I wanted to be able to re-run spatch on an already patched content. The
use case is to add new function prototypes manually and run spatch to
handle the associated #define automatically.

-- 
Jean-S?bastien P?dron

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: OpenPGP digital signature
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20141029/6bcfa88d/attachment.asc>

[Cocci] Addition of preprocessor statements with SmPL

[SF Markus Elfring]

> I wanted to be able to re-run spatch on an already patched content.
> The use case is to add new function prototypes manually and run spatch
> to handle the associated #define automatically.

Thanks for a bit more explanation.

I imagine that you might need to specify more source code as adjustment context
for a statement deletion to achieve a safer semantic patch.

Regards,
Markus

[Cocci] Parsing errors in several headers

[Julia Lawall]

> I wanted to be able to re-run spatch on an already patched content. The
> use case is to add new function prototypes manually and run spatch to
> handle the associated #define automatically.

I think that you could do the following:

@r@
identifier func;
@@
#define        func    BAR(func)

@s@
identifier r.func;
position p;
@@

func at p(...) {...}

@@
identifier func;
position p != s.p;
@@

+#define        func    BAR(func)
func(...) { ... }

julia

[Cocci] Addition of preprocessor statements with SmPL

[Jean-Sébastien Pédron]

On 29.10.2014 11:26, Julia Lawall wrote:
>> I wanted to be able to re-run spatch on an already patched content. The
>> use case is to add new function prototypes manually and run spatch to
>> handle the associated #define automatically.
> 
> I think that you could do the following:
> 
> @r@
> identifier func;
> @@
> #define        func    BAR(func)
> 
> @s@
> identifier r.func;
> position p;
> @@
> 
> func at p(...) {...}
> 
> @@
> identifier func;
> position p != s.p;
> @@
> 
> +#define        func    BAR(func)
> func(...) { ... }

I tried this semantic patch and it doesn't work: more #define are added
with each run.

FWIW, the "BAR" identifier is unique in my use case: I'm sure the
original headers won't have it. That's why I took a simple approach in
removing all previous #define. However, I'm willing to learn the correct
way to handle this for a future usage :)

I have an additional question: If there's a comment just above the
function declaration/implementation, the #define is added above the
comment. Is there a way to put it between the comment and the function?

-- 
Jean-S?bastien P?dron

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: OpenPGP digital signature
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20141029/ccdb8366/attachment-0001.asc>

[Cocci] Addition of preprocessor statements with SmPL

[Julia Lawall]

On Wed, 29 Oct 2014, Jean-S?bastien P?dron wrote:

> On 29.10.2014 11:26, Julia Lawall wrote:
> >> I wanted to be able to re-run spatch on an already patched content. The
> >> use case is to add new function prototypes manually and run spatch to
> >> handle the associated #define automatically.
> >
> > I think that you could do the following:
> >
> > @r@
> > identifier func;
> > @@
> > #define        func    BAR(func)
> >
> > @s@
> > identifier r.func;
> > position p;
> > @@
> >
> > func at p(...) {...}
> >
> > @@
> > identifier func;
> > position p != s.p;
> > @@
> >
> > +#define        func    BAR(func)
> > func(...) { ... }
>
> I tried this semantic patch and it doesn't work: more #define are added
> with each run.

OK, I will see what is wrong.

> FWIW, the "BAR" identifier is unique in my use case: I'm sure the
> original headers won't have it. That's why I took a simple approach in
> removing all previous #define. However, I'm willing to learn the correct
> way to handle this for a future usage :)
>
> I have an additional question: If there's a comment just above the
> function declaration/implementation, the #define is added above the
> comment. Is there a way to put it between the comment and the function?

No.  It assumes that the comment is very attached to the function
definition.

julia

[Cocci] Addition of preprocessor statements with SmPL

[Julia Lawall]

On Wed, 29 Oct 2014, Jean-S?bastien P?dron wrote:

> On 29.10.2014 11:26, Julia Lawall wrote:
> >> I wanted to be able to re-run spatch on an already patched content. The
> >> use case is to add new function prototypes manually and run spatch to
> >> handle the associated #define automatically.
> >
> > I think that you could do the following:
> >
> > @r@
> > identifier func;
> > @@
> > #define        func    BAR(func)
> >
> > @s@
> > identifier r.func;
> > position p;
> > @@
> >
> > func at p(...) {...}
> >
> > @@
> > identifier func;
> > position p != s.p;
> > @@
> >
> > +#define        func    BAR(func)
> > func(...) { ... }

Sorry, typo here.  It should be func at p(...) { ... }

What is more unpleasant is that at least in my version of Coccinelle, it
indents the function definition.

julia

[Cocci] Addition of preprocessor statements with SmPL

[Jean-Sébastien Pédron]

On 29.10.2014 12:07, Julia Lawall wrote:
> On Wed, 29 Oct 2014, Jean-S?bastien P?dron wrote:
> 
>> On 29.10.2014 11:26, Julia Lawall wrote:
>>>> I wanted to be able to re-run spatch on an already patched content. The
>>>> use case is to add new function prototypes manually and run spatch to
>>>> handle the associated #define automatically.
>>>
>>> I think that you could do the following:
>>>
>>> @r@
>>> identifier func;
>>> @@
>>> #define        func    BAR(func)
>>>
>>> @s@
>>> identifier r.func;
>>> position p;
>>> @@
>>>
>>> func at p(...) {...}
>>>
>>> @@
>>> identifier func;
>>> position p != s.p;
>>> @@
>>>
>>> +#define        func    BAR(func)
>>> func(...) { ... }
> 
> Sorry, typo here.  It should be func at p(...) { ... }

You're right, it's working now! Thank you :)

> What is more unpleasant is that at least in my version of Coccinelle, it
> indents the function definition.

Here, with 1.0.0rc19, I don't have this problem.

-- 
Jean-S?bastien P?dron

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: OpenPGP digital signature
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20141029/e4311a64/attachment.asc>

[Cocci] Addition of preprocessor statements with SmPL

[Jean-Sébastien Pédron]

On 29.10.2014 12:25, Jean-S?bastien P?dron wrote:
>> Sorry, typo here.  It should be func at p(...) { ... }
> 
> You're right, it's working now! Thank you :)

Here's the complete working semantic patch:

---8<---
@r@
identifier func;
@@

#define func    BAR(func)

@s1@
identifier r.func;
position p;
@@

func at p(...) {...}

@@
identifier func;
position p != s1.p;
@@

+#define        func    BAR(func)
func at p(...) { ... }

@s2@
type ret;
identifier r.func;
position p;
@@

ret func at p(...);

@@
type ret;
identifier func;
position p != s2.p;
@@

+#define        func    BAR(func)
ret func@p(...);
---8<---

Thank you very much for your help!

-- 
Jean-S?bastien P?dron

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: OpenPGP digital signature
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20141029/05a643e5/attachment.asc>

[Cocci] Parsing errors in several headers

[SF Markus Elfring]

> @@
> identifier func;
> position p != s.p;
> @@
> 
> +#define        func    BAR(func)
> func(...) { ... }

Will the shown position check really work when the variable p is not referenced
in the patch specification block?

Regards,
Markus

[Cocci] Addition of preprocessor statements with SmPL

[SF Markus Elfring]

> However, I fail to assemble those two patches in a single concise one.

How do you think about the following approach?

@Add_before_declaration@
identifier func;
type t;
@@
(
 #define func BAR(func)
 t func(...);
|
+#define func BAR(func)
 t func(...);
)

@Add_before_implementation@
identifier func;
type t;
@@
(
 #define func BAR(func)
 t func(...)
 { ... }
|
+#define func BAR(func)
 t func(...)
 { ... }
)


Do you need to express a detailed adjustment precondition for a safer semantic
patch?

Regards,
Markus

[Cocci] Addition of preprocessor statements with SmPL

[Julia Lawall]

On Wed, 29 Oct 2014, SF Markus Elfring wrote:

> > However, I fail to assemble those two patches in a single concise one.
>
> How do you think about the following approach?
>
> @Add_before_declaration@
> identifier func;
> type t;
> @@
> (
>  #define func BAR(func)
>  t func(...);
> |
> +#define func BAR(func)
>  t func(...);
> )
>
> @Add_before_implementation@
> identifier func;
> type t;
> @@
> (
>  #define func BAR(func)
>  t func(...)
>  { ... }
> |
> +#define func BAR(func)
>  t func(...)
>  { ... }
> )

This doesn't work.  There is no kind of disjunction that includes a
function definition in a branch.

julia

>
>
> Do you need to express a detailed adjustment precondition for a safer semantic
> patch?
>
> Regards,
> Markus
> _______________________________________________
> Cocci mailing list
> Cocci at systeme.lip6.fr
> https://systeme.lip6.fr/mailman/listinfo/cocci
>

[Cocci] Addition of preprocessor statements with SmPL

[SF Markus Elfring]

>> @Add_before_implementation@
>> identifier func;
>> type t;
>> @@
>> (
>>  #define func BAR(func)
>>  t func(...)
>>  { ... }
>> |
>> +#define func BAR(func)
>>  t func(...)
>>  { ... }
>> )
> 
> This doesn't work.

Interesting ...


> There is no kind of disjunction that includes a function definition in a branch.

Was I too optimistic that my suggestion with the specification of SmPL
alternatives should also work?

Regards,
Markus

[Cocci] Addition of preprocessor statements with SmPL

[Julia Lawall]

On Wed, 29 Oct 2014, SF Markus Elfring wrote:

> >> @Add_before_implementation@
> >> identifier func;
> >> type t;
> >> @@
> >> (
> >>  #define func BAR(func)
> >>  t func(...)
> >>  { ... }
> >> |
> >> +#define func BAR(func)
> >>  t func(...)
> >>  { ... }
> >> )
> >
> > This doesn't work.
>
> Interesting ...
>
>
> > There is no kind of disjunction that includes a function definition in a branch.
>
> Was I too optimistic that my suggestion with the specification of SmPL
> alternatives should also work?

You can see with spatch --parse-cocci what works and what does not.

julia