[cocci] Optimizing *.cocci rules by concat'ing them

[cocci] Optimizing *.cocci rules by concat'ing them

[Ævar Arnfjörð Bjarmason]

Are there reasons for why $subject is a generally bad idea, i.e. given a
set of *.cocci rules in a directory instead of something like:

	for c in cocci/*.cocci
	do
		spatch --sp-file $c --patch . *.c
	done

Doing:

	cat cocci/*.cocci >ALL.cocci
	spatch --sp-file ALL.cocci --patch . *.c

I'm aware that rules in general may rely on previous transformations,
such that the 3rd rule in a file might not fire unless the other 2 have
previously changed the code.

But aside from any such undeclared dependencies, which surely I'll have
trouble with either way. I.e. I'd get to the 2nd rule in A.cocci, apply
that to my sources, and only then on a second run would the 1st rule in
B.cocci fire, but it should have been in A.cocci to begin with, and
depending on the 2nd rule in that file.

Are there reasons for why this is a Bad Idea?

The motivation here (and this is probably burying the lede) is that I
implemented a "spatchcache", which can cache previous runs of "spatch"
in a local Redis (or with a custom command, if you don't have Redis, I
started out with a dumb FS-based cache).

It uses the approach of piggy-backing on GCC and Clang's ability to dump
a full list of all the files that a given *.c file depends on.

This is usually used to drive better incremental "make" support, but I'm
(ab)using it to implement an "spatchcache". If the hash of a the target
file *and* everything it depends on *and* spatch's path, flags etc. is
the same I can cache the output of "spatch".

See [1] for a series to implement this for git.git. I'd previously mused
on this list in [2] about this approach (but hadn't finished it up until
now).

But even then git's fairly slow in running "make coccicheck", and of
course it does nothing to speed up an initial run. So concat'ing the
*.cocci files we use seems like a viable option. But I thought I'd ask
first in case I'm running into somewell-known caveat.

Thanks!

1. https://lore.kernel.org/git/cover-v2-0.9-00000000000-20220831T205130Z-avarab@gmail.com/
2. https://lore.kernel.org/cocci/211116.86lf1op65q.gmgdl@evledraar.gmail.com/

Re: [cocci] Optimizing *.cocci rules by concat'ing them

[Julia Lawall]

[-- Attachment #1: Type: text/plain, Size: 2764 bytes --]



On Thu, 1 Sep 2022, Ævar Arnfjörð Bjarmason wrote:

> Are there reasons for why $subject is a generally bad idea, i.e. given a
> set of *.cocci rules in a directory instead of something like:
>
> 	for c in cocci/*.cocci
> 	do
> 		spatch --sp-file $c --patch . *.c
> 	done
>
> Doing:
>
> 	cat cocci/*.cocci >ALL.cocci
> 	spatch --sp-file ALL.cocci --patch . *.c
>
> I'm aware that rules in general may rely on previous transformations,
> such that the 3rd rule in a file might not fire unless the other 2 have
> previously changed the code.

I think that the main problem is the opposite of what you suggest.  Two
.cocci files might define rules that have the same names.

I don't see any other problems offhand.  If one rule makes a change that
somehow affects a later semantic patch, then that should be ok, assuming
that the change made by the first rule was actually a fix.

Another issue is that you can't mix rules that make changes with rules
that just search (*).  You need at least some command line options, like
-D patch in make coccicheck, to keep them apart.

julia

>
> But aside from any such undeclared dependencies, which surely I'll have
> trouble with either way. I.e. I'd get to the 2nd rule in A.cocci, apply
> that to my sources, and only then on a second run would the 1st rule in
> B.cocci fire, but it should have been in A.cocci to begin with, and
> depending on the 2nd rule in that file.
>
> Are there reasons for why this is a Bad Idea?
>
> The motivation here (and this is probably burying the lede) is that I
> implemented a "spatchcache", which can cache previous runs of "spatch"
> in a local Redis (or with a custom command, if you don't have Redis, I
> started out with a dumb FS-based cache).
>
> It uses the approach of piggy-backing on GCC and Clang's ability to dump
> a full list of all the files that a given *.c file depends on.
>
> This is usually used to drive better incremental "make" support, but I'm
> (ab)using it to implement an "spatchcache". If the hash of a the target
> file *and* everything it depends on *and* spatch's path, flags etc. is
> the same I can cache the output of "spatch".
>
> See [1] for a series to implement this for git.git. I'd previously mused
> on this list in [2] about this approach (but hadn't finished it up until
> now).
>
> But even then git's fairly slow in running "make coccicheck", and of
> course it does nothing to speed up an initial run. So concat'ing the
> *.cocci files we use seems like a viable option. But I thought I'd ask
> first in case I'm running into somewell-known caveat.
>
> Thanks!
>
> 1. https://lore.kernel.org/git/cover-v2-0.9-00000000000-20220831T205130Z-avarab@gmail.com/
> 2. https://lore.kernel.org/cocci/211116.86lf1op65q.gmgdl@evledraar.gmail.com/
>

Re: [cocci] Checking consequences from concatenation of SmPL rules

[Markus Elfring]

> Are there reasons for why this is a Bad Idea?


* Do you care for the uniqueness of SmPL rules?

* Special identifiers: initialize:python, finalize:python

* I recommend to handle operation modes for SmPL scripts in consistent ways.

* Would you like to reconsider risks for undesirable change mixtures?

* How do you think about previous feature requests?

  Example:
  2019-04-07
  Support for SmPL rule groups
  https://github.com/coccinelle/coccinelle/issues/164


Regards,
Markus