[cocci] using gcc & clang -MF to reduce spatch work (was: Using `parallel` [...])

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Sat, Mar 19 2022, Eric Wheeler wrote:

> Hi All,
>
> Just a quick tip for others (like me) who are new to Coccinelle:
>
> You might already use this, but if not, here's a hint for doing lots of 
> replacements in parallel:
>
> 	parallel -j24 spatch --sp-file smpl.cocci {} --in-place ::: *.c
>
> If the SmPL depends on interactions between files then `parallel` won't 
> work, but if the changes work per-file then it runs much faster with lots 
> of big .c files.
>
> In cases where you do need to run over lots of files and they _do_ 
> interact, you might get a Stack Overflow error.  In this case set 
> something like this for 1GB of stack space:
> 	ulimit -s $((1024*1024))
>
> In my case spatch needed 1GB of stack and 2.4GB of RAM.  It took a few 
> minutes and finished thousands of replacements!
>
> Julia, you might add this to documentation if you think it would be 
> useful.

To change the $subject to an alternate approach that may work for you,
and that I've been experimenting with.

If you use GCC or Clang to compile your C code you can use the -MF
option to emit a dependency tree for your *.c files, i.e. your compiler
is already doing all the work that spatch needs to re-do to find
includes.

But more importantly, if you have 100 files and change a *.h only used
by 10 of them, you'll only need to invoke spatch on those 10 files.

I had a question related to this here on-list in
<211116.86h7ccox6b.gmgdl@evledraar.gmail.com>, i.e. you can use
--no-includes and instead feed the list of files to be included to
spatch. If those files come from your dependencies made with -MF you
migth be able to assume they're correct.

Well, not if your *.o file is newer than your *.c, as it might have
grown new includes, but in this sub-example you'd make your *.c.cocci-ok
(or whatever) generation depend on the corresponding *.o, or would
otherwise compile first (which is much, much faster than spatch,
especially with ccache).

I don't have any stand-alone example for use, sorry, but basically in
make syntax something like:

	# get dependencies from our last compilation
	-inlude $(wildcard *.mak)
	# next compilation
	%.o: %.c
		$(CC) ... -MF $@.mak -MT $@ -MT $@.cocci-ok $<
	%.o.cocci-ok: %.o
		spatch ...

I just typed that up now, and there's sure to be syntax errors etc, but
I think you get the idea, i.e. you'll save yourself spatch work by
bootstrapping from your C compilation.

Also note that you don't need to run GNU make, if you build with
something else you can (with a tiny bit of munging) change the make
syntax it emits to just a list of files to depend on (the easiest way to
parse it being to invoke a make one-liner to spew it out for you).

I think it would be really useful if spatch could spew out the list of
headers it understood from a given file.

Depending on your compilation (e.g. ifdefs) the two may not 1=1 map, but
you could also do this CC trick in reverse, i.e. if you ran spatch first
you could feed the resulting dependency graph from spatch to your C
compiler.

In pracice I haven't found reason to worry about that lack of 1=1
mapping.