Re: [PATCH] coccicheck: optionally process every source file at once

[Duy Nguyen <pclouds@gmail.com>]

On Mon, May 6, 2019 at 12:13 PM Jeff King <peff@peff.net> wrote:
> This reduces the time required to run make coccicheck by a significant
> amount of time:
>
> Prior timing of make coccicheck
>   real    6m14.090s
>   user    25m2.606s
>   sys     1m22.919s
>
> New timing of make coccicheck
>   real    1m36.580s
>   user    7m55.933s
>   sys     0m18.219s
>
> This is nearly a 4x decrease in the time required to run make

Whoa.

> coccicheck. This is due to the overhead of restarting spatch for every
> file. By processing all files at once, we can amortize this startup cost
> across the total number of files, rather than paying it once per file.
>
> However, it comes at a cost. The RSS of each spatch process goes from
> ~50MB to ~1500MB (and peak memory usage may be even higher if make runs

1.5G should be fine. Trying...

Even with no -j, my htop's RES column goes up 6GB and put my laptop in
"swap every bit of memory out, including the bits handling the screen"
mode :( I don't think it was even the peak.

It's probably a bit too much to ask, but is it possible to handle N
files at a time (instead of all files), which consumes less memory and
runs a bit slower, but still better than the default mode? I can see
it already gets tricky doing complicated stuff in Makefile so "no" is
perfectly ok.

> multiple rule files in parallel due to "-j"). That's enough to make some
> systems (like our Travis build!) fail the whole process, or could make
> things slower due to swapping. So let's make the new behavior optional,
> and people with a lot of memory can choose to use it.
>
> [peff: modified Jacob's patch to make the behavior optional, since
>  people reported complications due to the memory use]
>
> Signed-off-by: Jacob Keller <jacob.keller@gmail.com>
> Signed-off-by: Jeff King <peff@peff.net>
-- 
Duy