Parallel compilation performance regression

Parallel compilation performance regression

[Derrick, Jonathan]

Hi David,

I've been experiencing a performance regression when running a parallel
compilation (eg, make -j72) on recent kernels.

I bisected it to this commit:

commit b667b867344301e24f21d4a4c844675ff61d89e1
Author: David Howells <dhowells@redhat.com>
Date:   Tue Sep 24 16:09:04 2019 +0100

    pipe: Advance tail pointer inside of wait spinlock in pipe_read()
    
    Advance the pipe ring tail pointer inside of wait spinlock in pipe_read()
    so that the pipe can be written into with kernel notifications from
    contexts where pipe->mutex cannot be taken.
    

Prior to this commit I got 70% or so thread saturation of cc1 and after
it rarely gets above 15% and would often drop down to 1-2 threads. It
doesn't look like a clean revert either. Looking at upstream, it seems
that some later code changed the wakeup. I'm not really sure how this
all fits into parallelized make.

Best
Jon

Re: Parallel compilation performance regression

[Matthew Wilcox]

On Thu, Jun 18, 2020 at 11:52:55PM +0000, Derrick, Jonathan wrote:
> Hi David,
> 
> I've been experiencing a performance regression when running a parallel
> compilation (eg, make -j72) on recent kernels.

I bet you're using a version of make which predates 4.3:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0ddad21d3e99c743a3aa473121dc5561679e26bb

Re: Parallel compilation performance regression

[Derrick, Jonathan]

On Thu, 2020-06-18 at 18:05 -0700, Matthew Wilcox wrote:
> On Thu, Jun 18, 2020 at 11:52:55PM +0000, Derrick, Jonathan wrote:
> > Hi David,
> > 
> > I've been experiencing a performance regression when running a parallel
> > compilation (eg, make -j72) on recent kernels.
> 
> I bet you're using a version of make which predates 4.3:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0ddad21d3e99c743a3aa473121dc5561679e26bb
> 

I am!

# make --version
GNU Make 4.2.1


Thank you Matthew!

Re: Parallel compilation performance regression

[Sedat Dilek]

On Sat, Jun 20, 2020 at 5:53 AM Derrick, Jonathan
<jonathan.derrick@intel.com> wrote:
>
> On Thu, 2020-06-18 at 18:05 -0700, Matthew Wilcox wrote:
> > On Thu, Jun 18, 2020 at 11:52:55PM +0000, Derrick, Jonathan wrote:
> > > Hi David,
> > >
> > > I've been experiencing a performance regression when running a parallel
> > > compilation (eg, make -j72) on recent kernels.
> >
> > I bet you're using a version of make which predates 4.3:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0ddad21d3e99c743a3aa473121dc5561679e26bb
> >
>
> I am!
>
> # make --version
> GNU Make 4.2.1
>
>
> Thank you Matthew!

Check your distribution and included patches on top of a vanilla make v4.2.
Debian had some important ones I described in [1].

When proposing "make version 4.3" for tc-build - an opportunity to
build a llvm-toolchain the easy way - we saw different numbers.
I pointed in [1] to the Linus patch Matthew did here.

Personally, with switching to Debian's make version 4.3-3 I have seen
no big differences when using "make -j3" to build Linux v5.7+.
That might be different with "make -j72"...
...can I have SSH access to this machine, please :-),

You forgot to tell which Linux version you use.

If you are interested please look at closed tc-build issue #72 for our analysis.
For tc-build it did not matter - (Debian's) make v4.2 had some
slightly better performance.

BTW, with that pipe improvements in Linux v5.7 I see some better
numbers when using pipebench to benchmark my devices:

Example: SanDisk iSSD 16GB

root# cat /dev/sdb | pipebench > /dev/null
Summary:
Piped   14.91 GB in 00h01m25.20s:  179.23 MB/second

Before: approx. 100MB/s

BTW, I heard of hyperfine benchmark tool the first time when dealing
with make performance (see [2]) in ClangBuiltLinux.
I would like to see some benchmark numbers with hyperfine from you if
you do not mind.

Stay OPEN minded and curious.

Thanks.

- Sedat -

[1] https://github.com/ClangBuiltLinux/tc-build/issues/72
[2] https://github.com/sharkdp/hyperfine
[3] https://github.com/sharkdp/hyperfine/releases