[PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Jiri Slaby]

Until the link-vmlinux.sh split (cf. the commit below), the linker was
run with jobserver set in MAKEFLAGS. After the split, the command in
Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
is lost.

Restore it as linkers working in parallel (esp. the LTO ones) make a use
of it.

Cc: Sedat Dilek <sedat.dilek@gmail.com>
Cc: Masahiro Yamada <masahiroy@kernel.org>
Cc: Michal Marek <michal.lkml@markovi.net>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
Signed-off-by: Jiri Slaby <jslaby@suse.cz>
---
 scripts/Makefile.vmlinux_o | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
index 3c97a1564947..d032f625b576 100644
--- a/scripts/Makefile.vmlinux_o
+++ b/scripts/Makefile.vmlinux_o
@@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
 endef
 
 vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
-	$(call if_changed_rule,ld_vmlinux.o)
+	+$(call if_changed_rule,ld_vmlinux.o)
 
 targets += vmlinux.o
 
-- 
2.36.1

Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Sedat Dilek]

On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <jslaby@suse.cz> wrote:
>
> Until the link-vmlinux.sh split (cf. the commit below), the linker was
> run with jobserver set in MAKEFLAGS. After the split, the command in
> Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> is lost.
>
> Restore it as linkers working in parallel (esp. the LTO ones) make a use
> of it.
>
> Cc: Sedat Dilek <sedat.dilek@gmail.com>
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Cc: Michal Marek <michal.lkml@markovi.net>
> Cc: Nick Desaulniers <ndesaulniers@google.com>
> Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
> Signed-off-by: Jiri Slaby <jslaby@suse.cz>

Nice catch.
( I have seen some slowdown in my build-time. Will test and report. )

Reviewed-by: Sedat Dilek <sedat.dilek@gmail.com>

-Sedat-

> ---
>  scripts/Makefile.vmlinux_o | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
> index 3c97a1564947..d032f625b576 100644
> --- a/scripts/Makefile.vmlinux_o
> +++ b/scripts/Makefile.vmlinux_o
> @@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
>  endef
>
>  vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
> -       $(call if_changed_rule,ld_vmlinux.o)
> +       +$(call if_changed_rule,ld_vmlinux.o)
>
>  targets += vmlinux.o
>
> --
> 2.36.1
>

Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Sedat Dilek]

On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <jslaby@suse.cz> wrote:
> >
> > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > run with jobserver set in MAKEFLAGS. After the split, the command in
> > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > is lost.
> >
> > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > of it.
> >
> > Cc: Sedat Dilek <sedat.dilek@gmail.com>
> > Cc: Masahiro Yamada <masahiroy@kernel.org>
> > Cc: Michal Marek <michal.lkml@markovi.net>
> > Cc: Nick Desaulniers <ndesaulniers@google.com>
> > Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
> > Signed-off-by: Jiri Slaby <jslaby@suse.cz>
>
> Nice catch.
> ( I have seen some slowdown in my build-time. Will test and report. )
>
> Reviewed-by: Sedat Dilek <sedat.dilek@gmail.com>
>

No measurable difference in build-time.

Tested-by: Sedat Dilek <sedat.dilek@gmail.com> # LLVM-14 (x86-64)

-Sedat-

>
> > ---
> >  scripts/Makefile.vmlinux_o | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
> > index 3c97a1564947..d032f625b576 100644
> > --- a/scripts/Makefile.vmlinux_o
> > +++ b/scripts/Makefile.vmlinux_o
> > @@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
> >  endef
> >
> >  vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
> > -       $(call if_changed_rule,ld_vmlinux.o)
> > +       +$(call if_changed_rule,ld_vmlinux.o)
> >
> >  targets += vmlinux.o
> >
> > --
> > 2.36.1
> >

Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Sedat Dilek]

On Fri, Jun 17, 2022 at 12:35 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >
> > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > >
> > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <jslaby@suse.cz> wrote:
> > > >
> > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > > is lost.
> > > >
> > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > > of it.
> > > >
> > > > Cc: Sedat Dilek <sedat.dilek@gmail.com>
> > > > Cc: Masahiro Yamada <masahiroy@kernel.org>
> > > > Cc: Michal Marek <michal.lkml@markovi.net>
> > > > Cc: Nick Desaulniers <ndesaulniers@google.com>
> > > > Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
> > > > Signed-off-by: Jiri Slaby <jslaby@suse.cz>
> > >
> > > Nice catch.
> > > ( I have seen some slowdown in my build-time. Will test and report. )
> > >
> > > Reviewed-by: Sedat Dilek <sedat.dilek@gmail.com>
> > >
> >
> > No measurable difference in build-time.
> >
> > Tested-by: Sedat Dilek <sedat.dilek@gmail.com> # LLVM-14 (x86-64)
> >
>
> [ CC Nathan and Sami ]
>
> I re-checked as my build-time takes approx. 15mins more than usual
> with Linux v5.17.
>
> [ Perf - seconds time elapsed ]
> v5.17: approx. 12500
> v5.19: approx. 13500
>
> +1.000 secs = +16,67mins
>
> I am still investigating.
>
> I use LLVM v14.0.4 and CONFIG_LTO_CLANG_THIN=y.
>
> When I check for...
>
> $ egrep 'Makefile.vmlinux_o|jobserver-exec|link-vmlinux.sh'
> build-log_5.19.0-rc2-2-amd64-clang14-lto.txt
> 61414:  sh scripts/link-vmlinux.sh "ld.lld" "-m elf_x86_64
> --thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5"
> "--emit-relocs --discard-none -z max-page-size=0x200000
> --build-id=sha1  -X --orphan-handling=warn";  true
> 61426:+ make -f ./scripts/Makefile.vmlinux_o
> 61427:  python3 ./scripts/jobserver-exec perl
> scripts/generate_initcall_order.pl arch/x86/kernel/head_64.o
> arch/x86/kernel/head64.o arch/x86/kernel/ebda.o
> arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a
> arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a
> fs/built-in.a ipc/built-in.a security/built-i
> n.a crypto/built-in.a block/built-in.a lib/built-in.a
> arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a
> drivers/built-in.a sound/built-in.a net/built-in.a virt/buil
> t-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a
> arch/x86/video/built-in.a > .tmp_initcalls.lds
> 61693:+ echo vmlinux: scripts/link-vmlinux.sh
>
> Plus vmlinux.o
>
> 61428:  ld.lld -m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm
> -import-instr-limit=5 -r -o vmlinux.o -T .tmp_initcalls.lds
> --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o
> arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o
> init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a
> certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a
> security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a
> arch/x86/lib/built-in.a  lib/lib.a  arch/x86/lib/lib.a
> drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a
> arch/x86/pci/built-in.a arch/x86/power/built-in.a
> arch/x86/video/built-in.a --no-whole-archive --start-group
> --end-group  ; ./tools/objtool/objtool  --hacks=jump_label
> --hacks=noinstr   --mcount  --orc  --retpoline    --static-call
> --uaccess  --link     --
> link vmlinux.o
>
> Pooh, truncated via Gmail? Attached as a file.
>
> Checking via ps...
>
> $ ps -ef | grep l[l]d > lld-jobserver.txt
>
> ...and top util shows me only one lld link-job is running (file attached).
>
> Unsure, if in my setup lld linker benefits from jobserver feature or not.
>
> Am I missing something?
>
> My kernel-config is attached.
>

Found this...

$ ld.lld-14 --help

  --thinlto-jobs=<value>  Number of ThinLTO jobs. Default to --threads=

-Sedat-

Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Masahiro Yamada]

(+LLVM list, Fangrui Song)

On Fri, Jun 17, 2022 at 7:41 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> On Fri, Jun 17, 2022 at 12:35 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >
> > On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > >
> > > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > > >
> > > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <jslaby@suse.cz> wrote:
> > > > >
> > > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > > > is lost.
> > > > >
> > > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > > > of i

Hi Jiri,

Please let me clarify first.

Here, is it OK to assume you are talking about Clang LTO
instead of GCC LTO because the latter is not upstreamed ?





I tested this patch but I did not see any performance change for Clang LTO.


[1] CONFIG_CLANG_LTO_FULL

   lld always runs sequential.
   It never runs in parallel even if you pass -j option to Make


[2] CONFIG_CLANG_LTO_THIN

   lld always runs in parallel even if you do not pass -j option

   In my machine, lld always allocated 12 threads.
   This is irrespective of the Make parallelisms.




One more thing, if a program wants to participate in
Make's jobserver, it must parse MAKEFLAGS, and extract
file descriptors to be used to communicate to the jobserver.

As a code example in the kernel tree,
scripts/jobserver-exec parses "MAKEFLAGS" and "--jobserver".




I grepped the lld source code, but it does not contain
"MAKEFLAGS" or "jobserver".


masahiro@oscar:~/ref/lld$ git remote  show origin
* remote origin
  Fetch URL: https://github.com/llvm-mirror/lld.git
  Push  URL: https://github.com/llvm-mirror/lld.git
  HEAD branch: master
  Remote branches:
    master     tracked
    release_36 tracked
    release_37 tracked
    release_38 tracked
    release_39 tracked
    release_40 tracked
    release_50 tracked
    release_60 tracked
    release_70 tracked
    release_80 tracked
    release_90 tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (up to date)
masahiro@oscar:~/ref/lld$ git grep MAKEFLAGS
masahiro@oscar:~/ref/lld$ git grep jobserver


So, in my research, LLD does not seem to support the jobserver.





If you are talking about GCC LTO, yes, the code
tries to parse "--jobserver-auth=" from the MAKEFLAGS
environment variable.  [1]

[1]:  https://github.com/gcc-mirror/gcc/blob/releases/gcc-12.1.0/gcc/lto-wrapper.cc#L1341


But, as you may know, GCC LTO works in a different way,
at least, we cannot do it before modpost.


-- 
Best Regards
Masahiro Yamada

Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Nick Desaulniers]

On Fri, Jun 17, 2022 at 3:35 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >
> > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > >
> > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <jslaby@suse.cz> wrote:
> > > >
> > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > > is lost.
> > > >
> > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > > of it.

TBH, I agree with Masahiro. I didn't understand this comment "esp. the
LTO ones."  Jiri, can you clarify what you mean here?

> [ CC Nathan and Sami ]
>
> I re-checked as my build-time takes approx. 15mins more than usual
> with Linux v5.17.
>
> [ Perf - seconds time elapsed ]
> v5.17: approx. 12500
> v5.19: approx. 13500
>
> +1.000 secs = +16,67mins

Does this patch make a difference?
https://lore.kernel.org/linux-kbuild/20220616195759.3214538-1-samitolvanen@google.com/

-- 
Thanks,
~Nick Desaulniers

Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Sedat Dilek]

On Fri, Jun 17, 2022 at 6:32 PM Nick Desaulniers
<ndesaulniers@google.com> wrote:
>
> On Fri, Jun 17, 2022 at 3:35 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >
> > On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > >
> > > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > > >
> > > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <jslaby@suse.cz> wrote:
> > > > >
> > > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > > > is lost.
> > > > >
> > > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > > > of it.
>
> TBH, I agree with Masahiro. I didn't understand this comment "esp. the
> LTO ones."  Jiri, can you clarify what you mean here?
>
> > [ CC Nathan and Sami ]
> >
> > I re-checked as my build-time takes approx. 15mins more than usual
> > with Linux v5.17.
> >
> > [ Perf - seconds time elapsed ]
> > v5.17: approx. 12500
> > v5.19: approx. 13500
> >
> > +1.000 secs = +16,67mins
>
> Does this patch make a difference?
> https://lore.kernel.org/linux-kbuild/20220616195759.3214538-1-samitolvanen@google.com/
>

Running a new build...

-sed@-

Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Nathan Chancellor]

On Fri, Jun 17, 2022 at 07:50:58PM +0200, Sedat Dilek wrote:
> On Fri, Jun 17, 2022 at 6:32 PM Nick Desaulniers
> <ndesaulniers@google.com> wrote:
> >
> > On Fri, Jun 17, 2022 at 3:35 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > >
> > > On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > > >
> > > > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > > > >
> > > > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <jslaby@suse.cz> wrote:
> > > > > >
> > > > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > > > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > > > > is lost.
> > > > > >
> > > > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > > > > of it.
> >
> > TBH, I agree with Masahiro. I didn't understand this comment "esp. the
> > LTO ones."  Jiri, can you clarify what you mean here?
> >
> > > [ CC Nathan and Sami ]
> > >
> > > I re-checked as my build-time takes approx. 15mins more than usual
> > > with Linux v5.17.
> > >
> > > [ Perf - seconds time elapsed ]
> > > v5.17: approx. 12500
> > > v5.19: approx. 13500
> > >
> > > +1.000 secs = +16,67mins

So roughly 7.5% increase if I did my math correctly?

> > Does this patch make a difference?
> > https://lore.kernel.org/linux-kbuild/20220616195759.3214538-1-samitolvanen@google.com/
> >
> 
> Running a new build...

I did a quick benchmark with Fedora's clang 14.0.0 and I didn't see an
increase of that proportion with your configuration (only about 1%,
which I would expect would be within the noise for new drivers and
such):

Benchmark 1: 5.17
  Time (abs ≡):        287.744 s               [User: 23497.818 s, System: 4444.602 s]

Benchmark 2: 5.19-rc2
  Time (abs ≡):        290.027 s               [User: 24567.290 s, System: 4834.458 s]

Summary
  '5.17' ran
    1.01 times faster than '5.19-rc2'

Not to say that there is not an issue but it might be more visible with
slower hardware.

Cheers,
Nathan

Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Fangrui Song]

On 2022-06-18, Masahiro Yamada wrote:
>(+LLVM list, Fangrui Song)

Thanks for tagging me. I'll clarify some stuff.

>On Fri, Jun 17, 2022 at 7:41 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>
>> On Fri, Jun 17, 2022 at 12:35 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>> >
>> > On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>> > >
>> > > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>> > > >
>> > > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <jslaby@suse.cz> wrote:
>> > > > >
>> > > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
>> > > > > run with jobserver set in MAKEFLAGS. After the split, the command in
>> > > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
>> > > > > is lost.
>> > > > >
>> > > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
>> > > > > of i
>
>Hi Jiri,
>
>Please let me clarify first.
>
>Here, is it OK to assume you are talking about Clang LTO
>instead of GCC LTO because the latter is not upstreamed ?
>
>
>
>
>
>I tested this patch but I did not see any performance change for Clang LTO.
>
>
>[1] CONFIG_CLANG_LTO_FULL
>
>   lld always runs sequential.
>   It never runs in parallel even if you pass -j option to Make

"lld always runs sequential" is not accurate. There are a number of
parallel linker passes.  ld.lld --threads= defaults to
llvm::hardware_concurrency (similar to
https://en.cppreference.com/w/cpp/thread/thread/hardware_concurrency,
but uses sched_getaffinity to compute the number of available cores).

"lld always runs sequential" is only correct only when --threads=1 is
specified or the system only provides one thread to the lld process.

I think people may be more interested in LTO parallelism here.  Regular
LTO (sometimes called full LTO when there is mixed-thin-and-regular LTO)
supports limited parallelism which applies to code generation, but not
IR-level optimization.  (IR-level optimization has many interprocedural
optimizations passes.  Splitting will make LTO less effective. Code
generation is per function, so parallelism does not regress
optimization.)

>
>[2] CONFIG_CLANG_LTO_THIN
>
>   lld always runs in parallel even if you do not pass -j option
>
>   In my machine, lld always allocated 12 threads.
>   This is irrespective of the Make parallelisms.
>
>
>
>
>One more thing, if a program wants to participate in
>Make's jobserver, it must parse MAKEFLAGS, and extract
>file descriptors to be used to communicate to the jobserver.
>
>As a code example in the kernel tree,
>scripts/jobserver-exec parses "MAKEFLAGS" and "--jobserver".
>
>
>I grepped the lld source code, but it does not contain
>"MAKEFLAGS" or "jobserver".

>masahiro@oscar:~/ref/lld$ git remote  show origin
>* remote origin
>  Fetch URL: https://github.com/llvm-mirror/lld.git
>  Push  URL: https://github.com/llvm-mirror/lld.git
>  HEAD branch: master
>  Remote branches:
>    master     tracked
>    release_36 tracked
>    release_37 tracked
>    release_38 tracked
>    release_39 tracked
>    release_40 tracked
>    release_50 tracked
>    release_60 tracked
>    release_70 tracked
>    release_80 tracked
>    release_90 tracked
>  Local branch configured for 'git pull':
>    master merges with remote master
>  Local ref configured for 'git push':
>    master pushes to master (up to date)
>masahiro@oscar:~/ref/lld$ git grep MAKEFLAGS
>masahiro@oscar:~/ref/lld$ git grep jobserver
>
>
>So, in my research, LLD does not seem to support the jobserver.


Correct. lld does not support GNU make's jobserver.  On the other hand,
I don't think the jobserver implementation supports flexible "give this
target N hardware concurrency". A heavy link target does not necessarily
get more resources than a quick target.

If a make target knows how many hardware concurrency it gets, we can
pass --threads= to lld. LTO easily takes 95+% link time, so LTO
parallelism may needs a dedicated setting. lld has --thinlto-jobs=.

>
>
>
>If you are talking about GCC LTO, yes, the code
>tries to parse "--jobserver-auth=" from the MAKEFLAGS
>environment variable.  [1]
>
>[1]:  https://github.com/gcc-mirror/gcc/blob/releases/gcc-12.1.0/gcc/lto-wrapper.cc#L1341
>
>
>But, as you may know, GCC LTO works in a different way,
>at least, we cannot do it before modpost.
>
>
>-- 
>Best Regards
>Masahiro Yamada
>

Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Sedat Dilek]

On Fri, Jun 17, 2022 at 9:34 PM Nathan Chancellor <nathan@kernel.org> wrote:
>
> On Fri, Jun 17, 2022 at 07:50:58PM +0200, Sedat Dilek wrote:
> > On Fri, Jun 17, 2022 at 6:32 PM Nick Desaulniers
> > <ndesaulniers@google.com> wrote:
> > >
> > > On Fri, Jun 17, 2022 at 3:35 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > > >
> > > > On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > > > >
> > > > > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > > > > >
> > > > > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <jslaby@suse.cz> wrote:
> > > > > > >
> > > > > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > > > > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > > > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > > > > > is lost.
> > > > > > >
> > > > > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > > > > > of it.
> > >
> > > TBH, I agree with Masahiro. I didn't understand this comment "esp. the
> > > LTO ones."  Jiri, can you clarify what you mean here?
> > >
> > > > [ CC Nathan and Sami ]
> > > >
> > > > I re-checked as my build-time takes approx. 15mins more than usual
> > > > with Linux v5.17.
> > > >
> > > > [ Perf - seconds time elapsed ]
> > > > v5.17: approx. 12500
> > > > v5.19: approx. 13500
> > > >
> > > > +1.000 secs = +16,67mins
>
> So roughly 7.5% increase if I did my math correctly?
>
> > > Does this patch make a difference?
> > > https://lore.kernel.org/linux-kbuild/20220616195759.3214538-1-samitolvanen@google.com/
> > >
> >
> > Running a new build...
>
> I did a quick benchmark with Fedora's clang 14.0.0 and I didn't see an
> increase of that proportion with your configuration (only about 1%,
> which I would expect would be within the noise for new drivers and
> such):
>
> Benchmark 1: 5.17
>   Time (abs ≡):        287.744 s               [User: 23497.818 s, System: 4444.602 s]
>
> Benchmark 2: 5.19-rc2
>   Time (abs ≡):        290.027 s               [User: 24567.290 s, System: 4834.458 s]
>
> Summary
>   '5.17' ran
>     1.01 times faster than '5.19-rc2'
>
> Not to say that there is not an issue but it might be more visible with
> slower hardware.
>

@Nick Desaulniers
I applied the patch.
Seen from build-time not really measurable.
Noticed some binary size reduction.

@Nathan Chancellor
Yeah, that I was thinking, too.
This is due to the number of driver (and modules) increased.

And of course the changes to kbuild-system.
It's hard to compare.

Surely, ancient CPU (10 years) plus HDD etc.

-Sedat-

Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Jiri Slaby]

On 17. 06. 22, 18:32, Nick Desaulniers wrote:
> On Fri, Jun 17, 2022 at 3:35 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>
>> On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>
>>> On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>>
>>>> On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <jslaby@suse.cz> wrote:
>>>>>
>>>>> Until the link-vmlinux.sh split (cf. the commit below), the linker was
>>>>> run with jobserver set in MAKEFLAGS. After the split, the command in
>>>>> Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
>>>>> is lost.
>>>>>
>>>>> Restore it as linkers working in parallel (esp. the LTO ones) make a use
>>>>> of it.
> 
> TBH, I agree with Masahiro. I didn't understand this comment "esp. the
> LTO ones."  Jiri, can you clarify what you mean here?

Sure. clang is not the only compiler/linker that can handle kernel ;). 
So while rebasing gcc-LTO-kernel-patches to 5.19-rc (to upstream them 
later), this broke. See:
https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/lto-wrapper.cc;h=26e06e77be4e0afb2bc3e913062a9c51cab5d205;hb=HEAD#l1336

gcc really parses MAKEFLAGS and looks for "--jobserver-auth=" there, if 
one passes specifies -flto=jobserver.

thanks,
-- 
js
suse labs

Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Masahiro Yamada]

On Tue, Jun 21, 2022 at 4:30 PM Jiri Slaby <jslaby@suse.cz> wrote:
>
> On 17. 06. 22, 18:32, Nick Desaulniers wrote:
> > On Fri, Jun 17, 2022 at 3:35 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >>
> >> On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >>>
> >>> On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >>>>
> >>>> On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <jslaby@suse.cz> wrote:
> >>>>>
> >>>>> Until the link-vmlinux.sh split (cf. the commit below), the linker was
> >>>>> run with jobserver set in MAKEFLAGS. After the split, the command in
> >>>>> Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> >>>>> is lost.
> >>>>>
> >>>>> Restore it as linkers working in parallel (esp. the LTO ones) make a use
> >>>>> of it.
> >
> > TBH, I agree with Masahiro. I didn't understand this comment "esp. the
> > LTO ones."  Jiri, can you clarify what you mean here?
>
> Sure. clang is not the only compiler/linker that can handle kernel ;).
> So while rebasing gcc-LTO-kernel-patches to 5.19-rc (to upstream them
> later), this broke. See:
> https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/lto-wrapper.cc;h=26e06e77be4e0afb2bc3e913062a9c51cab5d205;hb=HEAD#l1336
>
> gcc really parses MAKEFLAGS and looks for "--jobserver-auth=" there, if
> one passes specifies -flto=jobserver.
>
> thanks,
> --
> js
> suse labs



Then, this patch should go together (or after) the GCC LTO upstreaming.




I still do not understand why you are patching the pre-modpost linking, though.

In my understanding, Clang LTO and GCC LTO are implemented
in different layers.

Clang LTO is implemented in the linker (i.e. lld).

In contrast,  GCC LTO is implemented in the compiler (i.e. gcc itself)
The linker (i.e. BFD linker or Gold from GNU binutils project)
is agnostic about LTO.


Your are patching the relocatable link ($(LD) -r),
but I could not find the code parsing jobserver-auth
in the binutils-gdb repository.





-- 
Best Regards
Masahiro Yamada

Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Jiri Slaby]

On 21. 06. 22, 10:41, Masahiro Yamada wrote:
> On Tue, Jun 21, 2022 at 4:30 PM Jiri Slaby <jslaby@suse.cz> wrote:
>>
>> On 17. 06. 22, 18:32, Nick Desaulniers wrote:
>>> On Fri, Jun 17, 2022 at 3:35 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>>
>>>> On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>>>
>>>>> On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>>>>
>>>>>> On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <jslaby@suse.cz> wrote:
>>>>>>>
>>>>>>> Until the link-vmlinux.sh split (cf. the commit below), the linker was
>>>>>>> run with jobserver set in MAKEFLAGS. After the split, the command in
>>>>>>> Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
>>>>>>> is lost.
>>>>>>>
>>>>>>> Restore it as linkers working in parallel (esp. the LTO ones) make a use
>>>>>>> of it.
>>>
>>> TBH, I agree with Masahiro. I didn't understand this comment "esp. the
>>> LTO ones."  Jiri, can you clarify what you mean here?
>>
>> Sure. clang is not the only compiler/linker that can handle kernel ;).
>> So while rebasing gcc-LTO-kernel-patches to 5.19-rc (to upstream them
>> later), this broke. See:
>> https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/lto-wrapper.cc;h=26e06e77be4e0afb2bc3e913062a9c51cab5d205;hb=HEAD#l1336
>>
>> gcc really parses MAKEFLAGS and looks for "--jobserver-auth=" there, if
>> one passes specifies -flto=jobserver.
>>
>> thanks,
>> --
>> js
>> suse labs
> 
> 
> 
> Then, this patch should go together (or after) the GCC LTO upstreaming.

Hi,

OK, I will make it part of the gcc-lto series.

> I still do not understand why you are patching the pre-modpost linking, though.
> 
> In my understanding, Clang LTO and GCC LTO are implemented
> in different layers.
> 
> Clang LTO is implemented in the linker (i.e. lld).
> 
> In contrast,  GCC LTO is implemented in the compiler (i.e. gcc itself)
> The linker (i.e. BFD linker or Gold from GNU binutils project)
> is agnostic about LTO.

Right. First, gcc compiles the sources to an intermediate language. Then 
gcc-ld (LTO-)links the objects into a binary object. BFD ld/gold is 
invoked only from within gcc to actually link the binary.

> Your are patching the relocatable link ($(LD) -r),
> but I could not find the code parsing jobserver-auth
> in the binutils-gdb repository.

With gcc-lto patchset, gcc-ld is invoked in cmd_ld_vmlinux.o.

So you are right, making it part of the gcc-lto patchset makes perfect 
sense. First, to see the context. Second, upstream doesn't need the change.

thanks,
-- 
js
suse labs