clang thin-lto not working for aarch64 for v6.13

clang thin-lto not working for aarch64 for v6.13

[Yonghong Song]

Hi, Masahiro,

We are trying 6.13 kernel and found that for aarch64 thinlto not 
working. For example, for kernel/bpf/syscall.o, the compilation flags 
from .syscall.o.cmd are savedcmd_kernel/bpf/syscall.o := clang 
-Wp,-MMD,kernel/bpf/.syscall.o.d ... -D__KBUILD_MODNAME=kmod_syscall -c 
-o kernel/bpf/syscall.o kernel/bpf/syscall.c ; ld.lld -EL -maarch64elf 
-z norelro -mllvm -import-instr-limit=5 -z noexecstack -r -o 
kernel/bpf/.tmp_syscall.o kernel/bpf/syscall.o; mv 
kernel/bpf/.tmp_syscall.o kernel/bpf/syscall.o I did some bisecting and 
found the issue is due to ``` commit 
bede169618c68379e1be7ace14e8ac85b964a9ec Author: Masahiro Yamada 
<masahiroy@kernel.org> Date: Thu Nov 14 08:45:22 2024 +0900 kbuild: 
enable objtool for *.mod.o and additional kernel objects ``` In the 
above, for thinlto, we should not do ld.lld as compiler needs IR to do 
cross-file inlining. Searching the internet, I found that the issue has 
been reported e.g. in 
https://patchwork.kernel.org/project/linux-kbuild/patch/20241113234526.402738-3-masahiroy@kernel.org/ 
and you mentioned you will fix it. Do you have a fix somewhere? With 
this fix, deploying 6.13 in our production will cause performance 
regression and that is not what we want. Thanks! Yonghong

Re: clang thin-lto not working for aarch64 for v6.13

[Song Liu]

On Sun, Jan 26, 2025 at 7:48 PM Yonghong Song <yonghong.song@linux.dev> wrote:
>
> Hi, Masahiro,
>
> We are trying 6.13 kernel and found that for aarch64 thinlto not
> working. For example, for kernel/bpf/syscall.o, the compilation flags
> from .syscall.o.cmd are savedcmd_kernel/bpf/syscall.o := clang
> -Wp,-MMD,kernel/bpf/.syscall.o.d ... -D__KBUILD_MODNAME=kmod_syscall -c
> -o kernel/bpf/syscall.o kernel/bpf/syscall.c ; ld.lld -EL -maarch64elf
> -z norelro -mllvm -import-instr-limit=5 -z noexecstack -r -o
> kernel/bpf/.tmp_syscall.o kernel/bpf/syscall.o; mv
> kernel/bpf/.tmp_syscall.o kernel/bpf/syscall.o I did some bisecting and
> found the issue is due to ``` commit
> bede169618c68379e1be7ace14e8ac85b964a9ec Author: Masahiro Yamada
> <masahiroy@kernel.org> Date: Thu Nov 14 08:45:22 2024 +0900 kbuild:
> enable objtool for *.mod.o and additional kernel objects ``` In the
> above, for thinlto, we should not do ld.lld as compiler needs IR to do
> cross-file inlining. Searching the internet, I found that the issue has
> been reported e.g. in
> https://patchwork.kernel.org/project/linux-kbuild/patch/20241113234526.402738-3-masahiroy@kernel.org/

It appears the fix suggested by Nathan is already squashed with the
commit before being merged upstream. However, this causes another
issue. As Yonghong stated, after upstream commit
bede169618c68379e1be7ace14e8ac85b964a9ec, the linker runs on
individual .o file, which defeats the benefit of LTO.

IIUC,  the proper behavior is to do "cmd_ld_single" only for "is-single-obj-m"
case. However, after bede169618c68379e1be7ace14e8ac85b964a9ec,
the "is-single-obj-m" check is removed. I am not quite sure what is the
proper fix for this.

Masahiro,

Could you please help us with this?

Thanks,
Song

> and you mentioned you will fix it. Do you have a fix somewhere? With
> this fix, deploying 6.13 in our production will cause performance
> regression and that is not what we want. Thanks! Yonghong

Re: clang thin-lto not working for aarch64 for v6.13

[Song Liu]

Hi Masahiro and Nathan,

On Mon, Jan 27, 2025 at 9:50 AM Song Liu <song@kernel.org> wrote:
>
> On Sun, Jan 26, 2025 at 7:48 PM Yonghong Song <yonghong.song@linux.dev> wrote:
> >
> > Hi, Masahiro,
> >
> > We are trying 6.13 kernel and found that for aarch64 thinlto not
> > working. For example, for kernel/bpf/syscall.o, the compilation flags
> > from .syscall.o.cmd are savedcmd_kernel/bpf/syscall.o := clang
> > -Wp,-MMD,kernel/bpf/.syscall.o.d ... -D__KBUILD_MODNAME=kmod_syscall -c
> > -o kernel/bpf/syscall.o kernel/bpf/syscall.c ; ld.lld -EL -maarch64elf
> > -z norelro -mllvm -import-instr-limit=5 -z noexecstack -r -o
> > kernel/bpf/.tmp_syscall.o kernel/bpf/syscall.o; mv
> > kernel/bpf/.tmp_syscall.o kernel/bpf/syscall.o I did some bisecting and
> > found the issue is due to ``` commit
> > bede169618c68379e1be7ace14e8ac85b964a9ec Author: Masahiro Yamada
> > <masahiroy@kernel.org> Date: Thu Nov 14 08:45:22 2024 +0900 kbuild:
> > enable objtool for *.mod.o and additional kernel objects ``` In the
> > above, for thinlto, we should not do ld.lld as compiler needs IR to do
> > cross-file inlining. Searching the internet, I found that the issue has
> > been reported e.g. in
> > https://patchwork.kernel.org/project/linux-kbuild/patch/20241113234526.402738-3-masahiroy@kernel.org/
>
> It appears the fix suggested by Nathan is already squashed with the
> commit before being merged upstream. However, this causes another
> issue. As Yonghong stated, after upstream commit
> bede169618c68379e1be7ace14e8ac85b964a9ec, the linker runs on
> individual .o file, which defeats the benefit of LTO.

It appears we still have this issue in the latest upstream kernel. Reverting
bede169618c68379e1be7ace14e8ac85b964a9ec fixes the issue. But I
am not sure whether we can do that without also reverting
1b466b29a3bf02ed95f28682a975f41ae47bce7d.

Could you please share your suggestions on this?

Thanks,
Song

> IIUC,  the proper behavior is to do "cmd_ld_single" only for "is-single-obj-m"
> case. However, after bede169618c68379e1be7ace14e8ac85b964a9ec,
> the "is-single-obj-m" check is removed. I am not quite sure what is the
> proper fix for this.

[...]

Re: clang thin-lto not working for aarch64 for v6.13

[Nathan Chancellor]

Hi Song,

On Wed, Jan 29, 2025 at 03:04:56PM -0800, Song Liu wrote:
> On Mon, Jan 27, 2025 at 9:50 AM Song Liu <song@kernel.org> wrote:
> >
> > On Sun, Jan 26, 2025 at 7:48 PM Yonghong Song <yonghong.song@linux.dev> wrote:
> > >
> > > Hi, Masahiro,
> > >
> > > We are trying 6.13 kernel and found that for aarch64 thinlto not
> > > working. For example, for kernel/bpf/syscall.o, the compilation flags
> > > from .syscall.o.cmd are savedcmd_kernel/bpf/syscall.o := clang
> > > -Wp,-MMD,kernel/bpf/.syscall.o.d ... -D__KBUILD_MODNAME=kmod_syscall -c
> > > -o kernel/bpf/syscall.o kernel/bpf/syscall.c ; ld.lld -EL -maarch64elf
> > > -z norelro -mllvm -import-instr-limit=5 -z noexecstack -r -o
> > > kernel/bpf/.tmp_syscall.o kernel/bpf/syscall.o; mv
> > > kernel/bpf/.tmp_syscall.o kernel/bpf/syscall.o I did some bisecting and
> > > found the issue is due to ``` commit
> > > bede169618c68379e1be7ace14e8ac85b964a9ec Author: Masahiro Yamada
> > > <masahiroy@kernel.org> Date: Thu Nov 14 08:45:22 2024 +0900 kbuild:
> > > enable objtool for *.mod.o and additional kernel objects ``` In the
> > > above, for thinlto, we should not do ld.lld as compiler needs IR to do
> > > cross-file inlining. Searching the internet, I found that the issue has
> > > been reported e.g. in
> > > https://patchwork.kernel.org/project/linux-kbuild/patch/20241113234526.402738-3-masahiroy@kernel.org/
> >
> > It appears the fix suggested by Nathan is already squashed with the
> > commit before being merged upstream. However, this causes another
> > issue. As Yonghong stated, after upstream commit
> > bede169618c68379e1be7ace14e8ac85b964a9ec, the linker runs on
> > individual .o file, which defeats the benefit of LTO.
> 
> It appears we still have this issue in the latest upstream kernel. Reverting
> bede169618c68379e1be7ace14e8ac85b964a9ec fixes the issue. But I
> am not sure whether we can do that without also reverting
> 1b466b29a3bf02ed95f28682a975f41ae47bce7d.
> 
> Could you please share your suggestions on this?

I apologize for the radio silence on my end. I agree with you on the
root cause but I am not really sure what the solution here is either
(other than an outright revert) since I did not write the change so I
defer to Masahiro. Hopefully he can answer soon.

> > IIUC,  the proper behavior is to do "cmd_ld_single" only for "is-single-obj-m"
> > case. However, after bede169618c68379e1be7ace14e8ac85b964a9ec,
> > the "is-single-obj-m" check is removed. I am not quite sure what is the
> > proper fix for this.
> 
> [...]

Cheers,
Nathan