Re: Call graph dwarf unwinding fails with lld

Re: Call graph dwarf unwinding fails with lld

[Fangrui Song]

On 2022-05-25, Ian Rogers wrote:
>On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote:
>>
>> I have been tracking down an issue locally where `perf record
>> --call-graph=dwarf` stopped working in the sense that unwinding always
>> failed.
>>
>> Turns out it was related to `lld` - binaries linked with `lld` don't
>> seem to unwind correctly with `--call-graph=dwarf`. I've confirmed
>> that the `.eh_frame` sections are there, but I guess they are
>> different after using this linker.
>>
>> Has anyone run into this?
>
>I was unaware but thanks for filing the bug!
>
>> There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156
>>
>> ... but no activity since it was filed, though it is not clear if the
>> problem is really with lld or the unwinders.
>>
>> In a related question: perf supports more than one DWARF unwinding
>> implementation, depending on what libraries are available at build
>> time, right? Is there a way to select which unwinder is used at
>> runtime?
>
>I don't think so. There are two build options NO_LIBUNWIND=1 and
>NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the
>bug, confirmed no difference with the eh_frames and hand verified
>them, I also added -fno-omit-frame-pointer -O0 and -static. I've not
>got a good handle on the issue but my suspicion is that the
>eh_frame_hdr is broken - it is at least something that lld generates.
>If I change the eh_frame_hdr finding to some findable but unexpected
>offset like eh_frame here:
>https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286
>and then test with a working linker the unwinding is broken similarly
>to with lld. I was looking for a useful tool (objdump, llvm-objdump,
>readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one.
>Perhaps someone has a suggestion or maybe pahole can do it. Hopefully
>this gives a lead for you to dig into for the bug.
>
>Thanks,
>Ian

I am very familiar with lld and have read some nongnu libunwind code, so I guess I
may be an appropriate one debugging the issue, but I am puzzled about how to build tools/perf:

/tmp/out/custom1 has needed llvm-project tools, built from origin/main
ninja -C /tmp/out/custom1 clang lld llvm-{nm,size,strings,objcopy,objdump,readelf,ar,strip}

```
% git checkout master  # torvalds/linux
% PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 defconfig all
% PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 -C tools/perf
...
In file included from bench/../../arch/x86/lib/memcpy_64.S:8,
                  from bench/mem-memcpy-x86-64-asm.S:14:
/tmp/linux/x86_64/arch/x86/include/generated/asm/export.h:1:10: fatal error: asm-generic/export.h: No such file or directory
     1 | #include <asm-generic/export.h>
       |          ^~~~~~~~~~~~~~~~~~~~~~
```

I am running a 5.16 kernel so I have tried `git checkout v5.16`, no luck as well.
I cannot find useful information about "asm-generic/export.h: no such file or directory" on www:(

Re: Call graph dwarf unwinding fails with lld

[Ian Rogers]

On Wed, May 25, 2022 at 9:16 PM Fangrui Song <maskray@google.com> wrote:
>
> On 2022-05-25, Ian Rogers wrote:
> >On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote:
> >>
> >> I have been tracking down an issue locally where `perf record
> >> --call-graph=dwarf` stopped working in the sense that unwinding always
> >> failed.
> >>
> >> Turns out it was related to `lld` - binaries linked with `lld` don't
> >> seem to unwind correctly with `--call-graph=dwarf`. I've confirmed
> >> that the `.eh_frame` sections are there, but I guess they are
> >> different after using this linker.
> >>
> >> Has anyone run into this?
> >
> >I was unaware but thanks for filing the bug!
> >
> >> There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156
> >>
> >> ... but no activity since it was filed, though it is not clear if the
> >> problem is really with lld or the unwinders.
> >>
> >> In a related question: perf supports more than one DWARF unwinding
> >> implementation, depending on what libraries are available at build
> >> time, right? Is there a way to select which unwinder is used at
> >> runtime?
> >
> >I don't think so. There are two build options NO_LIBUNWIND=1 and
> >NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the
> >bug, confirmed no difference with the eh_frames and hand verified
> >them, I also added -fno-omit-frame-pointer -O0 and -static. I've not
> >got a good handle on the issue but my suspicion is that the
> >eh_frame_hdr is broken - it is at least something that lld generates.
> >If I change the eh_frame_hdr finding to some findable but unexpected
> >offset like eh_frame here:
> >https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286
> >and then test with a working linker the unwinding is broken similarly
> >to with lld. I was looking for a useful tool (objdump, llvm-objdump,
> >readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one.
> >Perhaps someone has a suggestion or maybe pahole can do it. Hopefully
> >this gives a lead for you to dig into for the bug.
> >
> >Thanks,
> >Ian
>
> I am very familiar with lld and have read some nongnu libunwind code, so I guess I
> may be an appropriate one debugging the issue, but I am puzzled about how to build tools/perf:
>
> /tmp/out/custom1 has needed llvm-project tools, built from origin/main
> ninja -C /tmp/out/custom1 clang lld llvm-{nm,size,strings,objcopy,objdump,readelf,ar,strip}
>
> ```
> % git checkout master  # torvalds/linux
> % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 defconfig all
> % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 -C tools/perf
> ...
> In file included from bench/../../arch/x86/lib/memcpy_64.S:8,
>                   from bench/mem-memcpy-x86-64-asm.S:14:
> /tmp/linux/x86_64/arch/x86/include/generated/asm/export.h:1:10: fatal error: asm-generic/export.h: No such file or directory
>      1 | #include <asm-generic/export.h>
>        |          ^~~~~~~~~~~~~~~~~~~~~~
> ```
>
> I am running a 5.16 kernel so I have tried `git checkout v5.16`, no luck as well.
> I cannot find useful information about "asm-generic/export.h: no such file or directory" on www:(

Arnaldo, can the tools/include headers be made hermetic?

You may have better luck with the development tree:
https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/
branch perf/core. The perf tool is backward compatible with old
kernels, unfortunately Debian packages it wrong which causes issues
everywhere.

Thanks,
Ian

Re: Call graph dwarf unwinding fails with lld

[Arnaldo Carvalho de Melo]

Em Wed, May 25, 2022 at 09:25:39PM -0700, Ian Rogers escreveu:
> On Wed, May 25, 2022 at 9:16 PM Fangrui Song <maskray@google.com> wrote:
> >
> > On 2022-05-25, Ian Rogers wrote:
> > >On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote:
> > >>
> > >> I have been tracking down an issue locally where `perf record
> > >> --call-graph=dwarf` stopped working in the sense that unwinding always
> > >> failed.
> > >>
> > >> Turns out it was related to `lld` - binaries linked with `lld` don't
> > >> seem to unwind correctly with `--call-graph=dwarf`. I've confirmed
> > >> that the `.eh_frame` sections are there, but I guess they are
> > >> different after using this linker.
> > >>
> > >> Has anyone run into this?
> > >
> > >I was unaware but thanks for filing the bug!
> > >
> > >> There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156
> > >>
> > >> ... but no activity since it was filed, though it is not clear if the
> > >> problem is really with lld or the unwinders.
> > >>
> > >> In a related question: perf supports more than one DWARF unwinding
> > >> implementation, depending on what libraries are available at build
> > >> time, right? Is there a way to select which unwinder is used at
> > >> runtime?
> > >
> > >I don't think so. There are two build options NO_LIBUNWIND=1 and
> > >NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the
> > >bug, confirmed no difference with the eh_frames and hand verified
> > >them, I also added -fno-omit-frame-pointer -O0 and -static. I've not
> > >got a good handle on the issue but my suspicion is that the
> > >eh_frame_hdr is broken - it is at least something that lld generates.
> > >If I change the eh_frame_hdr finding to some findable but unexpected
> > >offset like eh_frame here:
> > >https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286
> > >and then test with a working linker the unwinding is broken similarly
> > >to with lld. I was looking for a useful tool (objdump, llvm-objdump,
> > >readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one.
> > >Perhaps someone has a suggestion or maybe pahole can do it. Hopefully
> > >this gives a lead for you to dig into for the bug.
> > >
> > >Thanks,
> > >Ian
> >
> > I am very familiar with lld and have read some nongnu libunwind code, so I guess I
> > may be an appropriate one debugging the issue, but I am puzzled about how to build tools/perf:
> >
> > /tmp/out/custom1 has needed llvm-project tools, built from origin/main
> > ninja -C /tmp/out/custom1 clang lld llvm-{nm,size,strings,objcopy,objdump,readelf,ar,strip}
> >
> > ```
> > % git checkout master  # torvalds/linux
> > % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 defconfig all
> > % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 -C tools/perf

I never tried build both Linux and perf on the same O= directory, can
you please building perf on a separate directory to see if this changes
anything?

Also you don't need to build Linux proper to test these problems, right?

- Arnaldo

> > ...
> > In file included from bench/../../arch/x86/lib/memcpy_64.S:8,
> >                   from bench/mem-memcpy-x86-64-asm.S:14:
> > /tmp/linux/x86_64/arch/x86/include/generated/asm/export.h:1:10: fatal error: asm-generic/export.h: No such file or directory
> >      1 | #include <asm-generic/export.h>
> >        |          ^~~~~~~~~~~~~~~~~~~~~~
> > ```
> >
> > I am running a 5.16 kernel so I have tried `git checkout v5.16`, no luck as well.
> > I cannot find useful information about "asm-generic/export.h: no such file or directory" on www:(
> 
> Arnaldo, can the tools/include headers be made hermetic?
> 
> You may have better luck with the development tree:
> https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/
> branch perf/core. The perf tool is backward compatible with old
> kernels, unfortunately Debian packages it wrong which causes issues
> everywhere.
> 
> Thanks,
> Ian

-- 

- Arnaldo

Re: Call graph dwarf unwinding fails with lld

[Fāng-ruì Sòng]

On Thu, May 26, 2022 at 7:39 AM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> Em Wed, May 25, 2022 at 09:25:39PM -0700, Ian Rogers escreveu:
> > On Wed, May 25, 2022 at 9:16 PM Fangrui Song <maskray@google.com> wrote:
> > >
> > > On 2022-05-25, Ian Rogers wrote:
> > > >On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote:
> > > >>
> > > >> I have been tracking down an issue locally where `perf record
> > > >> --call-graph=dwarf` stopped working in the sense that unwinding always
> > > >> failed.
> > > >>
> > > >> Turns out it was related to `lld` - binaries linked with `lld` don't
> > > >> seem to unwind correctly with `--call-graph=dwarf`. I've confirmed
> > > >> that the `.eh_frame` sections are there, but I guess they are
> > > >> different after using this linker.
> > > >>
> > > >> Has anyone run into this?
> > > >
> > > >I was unaware but thanks for filing the bug!
> > > >
> > > >> There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156
> > > >>
> > > >> ... but no activity since it was filed, though it is not clear if the
> > > >> problem is really with lld or the unwinders.
> > > >>
> > > >> In a related question: perf supports more than one DWARF unwinding
> > > >> implementation, depending on what libraries are available at build
> > > >> time, right? Is there a way to select which unwinder is used at
> > > >> runtime?
> > > >
> > > >I don't think so. There are two build options NO_LIBUNWIND=1 and
> > > >NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the
> > > >bug, confirmed no difference with the eh_frames and hand verified
> > > >them, I also added -fno-omit-frame-pointer -O0 and -static. I've not
> > > >got a good handle on the issue but my suspicion is that the
> > > >eh_frame_hdr is broken - it is at least something that lld generates.
> > > >If I change the eh_frame_hdr finding to some findable but unexpected
> > > >offset like eh_frame here:
> > > >https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286
> > > >and then test with a working linker the unwinding is broken similarly
> > > >to with lld. I was looking for a useful tool (objdump, llvm-objdump,
> > > >readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one.
> > > >Perhaps someone has a suggestion or maybe pahole can do it. Hopefully
> > > >this gives a lead for you to dig into for the bug.
> > > >
> > > >Thanks,
> > > >Ian
> > >
> > > I am very familiar with lld and have read some nongnu libunwind code, so I guess I
> > > may be an appropriate one debugging the issue, but I am puzzled about how to build tools/perf:
> > >
> > > /tmp/out/custom1 has needed llvm-project tools, built from origin/main
> > > ninja -C /tmp/out/custom1 clang lld llvm-{nm,size,strings,objcopy,objdump,readelf,ar,strip}
> > >
> > > ```
> > > % git checkout master  # torvalds/linux
> > > % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 defconfig all
> > > % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 -C tools/perf
>
> I never tried build both Linux and perf on the same O= directory, can
> you please building perf on a separate directory to see if this changes
> anything?
>
> Also you don't need to build Linux proper to test these problems, right?
>
> - Arnaldo

Thank you! Using an O= without the kernel build fixes my build problem...
Then I managed to craft a patch:)
https://lore.kernel.org/linux-perf-users/20220527055217.452307-1-maskray@google.com/
("perf: Fix segbase for ld.lld linked objects")