Call graph dwarf unwinding fails with lld

Call graph dwarf unwinding fails with lld

[Travis Downs]

I have been tracking down an issue locally where `perf record
--call-graph=dwarf` stopped working in the sense that unwinding always
failed.

Turns out it was related to `lld` - binaries linked with `lld` don't
seem to unwind correctly with `--call-graph=dwarf`. I've confirmed
that the `.eh_frame` sections are there, but I guess they are
different after using this linker.

Has anyone run into this?

There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156

... but no activity since it was filed, though it is not clear if the
problem is really with lld or the unwinders.

In a related question: perf supports more than one DWARF unwinding
implementation, depending on what libraries are available at build
time, right? Is there a way to select which unwinder is used at
runtime?

Re: Call graph dwarf unwinding fails with lld

[Ian Rogers]

On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote:
>
> I have been tracking down an issue locally where `perf record
> --call-graph=dwarf` stopped working in the sense that unwinding always
> failed.
>
> Turns out it was related to `lld` - binaries linked with `lld` don't
> seem to unwind correctly with `--call-graph=dwarf`. I've confirmed
> that the `.eh_frame` sections are there, but I guess they are
> different after using this linker.
>
> Has anyone run into this?

I was unaware but thanks for filing the bug!

> There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156
>
> ... but no activity since it was filed, though it is not clear if the
> problem is really with lld or the unwinders.
>
> In a related question: perf supports more than one DWARF unwinding
> implementation, depending on what libraries are available at build
> time, right? Is there a way to select which unwinder is used at
> runtime?

I don't think so. There are two build options NO_LIBUNWIND=1 and
NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the
bug, confirmed no difference with the eh_frames and hand verified
them, I also added -fno-omit-frame-pointer -O0 and -static. I've not
got a good handle on the issue but my suspicion is that the
eh_frame_hdr is broken - it is at least something that lld generates.
If I change the eh_frame_hdr finding to some findable but unexpected
offset like eh_frame here:
https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286
and then test with a working linker the unwinding is broken similarly
to with lld. I was looking for a useful tool (objdump, llvm-objdump,
readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one.
Perhaps someone has a suggestion or maybe pahole can do it. Hopefully
this gives a lead for you to dig into for the bug.

Thanks,
Ian

Re: Call graph dwarf unwinding fails with lld

[Fāng-ruì Sòng]

On Wed, May 25, 2022 at 12:50 AM Ian Rogers <irogers@google.com> wrote:
>
> On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote:
> >
> > I have been tracking down an issue locally where `perf record
> > --call-graph=dwarf` stopped working in the sense that unwinding always
> > failed.
> >
> > Turns out it was related to `lld` - binaries linked with `lld` don't
> > seem to unwind correctly with `--call-graph=dwarf`. I've confirmed
> > that the `.eh_frame` sections are there, but I guess they are
> > different after using this linker.
> >
> > Has anyone run into this?
>
> I was unaware but thanks for filing the bug!
>
> > There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156
> >
> > ... but no activity since it was filed, though it is not clear if the
> > problem is really with lld or the unwinders.
> >
> > In a related question: perf supports more than one DWARF unwinding
> > implementation, depending on what libraries are available at build
> > time, right? Is there a way to select which unwinder is used at
> > runtime?
>
> I don't think so. There are two build options NO_LIBUNWIND=1 and
> NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the
> bug, confirmed no difference with the eh_frames and hand verified
> them, I also added -fno-omit-frame-pointer -O0 and -static. I've not
> got a good handle on the issue but my suspicion is that the
> eh_frame_hdr is broken - it is at least something that lld generates.
> If I change the eh_frame_hdr finding to some findable but unexpected
> offset like eh_frame here:
> https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286
> and then test with a working linker the unwinding is broken similarly
> to with lld. I was looking for a useful tool (objdump, llvm-objdump,
> readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one.
> Perhaps someone has a suggestion or maybe pahole can do it. Hopefully
> this gives a lead for you to dig into for the bug.
>
> Thanks,
> Ian

I left a comment on
https://github.com/llvm/llvm-project/issues/53156#issuecomment-1136777149
that the perf issue is related to unsupported `ld.lld --rosegments`
and `ld.bfd -z separate-code` layouts.

-- 
宋方睿

Re: Call graph dwarf unwinding fails with lld

[Ian Rogers]

On Wed, May 25, 2022 at 1:10 AM Fāng-ruì Sòng <maskray@google.com> wrote:
>
> On Wed, May 25, 2022 at 12:50 AM Ian Rogers <irogers@google.com> wrote:
> >
> > On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote:
> > >
> > > I have been tracking down an issue locally where `perf record
> > > --call-graph=dwarf` stopped working in the sense that unwinding always
> > > failed.
> > >
> > > Turns out it was related to `lld` - binaries linked with `lld` don't
> > > seem to unwind correctly with `--call-graph=dwarf`. I've confirmed
> > > that the `.eh_frame` sections are there, but I guess they are
> > > different after using this linker.
> > >
> > > Has anyone run into this?
> >
> > I was unaware but thanks for filing the bug!
> >
> > > There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156
> > >
> > > ... but no activity since it was filed, though it is not clear if the
> > > problem is really with lld or the unwinders.
> > >
> > > In a related question: perf supports more than one DWARF unwinding
> > > implementation, depending on what libraries are available at build
> > > time, right? Is there a way to select which unwinder is used at
> > > runtime?
> >
> > I don't think so. There are two build options NO_LIBUNWIND=1 and
> > NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the
> > bug, confirmed no difference with the eh_frames and hand verified
> > them, I also added -fno-omit-frame-pointer -O0 and -static. I've not
> > got a good handle on the issue but my suspicion is that the
> > eh_frame_hdr is broken - it is at least something that lld generates.
> > If I change the eh_frame_hdr finding to some findable but unexpected
> > offset like eh_frame here:
> > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286
> > and then test with a working linker the unwinding is broken similarly
> > to with lld. I was looking for a useful tool (objdump, llvm-objdump,
> > readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one.
> > Perhaps someone has a suggestion or maybe pahole can do it. Hopefully
> > this gives a lead for you to dig into for the bug.
> >
> > Thanks,
> > Ian
>
> I left a comment on
> https://github.com/llvm/llvm-project/issues/53156#issuecomment-1136777149
> that the perf issue is related to unsupported `ld.lld --rosegments`
> and `ld.bfd -z separate-code` layouts.

Thanks Fangrui! I commented on the bug (thanks for the help!) and I
think the bug is with libunwind assuming that sections are consecutive
in memory, at different offsets, etc. lld is playing games to reduce
file size and ultimately that's broken the perf unwinding. Perhaps
someone can figure out how to better drive libunwind from perf, but I
think it is wrong in its offset assumptions. libdw based unwinding is
no better from my testing, overall this is a fairly major
bug/regression.

Thanks,
Ian

> --
> 宋方睿

Re: Call graph dwarf unwinding fails with lld

[Fangrui Song]

On 2022-05-25, Ian Rogers wrote:
>On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote:
>>
>> I have been tracking down an issue locally where `perf record
>> --call-graph=dwarf` stopped working in the sense that unwinding always
>> failed.
>>
>> Turns out it was related to `lld` - binaries linked with `lld` don't
>> seem to unwind correctly with `--call-graph=dwarf`. I've confirmed
>> that the `.eh_frame` sections are there, but I guess they are
>> different after using this linker.
>>
>> Has anyone run into this?
>
>I was unaware but thanks for filing the bug!
>
>> There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156
>>
>> ... but no activity since it was filed, though it is not clear if the
>> problem is really with lld or the unwinders.
>>
>> In a related question: perf supports more than one DWARF unwinding
>> implementation, depending on what libraries are available at build
>> time, right? Is there a way to select which unwinder is used at
>> runtime?
>
>I don't think so. There are two build options NO_LIBUNWIND=1 and
>NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the
>bug, confirmed no difference with the eh_frames and hand verified
>them, I also added -fno-omit-frame-pointer -O0 and -static. I've not
>got a good handle on the issue but my suspicion is that the
>eh_frame_hdr is broken - it is at least something that lld generates.
>If I change the eh_frame_hdr finding to some findable but unexpected
>offset like eh_frame here:
>https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286
>and then test with a working linker the unwinding is broken similarly
>to with lld. I was looking for a useful tool (objdump, llvm-objdump,
>readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one.
>Perhaps someone has a suggestion or maybe pahole can do it. Hopefully
>this gives a lead for you to dig into for the bug.
>
>Thanks,
>Ian

I am very familiar with lld and have read some nongnu libunwind code, so I guess I
may be an appropriate one debugging the issue, but I am puzzled about how to build tools/perf:

/tmp/out/custom1 has needed llvm-project tools, built from origin/main
ninja -C /tmp/out/custom1 clang lld llvm-{nm,size,strings,objcopy,objdump,readelf,ar,strip}

```
% git checkout master  # torvalds/linux
% PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 defconfig all
% PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 -C tools/perf
...
In file included from bench/../../arch/x86/lib/memcpy_64.S:8,
                  from bench/mem-memcpy-x86-64-asm.S:14:
/tmp/linux/x86_64/arch/x86/include/generated/asm/export.h:1:10: fatal error: asm-generic/export.h: No such file or directory
     1 | #include <asm-generic/export.h>
       |          ^~~~~~~~~~~~~~~~~~~~~~
```

I am running a 5.16 kernel so I have tried `git checkout v5.16`, no luck as well.
I cannot find useful information about "asm-generic/export.h: no such file or directory" on www:(

Re: Call graph dwarf unwinding fails with lld

[Ian Rogers]

On Wed, May 25, 2022 at 9:16 PM Fangrui Song <maskray@google.com> wrote:
>
> On 2022-05-25, Ian Rogers wrote:
> >On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote:
> >>
> >> I have been tracking down an issue locally where `perf record
> >> --call-graph=dwarf` stopped working in the sense that unwinding always
> >> failed.
> >>
> >> Turns out it was related to `lld` - binaries linked with `lld` don't
> >> seem to unwind correctly with `--call-graph=dwarf`. I've confirmed
> >> that the `.eh_frame` sections are there, but I guess they are
> >> different after using this linker.
> >>
> >> Has anyone run into this?
> >
> >I was unaware but thanks for filing the bug!
> >
> >> There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156
> >>
> >> ... but no activity since it was filed, though it is not clear if the
> >> problem is really with lld or the unwinders.
> >>
> >> In a related question: perf supports more than one DWARF unwinding
> >> implementation, depending on what libraries are available at build
> >> time, right? Is there a way to select which unwinder is used at
> >> runtime?
> >
> >I don't think so. There are two build options NO_LIBUNWIND=1 and
> >NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the
> >bug, confirmed no difference with the eh_frames and hand verified
> >them, I also added -fno-omit-frame-pointer -O0 and -static. I've not
> >got a good handle on the issue but my suspicion is that the
> >eh_frame_hdr is broken - it is at least something that lld generates.
> >If I change the eh_frame_hdr finding to some findable but unexpected
> >offset like eh_frame here:
> >https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286
> >and then test with a working linker the unwinding is broken similarly
> >to with lld. I was looking for a useful tool (objdump, llvm-objdump,
> >readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one.
> >Perhaps someone has a suggestion or maybe pahole can do it. Hopefully
> >this gives a lead for you to dig into for the bug.
> >
> >Thanks,
> >Ian
>
> I am very familiar with lld and have read some nongnu libunwind code, so I guess I
> may be an appropriate one debugging the issue, but I am puzzled about how to build tools/perf:
>
> /tmp/out/custom1 has needed llvm-project tools, built from origin/main
> ninja -C /tmp/out/custom1 clang lld llvm-{nm,size,strings,objcopy,objdump,readelf,ar,strip}
>
> ```
> % git checkout master  # torvalds/linux
> % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 defconfig all
> % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 -C tools/perf
> ...
> In file included from bench/../../arch/x86/lib/memcpy_64.S:8,
>                   from bench/mem-memcpy-x86-64-asm.S:14:
> /tmp/linux/x86_64/arch/x86/include/generated/asm/export.h:1:10: fatal error: asm-generic/export.h: No such file or directory
>      1 | #include <asm-generic/export.h>
>        |          ^~~~~~~~~~~~~~~~~~~~~~
> ```
>
> I am running a 5.16 kernel so I have tried `git checkout v5.16`, no luck as well.
> I cannot find useful information about "asm-generic/export.h: no such file or directory" on www:(

Arnaldo, can the tools/include headers be made hermetic?

You may have better luck with the development tree:
https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/
branch perf/core. The perf tool is backward compatible with old
kernels, unfortunately Debian packages it wrong which causes issues
everywhere.

Thanks,
Ian

Re: Call graph dwarf unwinding fails with lld

[Arnaldo Carvalho de Melo]

Em Wed, May 25, 2022 at 09:25:39PM -0700, Ian Rogers escreveu:
> On Wed, May 25, 2022 at 9:16 PM Fangrui Song <maskray@google.com> wrote:
> >
> > On 2022-05-25, Ian Rogers wrote:
> > >On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote:
> > >>
> > >> I have been tracking down an issue locally where `perf record
> > >> --call-graph=dwarf` stopped working in the sense that unwinding always
> > >> failed.
> > >>
> > >> Turns out it was related to `lld` - binaries linked with `lld` don't
> > >> seem to unwind correctly with `--call-graph=dwarf`. I've confirmed
> > >> that the `.eh_frame` sections are there, but I guess they are
> > >> different after using this linker.
> > >>
> > >> Has anyone run into this?
> > >
> > >I was unaware but thanks for filing the bug!
> > >
> > >> There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156
> > >>
> > >> ... but no activity since it was filed, though it is not clear if the
> > >> problem is really with lld or the unwinders.
> > >>
> > >> In a related question: perf supports more than one DWARF unwinding
> > >> implementation, depending on what libraries are available at build
> > >> time, right? Is there a way to select which unwinder is used at
> > >> runtime?
> > >
> > >I don't think so. There are two build options NO_LIBUNWIND=1 and
> > >NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the
> > >bug, confirmed no difference with the eh_frames and hand verified
> > >them, I also added -fno-omit-frame-pointer -O0 and -static. I've not
> > >got a good handle on the issue but my suspicion is that the
> > >eh_frame_hdr is broken - it is at least something that lld generates.
> > >If I change the eh_frame_hdr finding to some findable but unexpected
> > >offset like eh_frame here:
> > >https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286
> > >and then test with a working linker the unwinding is broken similarly
> > >to with lld. I was looking for a useful tool (objdump, llvm-objdump,
> > >readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one.
> > >Perhaps someone has a suggestion or maybe pahole can do it. Hopefully
> > >this gives a lead for you to dig into for the bug.
> > >
> > >Thanks,
> > >Ian
> >
> > I am very familiar with lld and have read some nongnu libunwind code, so I guess I
> > may be an appropriate one debugging the issue, but I am puzzled about how to build tools/perf:
> >
> > /tmp/out/custom1 has needed llvm-project tools, built from origin/main
> > ninja -C /tmp/out/custom1 clang lld llvm-{nm,size,strings,objcopy,objdump,readelf,ar,strip}
> >
> > ```
> > % git checkout master  # torvalds/linux
> > % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 defconfig all
> > % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 -C tools/perf

I never tried build both Linux and perf on the same O= directory, can
you please building perf on a separate directory to see if this changes
anything?

Also you don't need to build Linux proper to test these problems, right?

- Arnaldo

> > ...
> > In file included from bench/../../arch/x86/lib/memcpy_64.S:8,
> >                   from bench/mem-memcpy-x86-64-asm.S:14:
> > /tmp/linux/x86_64/arch/x86/include/generated/asm/export.h:1:10: fatal error: asm-generic/export.h: No such file or directory
> >      1 | #include <asm-generic/export.h>
> >        |          ^~~~~~~~~~~~~~~~~~~~~~
> > ```
> >
> > I am running a 5.16 kernel so I have tried `git checkout v5.16`, no luck as well.
> > I cannot find useful information about "asm-generic/export.h: no such file or directory" on www:(
> 
> Arnaldo, can the tools/include headers be made hermetic?
> 
> You may have better luck with the development tree:
> https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/
> branch perf/core. The perf tool is backward compatible with old
> kernels, unfortunately Debian packages it wrong which causes issues
> everywhere.
> 
> Thanks,
> Ian

-- 

- Arnaldo

Re: Call graph dwarf unwinding fails with lld

[Fāng-ruì Sòng]

On Thu, May 26, 2022 at 7:39 AM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> Em Wed, May 25, 2022 at 09:25:39PM -0700, Ian Rogers escreveu:
> > On Wed, May 25, 2022 at 9:16 PM Fangrui Song <maskray@google.com> wrote:
> > >
> > > On 2022-05-25, Ian Rogers wrote:
> > > >On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote:
> > > >>
> > > >> I have been tracking down an issue locally where `perf record
> > > >> --call-graph=dwarf` stopped working in the sense that unwinding always
> > > >> failed.
> > > >>
> > > >> Turns out it was related to `lld` - binaries linked with `lld` don't
> > > >> seem to unwind correctly with `--call-graph=dwarf`. I've confirmed
> > > >> that the `.eh_frame` sections are there, but I guess they are
> > > >> different after using this linker.
> > > >>
> > > >> Has anyone run into this?
> > > >
> > > >I was unaware but thanks for filing the bug!
> > > >
> > > >> There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156
> > > >>
> > > >> ... but no activity since it was filed, though it is not clear if the
> > > >> problem is really with lld or the unwinders.
> > > >>
> > > >> In a related question: perf supports more than one DWARF unwinding
> > > >> implementation, depending on what libraries are available at build
> > > >> time, right? Is there a way to select which unwinder is used at
> > > >> runtime?
> > > >
> > > >I don't think so. There are two build options NO_LIBUNWIND=1 and
> > > >NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the
> > > >bug, confirmed no difference with the eh_frames and hand verified
> > > >them, I also added -fno-omit-frame-pointer -O0 and -static. I've not
> > > >got a good handle on the issue but my suspicion is that the
> > > >eh_frame_hdr is broken - it is at least something that lld generates.
> > > >If I change the eh_frame_hdr finding to some findable but unexpected
> > > >offset like eh_frame here:
> > > >https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286
> > > >and then test with a working linker the unwinding is broken similarly
> > > >to with lld. I was looking for a useful tool (objdump, llvm-objdump,
> > > >readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one.
> > > >Perhaps someone has a suggestion or maybe pahole can do it. Hopefully
> > > >this gives a lead for you to dig into for the bug.
> > > >
> > > >Thanks,
> > > >Ian
> > >
> > > I am very familiar with lld and have read some nongnu libunwind code, so I guess I
> > > may be an appropriate one debugging the issue, but I am puzzled about how to build tools/perf:
> > >
> > > /tmp/out/custom1 has needed llvm-project tools, built from origin/main
> > > ninja -C /tmp/out/custom1 clang lld llvm-{nm,size,strings,objcopy,objdump,readelf,ar,strip}
> > >
> > > ```
> > > % git checkout master  # torvalds/linux
> > > % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 defconfig all
> > > % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 -C tools/perf
>
> I never tried build both Linux and perf on the same O= directory, can
> you please building perf on a separate directory to see if this changes
> anything?
>
> Also you don't need to build Linux proper to test these problems, right?
>
> - Arnaldo

Thank you! Using an O= without the kernel build fixes my build problem...
Then I managed to craft a patch:)
https://lore.kernel.org/linux-perf-users/20220527055217.452307-1-maskray@google.com/
("perf: Fix segbase for ld.lld linked objects")