Re: Call graph dwarf unwinding fails with lld

[Ian Rogers <irogers@google.com>]

On Wed, May 25, 2022 at 1:10 AM Fāng-ruì Sòng <maskray@google.com> wrote:
>
> On Wed, May 25, 2022 at 12:50 AM Ian Rogers <irogers@google.com> wrote:
> >
> > On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote:
> > >
> > > I have been tracking down an issue locally where `perf record
> > > --call-graph=dwarf` stopped working in the sense that unwinding always
> > > failed.
> > >
> > > Turns out it was related to `lld` - binaries linked with `lld` don't
> > > seem to unwind correctly with `--call-graph=dwarf`. I've confirmed
> > > that the `.eh_frame` sections are there, but I guess they are
> > > different after using this linker.
> > >
> > > Has anyone run into this?
> >
> > I was unaware but thanks for filing the bug!
> >
> > > There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156
> > >
> > > ... but no activity since it was filed, though it is not clear if the
> > > problem is really with lld or the unwinders.
> > >
> > > In a related question: perf supports more than one DWARF unwinding
> > > implementation, depending on what libraries are available at build
> > > time, right? Is there a way to select which unwinder is used at
> > > runtime?
> >
> > I don't think so. There are two build options NO_LIBUNWIND=1 and
> > NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the
> > bug, confirmed no difference with the eh_frames and hand verified
> > them, I also added -fno-omit-frame-pointer -O0 and -static. I've not
> > got a good handle on the issue but my suspicion is that the
> > eh_frame_hdr is broken - it is at least something that lld generates.
> > If I change the eh_frame_hdr finding to some findable but unexpected
> > offset like eh_frame here:
> > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286
> > and then test with a working linker the unwinding is broken similarly
> > to with lld. I was looking for a useful tool (objdump, llvm-objdump,
> > readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one.
> > Perhaps someone has a suggestion or maybe pahole can do it. Hopefully
> > this gives a lead for you to dig into for the bug.
> >
> > Thanks,
> > Ian
>
> I left a comment on
> https://github.com/llvm/llvm-project/issues/53156#issuecomment-1136777149
> that the perf issue is related to unsupported `ld.lld --rosegments`
> and `ld.bfd -z separate-code` layouts.

Thanks Fangrui! I commented on the bug (thanks for the help!) and I
think the bug is with libunwind assuming that sections are consecutive
in memory, at different offsets, etc. lld is playing games to reduce
file size and ultimately that's broken the perf unwinding. Perhaps
someone can figure out how to better drive libunwind from perf, but I
think it is wrong in its offset assumptions. libdw based unwinding is
no better from my testing, overall this is a fairly major
bug/regression.

Thanks,
Ian

> --
> 宋方睿