* Re: Call graph dwarf unwinding fails with lld [not found] ` <CAP-5=fUobnt9anqeTaUt_EJVuMOpsxGR-Sa_e7qX=ZxqPDr+sA@mail.gmail.com> @ 2022-05-26 4:16 ` Fangrui Song 2022-05-26 4:25 ` Ian Rogers 0 siblings, 1 reply; 4+ messages in thread From: Fangrui Song @ 2022-05-26 4:16 UTC (permalink / raw) To: Ian Rogers; +Cc: Travis Downs, linux-perf-users, llvm On 2022-05-25, Ian Rogers wrote: >On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote: >> >> I have been tracking down an issue locally where `perf record >> --call-graph=dwarf` stopped working in the sense that unwinding always >> failed. >> >> Turns out it was related to `lld` - binaries linked with `lld` don't >> seem to unwind correctly with `--call-graph=dwarf`. I've confirmed >> that the `.eh_frame` sections are there, but I guess they are >> different after using this linker. >> >> Has anyone run into this? > >I was unaware but thanks for filing the bug! > >> There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156 >> >> ... but no activity since it was filed, though it is not clear if the >> problem is really with lld or the unwinders. >> >> In a related question: perf supports more than one DWARF unwinding >> implementation, depending on what libraries are available at build >> time, right? Is there a way to select which unwinder is used at >> runtime? > >I don't think so. There are two build options NO_LIBUNWIND=1 and >NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the >bug, confirmed no difference with the eh_frames and hand verified >them, I also added -fno-omit-frame-pointer -O0 and -static. I've not >got a good handle on the issue but my suspicion is that the >eh_frame_hdr is broken - it is at least something that lld generates. >If I change the eh_frame_hdr finding to some findable but unexpected >offset like eh_frame here: >https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286 >and then test with a working linker the unwinding is broken similarly >to with lld. I was looking for a useful tool (objdump, llvm-objdump, >readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one. >Perhaps someone has a suggestion or maybe pahole can do it. Hopefully >this gives a lead for you to dig into for the bug. > >Thanks, >Ian I am very familiar with lld and have read some nongnu libunwind code, so I guess I may be an appropriate one debugging the issue, but I am puzzled about how to build tools/perf: /tmp/out/custom1 has needed llvm-project tools, built from origin/main ninja -C /tmp/out/custom1 clang lld llvm-{nm,size,strings,objcopy,objdump,readelf,ar,strip} ``` % git checkout master # torvalds/linux % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 defconfig all % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 -C tools/perf ... In file included from bench/../../arch/x86/lib/memcpy_64.S:8, from bench/mem-memcpy-x86-64-asm.S:14: /tmp/linux/x86_64/arch/x86/include/generated/asm/export.h:1:10: fatal error: asm-generic/export.h: No such file or directory 1 | #include <asm-generic/export.h> | ^~~~~~~~~~~~~~~~~~~~~~ ``` I am running a 5.16 kernel so I have tried `git checkout v5.16`, no luck as well. I cannot find useful information about "asm-generic/export.h: no such file or directory" on www:( ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Call graph dwarf unwinding fails with lld 2022-05-26 4:16 ` Call graph dwarf unwinding fails with lld Fangrui Song @ 2022-05-26 4:25 ` Ian Rogers 2022-05-26 14:39 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 4+ messages in thread From: Ian Rogers @ 2022-05-26 4:25 UTC (permalink / raw) To: Fangrui Song, Arnaldo Carvalho de Melo Cc: Travis Downs, linux-perf-users, llvm On Wed, May 25, 2022 at 9:16 PM Fangrui Song <maskray@google.com> wrote: > > On 2022-05-25, Ian Rogers wrote: > >On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote: > >> > >> I have been tracking down an issue locally where `perf record > >> --call-graph=dwarf` stopped working in the sense that unwinding always > >> failed. > >> > >> Turns out it was related to `lld` - binaries linked with `lld` don't > >> seem to unwind correctly with `--call-graph=dwarf`. I've confirmed > >> that the `.eh_frame` sections are there, but I guess they are > >> different after using this linker. > >> > >> Has anyone run into this? > > > >I was unaware but thanks for filing the bug! > > > >> There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156 > >> > >> ... but no activity since it was filed, though it is not clear if the > >> problem is really with lld or the unwinders. > >> > >> In a related question: perf supports more than one DWARF unwinding > >> implementation, depending on what libraries are available at build > >> time, right? Is there a way to select which unwinder is used at > >> runtime? > > > >I don't think so. There are two build options NO_LIBUNWIND=1 and > >NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the > >bug, confirmed no difference with the eh_frames and hand verified > >them, I also added -fno-omit-frame-pointer -O0 and -static. I've not > >got a good handle on the issue but my suspicion is that the > >eh_frame_hdr is broken - it is at least something that lld generates. > >If I change the eh_frame_hdr finding to some findable but unexpected > >offset like eh_frame here: > >https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286 > >and then test with a working linker the unwinding is broken similarly > >to with lld. I was looking for a useful tool (objdump, llvm-objdump, > >readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one. > >Perhaps someone has a suggestion or maybe pahole can do it. Hopefully > >this gives a lead for you to dig into for the bug. > > > >Thanks, > >Ian > > I am very familiar with lld and have read some nongnu libunwind code, so I guess I > may be an appropriate one debugging the issue, but I am puzzled about how to build tools/perf: > > /tmp/out/custom1 has needed llvm-project tools, built from origin/main > ninja -C /tmp/out/custom1 clang lld llvm-{nm,size,strings,objcopy,objdump,readelf,ar,strip} > > ``` > % git checkout master # torvalds/linux > % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 defconfig all > % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 -C tools/perf > ... > In file included from bench/../../arch/x86/lib/memcpy_64.S:8, > from bench/mem-memcpy-x86-64-asm.S:14: > /tmp/linux/x86_64/arch/x86/include/generated/asm/export.h:1:10: fatal error: asm-generic/export.h: No such file or directory > 1 | #include <asm-generic/export.h> > | ^~~~~~~~~~~~~~~~~~~~~~ > ``` > > I am running a 5.16 kernel so I have tried `git checkout v5.16`, no luck as well. > I cannot find useful information about "asm-generic/export.h: no such file or directory" on www:( Arnaldo, can the tools/include headers be made hermetic? You may have better luck with the development tree: https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/ branch perf/core. The perf tool is backward compatible with old kernels, unfortunately Debian packages it wrong which causes issues everywhere. Thanks, Ian ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Call graph dwarf unwinding fails with lld 2022-05-26 4:25 ` Ian Rogers @ 2022-05-26 14:39 ` Arnaldo Carvalho de Melo 2022-05-27 6:05 ` Fāng-ruì Sòng 0 siblings, 1 reply; 4+ messages in thread From: Arnaldo Carvalho de Melo @ 2022-05-26 14:39 UTC (permalink / raw) To: Ian Rogers; +Cc: Fangrui Song, Travis Downs, linux-perf-users, llvm Em Wed, May 25, 2022 at 09:25:39PM -0700, Ian Rogers escreveu: > On Wed, May 25, 2022 at 9:16 PM Fangrui Song <maskray@google.com> wrote: > > > > On 2022-05-25, Ian Rogers wrote: > > >On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote: > > >> > > >> I have been tracking down an issue locally where `perf record > > >> --call-graph=dwarf` stopped working in the sense that unwinding always > > >> failed. > > >> > > >> Turns out it was related to `lld` - binaries linked with `lld` don't > > >> seem to unwind correctly with `--call-graph=dwarf`. I've confirmed > > >> that the `.eh_frame` sections are there, but I guess they are > > >> different after using this linker. > > >> > > >> Has anyone run into this? > > > > > >I was unaware but thanks for filing the bug! > > > > > >> There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156 > > >> > > >> ... but no activity since it was filed, though it is not clear if the > > >> problem is really with lld or the unwinders. > > >> > > >> In a related question: perf supports more than one DWARF unwinding > > >> implementation, depending on what libraries are available at build > > >> time, right? Is there a way to select which unwinder is used at > > >> runtime? > > > > > >I don't think so. There are two build options NO_LIBUNWIND=1 and > > >NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the > > >bug, confirmed no difference with the eh_frames and hand verified > > >them, I also added -fno-omit-frame-pointer -O0 and -static. I've not > > >got a good handle on the issue but my suspicion is that the > > >eh_frame_hdr is broken - it is at least something that lld generates. > > >If I change the eh_frame_hdr finding to some findable but unexpected > > >offset like eh_frame here: > > >https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286 > > >and then test with a working linker the unwinding is broken similarly > > >to with lld. I was looking for a useful tool (objdump, llvm-objdump, > > >readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one. > > >Perhaps someone has a suggestion or maybe pahole can do it. Hopefully > > >this gives a lead for you to dig into for the bug. > > > > > >Thanks, > > >Ian > > > > I am very familiar with lld and have read some nongnu libunwind code, so I guess I > > may be an appropriate one debugging the issue, but I am puzzled about how to build tools/perf: > > > > /tmp/out/custom1 has needed llvm-project tools, built from origin/main > > ninja -C /tmp/out/custom1 clang lld llvm-{nm,size,strings,objcopy,objdump,readelf,ar,strip} > > > > ``` > > % git checkout master # torvalds/linux > > % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 defconfig all > > % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 -C tools/perf I never tried build both Linux and perf on the same O= directory, can you please building perf on a separate directory to see if this changes anything? Also you don't need to build Linux proper to test these problems, right? - Arnaldo > > ... > > In file included from bench/../../arch/x86/lib/memcpy_64.S:8, > > from bench/mem-memcpy-x86-64-asm.S:14: > > /tmp/linux/x86_64/arch/x86/include/generated/asm/export.h:1:10: fatal error: asm-generic/export.h: No such file or directory > > 1 | #include <asm-generic/export.h> > > | ^~~~~~~~~~~~~~~~~~~~~~ > > ``` > > > > I am running a 5.16 kernel so I have tried `git checkout v5.16`, no luck as well. > > I cannot find useful information about "asm-generic/export.h: no such file or directory" on www:( > > Arnaldo, can the tools/include headers be made hermetic? > > You may have better luck with the development tree: > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/ > branch perf/core. The perf tool is backward compatible with old > kernels, unfortunately Debian packages it wrong which causes issues > everywhere. > > Thanks, > Ian -- - Arnaldo ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Call graph dwarf unwinding fails with lld 2022-05-26 14:39 ` Arnaldo Carvalho de Melo @ 2022-05-27 6:05 ` Fāng-ruì Sòng 0 siblings, 0 replies; 4+ messages in thread From: Fāng-ruì Sòng @ 2022-05-27 6:05 UTC (permalink / raw) To: Arnaldo Carvalho de Melo; +Cc: Ian Rogers, Travis Downs, linux-perf-users, llvm On Thu, May 26, 2022 at 7:39 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > > Em Wed, May 25, 2022 at 09:25:39PM -0700, Ian Rogers escreveu: > > On Wed, May 25, 2022 at 9:16 PM Fangrui Song <maskray@google.com> wrote: > > > > > > On 2022-05-25, Ian Rogers wrote: > > > >On Tue, May 24, 2022 at 1:50 PM Travis Downs <travis.downs@gmail.com> wrote: > > > >> > > > >> I have been tracking down an issue locally where `perf record > > > >> --call-graph=dwarf` stopped working in the sense that unwinding always > > > >> failed. > > > >> > > > >> Turns out it was related to `lld` - binaries linked with `lld` don't > > > >> seem to unwind correctly with `--call-graph=dwarf`. I've confirmed > > > >> that the `.eh_frame` sections are there, but I guess they are > > > >> different after using this linker. > > > >> > > > >> Has anyone run into this? > > > > > > > >I was unaware but thanks for filing the bug! > > > > > > > >> There's an open LLVM issue at https://github.com/llvm/llvm-project/issues/53156 > > > >> > > > >> ... but no activity since it was filed, though it is not clear if the > > > >> problem is really with lld or the unwinders. > > > >> > > > >> In a related question: perf supports more than one DWARF unwinding > > > >> implementation, depending on what libraries are available at build > > > >> time, right? Is there a way to select which unwinder is used at > > > >> runtime? > > > > > > > >I don't think so. There are two build options NO_LIBUNWIND=1 and > > > >NO_LIBDW_DWARF_UNWIND=1. I played around with the example from the > > > >bug, confirmed no difference with the eh_frames and hand verified > > > >them, I also added -fno-omit-frame-pointer -O0 and -static. I've not > > > >got a good handle on the issue but my suspicion is that the > > > >eh_frame_hdr is broken - it is at least something that lld generates. > > > >If I change the eh_frame_hdr finding to some findable but unexpected > > > >offset like eh_frame here: > > > >https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/util/unwind-libunwind-local.c?h=perf/core#n286 > > > >and then test with a working linker the unwinding is broken similarly > > > >to with lld. I was looking for a useful tool (objdump, llvm-objdump, > > > >readelf, dwarfdump) to look at the eh_frame_hdr but I didn't see one. > > > >Perhaps someone has a suggestion or maybe pahole can do it. Hopefully > > > >this gives a lead for you to dig into for the bug. > > > > > > > >Thanks, > > > >Ian > > > > > > I am very familiar with lld and have read some nongnu libunwind code, so I guess I > > > may be an appropriate one debugging the issue, but I am puzzled about how to build tools/perf: > > > > > > /tmp/out/custom1 has needed llvm-project tools, built from origin/main > > > ninja -C /tmp/out/custom1 clang lld llvm-{nm,size,strings,objcopy,objdump,readelf,ar,strip} > > > > > > ``` > > > % git checkout master # torvalds/linux > > > % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 defconfig all > > > % PATH=/tmp/out/custom1/bin:$PATH make O=/tmp/linux/x86_64 ARCH=x86_64 -j60 -C tools/perf > > I never tried build both Linux and perf on the same O= directory, can > you please building perf on a separate directory to see if this changes > anything? > > Also you don't need to build Linux proper to test these problems, right? > > - Arnaldo Thank you! Using an O= without the kernel build fixes my build problem... Then I managed to craft a patch:) https://lore.kernel.org/linux-perf-users/20220527055217.452307-1-maskray@google.com/ ("perf: Fix segbase for ld.lld linked objects") ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-05-27 6:05 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CAOBGo4zjkcX=ZQm1uYRDe9EjYqsTCyZY-Gf1C4XqMNYGFCcF+Q@mail.gmail.com>
[not found] ` <CAP-5=fUobnt9anqeTaUt_EJVuMOpsxGR-Sa_e7qX=ZxqPDr+sA@mail.gmail.com>
2022-05-26 4:16 ` Call graph dwarf unwinding fails with lld Fangrui Song
2022-05-26 4:25 ` Ian Rogers
2022-05-26 14:39 ` Arnaldo Carvalho de Melo
2022-05-27 6:05 ` Fāng-ruì Sòng
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox