From: Nathan Chancellor <nathan@kernel.org>
To: Jiri Olsa <jolsa@redhat.com>
Cc: Nick Desaulniers <ndesaulniers@google.com>,
Andrii Nakryiko <andrii.nakryiko@gmail.com>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andrii@kernel.org>,
Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>,
Yonghong Song <yhs@fb.com>,
John Fastabend <john.fastabend@gmail.com>,
KP Singh <kpsingh@kernel.org>,
Networking <netdev@vger.kernel.org>, bpf <bpf@vger.kernel.org>,
clang-built-linux <clang-built-linux@googlegroups.com>,
Veronika Kabatova <vkabatov@redhat.com>,
Jiri Olsa <jolsa@kernel.org>,
Arnaldo Carvalho de Melo <acme@redhat.com>,
Daniel Kiss <daniel.kiss@arm.com>
Subject: Re: FAILED unresolved symbol vfs_truncate on arm64 with LLVM
Date: Tue, 9 Feb 2021 16:15:03 -0700 [thread overview]
Message-ID: <20210209231503.GA19859@ubuntu-m3-large-x86> (raw)
In-Reply-To: <YCMBmNujLsMg0Q0q@krava>
On Tue, Feb 09, 2021 at 10:41:44PM +0100, Jiri Olsa wrote:
> On Tue, Feb 09, 2021 at 09:50:48PM +0100, Jiri Olsa wrote:
> > On Tue, Feb 09, 2021 at 12:09:31PM -0800, Nick Desaulniers wrote:
> > > On Tue, Feb 9, 2021 at 11:06 AM Jiri Olsa <jolsa@redhat.com> wrote:
> > > >
> > > > On Tue, Feb 09, 2021 at 05:13:42PM +0100, Jiri Olsa wrote:
> > > > > On Tue, Feb 09, 2021 at 04:09:36PM +0100, Jiri Olsa wrote:
> > > > >
> > > > > SNIP
> > > > >
> > > > > > > > > > DW_AT_prototyped (true)
> > > > > > > > > > DW_AT_type (0x01cfdfe4 "long int")
> > > > > > > > > > DW_AT_external (true)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Ok, the problem appears to be not in DWARF, but in mcount_loc data.
> > > > > > > > > vfs_truncate's address is not recorded as ftrace-attachable, and thus
> > > > > > > > > pahole ignores it. I don't know why this happens and it's quite
> > > > > > > > > strange, given vfs_truncate is just a normal global function.
> > > > > > >
> > > > > > > right, I can't see it in mcount adresses.. but it begins with instructions
> > > > > > > that appears to be nops, which would suggest it's traceable
> > > > > > >
> > > > > > > ffff80001031f430 <vfs_truncate>:
> > > > > > > ffff80001031f430: 5f 24 03 d5 hint #34
> > > > > > > ffff80001031f434: 1f 20 03 d5 nop
> > > > > > > ffff80001031f438: 1f 20 03 d5 nop
> > > > > > > ffff80001031f43c: 3f 23 03 d5 hint #25
> > > > > > >
> > > > > > > > >
> > > > > > > > > I'd like to understand this issue before we try to fix it, but there
> > > > > > > > > is at least one improvement we can make: pahole should check ftrace
> > > > > > > > > addresses only for static functions, not the global ones (global ones
> > > > > > > > > should be always attachable, unless they are special, e.g., notrace
> > > > > > > > > and stuff). We can easily check that by looking at the corresponding
> > > > > > > > > symbol. But I'd like to verify that vfs_truncate is ftrace-attachable
> > > > > >
> > > > > > I'm still trying to build the kernel.. however ;-)
> > > > >
> > > > > I finally reproduced.. however arm's not using mcount_loc
> > > > > but some other special section.. so it's new mess for me
> > > >
> > > > so ftrace data actualy has vfs_truncate address but with extra 4 bytes:
> > > >
> > > > ffff80001031f434
> > > >
> > > > real vfs_truncate address:
> > > >
> > > > ffff80001031f430 g F .text 0000000000000168 vfs_truncate
> > > >
> > > > vfs_truncate disasm:
> > > >
> > > > ffff80001031f430 <vfs_truncate>:
> > > > ffff80001031f430: 5f 24 03 d5 hint #34
> > > > ffff80001031f434: 1f 20 03 d5 nop
> > > > ffff80001031f438: 1f 20 03 d5 nop
> > > > ffff80001031f43c: 3f 23 03 d5 hint #25
> > > >
> > > > thats why we don't match it in pahole.. I checked few other functions
> > > > and some have the same problem and some match the function boundary
> > > >
> > > > those that match don't have that first hint instrucion, like:
> > > >
> > > > ffff800010321e40 <do_faccessat>:
> > > > ffff800010321e40: 1f 20 03 d5 nop
> > > > ffff800010321e44: 1f 20 03 d5 nop
> > > > ffff800010321e48: 3f 23 03 d5 hint #25
> > > >
> > > > any hints about hint instructions? ;-)
> > >
> > > aarch64 makes *some* newer instructions reuse the "hint" ie "nop"
> > > encoding space to make software backwards compatible on older hardware
> > > that doesn't support such instructions. Is this BTI, perhaps? (The
> > > function is perhaps the destination of an indirect call?)
> >
> > I see, I think we can't take ftrace addresses as start of the function
> > because there could be extra instruction(s) related to the call before
> > it like here
> >
> > we need to check ftrace address be within the function/symbol,
> > not exact start
>
> the build with gcc passed only because mcount data are all zeros
> and pahole falls back to 'not-ftrace' mode
>
> $ llvm-objdump -t build/aarch64-gcc/vmlinux | grep mcount
> ffff800011eb4840 g .init.data 0000000000000000 __stop_mcount_loc
> ffff800011e47d58 g .init.data 0000000000000000 __start_mcount_loc
>
> $ llvm-objdump -s build/aarch64-gcc/vmlinux
> ffff800011e47d50 00000000 00000000 00000000 00000000 ................
> ffff800011e47d60 00000000 00000000 00000000 00000000 ................
> ffff800011e47d70 00000000 00000000 00000000 00000000 ................
> ffff800011e47d80 00000000 00000000 00000000 00000000 ................
> ffff800011e47d90 00000000 00000000 00000000 00000000 ................
> ...
>
> we should check on why it's zero
>
> Nathan, any chance you could run kernel built with gcc and check on ftrace?
Sure, with GCC 10.2.0:
/ # cat /proc/version
Linux version 5.11.0-rc7 (nathan@ubuntu-m3-large-x86) (aarch64-linux-gcc (GCC) 10.2.0, GNU ld (GNU Binutils) 2.35) #1 SMP PREEMPT Tue Feb 9 16:04:19 MST 2021
/ # grep vfs_truncate /sys/kernel/debug/tracing/available_filter_functions
vfs_truncate
/ # zcat /proc/config.gz | grep "DEBUG_INFO_BTF\|FTRACE\|BPF"
# CONFIG_CGROUP_BPF is not set
CONFIG_BPF=y
# CONFIG_BPF_LSM is not set
CONFIG_BPF_SYSCALL=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
CONFIG_BPF_JIT_DEFAULT_ON=y
# CONFIG_BPF_PRELOAD is not set
# CONFIG_NETFILTER_XT_MATCH_BPF is not set
# CONFIG_BPFILTER is not set
# CONFIG_NET_CLS_BPF is not set
# CONFIG_NET_ACT_BPF is not set
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
# CONFIG_PSTORE_FTRACE is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_FTRACE=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
# CONFIG_FTRACE_SYSCALLS is not set
CONFIG_BPF_EVENTS=y
CONFIG_FTRACE_MCOUNT_RECORD=y
# CONFIG_FTRACE_RECORD_RECURSION is not set
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_TEST_BPF is not set
Cheers,
Nathan
> the build with clang fails because the ftrace data are there,
> but pahole takes them as 'start' of the function, which is wrong
>
> $ llvm-objdump -t build/aarch64/vmlinux | grep mcount
> ffff800011d27d10 g .init.data 0000000000000000 __start_mcount_loc
> ffff800011d90038 g .init.data 0000000000000000 __stop_mcount_loc
>
> $ llvm-objdump -s build/aarch64-gcc/vmlinux
> ffff800011d27d10 cc330110 0080ffff 1c340110 0080ffff .3.......4......
> ffff800011d27d20 6c340110 0080ffff 1004c111 0080ffff l4..............
> ffff800011d27d30 3804c111 0080ffff 6004c111 0080ffff 8.......`.......
> ffff800011d27d40 8804c111 0080ffff 0405c111 0080ffff ................
> ffff800011d27d50 3805c111 0080ffff 7c05c111 0080ffff 8.......|.......
> ...
>
> I think if we fix pahole to take check the ftrace address is
> within the processed function, we should be fine.. I'll try to
> send something soon
>
> jirka
>
next prev parent reply other threads:[~2021-02-10 0:04 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-09 3:44 FAILED unresolved symbol vfs_truncate on arm64 with LLVM Nathan Chancellor
2021-02-09 4:45 ` Andrii Nakryiko
2021-02-09 5:23 ` Nathan Chancellor
2021-02-09 6:09 ` Andrii Nakryiko
2021-02-09 6:13 ` Andrii Nakryiko
2021-02-09 6:56 ` Andrii Nakryiko
2021-02-09 7:49 ` Nathan Chancellor
2021-02-09 12:36 ` Jiri Olsa
2021-02-09 15:09 ` Jiri Olsa
2021-02-09 16:13 ` Jiri Olsa
2021-02-09 16:35 ` Nathan Chancellor
2021-02-09 17:07 ` Sedat Dilek
2021-02-09 17:12 ` Nick Desaulniers
2021-02-09 17:26 ` Sedat Dilek
2021-02-09 19:06 ` Jiri Olsa
2021-02-09 19:22 ` Jiri Olsa
2021-02-09 20:09 ` Nick Desaulniers
2021-02-09 20:50 ` Jiri Olsa
2021-02-09 21:41 ` Jiri Olsa
2021-02-09 23:15 ` Nathan Chancellor [this message]
2021-02-10 0:02 ` Nathan Chancellor
2021-02-10 0:49 ` Daniel Kiss
2021-02-10 11:34 ` David Laight
2021-02-10 12:32 ` Jiri Olsa
2021-02-09 20:59 ` Andrii Nakryiko
2021-02-09 21:55 ` Jiri Olsa
2021-02-09 22:00 ` Andrii Nakryiko
2021-02-10 13:26 ` Jiri Olsa
2021-02-10 18:02 ` Nathan Chancellor
2021-02-10 18:20 ` Andrii Nakryiko
2021-02-10 18:24 ` Sedat Dilek
2021-02-10 19:10 ` Jiri Olsa
2021-02-10 19:21 ` Andrii Nakryiko
2021-02-10 20:13 ` Jiri Olsa
2021-02-11 15:08 ` Jiri Olsa
2021-02-11 15:43 ` Sedat Dilek
2021-02-11 16:07 ` Jiri Olsa
2021-02-11 16:36 ` Sedat Dilek
2021-02-11 17:24 ` Nathan Chancellor
2021-02-11 19:59 ` Andrii Nakryiko
2021-02-11 21:47 ` Jiri Olsa
2021-02-12 16:38 ` Jiri Olsa
2021-02-12 19:22 ` Andrii Nakryiko
2021-02-12 21:29 ` Jiri Olsa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210209231503.GA19859@ubuntu-m3-large-x86 \
--to=nathan@kernel.org \
--cc=acme@redhat.com \
--cc=andrii.nakryiko@gmail.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=clang-built-linux@googlegroups.com \
--cc=daniel.kiss@arm.com \
--cc=daniel@iogearbox.net \
--cc=john.fastabend@gmail.com \
--cc=jolsa@kernel.org \
--cc=jolsa@redhat.com \
--cc=kafai@fb.com \
--cc=kpsingh@kernel.org \
--cc=ndesaulniers@google.com \
--cc=netdev@vger.kernel.org \
--cc=songliubraving@fb.com \
--cc=vkabatov@redhat.com \
--cc=yhs@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox