From: Masami Hiramatsu <mhiramat@kernel.org>
To: "Daniel Xu" <dxu@dxuuu.xyz>
Cc: linux-kernel@vger.kernel.org,
"bpf@vger.kernel.org" <bpf@vger.kernel.org>,
kuba@kernel.org, Steven Rostedt <rostedt@goodmis.org>
Subject: Re: Broken kretprobe stack traces
Date: Wed, 3 Mar 2021 13:48:28 +0900 [thread overview]
Message-ID: <20210303134828.39922eb167524bc7206c7880@kernel.org> (raw)
In-Reply-To: <1fed0793-391c-4c68-8d19-6dcd9017271d@www.fastmail.com>
Hi Daniel,
On Tue, 02 Mar 2021 17:15:13 -0800
"Daniel Xu" <dxu@dxuuu.xyz> wrote:
> Hi Masami,
>
> Jakub reported a bug with kretprobe stack traces -- wondering if you've gotten
> any bug reports related to stack traces being broken for kretprobes.
Yeah, stack dumper must check the stack entry is kretprobe'd and skip it.
For example, ftrace checks it as below.
/sys/kernel/debug/tracing # echo r vfs_read > kprobe_events
/sys/kernel/debug/tracing # echo stacktrace > events/kprobes/r_vfs_read_0/trigger
/sys/kernel/debug/tracing # echo 1 > events/kprobes/r_vfs_read_0/enable
/sys/kernel/debug/tracing # head -20 trace
# tracer: nop
#
# entries-in-buffer/entries-written: 15685/336094 #P:8
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
sh-132 [006] ...1 38.920352: <stack trace>
=> kretprobe_dispatcher
=> __kretprobe_trampoline_handler
=> trampoline_handler
=> [unknown/kretprobe'd]
=> [unknown/kretprobe'd]
=> __x64_sys_read
=> do_syscall_64
=> entry_SYSCALL_64_after_hwframe
>
> I think (can't prove) this used to work:
I'm not sure the bpftrace had correctly handled it or not.
>
> # bpftrace -e 'kretprobe:__tcp_retransmit_skb { @[kstack()] = count() }'
> Attaching 1 probe...
> ^C
>
> @[
> kretprobe_trampoline+0
> ]: 1
Would you know how the bpftrace stacktracer rewinds the stack entries?
FYI, ftrace does it in trace_seq_print_sym()@kernel/trace/trace_output.c
Thank you,
>
> fentry/fexit probes seem to work:
>
> # bpftrace -e 'kretfunc:__tcp_retransmit_skb { @[kstack()] = count() }'
> Attaching 1 probe...
> ^C
>
> @[
> ftrace_trampoline+10799
> bpf_get_stackid_raw_tp+121
> ftrace_trampoline+10799
> __tun_chr_ioctl.isra.0.cold+33312
> __tcp_retransmit_skb+5
> tcp_send_loss_probe+254
> tcp_write_timer_handler+394
> tcp_write_timer+149
> call_timer_fn+41
> __run_timers+493
> run_timer_softirq+25
> __softirqentry_text_start+207
> asm_call_sysvec_on_stack+18
> do_softirq_own_stack+55
> irq_exit_rcu+158
> sysvec_apic_timer_interrupt+54
> asm_sysvec_apic_timer_interrupt+18
> ]: 1
> @[
> ftrace_trampoline+10799
> bpf_get_stackid_raw_tp+121
> ftrace_trampoline+10799
> __tun_chr_ioctl.isra.0.cold+33312
> __tcp_retransmit_skb+5
> <...>
>
> which makes me suspect it's a kprobe specific issue.
>
> Thanks,
> Daniel
--
Masami Hiramatsu <mhiramat@kernel.org>
next prev parent reply other threads:[~2021-03-03 11:56 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-03 1:15 Broken kretprobe stack traces Daniel Xu
2021-03-03 4:48 ` Masami Hiramatsu [this message]
2021-03-03 14:26 ` Steven Rostedt
2021-03-03 19:58 ` Daniel Xu
2021-03-03 20:13 ` Daniel Xu
2021-03-03 20:37 ` Steven Rostedt
2021-03-04 2:18 ` Daniel Xu
2021-03-04 19:04 ` Daniel Xu
2021-03-04 13:19 ` Masami Hiramatsu
2021-03-04 15:22 ` [PATCH] kprobes: stacktrace: Recover the address changed by kretprobe kernel test robot
2021-03-04 17:37 ` kernel test robot
2021-03-04 20:25 ` kernel test robot
2021-03-03 19:57 ` Broken kretprobe stack traces Daniel Xu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210303134828.39922eb167524bc7206c7880@kernel.org \
--to=mhiramat@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=dxu@dxuuu.xyz \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox