All of lore.kernel.org
 help / color / mirror / Atom feed
From: Aleksa Sarai <cyphar@cyphar.com>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com>,
	Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>,
	"David S. Miller" <davem@davemloft.net>,
	Masami Hiramatsu <mhiramat@kernel.org>,
	Jonathan Corbet <corbet@lwn.net>,
	Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@redhat.com>,
	Arnaldo Carvalho de Melo <acme@kernel.org>,
	Alexander Shishkin <alexander.shishkin@linux.intel.com>,
	Jiri Olsa <jolsa@redhat.com>, Namhyung Kim <namhyung@kernel.org>,
	Shuah Khan <shuah@kernel.org>,
	Alexei Starovoitov <ast@kernel.org>,
	Daniel Borkmann <daniel@iogearbox.net>,
	Brendan Gregg <bgregg@netflix.com>,
	Christian Brauner <christian@brauner.io>,
	Aleksa Sarai <asarai@suse.de>,
	netdev@vger.kernel.org, linux-doc@vger.kernel.org,
	linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org,
	Josh Poimboeuf <jpoimboe@redhat.com>
Subject: Re: [PATCH v3 1/2] kretprobe: produce sane stack traces
Date: Thu, 8 Nov 2018 18:46:12 +1100	[thread overview]
Message-ID: <20181108074612.ldy6rozdpsdps6bf@yavin> (raw)
In-Reply-To: <20181106171501.59ccabbc@gandalf.local.home>

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]

On 2018-11-06, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Sun, 4 Nov 2018 22:59:13 +1100
> Aleksa Sarai <cyphar@cyphar.com> wrote:
> 
> > The same issue is present in __save_stack_trace
> > (arch/x86/kernel/stacktrace.c). This is likely the only reason that --
> > as Steven said -- stacktraces wouldn't work with ftrace-graph (and thus
> > with the refactor both of you are discussing).
> 
> By the way, I was playing with the the orc unwinder and stack traces
> from the function graph tracer return code, and got it working with the
> below patch. Caution, that patch also has a stack trace hardcoded in
> the return path of the function graph tracer, so you don't want to run
> function graph tracing without filtering.

Neat!

> diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c
> index 169b3c44ee97..aaeca73218cc 100644
> --- a/kernel/trace/trace_functions_graph.c
> +++ b/kernel/trace/trace_functions_graph.c
> @@ -242,13 +242,16 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret,
>  	trace->calltime = current->ret_stack[index].calltime;
>  	trace->overrun = atomic_read(&current->trace_overrun);
>  	trace->depth = index;
> +
> +	trace_dump_stack(0);

Right, this works because save_stack is not being passed a pt_regs. But if
you pass a pt_regs (as happens with bpf_getstackid -- which is what
spawned this discussion) then the top-most entry of the stack will still
be a trampoline because there is no ftrace_graph_ret_addr call.

(I'm struggling with how to fix this -- I can't figure out what retp
should be if you have a pt_regs. ->sp doesn't appear to work -- it's off
by a few bytes.)

I will attach what I have at the moment to hopefully explain what the
issue I've found is (re-using the kretprobe architecture but with the
shadow-stack idea).

-- 
Aleksa Sarai
Senior Software Engineer (Containers)
SUSE Linux GmbH
<https://www.cyphar.com/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

WARNING: multiple messages have this Message-ID (diff)
From: cyphar at cyphar.com (Aleksa Sarai)
Subject: [PATCH v3 1/2] kretprobe: produce sane stack traces
Date: Thu, 8 Nov 2018 18:46:12 +1100	[thread overview]
Message-ID: <20181108074612.ldy6rozdpsdps6bf@yavin> (raw)
In-Reply-To: <20181106171501.59ccabbc@gandalf.local.home>

On 2018-11-06, Steven Rostedt <rostedt at goodmis.org> wrote:
> On Sun, 4 Nov 2018 22:59:13 +1100
> Aleksa Sarai <cyphar at cyphar.com> wrote:
> 
> > The same issue is present in __save_stack_trace
> > (arch/x86/kernel/stacktrace.c). This is likely the only reason that --
> > as Steven said -- stacktraces wouldn't work with ftrace-graph (and thus
> > with the refactor both of you are discussing).
> 
> By the way, I was playing with the the orc unwinder and stack traces
> from the function graph tracer return code, and got it working with the
> below patch. Caution, that patch also has a stack trace hardcoded in
> the return path of the function graph tracer, so you don't want to run
> function graph tracing without filtering.

Neat!

> diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c
> index 169b3c44ee97..aaeca73218cc 100644
> --- a/kernel/trace/trace_functions_graph.c
> +++ b/kernel/trace/trace_functions_graph.c
> @@ -242,13 +242,16 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret,
>  	trace->calltime = current->ret_stack[index].calltime;
>  	trace->overrun = atomic_read(&current->trace_overrun);
>  	trace->depth = index;
> +
> +	trace_dump_stack(0);

Right, this works because save_stack is not being passed a pt_regs. But if
you pass a pt_regs (as happens with bpf_getstackid -- which is what
spawned this discussion) then the top-most entry of the stack will still
be a trampoline because there is no ftrace_graph_ret_addr call.

(I'm struggling with how to fix this -- I can't figure out what retp
should be if you have a pt_regs. ->sp doesn't appear to work -- it's off
by a few bytes.)

I will attach what I have at the moment to hopefully explain what the
issue I've found is (re-using the kretprobe architecture but with the
shadow-stack idea).

-- 
Aleksa Sarai
Senior Software Engineer (Containers)
SUSE Linux GmbH
<https://www.cyphar.com/>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.linaro.org/pipermail/linux-kselftest-mirror/attachments/20181108/fa93e719/attachment.sig>

WARNING: multiple messages have this Message-ID (diff)
From: cyphar@cyphar.com (Aleksa Sarai)
Subject: [PATCH v3 1/2] kretprobe: produce sane stack traces
Date: Thu, 8 Nov 2018 18:46:12 +1100	[thread overview]
Message-ID: <20181108074612.ldy6rozdpsdps6bf@yavin> (raw)
Message-ID: <20181108074612.z3YqD7MsbKf7vwF28o-LXEDbN6fEWTfRK2j6TTrVWR4@z> (raw)
In-Reply-To: <20181106171501.59ccabbc@gandalf.local.home>

On 2018-11-06, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Sun, 4 Nov 2018 22:59:13 +1100
> Aleksa Sarai <cyphar@cyphar.com> wrote:
> 
> > The same issue is present in __save_stack_trace
> > (arch/x86/kernel/stacktrace.c). This is likely the only reason that --
> > as Steven said -- stacktraces wouldn't work with ftrace-graph (and thus
> > with the refactor both of you are discussing).
> 
> By the way, I was playing with the the orc unwinder and stack traces
> from the function graph tracer return code, and got it working with the
> below patch. Caution, that patch also has a stack trace hardcoded in
> the return path of the function graph tracer, so you don't want to run
> function graph tracing without filtering.

Neat!

> diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c
> index 169b3c44ee97..aaeca73218cc 100644
> --- a/kernel/trace/trace_functions_graph.c
> +++ b/kernel/trace/trace_functions_graph.c
> @@ -242,13 +242,16 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret,
>  	trace->calltime = current->ret_stack[index].calltime;
>  	trace->overrun = atomic_read(&current->trace_overrun);
>  	trace->depth = index;
> +
> +	trace_dump_stack(0);

Right, this works because save_stack is not being passed a pt_regs. But if
you pass a pt_regs (as happens with bpf_getstackid -- which is what
spawned this discussion) then the top-most entry of the stack will still
be a trampoline because there is no ftrace_graph_ret_addr call.

(I'm struggling with how to fix this -- I can't figure out what retp
should be if you have a pt_regs. ->sp doesn't appear to work -- it's off
by a few bytes.)

I will attach what I have at the moment to hopefully explain what the
issue I've found is (re-using the kretprobe architecture but with the
shadow-stack idea).

-- 
Aleksa Sarai
Senior Software Engineer (Containers)
SUSE Linux GmbH
<https://www.cyphar.com/>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.linaro.org/pipermail/linux-kselftest-mirror/attachments/20181108/fa93e719/attachment.sig>

WARNING: multiple messages have this Message-ID (diff)
From: Aleksa Sarai <cyphar@cyphar.com>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com>,
	Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>,
	"David S. Miller" <davem@davemloft.net>,
	Masami Hiramatsu <mhiramat@kernel.org>,
	Jonathan Corbet <corbet@lwn.net>,
	Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@redhat.com>,
	Arnaldo Carvalho de Melo <acme@kernel.org>,
	Alexander Shishkin <alexander.shishkin@linux.intel.com>,
	Jiri Olsa <jolsa@redhat.com>, Namhyung Kim <namhyung@kernel.org>,
	Shuah Khan <shuah@kernel.org>,
	Alexei Starovoitov <ast@kernel.org>,
	Daniel Borkmann <daniel@iogearbox.net>,
	Brendan Gregg <bgregg@netflix.com>,
	Christian Brauner <christian@brauner.io>,
	Aleksa Sarai <asarai@suse.de>,
	netdev@vger.kernel.org, linux-doc@vger.kernel.org,
	linux-kernel@vger.kernel.org,
Subject: Re: [PATCH v3 1/2] kretprobe: produce sane stack traces
Date: Thu, 8 Nov 2018 18:46:12 +1100	[thread overview]
Message-ID: <20181108074612.ldy6rozdpsdps6bf@yavin> (raw)
In-Reply-To: <20181106171501.59ccabbc@gandalf.local.home>

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]

On 2018-11-06, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Sun, 4 Nov 2018 22:59:13 +1100
> Aleksa Sarai <cyphar@cyphar.com> wrote:
> 
> > The same issue is present in __save_stack_trace
> > (arch/x86/kernel/stacktrace.c). This is likely the only reason that --
> > as Steven said -- stacktraces wouldn't work with ftrace-graph (and thus
> > with the refactor both of you are discussing).
> 
> By the way, I was playing with the the orc unwinder and stack traces
> from the function graph tracer return code, and got it working with the
> below patch. Caution, that patch also has a stack trace hardcoded in
> the return path of the function graph tracer, so you don't want to run
> function graph tracing without filtering.

Neat!

> diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c
> index 169b3c44ee97..aaeca73218cc 100644
> --- a/kernel/trace/trace_functions_graph.c
> +++ b/kernel/trace/trace_functions_graph.c
> @@ -242,13 +242,16 @@ ftrace_pop_return_trace(struct ftrace_graph_ret *trace, unsigned long *ret,
>  	trace->calltime = current->ret_stack[index].calltime;
>  	trace->overrun = atomic_read(&current->trace_overrun);
>  	trace->depth = index;
> +
> +	trace_dump_stack(0);

Right, this works because save_stack is not being passed a pt_regs. But if
you pass a pt_regs (as happens with bpf_getstackid -- which is what
spawned this discussion) then the top-most entry of the stack will still
be a trampoline because there is no ftrace_graph_ret_addr call.

(I'm struggling with how to fix this -- I can't figure out what retp
should be if you have a pt_regs. ->sp doesn't appear to work -- it's off
by a few bytes.)

I will attach what I have at the moment to hopefully explain what the
issue I've found is (re-using the kretprobe architecture but with the
shadow-stack idea).

-- 
Aleksa Sarai
Senior Software Engineer (Containers)
SUSE Linux GmbH
<https://www.cyphar.com/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

  reply	other threads:[~2018-11-08  7:46 UTC|newest]

Thread overview: 137+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-11-01  8:35 [PATCH v3 0/2] kretprobe: produce sane stack traces Aleksa Sarai
2018-11-01  8:35 ` Aleksa Sarai
2018-11-01  8:35 ` cyphar
2018-11-01  8:35 ` [PATCH v3 1/2] " Aleksa Sarai
2018-11-01  8:35   ` Aleksa Sarai
2018-11-01  8:35   ` cyphar
2018-11-01 15:20   ` Masami Hiramatsu
2018-11-01 15:20     ` Masami Hiramatsu
2018-11-01 15:20     ` Masami Hiramatsu
2018-11-01 15:20     ` mhiramat
2018-11-01 21:13     ` Aleksa Sarai
2018-11-01 21:13       ` Aleksa Sarai
2018-11-01 21:13       ` Aleksa Sarai
2018-11-01 21:13       ` cyphar
2018-11-02  3:04       ` Masami Hiramatsu
2018-11-02  3:04         ` Masami Hiramatsu
2018-11-02  3:04         ` Masami Hiramatsu
2018-11-02  3:04         ` mhiramat
2018-11-02  4:37         ` Aleksa Sarai
2018-11-02  4:37           ` Aleksa Sarai
2018-11-02  4:37           ` Aleksa Sarai
2018-11-02  4:37           ` cyphar
2018-11-03 12:47           ` Masami Hiramatsu
2018-11-03 12:47             ` Masami Hiramatsu
2018-11-03 12:47             ` Masami Hiramatsu
2018-11-03 12:47             ` mhiramat
2018-11-02  0:47   ` Steven Rostedt
2018-11-02  0:47     ` Steven Rostedt
2018-11-02  0:47     ` Steven Rostedt
2018-11-02  0:47     ` rostedt
2018-11-02  5:05     ` Aleksa Sarai
2018-11-02  5:05       ` Aleksa Sarai
2018-11-02  5:05       ` Aleksa Sarai
2018-11-02  5:05       ` cyphar
2018-11-02  6:59       ` Aleksa Sarai
2018-11-02  6:59         ` Aleksa Sarai
2018-11-02  6:59         ` Aleksa Sarai
2018-11-02  6:59         ` cyphar
2018-11-02 13:16         ` Steven Rostedt
2018-11-02 13:16           ` Steven Rostedt
2018-11-02 13:16           ` Steven Rostedt
2018-11-02 13:16           ` rostedt
2018-11-02 15:43           ` Josh Poimboeuf
2018-11-02 15:43             ` Josh Poimboeuf
2018-11-02 15:43             ` Josh Poimboeuf
2018-11-02 15:43             ` jpoimboe
2018-11-02 16:13             ` Steven Rostedt
2018-11-02 16:13               ` Steven Rostedt
2018-11-02 16:13               ` Steven Rostedt
2018-11-02 16:13               ` rostedt
2018-11-03 13:00               ` Masami Hiramatsu
2018-11-03 13:00                 ` Masami Hiramatsu
2018-11-03 13:00                 ` Masami Hiramatsu
2018-11-03 13:00                 ` mhiramat
2018-11-03 13:13                 ` Steven Rostedt
2018-11-03 13:13                   ` Steven Rostedt
2018-11-03 13:13                   ` Steven Rostedt
2018-11-03 13:13                   ` rostedt
2018-11-03 16:34                   ` Masami Hiramatsu
2018-11-03 16:34                     ` Masami Hiramatsu
2018-11-03 16:34                     ` Masami Hiramatsu
2018-11-03 16:34                     ` mhiramat
2018-11-03 17:30                     ` Steven Rostedt
2018-11-03 17:30                       ` Steven Rostedt
2018-11-03 17:30                       ` Steven Rostedt
2018-11-03 17:30                       ` rostedt
2018-11-03 17:33                       ` Steven Rostedt
2018-11-03 17:33                         ` Steven Rostedt
2018-11-03 17:33                         ` Steven Rostedt
2018-11-03 17:33                         ` rostedt
2018-11-04  2:25                       ` Masami Hiramatsu
2018-11-04  2:25                         ` Masami Hiramatsu
2018-11-04  2:25                         ` Masami Hiramatsu
2018-11-04  2:25                         ` mhiramat
2018-11-03  7:02           ` Aleksa Sarai
2018-11-03  7:02             ` Aleksa Sarai
2018-11-03  7:02             ` Aleksa Sarai
2018-11-03  7:02             ` cyphar
2018-11-04 11:59             ` Aleksa Sarai
2018-11-04 11:59               ` Aleksa Sarai
2018-11-04 11:59               ` Aleksa Sarai
2018-11-04 11:59               ` cyphar
2018-11-06 22:15               ` Steven Rostedt
2018-11-06 22:15                 ` Steven Rostedt
2018-11-06 22:15                 ` Steven Rostedt
2018-11-06 22:15                 ` rostedt
2018-11-08  7:46                 ` Aleksa Sarai [this message]
2018-11-08  7:46                   ` Aleksa Sarai
2018-11-08  7:46                   ` Aleksa Sarai
2018-11-08  7:46                   ` cyphar
2018-11-08  8:04                   ` Aleksa Sarai
2018-11-08  8:04                     ` Aleksa Sarai
2018-11-08  8:04                     ` Aleksa Sarai
2018-11-08  8:04                     ` cyphar
2018-11-08 14:44                     ` Josh Poimboeuf
2018-11-08 14:44                       ` Josh Poimboeuf
2018-11-08 14:44                       ` Josh Poimboeuf
2018-11-08 14:44                       ` jpoimboe
2018-11-09  7:26                       ` Masami Hiramatsu
2018-11-09  7:26                         ` Masami Hiramatsu
2018-11-09  7:26                         ` Masami Hiramatsu
2018-11-09  7:26                         ` mhiramat
2018-11-09 15:10                         ` Aleksa Sarai
2018-11-09 15:10                           ` Aleksa Sarai
2018-11-09 15:10                           ` Aleksa Sarai
2018-11-09 15:10                           ` asarai
2018-11-09  7:15                     ` Masami Hiramatsu
2018-11-09  7:15                       ` Masami Hiramatsu
2018-11-09  7:15                       ` Masami Hiramatsu
2018-11-09  7:15                       ` mhiramat
2018-11-09 15:06                       ` Aleksa Sarai
2018-11-09 15:06                         ` Aleksa Sarai
2018-11-09 15:06                         ` Aleksa Sarai
2018-11-09 15:06                         ` asarai
2018-11-10 15:31                         ` Masami Hiramatsu
2018-11-10 15:31                           ` Masami Hiramatsu
2018-11-10 15:31                           ` Masami Hiramatsu
2018-11-10 15:31                           ` mhiramat
2018-11-12 10:38                           ` Aleksa Sarai
2018-11-12 10:38                             ` Aleksa Sarai
2018-11-12 10:38                             ` Aleksa Sarai
2018-11-12 10:38                             ` asarai
2018-11-03 13:23           ` Masami Hiramatsu
2018-11-03 13:23             ` Masami Hiramatsu
2018-11-03 13:23             ` Masami Hiramatsu
2018-11-03 13:23             ` mhiramat
2018-11-02  7:58       ` Aleksa Sarai
2018-11-02  7:58         ` Aleksa Sarai
2018-11-02  7:58         ` Aleksa Sarai
2018-11-02  7:58         ` cyphar
2018-11-02  4:01   ` kbuild test robot
2018-11-02  4:01     ` kbuild test robot
2018-11-02  4:01     ` kbuild test robot
2018-11-02  4:01     ` lkp
2018-11-01  8:35 ` [PATCH v3 2/2] trace: remove kretprobed checks Aleksa Sarai
2018-11-01  8:35   ` Aleksa Sarai
2018-11-01  8:35   ` cyphar

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20181108074612.ldy6rozdpsdps6bf@yavin \
    --to=cyphar@cyphar.com \
    --cc=acme@kernel.org \
    --cc=alexander.shishkin@linux.intel.com \
    --cc=anil.s.keshavamurthy@intel.com \
    --cc=asarai@suse.de \
    --cc=ast@kernel.org \
    --cc=bgregg@netflix.com \
    --cc=christian@brauner.io \
    --cc=corbet@lwn.net \
    --cc=daniel@iogearbox.net \
    --cc=davem@davemloft.net \
    --cc=jolsa@redhat.com \
    --cc=jpoimboe@redhat.com \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=mhiramat@kernel.org \
    --cc=mingo@redhat.com \
    --cc=namhyung@kernel.org \
    --cc=naveen.n.rao@linux.vnet.ibm.com \
    --cc=netdev@vger.kernel.org \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=shuah@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.