how to get a kernel callgraph starting from a syscall

how to get a kernel callgraph starting from a syscall

[Francis Moreau]

Hello,

Sorry if the question is dumb but I'm a bit lost about this.

I know that ftrace exists but I think that perf can do this too: I
have a userspace application which is calling a libc C function (read)
which ends up doing a sys_read.

For the sys_read, I'd like to see what's happening in the kernel by
displaying the call graph.

Can perf do that ?

If so, would it be possible to point me out the perf command to use ?

Thanks
-- 
Francis

Re: how to get a kernel callgraph starting from a syscall

[David Ahern]

On 04/21/11 01:46, Francis Moreau wrote:
> Hello,
> 
> Sorry if the question is dumb but I'm a bit lost about this.
> 
> I know that ftrace exists but I think that perf can do this too: I
> have a userspace application which is calling a libc C function (read)
> which ends up doing a sys_read.
> 
> For the sys_read, I'd like to see what's happening in the kernel by
> displaying the call graph.
> 
> Can perf do that ?

Have you tried 'perf record -g'?

David


> 
> If so, would it be possible to point me out the perf command to use ?
> 
> Thanks

Re: how to get a kernel callgraph starting from a syscall

[Francis Moreau]

On Thu, Apr 21, 2011 at 5:23 PM, David Ahern <dsahern@gmail.com> wrote:
>
>
> On 04/21/11 01:46, Francis Moreau wrote:
>> Hello,
>>
>> Sorry if the question is dumb but I'm a bit lost about this.
>>
>> I know that ftrace exists but I think that perf can do this too: I
>> have a userspace application which is calling a libc C function (read)
>> which ends up doing a sys_read.
>>
>> For the sys_read, I'd like to see what's happening in the kernel by
>> displaying the call graph.
>>
>> Can perf do that ?
>
> Have you tried 'perf record -g'?
>

Well, I don't see how it can be done with perf record -g.

Let's say I'm interested to see the path taken in the kernel when a
process is calling sys_sleep:

  $ perf record -g sleep 1

Then I can use:

  $ perf report -g

But I don't see what I am asking for.

What am I missing ?

Thanks
-- 
Francis

Re: how to get a kernel callgraph starting from a syscall

[David Ahern]

On 04/21/11 12:48, Francis Moreau wrote:
> On Thu, Apr 21, 2011 at 5:23 PM, David Ahern <dsahern@gmail.com> wrote:
>>
>>
>> On 04/21/11 01:46, Francis Moreau wrote:
>>> Hello,
>>>
>>> Sorry if the question is dumb but I'm a bit lost about this.
>>>
>>> I know that ftrace exists but I think that perf can do this too: I
>>> have a userspace application which is calling a libc C function (read)
>>> which ends up doing a sys_read.
>>>
>>> For the sys_read, I'd like to see what's happening in the kernel by
>>> displaying the call graph.
>>>
>>> Can perf do that ?
>>
>> Have you tried 'perf record -g'?
>>
> 
> Well, I don't see how it can be done with perf record -g.
> 
> Let's say I'm interested to see the path taken in the kernel when a
> process is calling sys_sleep:
> 
>   $ perf record -g sleep 1
> 
> Then I can use:
> 
>   $ perf report -g
> 
> But I don't see what I am asking for.
> 
> What am I missing ?

You want the individual samples? if so, perf script dumps them and you
can see the callchain for each sys_read.

David

> 
> Thanks

Re: how to get a kernel callgraph starting from a syscall

[Francis Moreau]

On Thu, Apr 21, 2011 at 9:49 PM, David Ahern <dsahern@gmail.com> wrote:
>
> You want the individual samples? if so, perf script dumps them and you
> can see the callchain for each sys_read.
>

hmm, I don't see what you mean by individual samples.

I don't think sampling can help in my case (tracing the callchain of
one syscall).

Could you give me an example of what you have in mind ?

Thanks
-- 
Francis

Re: how to get a kernel callgraph starting from a syscall

[David Ahern]

On 04/21/11 14:07, Francis Moreau wrote:
> On Thu, Apr 21, 2011 at 9:49 PM, David Ahern <dsahern@gmail.com> wrote:
>>
>> You want the individual samples? if so, perf script dumps them and you
>> can see the callchain for each sys_read.
>>
> 
> hmm, I don't see what you mean by individual samples.
> 
> I don't think sampling can help in my case (tracing the callchain of
> one syscall).
> 
> Could you give me an example of what you have in mind ?
> 
> Thanks

I think I get it now. 'perf record' for syscall event stops at sys_read
as that is the point the event is generated. You want where the kernel
goes starting with sys_read. In that case you are limited to the
pre-existing tracepoints (see 'perf list -e tracepoints') to trigger an
event and back trace, or if the read causes the processes to block you
can you use the context-switch event or sched_switch trace point.

You could also insert probe points using perf probe. Lin Ming posted an
example of this a couple of weeks ago:
# ./perf probe -k ~/vmlinux -s ~/linux-2.6/  find_get_page
# ./perf record -e probe:find_get_page -f -g -a

David

Re: how to get a kernel callgraph starting from a syscall

[Francis Moreau]

On Fri, Apr 22, 2011 at 2:47 AM, David Ahern <dsahern@gmail.com> wrote:
> On 04/21/11 14:07, Francis Moreau wrote:
>> On Thu, Apr 21, 2011 at 9:49 PM, David Ahern <dsahern@gmail.com> wrote:
>>>
>>> You want the individual samples? if so, perf script dumps them and you
>>> can see the callchain for each sys_read.
>>>
>>
>> hmm, I don't see what you mean by individual samples.
>>
>> I don't think sampling can help in my case (tracing the callchain of
>> one syscall).
>>
>> Could you give me an example of what you have in mind ?
>>
>> Thanks
>
> I think I get it now. 'perf record' for syscall event stops at sys_read
> as that is the point the event is generated. You want where the kernel
> goes starting with sys_read. In that case you are limited to the
> pre-existing tracepoints (see 'perf list -e tracepoints') to trigger an
> event and back trace, or if the read causes the processes to block you
> can you use the context-switch event or sched_switch trace point.
>
> You could also insert probe points using perf probe. Lin Ming posted an
> example of this a couple of weeks ago:
> # ./perf probe -k ~/vmlinux -s ~/linux-2.6/  find_get_page
> # ./perf record -e probe:find_get_page -f -g -a
>

Thanks for you answer.

I think I'll use probe.

Maybe it would be nice to have ftrace integrated in perf ?

Thanks
-- 
Francis

Re: how to get a kernel callgraph starting from a syscall

[Francis Moreau]

On Fri, Apr 22, 2011 at 7:29 AM, Francis Moreau <francis.moro@gmail.com> wrote:
> On Fri, Apr 22, 2011 at 2:47 AM, David Ahern <dsahern@gmail.com> wrote:
>> On 04/21/11 14:07, Francis Moreau wrote:
>>> On Thu, Apr 21, 2011 at 9:49 PM, David Ahern <dsahern@gmail.com> wrote:
>>>>
>>>> You want the individual samples? if so, perf script dumps them and you
>>>> can see the callchain for each sys_read.
>>>>
>>>
>>> hmm, I don't see what you mean by individual samples.
>>>
>>> I don't think sampling can help in my case (tracing the callchain of
>>> one syscall).
>>>
>>> Could you give me an example of what you have in mind ?
>>>
>>> Thanks
>>
>> I think I get it now. 'perf record' for syscall event stops at sys_read
>> as that is the point the event is generated. You want where the kernel
>> goes starting with sys_read. In that case you are limited to the
>> pre-existing tracepoints (see 'perf list -e tracepoints') to trigger an
>> event and back trace, or if the read causes the processes to block you
>> can you use the context-switch event or sched_switch trace point.
>>
>> You could also insert probe points using perf probe. Lin Ming posted an
>> example of this a couple of weeks ago:
>> # ./perf probe -k ~/vmlinux -s ~/linux-2.6/  find_get_page
>> # ./perf record -e probe:find_get_page -f -g -a
>>
>
> Thanks for you answer.
>
> I think I'll use probe.


Thinking more about it, I think that both solution (probe or
tracepoint) are not adapted.

Because I'm interested in discovering the path taken by the kernel
from a given syscall, therefore I don't know this path. However both
solution imply that the path is known since you need to insert some
'probes' in this path to see it...

-- 
Francis

Re: how to get a kernel callgraph starting from a syscall

[Stefan Hajnoczi]

On Fri, Apr 22, 2011 at 12:46 PM, Francis Moreau <francis.moro@gmail.com> wrote:
> On Fri, Apr 22, 2011 at 7:29 AM, Francis Moreau <francis.moro@gmail.com> wrote:
>> On Fri, Apr 22, 2011 at 2:47 AM, David Ahern <dsahern@gmail.com> wrote:
>>> On 04/21/11 14:07, Francis Moreau wrote:
>>>> On Thu, Apr 21, 2011 at 9:49 PM, David Ahern <dsahern@gmail.com> wrote:
>>>>>
>>>>> You want the individual samples? if so, perf script dumps them and you
>>>>> can see the callchain for each sys_read.
>>>>>
>>>>
>>>> hmm, I don't see what you mean by individual samples.
>>>>
>>>> I don't think sampling can help in my case (tracing the callchain of
>>>> one syscall).
>>>>
>>>> Could you give me an example of what you have in mind ?
>>>>
>>>> Thanks
>>>
>>> I think I get it now. 'perf record' for syscall event stops at sys_read
>>> as that is the point the event is generated. You want where the kernel
>>> goes starting with sys_read. In that case you are limited to the
>>> pre-existing tracepoints (see 'perf list -e tracepoints') to trigger an
>>> event and back trace, or if the read causes the processes to block you
>>> can you use the context-switch event or sched_switch trace point.
>>>
>>> You could also insert probe points using perf probe. Lin Ming posted an
>>> example of this a couple of weeks ago:
>>> # ./perf probe -k ~/vmlinux -s ~/linux-2.6/  find_get_page
>>> # ./perf record -e probe:find_get_page -f -g -a
>>>
>>
>> Thanks for you answer.
>>
>> I think I'll use probe.
>
>
> Thinking more about it, I think that both solution (probe or
> tracepoint) are not adapted.
>
> Because I'm interested in discovering the path taken by the kernel
> from a given syscall, therefore I don't know this path. However both
> solution imply that the path is known since you need to insert some
> 'probes' in this path to see it...

You can use "Dynamic ftrace with the function graph tracer"
(set_graph_function):
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/trace/ftrace.txt;hb=HEAD#l1664

I don't know of way to do this using perf(1).

Stefan

Re: how to get a kernel callgraph starting from a syscall

[Francis Moreau]

On Sat, Apr 23, 2011 at 10:40 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Fri, Apr 22, 2011 at 12:46 PM, Francis Moreau <francis.moro@gmail.com> wrote:
>> On Fri, Apr 22, 2011 at 7:29 AM, Francis Moreau <francis.moro@gmail.com> wrote:
>>> On Fri, Apr 22, 2011 at 2:47 AM, David Ahern <dsahern@gmail.com> wrote:
>>>> On 04/21/11 14:07, Francis Moreau wrote:
>>>>> On Thu, Apr 21, 2011 at 9:49 PM, David Ahern <dsahern@gmail.com> wrote:
>>>>>>
>>>>>> You want the individual samples? if so, perf script dumps them and you
>>>>>> can see the callchain for each sys_read.
>>>>>>
>>>>>
>>>>> hmm, I don't see what you mean by individual samples.
>>>>>
>>>>> I don't think sampling can help in my case (tracing the callchain of
>>>>> one syscall).
>>>>>
>>>>> Could you give me an example of what you have in mind ?
>>>>>
>>>>> Thanks
>>>>
>>>> I think I get it now. 'perf record' for syscall event stops at sys_read
>>>> as that is the point the event is generated. You want where the kernel
>>>> goes starting with sys_read. In that case you are limited to the
>>>> pre-existing tracepoints (see 'perf list -e tracepoints') to trigger an
>>>> event and back trace, or if the read causes the processes to block you
>>>> can you use the context-switch event or sched_switch trace point.
>>>>
>>>> You could also insert probe points using perf probe. Lin Ming posted an
>>>> example of this a couple of weeks ago:
>>>> # ./perf probe -k ~/vmlinux -s ~/linux-2.6/  find_get_page
>>>> # ./perf record -e probe:find_get_page -f -g -a
>>>>
>>>
>>> Thanks for you answer.
>>>
>>> I think I'll use probe.
>>
>>
>> Thinking more about it, I think that both solution (probe or
>> tracepoint) are not adapted.
>>
>> Because I'm interested in discovering the path taken by the kernel
>> from a given syscall, therefore I don't know this path. However both
>> solution imply that the path is known since you need to insert some
>> 'probes' in this path to see it...
>
> You can use "Dynamic ftrace with the function graph tracer"
> (set_graph_function):
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/trace/ftrace.txt;hb=HEAD#l1664
>
> I don't know of way to do this using perf(1).

I don't know either but since perf interface is more convenient than
sysfs one, I was wondering if that was possible with perf.

Thanks
-- 
Francis