perf not capturing stack traces

perf not capturing stack traces

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 8765 bytes --]

Hi,

I'm using v3.19-rc5 on an ARM Cortex A9 board. Unfortunately, perf is
not able to capture any stack traces even though I CONFIG_STACKTRACE,
CONFIG_FRAME_POINTER and CONFIG_ARM_UNWIND all enabled.

I wonder if anybody else is facing similar issues.

Here's console grab:

perf record -e 'sched:*' -ag sleep 5  && perf report --stdio
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.092 MB perf.data (~4024 samples) ]
# To display the perf.data header info, please use --header/--header-only option
#
# Samples: 0  of event 'sched:sched_kthread_stop'
# Event count (approx.): 0
#
# Children      Self  Command  Shared Object  Symbol
# ........  ........  .......  .............  ......
#


# Samples: 0  of event 'sched:sched_kthread_stop_ret'
# Event count (approx.): 0
#
# Children      Self  Command  Shared Object  Symbol
# ........  ........  .......  .............  ......
#


# Samples: 62  of event 'sched:sched_wakeup'
# Event count (approx.): 62
#
# Children      Self  Command      Shared Object  Symbol        
# ........  ........  ...........  .............  ..............
:#
    74.19%    74.19%  swapper      [unknown]      [.] 00000000  
                |
                ---0

    11.29%    11.29%  kworker/0:2  [unknown]      [.] 00000000  
            |
            ---0

     9.68%     9.68%  perf         [unknown]      [.] 00000000  
                   |
                   ---0

     3.23%     3.23%  sleep        [unknown]      [.] 00000000  
                  |
                  ---0

     1.61%     1.61%  ksoftirqd/0  [unknown]      [.] 00000000  
            |
            ---0



:# Samples: 0  of event 'sched:sched_wakeup_new'
# Event count (approx.): 0
#
# Children      Self  Command  Shared Object  Symbol
# ........  ........  .......  .............  ......
#


# Samples: 102  of event 'sched:sched_switch'
# Event count (approx.): 102
#
# Children      Self  Command       Shared Object  Symbol        
# ........  ........  ............  .............  ..............
#
    31.37%    31.37%  swapper       [unknown]      [.] 00000000  
                 |
                 ---0

    20.59%    20.59%  kworker/0:2   [unknown]      [.] 00000000  
             |
             ---0

    15.69%    15.69%  ksoftirqd/0   [unknown]      [.] 00000000  
:             |
             ---0

    11.76%    11.76%  perf          [unknown]      [.] 00000000  
                    |
                    ---0

    11.76%    11.76%  sleep         [unknown]      [.] 00000000  
                   |
                   ---0

     7.84%     7.84%  rcu_sched     [unknown]      [.] 00000000  
               |
               ---0

     0.98%     0.98%  kworker/u2:1  [unknown]      [.] 00000000  
            |
            ---0



# Samples: 0  of event 'sched:sched_migrate_task'
# Event count (approx.): 0
:#
# Children      Self  Command  Shared Object  Symbol
# ........  ........  .......  .............  ......
#


# Samples: 0  of event 'sched:sched_process_free'
# Event count (approx.): 0
#
# Children      Self  Command  Shared Object  Symbol
# ........  ........  .......  .............  ......
#


# Samples: 1  of event 'sched:sched_process_exit'
# Event count (approx.): 1
#
# Children      Self  Command  Shared Object  Symbol        
# ........  ........  .......  .............  ..............
#
   100.00%   100.00%  sleep    [unknown]      [.] 00000000  
              |
              ---0
:


# Samples: 0  of event 'sched:sched_wait_task'
# Event count (approx.): 0
#
# Children      Self  Command  Shared Object  Symbol
# ........  ........  .......  .............  ......
#


# Samples: 0  of event 'sched:sched_process_wait'
# Event count (approx.): 0
#
# Children      Self  Command  Shared Object  Symbol
# ........  ........  .......  .............  ......
#


# Samples: 0  of event 'sched:sched_process_fork'
# Event count (approx.): 0
#
# Children      Self  Command  Shared Object  Symbol
:# ........  ........  .......  .............  ......
#


# Samples: 1  of event 'sched:sched_process_exec'
# Event count (approx.): 1
#
# Children      Self  Command  Shared Object  Symbol        
# ........  ........  .......  .............  ..............
#
   100.00%   100.00%  sleep    [unknown]      [.] 00000000  
              |
              ---0



# Samples: 70  of event 'sched:sched_stat_wait'
# Event count (approx.): 17926123
#
# Children      Self  Command      Shared Object  Symbol        
# ........  ........  ...........  .............  ..............
#
    33.13%    33.13%  ksoftirqd/0  [unknown]      [.] 00000000  
:            |
            ---0

    24.83%    24.83%  perf         [unknown]      [.] 00000000  
                   |
                   ---0

    21.44%    21.44%  sleep        [unknown]      [.] 00000000  
                  |
                  ---0

    20.60%    20.60%  rcu_sched    [unknown]      [.] 00000000  
              |
              ---0

     0.00%     0.00%  kworker/0:2  [unknown]      [.] 00000000  
            |
            ---0

     0.00%     0.00%  swapper      [unknown]      [.] 00000000  
                |
                ---0

:

# Samples: 43  of event 'sched:sched_stat_sleep'
# Event count (approx.): 119659990589
#
# Children      Self  Command  Shared Object  Symbol        
# ........  ........  .......  .............  ..............
#
    95.01%    95.01%  swapper  [unknown]      [.] 00000000  
            |
            ---0

     4.93%     4.93%  sleep    [unknown]      [.] 00000000  
              |
              ---0

     0.06%     0.06%  perf     [unknown]      [.] 00000000  
               |
               ---0



# Samples: 7  of event 'sched:sched_stat_iowait'
:# Event count (approx.): 37447751
#
# Children      Self  Command      Shared Object  Symbol        
# ........  ........  ...........  .............  ..............
#
   100.00%   100.00%  kworker/0:2  [unknown]      [.] 00000000  
            |
            ---0



# Samples: 19  of event 'sched:sched_stat_blocked'
# Event count (approx.): 42514750
#
# Children      Self  Command      Shared Object  Symbol        
# ........  ........  ...........  .............  ..............
#
    88.08%    88.08%  kworker/0:2  [unknown]      [.] 00000000  
            |
            ---0

     7.24%     7.24%  swapper      [unknown]      [.] 00000000  
                |
:                ---0

     3.81%     3.81%  ksoftirqd/0  [unknown]      [.] 00000000  
            |
            ---0

     0.86%     0.86%  perf         [unknown]      [.] 00000000  
                   |
                   ---0



# Samples: 79  of event 'sched:sched_stat_runtime'
# Event count (approx.): 46820375
#
# Children      Self  Command       Shared Object  Symbol        
# ........  ........  ............  .............  ..............
#
    27.85%    27.85%  perf          [unknown]      [.] 00000000  
                    |
                    ---0

    27.48%    27.48%  kworker/0:2   [unknown]      [.] 00000000  
:             |
             ---0

    20.79%    20.79%  sleep         [unknown]      [.] 00000000  
                   |
                   ---0

    12.51%    12.51%  ksoftirqd/0   [unknown]      [.] 00000000  
             |
             ---0

     8.96%     8.96%  rcu_sched     [unknown]      [.] 00000000  
               |
               ---0

     2.41%     2.41%  kworker/u2:1  [unknown]      [.] 00000000  
            |
            ---0



# Samples: 0  of event 'sched:sched_pi_setprio'
# Event count (approx.): 0
:#
# Children      Self  Command  Shared Object  Symbol
# ........  ........  .......  .............  ......
#


# Samples: 0  of event 'sched:sched_move_numa'
# Event count (approx.): 0
#
# Children      Self  Command  Shared Object  Symbol
# ........  ........  .......  .............  ......
#


# Samples: 0  of event 'sched:sched_stick_numa'
# Event count (approx.): 0
#
# Children      Self  Command  Shared Object  Symbol
# ........  ........  .......  .............  ......
#


# Samples: 0  of event 'sched:sched_swap_numa'
:# Event count (approx.): 0
#
# Children      Self  Command  Shared Object  Symbol
# ........  ........  .......  .............  ......
#


# Samples: 0  of event 'sched:sched_wake_idle_without_ipi'
# Event count (approx.): 0
#
# Children      Self  Command  Shared Object  Symbol
# ........  ........  .......  .............  ......
#

-- 
balbi

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: perf not capturing stack traces

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 1057 bytes --]

Hi,

On Fri, Jan 23, 2015 at 01:51:28PM -0600, Felipe Balbi wrote:
> Hi,
> 
> I'm using v3.19-rc5 on an ARM Cortex A9 board. Unfortunately, perf is
> not able to capture any stack traces even though I CONFIG_STACKTRACE,
> CONFIG_FRAME_POINTER and CONFIG_ARM_UNWIND all enabled.
> 
> I wonder if anybody else is facing similar issues.
> 
> Here's console grab:

perf features:

Auto-detecting system features:
...                         dwarf: [ on  ]
...                         glibc: [ on  ]
...                          gtk2: [ OFF ]
...                      libaudit: [ on  ]
...                        libbfd: [ on  ]
...                        libelf: [ on  ]
...                       libnuma: [ OFF ]
...                       libperl: [ on  ]
...                     libpython: [ on  ]
...                      libslang: [ on  ]
...                     libunwind: [ on  ]
...            libdw-dwarf-unwind: [ on  ]
...                          zlib: [ on  ]
...     DWARF post unwind library: libunwind

-- 
balbi

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: perf not capturing stack traces

[Arnaldo Carvalho de Melo]

Em Fri, Jan 23, 2015 at 01:51:28PM -0600, Felipe Balbi escreveu:
> I'm using v3.19-rc5 on an ARM Cortex A9 board. Unfortunately, perf is
> not able to capture any stack traces even though I CONFIG_STACKTRACE,
> CONFIG_FRAME_POINTER and CONFIG_ARM_UNWIND all enabled.
> 
> I wonder if anybody else is facing similar issues.

Did it work with a previous kernel?

- Arnaldo

Re: perf not capturing stack traces

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]

On Fri, Jan 23, 2015 at 05:59:59PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Fri, Jan 23, 2015 at 01:51:28PM -0600, Felipe Balbi escreveu:
> > I'm using v3.19-rc5 on an ARM Cortex A9 board. Unfortunately, perf is
> > not able to capture any stack traces even though I CONFIG_STACKTRACE,
> > CONFIG_FRAME_POINTER and CONFIG_ARM_UNWIND all enabled.
> > 
> > I wonder if anybody else is facing similar issues.
> 
> Did it work with a previous kernel?

haven't checked with this board. First time I tried to use perf for
anything.

-- 
balbi

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: perf not capturing stack traces

[Arnaldo Carvalho de Melo]

Em Fri, Jan 23, 2015 at 04:37:45PM -0600, Felipe Balbi escreveu:
> On Fri, Jan 23, 2015 at 05:59:59PM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Fri, Jan 23, 2015 at 01:51:28PM -0600, Felipe Balbi escreveu:
> > > I'm using v3.19-rc5 on an ARM Cortex A9 board. Unfortunately, perf is
> > > not able to capture any stack traces even though I CONFIG_STACKTRACE,
> > > CONFIG_FRAME_POINTER and CONFIG_ARM_UNWIND all enabled.
> > > 
> > > I wonder if anybody else is facing similar issues.
> > 
> > Did it work with a previous kernel?
> 
> haven't checked with this board. First time I tried to use perf for
> anything.

And it is the first time I've heard about such a problem, not dealing
with ARM myself, so I guess this is a case for looking if there was some
previous kernel where this worked, then trying to bisect where the
problem was introduced :-\

- Arnaldo

Re: perf not capturing stack traces

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 1101 bytes --]

Hi,

On Sat, Jan 24, 2015 at 12:12:04PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Fri, Jan 23, 2015 at 04:37:45PM -0600, Felipe Balbi escreveu:
> > On Fri, Jan 23, 2015 at 05:59:59PM -0300, Arnaldo Carvalho de Melo wrote:
> > > Em Fri, Jan 23, 2015 at 01:51:28PM -0600, Felipe Balbi escreveu:
> > > > I'm using v3.19-rc5 on an ARM Cortex A9 board. Unfortunately, perf is
> > > > not able to capture any stack traces even though I CONFIG_STACKTRACE,
> > > > CONFIG_FRAME_POINTER and CONFIG_ARM_UNWIND all enabled.
> > > > 
> > > > I wonder if anybody else is facing similar issues.
> > > 
> > > Did it work with a previous kernel?
> > 
> > haven't checked with this board. First time I tried to use perf for
> > anything.
> 
> And it is the first time I've heard about such a problem, not dealing
> with ARM myself, so I guess this is a case for looking if there was some
> previous kernel where this worked, then trying to bisect where the
> problem was introduced :-\

yeah, I'll try a few older kernels, also see if I can reproduce on other
boards.

cheers

-- 
balbi

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: perf not capturing stack traces

[Russell King - ARM Linux]

On Sat, Jan 24, 2015 at 04:23:42PM -0600, Felipe Balbi wrote:
> yeah, I'll try a few older kernels, also see if I can reproduce on other
> boards.

Perf works for me with CONFIG_FRAME_POINTER=y, but that's only for kernel
space, and for userspace where the programs have been built for ARM mode
with frame pointers.

The kernel may work without CONFIG_FRAME_POINTER set, but I've never
tested that, and I'd suggest that (given my experience looking at oops
dumps) it's not all that reliable.

Lastly, userspace without frame pointers is pretty much hopeless.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Re: perf not capturing stack traces

[Will Deacon]

On Sun, Jan 25, 2015 at 03:56:52PM +0000, Russell King - ARM Linux wrote:
> On Sat, Jan 24, 2015 at 04:23:42PM -0600, Felipe Balbi wrote:
> > yeah, I'll try a few older kernels, also see if I can reproduce on other
> > boards.
> 
> Perf works for me with CONFIG_FRAME_POINTER=y, but that's only for kernel
> space, and for userspace where the programs have been built for ARM mode
> with frame pointers.
> 
> The kernel may work without CONFIG_FRAME_POINTER set, but I've never
> tested that, and I'd suggest that (given my experience looking at oops
> dumps) it's not all that reliable.
> 
> Lastly, userspace without frame pointers is pretty much hopeless.

FWIW, perf can now use libunwind for unwinding the userspace side of
things, so it's not quite as bad as it used to be. For the kernel side,
if the unwinder isn't working properly it would be nice to know *why*,
but I agree that it tends to be far flakier than the frame-pointer method.

Will

Re: perf not capturing stack traces

[Arnaldo Carvalho de Melo]

Em Mon, Jan 26, 2015 at 10:27:11AM +0000, Will Deacon escreveu:
> On Sun, Jan 25, 2015 at 03:56:52PM +0000, Russell King - ARM Linux wrote:
> > On Sat, Jan 24, 2015 at 04:23:42PM -0600, Felipe Balbi wrote:
> > > yeah, I'll try a few older kernels, also see if I can reproduce on other
> > > boards.
> > 
> > Perf works for me with CONFIG_FRAME_POINTER=y, but that's only for kernel
> > space, and for userspace where the programs have been built for ARM mode
> > with frame pointers.
> > 
> > The kernel may work without CONFIG_FRAME_POINTER set, but I've never
> > tested that, and I'd suggest that (given my experience looking at oops
> > dumps) it's not all that reliable.
> > 
> > Lastly, userspace without frame pointers is pretty much hopeless.
> 
> FWIW, perf can now use libunwind for unwinding the userspace side of
> things, so it's not quite as bad as it used to be. For the kernel side,
> if the unwinder isn't working properly it would be nice to know *why*,
> but I agree that it tends to be far flakier than the frame-pointer method.

Any idea why, with userspace using frame pointers, perf doesn't go all
the way from kernel to userspace main() (or whatever is the endpoint),
as Russel stated?

- Arnaldo

Re: perf not capturing stack traces

[Will Deacon]

On Mon, Jan 26, 2015 at 12:12:43PM +0000, Arnaldo Carvalho de Melo wrote:
> Em Mon, Jan 26, 2015 at 10:27:11AM +0000, Will Deacon escreveu:
> > On Sun, Jan 25, 2015 at 03:56:52PM +0000, Russell King - ARM Linux wrote:
> > > On Sat, Jan 24, 2015 at 04:23:42PM -0600, Felipe Balbi wrote:
> > > > yeah, I'll try a few older kernels, also see if I can reproduce on other
> > > > boards.
> > > 
> > > Perf works for me with CONFIG_FRAME_POINTER=y, but that's only for kernel
> > > space, and for userspace where the programs have been built for ARM mode
> > > with frame pointers.
> > > 
> > > The kernel may work without CONFIG_FRAME_POINTER set, but I've never
> > > tested that, and I'd suggest that (given my experience looking at oops
> > > dumps) it's not all that reliable.
> > > 
> > > Lastly, userspace without frame pointers is pretty much hopeless.
> > 
> > FWIW, perf can now use libunwind for unwinding the userspace side of
> > things, so it's not quite as bad as it used to be. For the kernel side,
> > if the unwinder isn't working properly it would be nice to know *why*,
> > but I agree that it tends to be far flakier than the frame-pointer method.
> 
> Any idea why, with userspace using frame pointers, perf doesn't go all
> the way from kernel to userspace main() (or whatever is the endpoint),
> as Russel stated?

Did he state that? I thought he was just saying that he couldn't unwind
userspace when *not* using frame pointers, which requires you to link
a recent perf with a bleeding-edge libunwind.

Will

Re: perf not capturing stack traces

[Arnaldo Carvalho de Melo]

Em Mon, Jan 26, 2015 at 12:16:21PM +0000, Will Deacon escreveu:
> On Mon, Jan 26, 2015 at 12:12:43PM +0000, Arnaldo Carvalho de Melo wrote:
> > Em Mon, Jan 26, 2015 at 10:27:11AM +0000, Will Deacon escreveu:
> > > On Sun, Jan 25, 2015 at 03:56:52PM +0000, Russell King - ARM Linux wrote:
> > > > On Sat, Jan 24, 2015 at 04:23:42PM -0600, Felipe Balbi wrote:
> > > > > yeah, I'll try a few older kernels, also see if I can reproduce on other
> > > > > boards.
> > > > 
> > > > Perf works for me with CONFIG_FRAME_POINTER=y, but that's only for kernel
> > > > space, and for userspace where the programs have been built for ARM mode
> > > > with frame pointers.
> > > > 
> > > > The kernel may work without CONFIG_FRAME_POINTER set, but I've never
> > > > tested that, and I'd suggest that (given my experience looking at oops
> > > > dumps) it's not all that reliable.
> > > > 
> > > > Lastly, userspace without frame pointers is pretty much hopeless.
> > > 
> > > FWIW, perf can now use libunwind for unwinding the userspace side of
> > > things, so it's not quite as bad as it used to be. For the kernel side,
> > > if the unwinder isn't working properly it would be nice to know *why*,
> > > but I agree that it tends to be far flakier than the frame-pointer method.
> > 
> > Any idea why, with userspace using frame pointers, perf doesn't go all
> > the way from kernel to userspace main() (or whatever is the endpoint),
> > as Russel stated?
> 
> Did he state that? I thought he was just saying that he couldn't unwind
> userspace when *not* using frame pointers, which requires you to link

sorry, yeah I misunderstood, so all is good for kernel + userspace when
both are built with frame pointers :-)

"The only for kernel space" caught my attention, I thought that he said
that it works only for the kernel with frame pointers, even with a
userspace built with fp.

- Arnaldo

> a recent perf with a bleeding-edge libunwind.
> 
> Will

Re: perf not capturing stack traces

[Russell King - ARM Linux]

On Mon, Jan 26, 2015 at 09:12:43AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Mon, Jan 26, 2015 at 10:27:11AM +0000, Will Deacon escreveu:
> > On Sun, Jan 25, 2015 at 03:56:52PM +0000, Russell King - ARM Linux wrote:
> > > On Sat, Jan 24, 2015 at 04:23:42PM -0600, Felipe Balbi wrote:
> > > > yeah, I'll try a few older kernels, also see if I can reproduce on other
> > > > boards.
> > > 
> > > Perf works for me with CONFIG_FRAME_POINTER=y, but that's only for kernel
> > > space, and for userspace where the programs have been built for ARM mode
> > > with frame pointers.
> > > 
> > > The kernel may work without CONFIG_FRAME_POINTER set, but I've never
> > > tested that, and I'd suggest that (given my experience looking at oops
> > > dumps) it's not all that reliable.
> > > 
> > > Lastly, userspace without frame pointers is pretty much hopeless.
> > 
> > FWIW, perf can now use libunwind for unwinding the userspace side of
> > things, so it's not quite as bad as it used to be. For the kernel side,
> > if the unwinder isn't working properly it would be nice to know *why*,
> > but I agree that it tends to be far flakier than the frame-pointer method.
> 
> Any idea why, with userspace using frame pointers, perf doesn't go all
> the way from kernel to userspace main() (or whatever is the endpoint),
> as Russel stated?
           ^ *growl*

I've rebuilt userspace code which I've been working on in with a bunch of
flags which makes it use frame pointers in ARM mode, and perf does seem
to be capable of that; in that case, perf_callchain_user() can walk the
linked set of frames.

However, if glibc is built for thumb2 or doesn't contain frame pointers,
userspace tracing pretty much stops after you hit the first function in
userspace.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Re: perf not capturing stack traces

[Arnaldo Carvalho de Melo]

Em Mon, Jan 26, 2015 at 01:54:06PM +0000, Russell King - ARM Linux escreveu:
> On Mon, Jan 26, 2015 at 09:12:43AM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Mon, Jan 26, 2015 at 10:27:11AM +0000, Will Deacon escreveu:
> > > FWIW, perf can now use libunwind for unwinding the userspace side of
> > > things, so it's not quite as bad as it used to be. For the kernel side,
> > > if the unwinder isn't working properly it would be nice to know *why*,
> > > but I agree that it tends to be far flakier than the frame-pointer method.

> > Any idea why, with userspace using frame pointers, perf doesn't go all
> > the way from kernel to userspace main() (or whatever is the endpoint),
> > as Russel stated?
>            ^ *growl*

I misunderstood, as corrected on another message, sorry.
 
> I've rebuilt userspace code which I've been working on in with a bunch of
> flags which makes it use frame pointers in ARM mode, and perf does seem
> to be capable of that; in that case, perf_callchain_user() can walk the
> linked set of frames.
> 
> However, if glibc is built for thumb2 or doesn't contain frame pointers,
> userspace tracing pretty much stops after you hit the first function in
> userspace.

Right.

- Arnaldo

Re: perf not capturing stack traces

[Russell King - ARM Linux]

On Mon, Jan 26, 2015 at 10:27:11AM +0000, Will Deacon wrote:
> On Sun, Jan 25, 2015 at 03:56:52PM +0000, Russell King - ARM Linux wrote:
> > On Sat, Jan 24, 2015 at 04:23:42PM -0600, Felipe Balbi wrote:
> > > yeah, I'll try a few older kernels, also see if I can reproduce on other
> > > boards.
> > 
> > Perf works for me with CONFIG_FRAME_POINTER=y, but that's only for kernel
> > space, and for userspace where the programs have been built for ARM mode
> > with frame pointers.
> > 
> > The kernel may work without CONFIG_FRAME_POINTER set, but I've never
> > tested that, and I'd suggest that (given my experience looking at oops
> > dumps) it's not all that reliable.
> > 
> > Lastly, userspace without frame pointers is pretty much hopeless.
> 
> FWIW, perf can now use libunwind for unwinding the userspace side of
> things, so it's not quite as bad as it used to be. For the kernel side,
> if the unwinder isn't working properly it would be nice to know *why*,
> but I agree that it tends to be far flakier than the frame-pointer method.

I don't see how userspace could be unwound without capturing the entire
userspace stack on every perf event - and that could be a considerable
size.  We have no way to know within the kernel which words on the
userspace stack are part of the callchain and which aren't - the only
way we'd know is by loading the userspace's unwind tables, having the
kernel parsing them and generate a list of functions.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Re: perf not capturing stack traces

[Arnaldo Carvalho de Melo]

Em Mon, Jan 26, 2015 at 01:51:23PM +0000, Russell King - ARM Linux escreveu:
> On Mon, Jan 26, 2015 at 10:27:11AM +0000, Will Deacon wrote:
> > On Sun, Jan 25, 2015 at 03:56:52PM +0000, Russell King - ARM Linux wrote:
> > > On Sat, Jan 24, 2015 at 04:23:42PM -0600, Felipe Balbi wrote:
> > > > yeah, I'll try a few older kernels, also see if I can reproduce on other
> > > > boards.

> > > Perf works for me with CONFIG_FRAME_POINTER=y, but that's only for kernel
> > > space, and for userspace where the programs have been built for ARM mode
> > > with frame pointers.

> > > The kernel may work without CONFIG_FRAME_POINTER set, but I've never
> > > tested that, and I'd suggest that (given my experience looking at oops
> > > dumps) it's not all that reliable.

> > > Lastly, userspace without frame pointers is pretty much hopeless.

> > FWIW, perf can now use libunwind for unwinding the userspace side of
> > things, so it's not quite as bad as it used to be. For the kernel side,
> > if the unwinder isn't working properly it would be nice to know *why*,
> > but I agree that it tends to be far flakier than the frame-pointer method.

> I don't see how userspace could be unwound without capturing the entire
> userspace stack on every perf event - and that could be a considerable
> size.  We have no way to know within the kernel which words on the

That is what you can do using 'perf record --call-graph dwarf':

    -g               enables call-graph recording
        --call-graph <mode[,dump_size]>
                     setup and enables call-graph (stack chain/backtrace)
                     recording: fp dwarf

That will use:

        PERF_SAMPLE_REGS_USER                   = 1U << 12,
        PERF_SAMPLE_STACK_USER                  = 1U << 13,

struct perf_event_attr {
<SNIP>
        /*
         * Defines set of user regs to dump on samples.
         * See asm/perf_regs.h for details.
         */
        __u64   sample_regs_user;

        /*
         * Defines size of the user stack to dump on samples.
         */
        __u32   sample_stack_user;
<SNIP>
}

> userspace stack are part of the callchain and which aren't - the only
> way we'd know is by loading the userspace's unwind tables, having the
> kernel parsing them and generate a list of functions.

Or deferring it to userspace to do that later.

- Arnaldo