[Questions] How to run 'perf top' on ARM to profile kernel functions with modules loaded

[Questions] How to run 'perf top' on ARM to profile kernel functions with modules loaded

[Ming Lei]

Hi,

'perf top' can't display any functions when modules are loaded on ARM.

But if all modules are unloaded, everything is OK and kernel functions can be
displayed.

So how can I make 'perf top' to profile kernel functions when modules are loaded
on ARM?

thanks,
-- 
Lei Ming

[Questions] How to run 'perf top' on ARM to profile kernel functions with modules loaded

[Ming Lei]

2010/11/11 Ming Lei <tom.leiming@gmail.com>:
> Hi,
>
> 'perf top' can't display any functions when modules are loaded on ARM.

Sorry, should be can't display any kernel functions when modules are loaded
on ARM.

Also, when modules are loaded, 'perf report' can't display functions names,
follows result of 'perf report':

root at beagleboard:~# perf report
# Events: 4K cycles
#
# Overhead      Command      Shared Object                  Symbol
# ........  ...........  .................  ......................
#
    97.36%      swapper  [musb_hdrc]        [k]  10466c4
     2.19%         perf  [musb_hdrc]        [k]  10da070
     0.09%  ksoftirqd/0  [musb_hdrc]        [k]  10db998
     0.06%         perf  libc-2.9.so        [.] memchr
     0.04%         perf  libc-2.9.so        [.]    663c0
     0.02%         perf  perf               [.]     2e80
     0.02%         perf  libc-2.9.so        [.] closedir
     0.02%         perf  libc-2.9.so        [.] __read
     0.02%  flush-179:0  [musb_hdrc]        [k]  10ff124
     0.02%         perf  libc-2.9.so        [.] memset
     0.02%         perf  libc-2.9.so        [.] __memcpy_chk
     0.02%         perf  perf               [.] event__synthesize_comm
     0.02%         perf           ffff0fc4  [.] ffff0fc4
     0.02%         perf  perf               [.] hex2u64
     0.02%         perf  libc-2.9.so        [.] _IO_vfprintf
     0.02%         perf  libc-2.9.so        [.] fgets
     0.02%         perf  libpthread-2.9.so  [.] flockfile
     0.02%         perf  perf               [.] kallsyms__parse
     0.02%         perf  libc-2.9.so        [.] _IO_getline_info

Any suggestions or comments?

thanks,
-- 
Lei Ming

[Questions] How to run 'perf top' on ARM to profile kernel functions with modules loaded

[Ingo Molnar]

* Ming Lei <tom.leiming@gmail.com> wrote:

> 2010/11/11 Ming Lei <tom.leiming@gmail.com>:
> > Hi,
> >
> > 'perf top' can't display any functions when modules are loaded on ARM.
> 
> Sorry, should be can't display any kernel functions when modules are loaded
> on ARM.
> 
> Also, when modules are loaded, 'perf report' can't display functions names,
> follows result of 'perf report':
> 
> root at beagleboard:~# perf report
> # Events: 4K cycles
> #
> # Overhead      Command      Shared Object                  Symbol
> # ........  ...........  .................  ......................
> #
>     97.36%      swapper  [musb_hdrc]        [k]  10466c4
>      2.19%         perf  [musb_hdrc]        [k]  10da070
>      0.09%  ksoftirqd/0  [musb_hdrc]        [k]  10db998
>      0.06%         perf  libc-2.9.so        [.] memchr
>      0.04%         perf  libc-2.9.so        [.]    663c0
>      0.02%         perf  perf               [.]     2e80
>      0.02%         perf  libc-2.9.so        [.] closedir
>      0.02%         perf  libc-2.9.so        [.] __read
>      0.02%  flush-179:0  [musb_hdrc]        [k]  10ff124
>      0.02%         perf  libc-2.9.so        [.] memset
>      0.02%         perf  libc-2.9.so        [.] __memcpy_chk
>      0.02%         perf  perf               [.] event__synthesize_comm
>      0.02%         perf           ffff0fc4  [.] ffff0fc4
>      0.02%         perf  perf               [.] hex2u64
>      0.02%         perf  libc-2.9.so        [.] _IO_vfprintf
>      0.02%         perf  libc-2.9.so        [.] fgets
>      0.02%         perf  libpthread-2.9.so  [.] flockfile
>      0.02%         perf  perf               [.] kallsyms__parse
>      0.02%         perf  libc-2.9.so        [.] _IO_getline_info
> 
> Any suggestions or comments?

Must be some bug in the ELF symbol handling code - we've had many cases of quirks 
and special cases there. Please run 'perf archive' and send the resulting 
perf.data.tar.bz2 to Arnaldo in private mail. (the file will be big)

Thanks,

	Ingo

[Questions] How to run 'perf top' on ARM to profile kernel functions with modules loaded

[Kyungmin Park]

On Thu, Nov 11, 2010 at 4:52 PM, Ming Lei <tom.leiming@gmail.com> wrote:
> 2010/11/11 Ming Lei <tom.leiming@gmail.com>:
>> Hi,
>>
>> 'perf top' can't display any functions when modules are loaded on ARM.
>
> Sorry, should be can't display any kernel functions when modules are loaded
> on ARM.
>
> Also, when modules are loaded, 'perf report' can't display functions names,
> follows result of 'perf report':
>

Hi,

I tried to build the perf as static and build success on ARM. But
there's no display and no top results also.
How do you build the perf on ARM? any hints or clues?

Thank you,
Kyungmin Park
> root at beagleboard:~# perf report
> # Events: 4K cycles
> #
> # Overhead ? ? ?Command ? ? ?Shared Object ? ? ? ? ? ? ? ? ?Symbol
> # ........ ?........... ?................. ?......................
> #
> ? ?97.36% ? ? ?swapper ?[musb_hdrc] ? ? ? ?[k] ?10466c4
> ? ? 2.19% ? ? ? ? perf ?[musb_hdrc] ? ? ? ?[k] ?10da070
> ? ? 0.09% ?ksoftirqd/0 ?[musb_hdrc] ? ? ? ?[k] ?10db998
> ? ? 0.06% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] memchr
> ? ? 0.04% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] ? ?663c0
> ? ? 0.02% ? ? ? ? perf ?perf ? ? ? ? ? ? ? [.] ? ? 2e80
> ? ? 0.02% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] closedir
> ? ? 0.02% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] __read
> ? ? 0.02% ?flush-179:0 ?[musb_hdrc] ? ? ? ?[k] ?10ff124
> ? ? 0.02% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] memset
> ? ? 0.02% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] __memcpy_chk
> ? ? 0.02% ? ? ? ? perf ?perf ? ? ? ? ? ? ? [.] event__synthesize_comm
> ? ? 0.02% ? ? ? ? perf ? ? ? ? ? ffff0fc4 ?[.] ffff0fc4
> ? ? 0.02% ? ? ? ? perf ?perf ? ? ? ? ? ? ? [.] hex2u64
> ? ? 0.02% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] _IO_vfprintf
> ? ? 0.02% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] fgets
> ? ? 0.02% ? ? ? ? perf ?libpthread-2.9.so ?[.] flockfile
> ? ? 0.02% ? ? ? ? perf ?perf ? ? ? ? ? ? ? [.] kallsyms__parse
> ? ? 0.02% ? ? ? ? perf ?libc-2.9.so ? ? ? ?[.] _IO_getline_info
>
> Any suggestions or comments?
>
> thanks,
> --
> Lei Ming
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>

[Questions] How to run 'perf top' on ARM to profile kernel functions with modules loaded

[Ming Lei]

>2010/11/11 Ingo Molnar <mingo@elte.hu>:
> Must be some bug in the ELF symbol handling code - we've had many cases of quirks
> and special cases there. Please run 'perf archive' and send the resulting
> perf.data.tar.bz2 to Arnaldo in private mail. (the file will be big)

Sent to Arnaldo already.

2010/11/11 Kyungmin Park <kmpark@infradead.org>:
>
> I tried to build the perf as static and build success on ARM. But
> there's no display and no top results also.
> How do you build the perf on ARM? any hints or clues?

Cross-compiling perf tool is very painful, so I choose to compile
perf in Angstom on beagle-xm(ARM) directly, isn't it fun? :-)

thanks,
-- 
Lei Ming

[Questions] How to run 'perf top' on ARM to profile kernel functions with modules loaded

[Jamie Iles]

On Thu, Nov 11, 2010 at 03:52:07PM +0800, Ming Lei wrote:
> 2010/11/11 Ming Lei <tom.leiming@gmail.com>:
> > Hi,
> >
> > 'perf top' can't display any functions when modules are loaded on ARM.
> 
> Sorry, should be can't display any kernel functions when modules are loaded
> on ARM.
I've noticed this myself and I spent a bit of time looking into it last week 
but didn't get too far. I did wonder if it was something to do with the map 
fixups after loading the modules. If you look at the raw events after a perf 
record with 'perf report -D' then the last module has a size that takes it to 
0xFFFFFFFF which overlaps with the rest of the kernel symbols.

On x86 it looks like the modules are loaded after the kernel image in VM 
unlike ARM.

Jamie

[Questions] How to run 'perf top' on ARM to profile kernel functions with modules loaded

[Peter Zijlstra]

On Thu, 2010-11-11 at 09:17 +0000, Jamie Iles wrote:
> On Thu, Nov 11, 2010 at 03:52:07PM +0800, Ming Lei wrote:
> > 2010/11/11 Ming Lei <tom.leiming@gmail.com>:
> > > Hi,
> > >
> > > 'perf top' can't display any functions when modules are loaded on ARM.
> > 
> > Sorry, should be can't display any kernel functions when modules are loaded
> > on ARM.
> I've noticed this myself and I spent a bit of time looking into it last week 
> but didn't get too far. I did wonder if it was something to do with the map 
> fixups after loading the modules. If you look at the raw events after a perf 
> record with 'perf report -D' then the last module has a size that takes it to 
> 0xFFFFFFFF which overlaps with the rest of the kernel symbols.
> 
> On x86 it looks like the modules are loaded after the kernel image in VM 
> unlike ARM.

Order shouldn't matter, but that overlap is very likely what kills it.
That module size is buggy.

[Questions] How to run 'perf top' on ARM to profile kernel functions with modules loaded

[Jamie Iles]

On Thu, Nov 11, 2010 at 01:06:16PM +0100, Peter Zijlstra wrote:
> On Thu, 2010-11-11 at 09:17 +0000, Jamie Iles wrote:
> > On Thu, Nov 11, 2010 at 03:52:07PM +0800, Ming Lei wrote:
> > > 2010/11/11 Ming Lei <tom.leiming@gmail.com>:
> > > > Hi,
> > > >
> > > > 'perf top' can't display any functions when modules are loaded on ARM.
> > > 
> > > Sorry, should be can't display any kernel functions when modules are loaded
> > > on ARM.
> > I've noticed this myself and I spent a bit of time looking into it last week 
> > but didn't get too far. I did wonder if it was something to do with the map 
> > fixups after loading the modules. If you look at the raw events after a perf 
> > record with 'perf report -D' then the last module has a size that takes it to 
> > 0xFFFFFFFF which overlaps with the rest of the kernel symbols.
> > 
> > On x86 it looks like the modules are loaded after the kernel image in VM 
> > unlike ARM.
> 
> Order shouldn't matter, but that overlap is very likely what kills it.
> That module size is buggy.
I'm guessing that its in util/symbol.c:__map_groups__fixup_end():

	for (nd = rb_next(prevnd); nd; nd = rb_next(nd)) {
                prev = curr;
                curr = rb_entry(nd, struct map, rb_node);
                prev->end = curr->start - 1;
        }

        /*
         * We still haven't the actual symbols, so guess the
         * last map final address.
         */
        curr->end = ~0UL;

I've noticed that if I change the symbol_conf.{use_modules,try_vmlinux_path} 
then I can get perf events for kernel symbols from /proc/kallsyms but nothing 
for the modules.

Jamie

[Questions] How to run 'perf top' on ARM to profile kernel functions with modules loaded

[Peter Zijlstra]

On Thu, 2010-11-11 at 12:21 +0000, Jamie Iles wrote:
> On Thu, Nov 11, 2010 at 01:06:16PM +0100, Peter Zijlstra wrote:
> > On Thu, 2010-11-11 at 09:17 +0000, Jamie Iles wrote:
> > > On Thu, Nov 11, 2010 at 03:52:07PM +0800, Ming Lei wrote:
> > > > 2010/11/11 Ming Lei <tom.leiming@gmail.com>:
> > > > > Hi,
> > > > >
> > > > > 'perf top' can't display any functions when modules are loaded on ARM.
> > > > 
> > > > Sorry, should be can't display any kernel functions when modules are loaded
> > > > on ARM.
> > > I've noticed this myself and I spent a bit of time looking into it last week 
> > > but didn't get too far. I did wonder if it was something to do with the map 
> > > fixups after loading the modules. If you look at the raw events after a perf 
> > > record with 'perf report -D' then the last module has a size that takes it to 
> > > 0xFFFFFFFF which overlaps with the rest of the kernel symbols.
> > > 
> > > On x86 it looks like the modules are loaded after the kernel image in VM 
> > > unlike ARM.
> > 
> > Order shouldn't matter, but that overlap is very likely what kills it.
> > That module size is buggy.
> I'm guessing that its in util/symbol.c:__map_groups__fixup_end():
> 
> 	for (nd = rb_next(prevnd); nd; nd = rb_next(nd)) {
>                 prev = curr;
>                 curr = rb_entry(nd, struct map, rb_node);
>                 prev->end = curr->start - 1;
>         }
> 
>         /*
>          * We still haven't the actual symbols, so guess the
>          * last map final address.
>          */
>         curr->end = ~0UL;
> 
> I've noticed that if I change the symbol_conf.{use_modules,try_vmlinux_path} 
> then I can get perf events for kernel symbols from /proc/kallsyms but nothing 
> for the modules.

Hrm,.. depending on how we load things this might be easy or hard to
fix. If we load the module symbols after having loaded the kernel
symbols we can replace ~0ULL with the address before the next symbol.

If however we load modules first we're in a bind and should look at
changing the load order to make the first suggestion work.

[Questions] How to run 'perf top' on ARM to profile kernel functions with modules loaded

[Ming Lei]

2010/11/11 Peter Zijlstra <a.p.zijlstra@chello.nl>:
> On Thu, 2010-11-11 at 12:21 +0000, Jamie Iles wrote:
>> On Thu, Nov 11, 2010 at 01:06:16PM +0100, Peter Zijlstra wrote:
>> > On Thu, 2010-11-11 at 09:17 +0000, Jamie Iles wrote:
>> > > On Thu, Nov 11, 2010 at 03:52:07PM +0800, Ming Lei wrote:
>> > > > 2010/11/11 Ming Lei <tom.leiming@gmail.com>:
>> > > > > Hi,
>> > > > >
>> > > > > 'perf top' can't display any functions when modules are loaded on ARM.
>> > > >
>> > > > Sorry, should be can't display any kernel functions when modules are loaded
>> > > > on ARM.
>> > > I've noticed this myself and I spent a bit of time looking into it last week
>> > > but didn't get too far. I did wonder if it was something to do with the map
>> > > fixups after loading the modules. If you look at the raw events after a perf
>> > > record with 'perf report -D' then the last module has a size that takes it to
>> > > 0xFFFFFFFF which overlaps with the rest of the kernel symbols.
>> > >
>> > > On x86 it looks like the modules are loaded after the kernel image in VM
>> > > unlike ARM.
>> >
>> > Order shouldn't matter, but that overlap is very likely what kills it.
>> > That module size is buggy.
>> I'm guessing that its in util/symbol.c:__map_groups__fixup_end():
>>
>> ? ? ? for (nd = rb_next(prevnd); nd; nd = rb_next(nd)) {
>> ? ? ? ? ? ? ? ? prev = curr;
>> ? ? ? ? ? ? ? ? curr = rb_entry(nd, struct map, rb_node);
>> ? ? ? ? ? ? ? ? prev->end = curr->start - 1;
>> ? ? ? ? }
>>
>> ? ? ? ? /*
>> ? ? ? ? ?* We still haven't the actual symbols, so guess the
>> ? ? ? ? ?* last map final address.
>> ? ? ? ? ?*/
>> ? ? ? ? curr->end = ~0UL;
>>
>> I've noticed that if I change the symbol_conf.{use_modules,try_vmlinux_path}
>> then I can get perf events for kernel symbols from /proc/kallsyms but nothing
>> for the modules.
>
> Hrm,.. depending on how we load things this might be easy or hard to
> fix. If we load the module symbols after having loaded the kernel
> symbols we can replace ~0ULL with the address before the next symbol.

If possible, could you post a patch so that I can verify if the issue is caused
by ~0ULL?

> If however we load modules first we're in a bind and should look at
> changing the load order to make the first suggestion work.
>


thanks,
-- 
Lei Ming