perf confused by modules being loaded in between addresses in /proc/kallsyms

perf confused by modules being loaded in between addresses in /proc/kallsyms

[Michael Hudson-Doyle]

Hi,

On arm, probably since "ARM: use linker magic for vectors and vector
stubs" (although I haven't checked this), perf report tends to allocate
all kernel time to the module loaded at the highest address.

This turns out to be because the "perf_event__synthesize_modules"
generates a PERF_RECORD_MMAP for the last module thats runs from its
start to the end of the address space.

This in turn turns out be because the first symbol in /proc/kallsyms is
now at 0:

# head -n 2 /proc/kallsyms
00000000 t __vectors_start
c00081c0 T asm_do_IRQ

/This/ means that the kallsyms entry in machine->kmaps[MAP__FUNCTION] is
now the first entry rather than the last in the map, so the last module
is the last in the map and so its "end" stays as ~0ULL.

I'm told that there is no particularly good reason for __vectors_start
to be in kallsyms at 0, but in any case this seems like a bug in the
user space tool.  I notice that there is code to split the kallsyms
symbols around symbols from modules (dso__split_kallsyms) but this
doesn't seem to be called for record, only report.  Is the fix as simple
as making sure this happens for record as well?  I don't feel qualified
to answer that.

Cheers,
mwh

Re: perf confused by modules being loaded in between addresses in /proc/kallsyms

[Ming Lei]

Hi Michael,

On Wed, Oct 30, 2013 at 8:53 AM, Michael Hudson-Doyle
<michael.hudson@linaro.org> wrote:
> Hi,
>
> On arm, probably since "ARM: use linker magic for vectors and vector
> stubs" (although I haven't checked this), perf report tends to allocate
> all kernel time to the module loaded at the highest address.
>
> This turns out to be because the "perf_event__synthesize_modules"
> generates a PERF_RECORD_MMAP for the last module thats runs from its
> start to the end of the address space.
>
> This in turn turns out be because the first symbol in /proc/kallsyms is
> now at 0:
>
> # head -n 2 /proc/kallsyms
> 00000000 t __vectors_start

I think it is the root cause, so I posted the below patch to
remove these symbols from /proc/kallsyms days ago:

         http://marc.info/?l=linux-kernel&m=138297540711321&w=2

On ARM, this symbol of zero address is only for generating code, and
won't be run really by CPU, so it shouldn't be in /proc/kallsyms.

> c00081c0 T asm_do_IRQ
>
> /This/ means that the kallsyms entry in machine->kmaps[MAP__FUNCTION] is
> now the first entry rather than the last in the map, so the last module
> is the last in the map and so its "end" stays as ~0ULL.
>
> I'm told that there is no particularly good reason for __vectors_start
> to be in kallsyms at 0, but in any case this seems like a bug in the
> user space tool.  I notice that there is code to split the kallsyms

I am wondering if it is bug in perf utility, since perf may use /proc/kallsyms
to figure out the start address of symbols in kernel address space, but with
the zero address, looks not easy to figure out the real start address of
symbols inside kernel if vmlinux file is missed, then can't parse kernel
IP any more.


Thanks,
-- 
Ming Lei