Re: Re: [PATCH perf/core 1/2] [BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance

[Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>]

(2015/01/30 23:21), Arnaldo Carvalho de Melo wrote:
> Em Fri, Jan 30, 2015 at 06:37:44PM +0900, Masami Hiramatsu escreveu:
>> Fix to handle optimized no-inline functions which have
>> only function definition but no actual instance at
>> that point. To fix this problem, we need to find actual
>> instance of the function.
>>
>> Without this patch:
>>   ----
>>   # perf probe -a __up
>>   Failed to get entry address of __up.
>>     Error: Failed to add events.
>>   # perf probe -L __up
>>   Specified source line is not found.
>>     Error: Failed to show lines.
>>   ----
>>
>> With this patch:
>>   ----
>>   # perf probe -a __up
>>   Added new event:
>>     probe:__up           (on __up)
>>
>>   You can now use it in all perf tools, such as:
>>
>>           perf record -e probe:__up -aR sleep 1
>>
>>   # perf probe -L __up
>>   <__up@/home/fedora/ksrc/linux-3/kernel/locking/semaphore.c:0>
>>         0  static noinline void __sched __up(struct semaphore *sem)
>>            {
>>                   struct semaphore_waiter *waiter = list_first_entry(&sem->wait_
>>                                                           struct semaphore_waite
>>         4         list_del(&waiter->list);
>>         5         waiter->up = true;
>>         6         wake_up_process(waiter->task);
>>         7  }
>>   ----
> 
> Since __up here was built in some other way, I looked for another
> 'noinline' function to try, and it failed here:
> 
>   [root@zoo ~]# perf probe -L vmalloc_fault | head -5
>   <vmalloc_fault@/home/git/linux/arch/x86/mm/fault.c:0>
>         0  static noinline int vmalloc_fault(unsigned long address)
>         1  {
>            	pgd_t *pgd, *pgd_ref;
>            	pud_t *pud, *pud_ref;
>   [root@zoo ~]# perf probe vmalloc_fault
>   Added new event:
>   Failed to write event: Invalid argument
>     Error: Failed to add events.
> 
> ---->>>>>> APPLY THE PATCH, rebuild and try again:
> 
>   [root@zoo ~]# perf probe vmalloc_fault
>   Added new event:
>   Failed to write event: Invalid argument
>     Error: Failed to add events.
>   [root@zoo ~]# perf probe -v vmalloc_fault
>   probe-definition(0): vmalloc_fault 
>   symbol:vmalloc_fault file:(null) line:0 offset:0 return:0 lazy:(null)
>   0 arguments
>   Looking at the vmlinux_path (7 entries long)
>   Using /lib/modules/3.19.0-rc6+/build/vmlinux for symbols
>   Open Debuginfo file: /lib/modules/3.19.0-rc6+/build/vmlinux
>   Try to find probe point from debuginfo.
>   Probe point found: vmalloc_fault+0
>   Found 1 probe_trace_events.
>   Opening /sys/kernel/debug/tracing/kprobe_events write=1
>   Added new event:
>   Writing event: p:probe/vmalloc_fault _text+289600
>   Failed to write event: Invalid argument
>     Error: Failed to add events. Reason: Invalid argument (Code: -22)
>   [root@zoo ~]# grep -w vmalloc_fault /proc/kallsyms
>   ffffffff81046b40 t vmalloc_fault
>   [root@zoo ~]# grep -w _text /proc/kallsyms 
>   ffffffff81000000 T _text
>   [root@zoo ~]# python 
>   Python 2.7.5 (default, Nov  3 2014, 14:26:24) 
>   [GCC 4.8.3 20140911 (Red Hat 4.8.3-7)] on linux2
>   Type "help", "copyright", "credits" or "license" for more information.
>   >>> 0xffffffff81046b40 - 0xffffffff81000000
>   289600L
>   >>> 
>   [root@zoo ~]# readelf -s /lib/modules/3.19.0-rc6+/build/vmlinux | egrep -w _text\|vmalloc_fault
>     3499: ffffffff81046b40   410 FUNC    LOCAL  DEFAULT    1 vmalloc_fault
>    48873: ffffffff81000000     0 NOTYPE  GLOBAL DEFAULT    1 _text
>   [root@zoo ~]# 
> 
> ---------------------------------------------------------------------------
> 
> So perhaps I need a 'noinline' _and_ '__sched', so that I have a "optimized
> no-inline functions which have only function definition but no actual instance
> at that point"?

Ok, actually vmalloc_fault is marked as a nokprobe symbol.

$ grep vmalloc_fault -r arch/x86/
arch/x86/mm/fault.c:static noinline int vmalloc_fault(unsigned long address)
arch/x86/mm/fault.c:NOKPROBE_SYMBOL(vmalloc_fault);
arch/x86/mm/fault.c:static noinline int vmalloc_fault(unsigned long address)
arch/x86/mm/fault.c:NOKPROBE_SYMBOL(vmalloc_fault);
arch/x86/mm/fault.c:                    if (vmalloc_fault(address) >= 0)

All the symbols which marked by NOKPROBE_SYMBOL macro can not be
kprobed. And we now have <debugfs>/kprobes/blacklist special file
to get all nokprobe symbols.
I'll make another perf-probe patch which looks the blacklist before
trying define new events.

Thank you,

> 
> Looking for one, will post here after objdump -dS finishes...
> 
> - Arnaldo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com