Re: Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

[Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>]

(2015/03/03 12:05), Arnaldo Carvalho de Melo wrote:
> Em Mon, Mar 02, 2015 at 11:45:12PM -0300, Arnaldo Carvalho de Melo escreveu:
>> Em Tue, Mar 03, 2015 at 11:39:02AM +0900, Masami Hiramatsu escreveu:
>>> (2015/03/03 0:46), Arnaldo Carvalho de Melo wrote:
>>>> Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu:
>>>>> With this patch;
>>>>>   -----
>>>>>   # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
>>>>>   Available variables at malloc
>>>>>           @<__libc_malloc+0>
>>>>>                   size_t  bytes
>>>>>   # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
>>>>>   Added new event:
>>>>>     probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)
>>>>>
>>>>>   You can now use it in all perf tools, such as:
>>>>>
>>>>>           perf record -e probe_libc:malloc -aR sleep 1
>>>>
>>>>> Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
>>>>
>>>> Humm, not working for me, after the patch:
>>>>
>>>> [root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc
>>>> Available variables at malloc
>>>>         @<__malloc_check_init+96>
>>>>                 (No matched variables)
>>>
>>
>> Will try after a 'make build-test' finishes for the current batch
> 

Thank you for checking this,

> [root@ssdandy ~]# perf probe -vvv -x /usr/lib64/libc-2.17.so -V malloc
> probe-definition(0): malloc
> symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null)
> 0 arguments
> Open Debuginfo file: /usr/lib/debug/usr/lib64/libc-2.17.so.debug
> Searching variables at malloc
> Symbol malloc address found : 800c0
> Get 2611 lines from this CU

Hmm, something wrong at here.

> Probe point found: __malloc_check_init+96

This seems that the debuginfo is a bit odd.
Could you also run eu-addr2line for 0x800c0 and 0x800b0?

I'm using CentOS7 with a bit newer glibc rpms.

# rpm -q glibc glibc-debuginfo
glibc-2.17-55.el7_0.5.x86_64
glibc-debuginfo-2.17-55.el7_0.5.x86_64

And I got following results.

[mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800c0
__libc_malloc
/usr/src/debug/glibc-2.17-c758a686/malloc/malloc.c:2855
[mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800b0
__malloc_check_init
/usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:75

So, 0x800b0 correctly points __malloc_check_init+96 but its address is not 0x800c0.

Thank you,


> Available variables at malloc
>         @<__malloc_check_init+96>
>                 (No matched variables)
> [root@ssdandy ~]#
> 
> If I add one more 'v' I get the symtabs as read by symbol-elf.c and this is
> what is there for the malloc routines:

Yeah, it seems symbol maps works fine. Debuginfo analysis failed.

> 
> [root@ssdandy ~]# grep malloc /tmp/4
> probe-definition(0): malloc
> symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null)
> Searching variables at malloc
> symbol__new: ptmalloc_lock_all 0x7b060-0x7b173
> symbol__new: malloc_atfork 0x801c0-0x802dd
> symbol__new: ptmalloc_unlock_all2 0x7b180-0x7b21e
> symbol__new: ptmalloc_unlock_all 0x7ba90-0x7bb3e
> symbol__new: malloc_printerr 0x7bb40-0x7bc28
> symbol__new: malloc_consolidate 0x7c3d0-0x7c9be
> symbol__new: _int_malloc 0x7dee0-0x7f32b
> symbol__new: malloc_check 0x7f330-0x7f44a
> symbol__new: ptmalloc_init.part.8 0x80fe0-0x813d7
> symbol__new: ptmalloc_init 0x813e0-0x813f5
> symbol__new: malloc_hook_ini 0x81400-0x8143c
> symbol__new: mallochook 0x82fc0-0x831af
> symbol__new: tr_mallochook 0x83fc0-0x8407d
> symbol__new: __malloc_set_state 0x814f0-0x819f6
> symbol__new: __malloc_usable_size 0x80d30-0x80e0e
> symbol__new: __malloc_trim 0x81d50-0x81fac
> symbol__new: __GI___libc_malloc 0x800c0-0x801b5
> symbol__new: __malloc_get_state 0x802e0-0x804db
> symbol__new: __malloc_stats 0x820c0-0x822aa
> symbol__new: __malloc_check_init 0x80050-0x800bb
> symbol__new: __malloc 0x800c0-0x801b5
> symbol__new: malloc_set_state 0x814f0-0x819f6
> symbol__new: malloc 0x800c0-0x801b5
> symbol__new: malloc_info 0x82320-0x8240f
> symbol__new: malloc_trim 0x81d50-0x81fac
> symbol__new: malloc_usable_size 0x80d30-0x80e0e
> symbol__new: malloc_get_state 0x802e0-0x804db
> symbol__new: __libc_malloc 0x800c0-0x801b5
> symbol__new: malloc_stats 0x820c0-0x822aa
> symbol__new: malloc@plt 0x1f300-0x1f310
> Symbol malloc address found : 800c0
> Probe point found: __malloc_check_init+96
> [root@ssdandy ~]#
>  
>>>> [root@ssdandy ~]#
>>>>
>>>> And then the one asking for 'bytes' to be collectd fails.
>>>>
>>>> After processing the other patches I'll try to debug this...
>>>>
>>>> [root@ssdandy ~]# cat /etc/redhat-release 
>>>> Red Hat Enterprise Linux Server release 7.0 (Maipo)
>>>> [root@ssdandy ~]# rpm -q glibc glibc-debuginfo
>>>> glibc-2.17-55.el7_0.3.x86_64
>>>> glibc-debuginfo-2.17-55.el7_0.1.x86_64
>>>                               ^^^ why is this different from the glibc version??
>>>
>>>> [root@ssdandy ~]#
>>>> [acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc
>>> [...]
>>>>   4849: 0000000000080050   107 FUNC    LOCAL  DEFAULT   12 __malloc_check_init
>>>
>>> __malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0
>>>
>>>>   5351: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __malloc
>>>>   1170: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc@@GLIBC_2.2.5
>>>
>>> that is not 0x800c0, so something goes wrong when translating the address
>>> to probe point. Could you check the patch 1/4 was applied?
>>> Actually there is a bug in the routine which gets the probe point from
>>> address. 1/4 fixes it.
>>
>> This one?
>>
>> commit 0104fe69e0287cf3635657b4c6b26a18e0091697
>> Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
>> Date:   Mon Mar 2 21:49:46 2015 +0900
>>
>>     perf probe: Remove bias offset to find probe point by address
>>     
>>     Remove bias offset to find probe point by address.
>>
>> ----------------
>>
>> Yes, it is applied.
>>
>> - Arnaldo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com