Re: Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

[Arnaldo Carvalho de Melo <acme@kernel.org>]

Em Tue, Mar 03, 2015 at 01:11:17PM +0900, Masami Hiramatsu escreveu:
> (2015/03/03 12:05), Arnaldo Carvalho de Melo wrote:
> > Em Mon, Mar 02, 2015 at 11:45:12PM -0300, Arnaldo Carvalho de Melo escreveu:
> >> Em Tue, Mar 03, 2015 at 11:39:02AM +0900, Masami Hiramatsu escreveu:
> >>> (2015/03/03 0:46), Arnaldo Carvalho de Melo wrote:
> >>>> Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu:
> >>>>> With this patch;
> >>>>>   -----
> >>>>>   # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
> >>>>>   Available variables at malloc
> >>>>>           @<__libc_malloc+0>
> >>>>>                   size_t  bytes
> >>>>>   # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
> >>>>>   Added new event:
> >>>>>     probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)
> >>>>>
> >>>>>   You can now use it in all perf tools, such as:
> >>>>>
> >>>>>           perf record -e probe_libc:malloc -aR sleep 1
> >>>>
> >>>>> Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
> >>>>
> >>>> Humm, not working for me, after the patch:
> >>>>
> >>>> [root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc
> >>>> Available variables at malloc
> >>>>         @<__malloc_check_init+96>
> >>>>                 (No matched variables)
> >>>
> >>
> >> Will try after a 'make build-test' finishes for the current batch
> > 
> 
> Thank you for checking this,
> 
> > [root@ssdandy ~]# perf probe -vvv -x /usr/lib64/libc-2.17.so -V malloc
> > probe-definition(0): malloc
> > symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null)
> > 0 arguments
> > Open Debuginfo file: /usr/lib/debug/usr/lib64/libc-2.17.so.debug
> > Searching variables at malloc
> > Symbol malloc address found : 800c0
> > Get 2611 lines from this CU
> 
> Hmm, something wrong at here.
> 
> > Probe point found: __malloc_check_init+96
> 
> This seems that the debuginfo is a bit odd.
> Could you also run eu-addr2line for 0x800c0 and 0x800b0?
> 
> I'm using CentOS7 with a bit newer glibc rpms.
> 
> # rpm -q glibc glibc-debuginfo
> glibc-2.17-55.el7_0.5.x86_64
> glibc-debuginfo-2.17-55.el7_0.5.x86_64
> 
> And I got following results.
> 
> [mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800c0
> __libc_malloc
> /usr/src/debug/glibc-2.17-c758a686/malloc/malloc.c:2855
> [mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800b0
> __malloc_check_init
> /usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:75
> 
> So, 0x800b0 correctly points __malloc_check_init+96 but its address is not 0x800c0.

[acme@ssdandy linux]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800c0
malloc
??:0
[acme@ssdandy linux]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800b0
__malloc_check_init
??:0
[acme@ssdandy linux]$ 

If I do it over the debuginfo files:

[acme@ssdandy linux]$ addr2line -fi -e /usr/lib/debug/usr/lib64/libc-2.17.so.debug  0x800c0
__malloc_check_init
/usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:75
[acme@ssdandy linux]$ addr2line -fi -e /usr/lib/debug/usr/lib64/libc-2.17.so.debug  0x800b0
__malloc_check_init
/usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:82
[acme@ssdandy linux]$ 

Tomorrow I'll check for newer packages or any reports about problems
with debuginfo files, please let me know if you need any further info,

Off to bed, zzZzz

- Arnaldo
 
> Thank you,
> 
> 
> > Available variables at malloc
> >         @<__malloc_check_init+96>
> >                 (No matched variables)
> > [root@ssdandy ~]#
> > 
> > If I add one more 'v' I get the symtabs as read by symbol-elf.c and this is
> > what is there for the malloc routines:
> 
> Yeah, it seems symbol maps works fine. Debuginfo analysis failed.
> 
> > 
> > [root@ssdandy ~]# grep malloc /tmp/4
> > probe-definition(0): malloc
> > symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null)
> > Searching variables at malloc
> > symbol__new: ptmalloc_lock_all 0x7b060-0x7b173
> > symbol__new: malloc_atfork 0x801c0-0x802dd
> > symbol__new: ptmalloc_unlock_all2 0x7b180-0x7b21e
> > symbol__new: ptmalloc_unlock_all 0x7ba90-0x7bb3e
> > symbol__new: malloc_printerr 0x7bb40-0x7bc28
> > symbol__new: malloc_consolidate 0x7c3d0-0x7c9be
> > symbol__new: _int_malloc 0x7dee0-0x7f32b
> > symbol__new: malloc_check 0x7f330-0x7f44a
> > symbol__new: ptmalloc_init.part.8 0x80fe0-0x813d7
> > symbol__new: ptmalloc_init 0x813e0-0x813f5
> > symbol__new: malloc_hook_ini 0x81400-0x8143c
> > symbol__new: mallochook 0x82fc0-0x831af
> > symbol__new: tr_mallochook 0x83fc0-0x8407d
> > symbol__new: __malloc_set_state 0x814f0-0x819f6
> > symbol__new: __malloc_usable_size 0x80d30-0x80e0e
> > symbol__new: __malloc_trim 0x81d50-0x81fac
> > symbol__new: __GI___libc_malloc 0x800c0-0x801b5
> > symbol__new: __malloc_get_state 0x802e0-0x804db
> > symbol__new: __malloc_stats 0x820c0-0x822aa
> > symbol__new: __malloc_check_init 0x80050-0x800bb
> > symbol__new: __malloc 0x800c0-0x801b5
> > symbol__new: malloc_set_state 0x814f0-0x819f6
> > symbol__new: malloc 0x800c0-0x801b5
> > symbol__new: malloc_info 0x82320-0x8240f
> > symbol__new: malloc_trim 0x81d50-0x81fac
> > symbol__new: malloc_usable_size 0x80d30-0x80e0e
> > symbol__new: malloc_get_state 0x802e0-0x804db
> > symbol__new: __libc_malloc 0x800c0-0x801b5
> > symbol__new: malloc_stats 0x820c0-0x822aa
> > symbol__new: malloc@plt 0x1f300-0x1f310
> > Symbol malloc address found : 800c0
> > Probe point found: __malloc_check_init+96
> > [root@ssdandy ~]#
> >  
> >>>> [root@ssdandy ~]#
> >>>>
> >>>> And then the one asking for 'bytes' to be collectd fails.
> >>>>
> >>>> After processing the other patches I'll try to debug this...
> >>>>
> >>>> [root@ssdandy ~]# cat /etc/redhat-release 
> >>>> Red Hat Enterprise Linux Server release 7.0 (Maipo)
> >>>> [root@ssdandy ~]# rpm -q glibc glibc-debuginfo
> >>>> glibc-2.17-55.el7_0.3.x86_64
> >>>> glibc-debuginfo-2.17-55.el7_0.1.x86_64
> >>>                               ^^^ why is this different from the glibc version??
> >>>
> >>>> [root@ssdandy ~]#
> >>>> [acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc
> >>> [...]
> >>>>   4849: 0000000000080050   107 FUNC    LOCAL  DEFAULT   12 __malloc_check_init
> >>>
> >>> __malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0
> >>>
> >>>>   5351: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __malloc
> >>>>   1170: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc@@GLIBC_2.2.5
> >>>
> >>> that is not 0x800c0, so something goes wrong when translating the address
> >>> to probe point. Could you check the patch 1/4 was applied?
> >>> Actually there is a bug in the routine which gets the probe point from
> >>> address. 1/4 fixes it.
> >>
> >> This one?
> >>
> >> commit 0104fe69e0287cf3635657b4c6b26a18e0091697
> >> Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
> >> Date:   Mon Mar 2 21:49:46 2015 +0900
> >>
> >>     perf probe: Remove bias offset to find probe point by address
> >>     
> >>     Remove bias offset to find probe point by address.
> >>
> >> ----------------
> >>
> >> Yes, it is applied.
> >>
> >> - Arnaldo
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> > 
> 
> 
> -- 
> Masami HIRAMATSU
> Software Platform Research Dept. Linux Technology Research Center
> Hitachi, Ltd., Yokohama Research Laboratory
> E-mail: masami.hiramatsu.pt@hitachi.com
>