Re: No source code for static library (compiled with -g -ggdb)

[Mark Davis <markdavisinboston@gmail.com>]

Milian, this is a very slow reply -- but I did want to thank you for
your feedback. I did remove the -fno-inline flag when profiling and it
does give more useful results.

On Mon, May 2, 2016 at 3:58 AM, Milian Wolff <milian.wolff@kdab.com> wrote:
> On Freitag, 29. April 2016 09:26:44 CEST Mark Davis wrote:
>> It turns out that I was incorrect about diagnosing this problem, and
>> this is probably just due to my misunderstanding of how to use
>> annotations in perf. (Sorry, I'm a new user to perf and trying to get
>> my head around it.) I do have a question below.
>>
>> Recall my example from above:
>>
>> execute_native_thread_routine
>> std::_Bind_simple<int (*(int, char**, Thread*))(int, char**,
>> Thread*)>::operator()()
>> main(int, char**)
>> my_func_A
>> my_func_B
>> malloc
>>
>> I was trying to jump into my_func_B to see the annotation of it, and
>> it kept jumping me into the annotation of malloc. That is, I wanted to
>> see *where* malloc was being called from my_func_B, and the hottest
>> areas of my_func_B. But, when I type "a" when on my_func_B (or drill
>> in with the arrows or enter key), it always takes me to the annotation
>> of malloc (i.e., the leaf node in the callchain). I figured this out
>> by filtering to the DSO of my system (which made the leaf node always
>> be a symbol in my system as opposed to some system library, etc.) and
>> it shows that my symbols have debug info, etc.
>>
>> But if I wanted to see the annotation of my_func_B, how would I? Is it
>> possible that I'm not able to see this simply because I don't have
>> enough samples of it (and all the real time is spent down in malloc)?
>> Or is there some other reason? Arnaldo, what exactly determines if
>> your arrows show up when pressing V in perf v4.1?
>
> I'm not really sure, but I seem to remember this "bug" in older perf versions.
> I see in the other thread that you manually installed a newer perf, but could
> you check whether the following works with the older versions:
>
> - navigate to the function you want to annotate (my_func_B)
> - instead of pressing 'a', press 'arrow right'
> - in the menu that shows up, select "annotate"
>
> I /think/ that 'a' used to be the "annotate sample entry point" hotkey as you
> encounter, whereas the submenu always allowed one to annotate individual
> frames of the sample callstack.
>
>> Furthermore, is there some other feature of perf_events that I should
>> try (perhaps tracing?) to get more fidelity? My programs are
>> relatively short-running, so I can afford to wait a bit longer.
>
> Tracing will require you to add tracepoints via uprobe manually, which is
> super time consuming - the changeset for Systemtap trace point support in perf
> has not yet been merged (bummer!).
>
> I suggest you try to increase the sampling rate if you are interested in more
> details of short-running apps.
>
>> Thank you for any input you can provide as I'm learning perf_events.
>
> On more suggestion from my side: In your original mail you said you compile
> your code with "-g -ggdb -fno-inline -O2 -fno-omit-frame-pointer". I think
> this will produce potentially extremely misleading results, due to you
> disabling the arguably most useful optimization feature of a compiler there
> is: "-fno-inline"
>
> I /really/ urge you to remove this check and actually profile a "real" world
> example that is build with "-g -O2", i.e. with inlining enabled. Yes, the
> callstacks /may/ be harder to grasp, but at the same time you actually profile
> what the real app is doing. In C++ code, esp. with templates, I've seen
> hotspots disappear just due to inlining and the avalanche effect they can
> trigger in a good optimizer, e.g. by enabling deadcode elimination etc.
>
> Cheers
>
>> On Fri, Apr 29, 2016 at 8:28 AM, Mark Davis <markdavisinboston@gmail.com>
> wrote:
>> > Arnaldo, that's a really nice feature. Unfortunately, I'm currently
>> > running on an older version of perf (3.13.11-ckt18) and it's a bit of
>> > a pain to modify things like this on the cluster I'm running on
>> > (although I can if I have something I absolutely need). So, for me
>> > when I run verbose mode I simply get the DSO names in the callchains,
>> > but no arrows. I would assume, if I had a newer version of perf, that
>> > I would get no arrows for the symbols I want to annotate, given the
>> > behavior I'm seeing.
>> >
>> > Anything else I can try? Does anyone have any ideas about why this may
>> > be happening? Again, this is only happening for symbols in my static
>> > library, and I'm creating this library using:
>> >
>> > ar -cvrs $@ $(OBJS)
>> >
>> > (I'm not an expert on ar, so I could be doing this wrong?)
>> >
>> > On Thu, Apr 28, 2016 at 6:16 PM, Arnaldo Carvalho de Melo
>> >
>> > <arnaldo.melo@gmail.com> wrote:
>> >> Em Thu, Apr 28, 2016 at 01:44:16PM -0400, Mark Davis escreveu:
>> >>> Hello, I'm using perf record and perf report to profile an application
>> >>> which is made from a static (.a) library file that I made as well as a
>> >>> handful of C++ files that are not in the static library. They are all
>> >>> compiled with "-g -ggdb -fno-inline -O2 -fno-omit-frame-pointer" and
>> >>> all linked together to create the application.
>> >>>
>> >>> When I use perf report (in interactive/tui mode), when I annotate the
>> >>> symbols that are in the C++ files (not the static library), I see the
>> >>> source code intermingled with the assembly (this is what I want). I
>> >>> can do this at any layer of the call stack. However, when I drill into
>> >>> the symbols that are from the static library, I just see the assembly
>> >>> of the leaf symbol. I don't expect to see the source in this case, as
>> >>> the leaf symbol is a low-level library from libc; but, perf report
>> >>> won't let me drill into (i.e., annotate) any non-leaf symbols.
>> >>>
>> >>> Example:
>> >>> execute_native_thread_routine
>> >>> std::_Bind_simple<int (*(int, char**, Thread*))(int, char**,
>> >>> Thread*)>::operator()()
>> >>> main(int, char**)
>> >>> my_func_A
>> >>> my_func_B
>> >>> malloc
>> >>
>> >> What happens when you press 'V'? Here, say for this callchain:
>> >>
>> >> -    0.73%  gnome-shell      libgobject-2.0.so.0.4600.2          [.]
>> >> handlers_find>>
>> >>    - handlers_find
>> >>
>> >>       - 0.68% signal_handlers_foreach_matched_R
>> >>
>> >>            g_signal_handlers_disconnect_matched
>> >>            0x7f71fe809a80
>> >>            g_object_unref
>> >>
>> >>          - st_widget_get_theme_node
>> >>
>> >>               0.58% 0x7f71fe813429
>> >>
>> >> And I press V, I get:
>> >>
>> >> -    0.73%  gnome-shell      libgobject-2.0.so.0.4600.2          [.]
>> >> handlers_find>>
>> >>    -→handlers_find libgobject-2.0.so.0.4600.2
>> >>
>> >>       - 0.68% signal_handlers_foreach_matched_R
>> >>       libgobject-2.0.so.0.4600.2
>> >>
>> >>            g_signal_handlers_disconnect_matched
>> >>            libgobject-2.0.so.0.4600.2
>> >>            0x7f71fe809a80 libgnome-shell.so
>> >>
>> >>           →g_object_unref libgobject-2.0.so.0.4600.2
>> >>
>> >>          -→st_widget_get_theme_node libgnome-shell.so
>> >>
>> >>               0.58% 0x7f71fe813429 libgnome-shell.so
>> >>
>> >> See those arrows? They point to the places where you can annotate, the
>> >> others can't because they had no samples or were unresolved.
>> >>
>> >> With some changes I think we can show the ones without samples, but that
>> >> remains to be done.
>> >>
>> >> Also, this is with:
>> >>
>> >> [root@jouet ~]# perf --version
>> >> perf version 4.6.rc4.ga453697
>> >>
>> >> This feature was introduced in:
>> >>
>> >> commit 70e972788888315851a5c1b4982efbcafcd03b5b
>> >> Author: Arnaldo Carvalho de Melo <acme@redhat.com>
>> >> Date:   Thu Mar 19 16:07:21 2015 -0300
>> >>
>> >>     perf hists browser: Indicate which callchain entries are annotated
>> >>
>> >>     Now that we can annotate entries in a callchain, show which ones have
>> >>     an
>> >>     associated symbol and samples, by adding a right arrow just before
>> >>     the
>> >>     symbol name when in verbose mode.
>> >>
>> >>     To toggle verbose mode press 'V'.
>> >>
>> >> ---------------------------------
>> >>
>> >> Which is:
>> >>
>> >> [acme@jouet linux]$ git tag --contains
>> >> 70e972788888315851a5c1b4982efbcafcd03b5b | grep ^v4 | head -1 v4.1
>> >> [acme@jouet linux]$
>> >>
>> >> Thanks,
>> >>
>> >> - Arnaldo
>> >>
>> >>> When I annotate my_func_A and my_func_B, it just jumps to the
>> >>> annotation of malloc (I see this listed at the top of the annotation
>> >>> view). Note that my_func_A and my_func_B are both defined in the
>> >>> static library that I'm linking in. However, when I do a similar thing
>> >>> with a different stack where I'm annotating a function that's not from
>> >>> the static library (but in the regular C++ files), it works fine.
>> >>>
>> >>> Here's what I'm doing to create my static lib:
>> >>> ar -cvrs $@ $(OBJS)
>> >>>
>> >>>
>> >>> I did confirm with nm --debug-syms that my static lib has debugging
>> >>> symbols in it:
>> >>>
>> >>> U __assert_fail
>> >>> 0000000000000000 b .bss
>> >>> 0000000000000000 n .comment
>> >>> U __cxa_atexit
>> >>> 0000000000000000 d .data
>> >>> 0000000000000000 N .debug_abbrev
>> >>> 0000000000000000 N .debug_aranges
>> >>> 0000000000000000 N .debug_info
>> >>> 0000000000000000 N .debug_line
>> >>> 0000000000000000 N .debug_loc
>> >>> 0000000000000000 N .debug_ranges
>> >>> 0000000000000000 N .debug_str
>> >>> U __dso_handle
>> >>> 0000000000000000 r .eh_frame
>> >>> U exit
>> >>>
>> >>>
>> >>> How can I compile and link my application and configure perf report so
>> >>> I can drill in like this on functions defined in a static library with
>> >>> debug symbols?
>> >>>
>> >>> Thank you,
>> >>> Mark
>
>
>
> --
> Milian Wolff | milian.wolff@kdab.com | Software Engineer
> KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
> Tel: +49-30-521325470
> KDAB - The Qt Experts