Re: Trying to understand some strange samples in a perf profile

[Milian Wolff <milian.wolff@kdab.com>]

[-- Attachment #1: Type: text/plain, Size: 5427 bytes --]

On Thursday, December 21, 2017 5:02:09 PM CET Hadrien Grasland wrote:
> Hey,
> 
> First of all, let me be even more aggressive with e-mail quote removal :)
> 
> > <snip>
> > 
> >>>> Can you help me check if my understanding of the other samples is
> >>>> correct, and give me some clues about what could happen for these last
> >>>> few weird ones?
> >>>> 
> >>>> I can provide perf.data files and archives on demand, but since the
> >>>> original is a rather big upload/download ( 2.7 GB ), I thought I would
> >>>> wait until you can give me some specific requests or a way to produce
> >>>> skimmed datasets with only the samples that you are looking for.
> >>> 
> >>> The perf.data file wouldn't be enough, we'd need access to all of the
> >>> so/dbg files too. perf archive could be used for that purpose, but I've
> >>> seen many situations where not all DSOs encountered had a build-id, and
> >>> thus where not put into the archive...
> >> 
> >> Yes, back when I discussed the ffffffffffffffff samples with Jiri on
> >> IRC, we had this problem that perf archive failed to include all
> >> necessary DSOs for analysis. So I'm not sure if it would work here.
> >> 
> >> If there is anything else which I can do to help you here, just tell me!
> > 
> > I never took the time to debug this, but I personally consider this a big
> > problem with perf. These broken backtraces make analysis often much harder
> > compared to others tools like VTune which apparently do not suffer from
> > this issue. At least there, I have never seen it - maybe it's just less
> > of a problem there for some reason.
> > 
> > So to debug this, I guess we need to sit down and actually find a way to
> > reproduce the issue with a smaller test case. I actually have something
> > that is much smaller than your code base and also shows this issue
> > sometimes:
> > 
> > https://github.com/KDAB/hotspot/blob/master/tests/test-clients/cpp-inlinin
> > g/ main.cpp
> > 
> > perf record --call-graph dwarf -c 1000000 ./a.out
> > ..
> > 
> > cpp-inlining 25448 258803.904661:     100000 cycles:uppp:
> >          ffffffff815fd197 [unknown] ([unknown])
> > 
> > ..
> > 
> > cpp-inlining 25448 258803.907014:     100000 cycles:uppp:
> >                     23783 __hypot_finite (/usr/lib/libm-2.26.so)
> >          
> >          40e3491ebf75199b [unknown] ([unknown])
> > 
> > ..
> > 
> > cpp-inlining 25448 258804.312048:     100000 cycles:uppp:
> >                     23783 __hypot_finite (/usr/lib/libm-2.26.so)
> >          
> >          40f204e9aa06ab79 [unknown] ([unknown])
> > 
> > Interestingly, the last broken state repeats, and the address in libm is
> > always the same (0x23783). So that's why I believe that the problem might
> > be in libunwind/libdw or in the DWARF emitter in GCC/Clang... We'd have
> > to debug this. I guess for that we'd have to do something along the
> > following:
> > 
> > - build a minimal "unwinder" application for libunwind e.g.
> > - notify libunwind about all mmap events
> > - install the register and stack state from the per.data file
> > - try to unwind, see what happens
> > 
> > This should give us a MWE that we could also show to Dave Watson and
> > others
> > from libunwind. Similarly, we can easily check what libdwfl does there and
> > could show it to Mark Wielaard and others from elfutils. I'm pretty sure
> > that together we can then figure out what to blame:
> > 
> > - are the register values correct?
> > - is the stack state correct?
> > - is the DWARF info correct?
> > - ...?
> > 
> > It sounds like a lot of work, but it's certainly doable. Fixing this issue
> > would make me very happy, and I bet a lot of other users would appreciate
> > this too. Right now, I often have to tell people who use perf "deal with
> > broken backtraces, try to guesstimate from the surroundings what actually
> > happened". But this is of course far from optimal!
> 
> This discussion is getting a bit far from my technological comfort zone,
> and I am very likely to say something stupid or repeat what you just
> said in a different way. But if I look at the libunwind documentation, I
> see that there seems to be a way to attach to a process and generate a
> backtrace of it like a debugger would.
> 
> If so, couldn't we bisect the issue and discriminate whether this is
> perf's fault by just using this capability to build a minimal poor man
> profiler, which uses libunwind to print a stack trace of an application
> at regular intervals? To the best of my understanding, if we get broken
> stack traces there, the blame would lie on the side of libunwind or the
> compiler-generated DWARF info, whereas if we get correct stack traces,
> the blame would lie on perf's facilities for copying register and stack
> state.
> 
> Am I overlooking something here?

No, this is pretty much what I had in mind too. The only difference is where 
to get the stack from for unwinding. Attaching with libunwind is one option, 
but I'm unsure whether we can "hit" the troublesome addresses in libm e.g. 
easily. My idea was to take the stack copy we have in perf.data attached to 
the sample event. Anyhow, both approaches sound useful, and having both would 
of course be the ideal scenario as we could where it breaks.

Bye

-- 
Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3826 bytes --]