about cpu_profiler

about cpu_profiler

[Colin McCabe]

tl;dr: cpu_profiler may randomly crash your program on x86_64.
The problem "should" be resolved in the "future".

Full text from google-perftools-1.7/README:

> 2) On x86-64 64-bit systems, while tcmalloc itself works fine, the
> cpu-profiler tool is unreliable: it will sometimes work, but sometimes
> cause a segfault.  I'll explain the problem first, and then some
> workarounds.
>
> Note that this only affects the cpu-profiler, which is a
> google-perftools feature you must turn on manually by setting the
> CPUPROFILE environment variable.  If you do not turn on cpu-profiling,
> you shouldn't see any crashes due to perftools.
>
> The gory details: The underlying problem is in the backtrace()
> function, which is a built-in function in libc.
> Backtracing is fairly straightforward in the normal case, but can run
> into problems when having to backtrace across a signal frame.
> Unfortunately, the cpu-profiler uses signals in order to register a
> profiling event, so every backtrace that the profiler does crosses a
> signal frame.
>
> In our experience, the only time there is trouble is when the signal
> fires in the middle of pthread_mutex_lock.  pthread_mutex_lock is
> called quite a bit from system libraries, particularly at program
> startup and when creating a new thread.
>
> The solution: The dwarf debugging format has support for 'cfi
> annotations', which make it easy to recognize a signal frame.  Some OS
> distributions, such as Fedora and gentoo 2007.0, already have added
> cfi annotations to their libc.  A future version of libunwind should
> recognize these annotations; these systems should not see any
> crashses.
>
> Workarounds: If you see problems with crashes when running the
> cpu-profiler, consider inserting ProfilerStart()/ProfilerStop() into
> your code, rather than setting CPUPROFILE.  This will profile only
> those sections of the codebase.  Though we haven't done much testing,
> in theory this should reduce the chance of crashes by limiting the
> signal generation to only a small part of the codebase.  Ideally, you
> would not use ProfilerStart()/ProfilerStop() around code that spawns
> new threads, or is otherwise likely to cause a call to
> pthread_mutex_lock!
>
> ---
> 11 January 2011

I'm hoping that this is out of date?

Colin

Re: about cpu_profiler

[Gregory Farnum]

On Friday, March 4, 2011 at 12:08 PM, Colin McCabe wrote: 
> tl;dr: cpu_profiler may randomly crash your program on x86_64.
> The problem "should" be resolved in the "future".

From what I've read, it's unfortunately not out of date. However, it manifests itself as a crash so it's not the reason that we can't get debugging output, unless there's something I'm missing. :)
And even if it turns out to be very unstable on x86-64, the value of a good cpu profiler is high enough that we can set up x86 boxes or VMs...
-Greg 

Re: about cpu_profiler

[Sage Weil]

On Fri, 4 Mar 2011, Gregory Farnum wrote:
> On Friday, March 4, 2011 at 12:08 PM, Colin McCabe wrote: 
> > tl;dr: cpu_profiler may randomly crash your program on x86_64.
> > The problem "should" be resolved in the "future".
> 
> From what I've read, it's unfortunately not out of date. However, it 
> manifests itself as a crash so it's not the reason that we can't get 
> debugging output, unless there's something I'm missing. :) And even if 
> it turns out to be very unstable on x86-64, the value of a good cpu 
> profiler is high enough that we can set up x86 boxes or VMs... 

Yeah.  If it's not going to be reliable, though, it's not worth investing 
time/debugging effort into making it (sort of) work.

I suspect oprofile is the way to go here...

sage

Re: about cpu_profiler

[Colin McCabe]

I'm testing out oprofile now. So far, it seems relatively painless,
and it is a system-wide profiler. It seems to have some kernel hooks
too...

Colin


On Fri, Mar 4, 2011 at 12:40 PM, Gregory Farnum
<gregory.farnum@dreamhost.com> wrote:
>
> On Friday, March 4, 2011 at 12:08 PM, Colin McCabe wrote:
>> tl;dr: cpu_profiler may randomly crash your program on x86_64.
>> The problem "should" be resolved in the "future".
>
> From what I've read, it's unfortunately not out of date. However, it manifests itself as a crash so it's not the reason that we can't get debugging output, unless there's something I'm missing. :)
> And even if it turns out to be very unstable on x86-64, the value of a good cpu profiler is high enough that we can set up x86 boxes or VMs...
> -Greg
>
>
>
>

Re: about cpu_profiler

[Colin McCabe]

I wrote a wiki page about using oprofile with Ceph. It's at:

http://ceph.newdream.net/wiki/Cpu_profiling

I guess we should give this a try next time we have a CPU load problem.

Apparently you can get really detailed with oprofile and measure
things like number of CPU pipeline stalls and cache line misses. I
didn't do any of that-- just used the simple counter.

Colin


On Fri, Mar 4, 2011 at 12:47 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> I'm testing out oprofile now. So far, it seems relatively painless,
> and it is a system-wide profiler. It seems to have some kernel hooks
> too...
>
> Colin
>
>
> On Fri, Mar 4, 2011 at 12:40 PM, Gregory Farnum
> <gregory.farnum@dreamhost.com> wrote:
>>
>> On Friday, March 4, 2011 at 12:08 PM, Colin McCabe wrote:
>>> tl;dr: cpu_profiler may randomly crash your program on x86_64.
>>> The problem "should" be resolved in the "future".
>>
>> From what I've read, it's unfortunately not out of date. However, it manifests itself as a crash so it's not the reason that we can't get debugging output, unless there's something I'm missing. :)
>> And even if it turns out to be very unstable on x86-64, the value of a good cpu profiler is high enough that we can set up x86 boxes or VMs...
>> -Greg
>>
>>
>>
>>
>