Re: perf for analyzing userspace contention

[Arun Sharma <asharma@fb.com>]

On Thu, Oct 13, 2011 at 11:36:23PM -0700, Roland Dreier wrote:
> Hi everyone,
> 
> Hope this question from a kernel hacker about profiling userspace
> isn't too dumb...
> 
> Anyway, suppose I have a multithreaded userspace app that uses a bunch of
> pthread_mutexes, and I want to figure out which locks are hot and/or heavily
> contended.  What's the best way to do that?  Is perf the right, or is there
> something better?  (This seems like such an obvious thing to want that there
> must be some good way to get this data, I hope)

Sampling on sys_futex entry/exit is a good start. But it doesn't tell
you if the thread spent 1us or 1ms waiting on the futex. We really need
to add weights to the profile based on sleep times (or other criteria).

You might want to follow further discussion in the thread with the
subject "Profiling sleep times?".

perf record -ag -e cs -- sleep 1

is a more general version of this. It helps in understanding what's
causing the thread to give up CPU voluntarily (you might have to filter
the output a bit).

 -Arun