Re: spin_lock cause ?

[David Ahern <dsahern@gmail.com>]

On 6/1/16 2:24 PM, Rick Jones wrote:
> On 06/01/2016 12:25 PM, M Kelly wrote:
>> I think I recall you from hp, years ago when I was there :-)
>> Thanks for the info.  I suspect we are too heavy on
>> pthread_mutex_lock()/unlock()
>
> HP-UX has some ways/tools to track mutex contention.  Some extensions to
> give names to mutexes and then get stats on how often one went to grab
> them and it was held etc etc.  Been ages since I used them - I'd hope
> there was something similar for Linux but I've not played with threads
> in a very long time.

If the kernel is new enough (3.8 or 3.10, I forget when userspace return 
probes were added) you can use probes + futex system call to analyze 
lock contention.

 From an old script for x86:

PL=/lib64/libpthread.so.0
perf probe -x ${PL} -a 'mutex_lock=pthread_mutex_lock addr=%di'
perf probe -x ${PL} -a 'mutex_trylock=pthread_mutex_trylock addr=%di'
perf probe -x ${PL} -a 'mutex_timedlock=pthread_mutex_timedlock addr=%di'
perf probe -x ${PL} -a 'mutex_unlock=pthread_mutex_unlock addr=%di'

perf probe -x ${PL} -a 'mutex_lock_ret=pthread_mutex_lock%return ret=%ax'
perf probe -x ${PL} -a 'mutex_trylock_ret=pthread_mutex_trylock%return 
ret=%ax'
perf probe -x ${PL} -a 
'mutex_timedlock_ret=pthread_mutex_timedlock%return ret=%ax'


perf record -a -e probe_libpthread:* -e 
syscalls:sys_enter_futex,syscalls:sys_exit_futex -- <workload spec>

perf script -g python

--> edit perf-script.py for analysis of interest.

perf script -s perf-script.py

e.g., you can track the thread id holding a lock and for a how long, how 
many contentions, who got the lock when it was released, etc.

Many years ago I used the above to understand a nasty apparent deadlock 
in a process.