Profile the waiting time for lock or lock stats for one application

Profile the waiting time for lock or lock stats for one application

[Junjie Qian]

Dear all,

I am working on one project, which tries to profile the lock stats for one application, such as total waiting time for locks or how many locks happens during the execution.

I did some research, which shows "perf lock" may help to report such information, but I am getting the error showing as "tracepoint lock:lock_acquire is not enabled. Are CONFIG_LOCKDEP and CONFIG_LOCK_STAT enabled?". Does this mean, I need to recompile the kernel to have this option?
Could anyone give me some link for this?

I also got this link "http://article.gmane.org/gmane.linux.kernel.perf.user/770/match=perf+lock", talking about profile the sleep times, but I cannot find how to use this.

Any help would be great appreciated.

Thanks!
Best
Junjie

Re: Profile the waiting time for lock or lock stats for one application

[Junjie Qian]

Hi all,

This is a follow up question about "perf lock".
I recompile the kernel, with both CONFIG_LOCKDEP and CONFIG_LOCK_STAT are "y". But when I run the command "perf lock record -- sleep 10", the error information still shows as:
tracepoint lock:lock_acquire is not enabled. Are CONFIG_LOCKDEP and CONFIG_LOCK_STAT enabled?

Could you give me some suggestions on this?

Thank you
Junjie Qian

Re: Profile the waiting time for lock or lock stats for one application

[David Ahern]

On 2/26/14, 8:17 PM, Junjie Qian wrote:
> Hi all,
>
> This is a follow up question about "perf lock".
> I recompile the kernel, with both CONFIG_LOCKDEP and CONFIG_LOCK_STAT are "y". But when I run the command "perf lock record -- sleep 10", the error information still shows as:
> tracepoint lock:lock_acquire is not enabled. Are CONFIG_LOCKDEP and CONFIG_LOCK_STAT enabled?
>
> Could you give me some suggestions on this?

are you running as root? If you are running perf as a normal user is the 
events directory accessible -- /sys/kernel/debug/tracing? Does 
/sys/kernel/debug/tracing/events/lock exist?

David

Re: Profile the waiting time for lock or lock stats for one application

[Rodrigo Campos]

On Tue, Feb 25, 2014 at 10:13:11AM -0800, Junjie Qian wrote:
> Dear all,
> 
> I am working on one project, which tries to profile the lock stats for one application, such as total waiting time for locks or how many locks happens during the execution.

perf lock wouldn't help with that, IIUC. perf lock is for locks inside the
kernel.

There are ways (not sure if the patch is in) to track for lock contension using
perf, but not sure (I think not easy at least) if the lock is not contended. As,
if using glibc, lot of locks are implemented using futex and then solved
entirely in userspace lot of times (at least without contention).

Also, about stats such as the total waiting time, take into account that if you
have more threads than cores, there WILL be waiting time you can't reduce. For
some experiments I did (with #threads >> #cores) trying to measure waiting time
was pointeless, as that didn't help us in any way. And in some scenarios (cpu
hungry tasks, for example), it's not important to have threads waiting if all
your cores are active "most of the time".

So, for me (for my workloads), the waiting time was pointless and it was WAY
more useful trying to see "wasted" time. As I was trying to optimize a library,
I did some instrumentation on it to get the CPU time used by the library and
compare it against #cores * elapsed_time.





Hope it helps,
Rodrigo

Re: Profile the waiting time for lock or lock stats for one application

[Junjie Qian]

Dear David,

Thank you for this suggestion, and sorry for late reply.
I have tried this yet, since I am using lock_stat to analyze the system lock info, and I will let you know if I have any updates on this.
 
Thanks!
Best
Junjie



On Wednesday, February 26, 2014 9:29 PM, David Ahern <dsahern@gmail.com> wrote:
On 2/26/14, 8:17 PM, Junjie Qian wrote:

> Hi all,
>
> This is a follow up question about "perf lock".
> I recompile the kernel, with both CONFIG_LOCKDEP and CONFIG_LOCK_STAT are "y". But when I run the command "perf lock record -- sleep 10", the error information still shows as:
> tracepoint lock:lock_acquire is not enabled. Are CONFIG_LOCKDEP and CONFIG_LOCK_STAT enabled?
>
> Could you give me some suggestions on this?

are you running as root? If you are running perf as a normal user is the 
events directory accessible -- /sys/kernel/debug/tracing? Does 
/sys/kernel/debug/tracing/events/lock exist?

David

--
To unsubscribe from this list: send the line "unsubscribe linux-perf-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Profile the waiting time for lock or lock stats for one application

[David Ahern]

On 2/28/14, 11:43 AM, Rodrigo Campos wrote:
> There are ways (not sure if the patch is in) to track for lock contension using
> perf, but not sure (I think not easy at least) if the lock is not contended. As,
> if using glibc, lot of locks are implemented using futex and then solved
> entirely in userspace lot of times (at least without contention).

What patch are you referring to? You can trace system calls to catch all 
contention and wakeups. You can put probes at lock entry, exit and 
unlock entry to track lock accesses by address.

I started working on this end of last year, just haven't had time to 
work on the analysis side -- statistical analysis of lock times with 
ability to see individual events. All I have right now is python script 
that pretty prints the events.

David

Re: Profile the waiting time for lock or lock stats for one application

[Rodrigo Campos]

On Sat, Mar 01, 2014 at 08:30:47AM -0700, David Ahern wrote:
> On 2/28/14, 11:43 AM, Rodrigo Campos wrote:
> >There are ways (not sure if the patch is in) to track for lock contension using
> >perf, but not sure (I think not easy at least) if the lock is not contended. As,
> >if using glibc, lot of locks are implemented using futex and then solved
> >entirely in userspace lot of times (at least without contention).
> 
> What patch are you referring to? You can trace system calls to catch

I think one from you, from late last year. I'm not sure and I'm not on my
notebook right now. Something to trace the futex syscall or something like that
? That can help to understand lock contention on userspace





Thanks,
Rodrigo

Re: Profile the waiting time for lock or lock stats for one application

[Junjie Qian]

Hi,

Really appreciate your help on this issue!
1. using root solve my problem with "perf lock", thank you very much!
2. about the discussion on locks in user-space. I discussed this with my advisor, and found that lock contention does not mean waiting time or idle cpu-cycles sometime, for example, spin-lock in user-space that all threads/cores are still running but useless work and this cannot be detected as either system lock or idle-cpu-cycles. 
The way, I am trying to do this, now is insert the counter into the application layer once the lock happens start the counting and I prepare to monitor the threads in queue to detect the lock.

I will let you know if I have any updates on this.
Again thank you all for your help and discussions.
 
Thanks!
Best
Junjie



On Saturday, March 1, 2014 10:04 AM, Rodrigo Campos <rodrigo@sdfg.com.ar> wrote:
On Sat, Mar 01, 2014 at 08:30:47AM -0700, David Ahern wrote:
> On 2/28/14, 11:43 AM, Rodrigo Campos wrote:
> >There are ways (not sure if the patch is in) to track for lock contension using
> >perf, but not sure (I think not easy at least) if the lock is not contended. As,
> >if using glibc, lot of locks are implemented using futex and then solved
> >entirely in userspace lot of times (at least without contention).
> 
> What patch are you referring to? You can trace system calls to catch

I think one from you, from late last year. I'm not sure and I'm not on my
notebook right now. Something to trace the futex syscall or something like that
? That can help to understand lock contention on userspace





Thanks,

Rodrigo

Re: Profile the waiting time for lock or lock stats for one application

[Rodrigo Campos]

On Mon, Mar 03, 2014 at 08:11:12AM -0800, Junjie Qian wrote:
> Hi,
> 
> Really appreciate your help on this issue!
> 1. using root solve my problem with "perf lock", thank you very much!
> 2. about the discussion on locks in user-space. I discussed this with my advisor, and found that lock contention does not mean waiting time or idle cpu-cycles sometime, for example, spin-lock in user-space that all threads/cores are still running but useless work and this cannot be detected as either system lock or idle-cpu-cycles. 

Yeah, with mutexes usually there is some spinning before waiting for the lock.

> The way, I am trying to do this, now is insert the counter into the application layer once the lock happens start the counting and I prepare to monitor the threads in queue to detect the lock.

clock_gettime() with some clocks is implemented as a VDSO, so it should be
pretty light to get that info if you need it.
And thread local variables can be very useful too probably :)

> I will let you know if I have any updates on this.

Great :)




Thanks a lot,
Rodrigo