Re: [ltt-dev] trace a futex

Re: [ltt-dev] trace a futex

[Mathieu Desnoyers]

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]

* Jan Kiszka (jan.kiszka@web.de) wrote:
> Mathieu Desnoyers wrote:
> > * Gian Lorenzo Meocci (glmeocci@gmail.com) wrote:
> >> Hi Mathieu,
> >>
> >> thanks for your reply.
> >>
> >> I want specify that:
> >> 1) I am already patching glibc (and, of course, nptl/pthread_*)
> >> 2) I already add two event to pthread_mutex_lock. The first at the
> >> beginning of the function and the second after all return 0 presented
> >> on that function. But those two events are not enough to establish if
> >> a pthread_mutex_lock has been blocking.
> >> In fact I know only the time spent on pthread_mutex_lock. If this time
> >> is little probably I hold the mutex otherwise I was been descheduled.
> >>
> >> So thanks a lot again,
> >>
> >>
> > 
> > Ok, then you will probably want to correlate your information with :
> > 
> > - scheduling activity regarding your threads
> > - system call entry events, especially sys_futex. Note that a thread
> >   calling sys_futex won't _necessarily_ be put to sleep.. it may still
> >   be able to take the lock relatively quickly.
> > 
> > If you need more than that, we may think of instrumenting futex.c, but I
> > am not sure this is required.
> 
> Haven't checked the state of instrumentation recently: Is the futex
> operation visible in the trace now? It used to be not, and we often had
> to guess the reason for sys_futex (wake, wait, pi or not pi, etc.) from
> the context - or add ad-hoc instrumentation.
> 
> Jan
> 

It still isn't instrumented, and I think it would be good to add such
instrumentation.

Have a look at the patch done by K. Prasad for futex.c : it should
probably be updated so it uses tracepoints instead of markers. Anyone
would like to do this ?

See http://lkml.org/lkml/2008/4/15/125
"[RFC PATCH 0/2] Debugging infrastructure for Futexes using Markers"

Mathieu

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [ltt-dev] trace a futex

[K.Prasad]

On Wed, Dec 03, 2008 at 12:26:40AM -0500, Mathieu Desnoyers wrote:
> * Jan Kiszka (jan.kiszka@web.de) wrote:
> > Mathieu Desnoyers wrote:
> > > * Gian Lorenzo Meocci (glmeocci@gmail.com) wrote:
> > >> Hi Mathieu,
> > >>
> > >> thanks for your reply.
> > >>
> > >> I want specify that:
> > >> 1) I am already patching glibc (and, of course, nptl/pthread_*)
> > >> 2) I already add two event to pthread_mutex_lock. The first at the
> > >> beginning of the function and the second after all return 0 presented
> > >> on that function. But those two events are not enough to establish if
> > >> a pthread_mutex_lock has been blocking.
> > >> In fact I know only the time spent on pthread_mutex_lock. If this time
> > >> is little probably I hold the mutex otherwise I was been descheduled.
> > >>
> > >> So thanks a lot again,
> > >>
> > >>
> > > 
> > > Ok, then you will probably want to correlate your information with :
> > > 
> > > - scheduling activity regarding your threads
> > > - system call entry events, especially sys_futex. Note that a thread
> > >   calling sys_futex won't _necessarily_ be put to sleep.. it may still
> > >   be able to take the lock relatively quickly.
> > > 
> > > If you need more than that, we may think of instrumenting futex.c, but I
> > > am not sure this is required.
> > 
> > Haven't checked the state of instrumentation recently: Is the futex
> > operation visible in the trace now? It used to be not, and we often had
> > to guess the reason for sys_futex (wake, wait, pi or not pi, etc.) from
> > the context - or add ad-hoc instrumentation.
> > 
> > Jan
> > 
> 
> It still isn't instrumented, and I think it would be good to add such
> instrumentation.
> 
> Have a look at the patch done by K. Prasad for futex.c : it should
> probably be updated so it uses tracepoints instead of markers. Anyone
> would like to do this ?
> 
> See http://lkml.org/lkml/2008/4/15/125
> "[RFC PATCH 0/2] Debugging infrastructure for Futexes using Markers"
> 
> Mathieu
> 
> -- 

The patches pointed out at http://lkml.org/lkml/2008/4/15/125 aimed to
introduce static probes in the form of kernel markers in the futex
subsystem. Apart from tracing function boundaries and return values,
they were intended to be helpful in gathering performance measurements
and turn-around times for futex operations.

The patches did not elicit a favourable response then, and haven't been
converted to tracepoints (owing to the format string parameter
requirements for trace_mark and the fine-granular information being
exported by the proposed markers). I doubt if the community opinion has
changed now to accept the patch, if converted to tracepoints.

On a related note, just thinking if the proposed marker patch submission
from Google (http://lwn.net/Articles/298685/) has been sent to/accepted by
the community folks? It would be interesting to note details such as
granularity, level of information exported, number and choice of probe
points in the patch that is accepted by the community.

Thanks,
K.Prasad

Re: [ltt-dev] trace a futex

[Peter Zijlstra]

FWIW, the ftrace infrastructure has on many an occasion (even before it
was called ftrace and specific to -rt) helped in debugging and fixing
futex races.

Re: [ltt-dev] trace a futex

[Mathieu Desnoyers]

* Peter Zijlstra (peterz@infradead.org) wrote:
> 
> FWIW, the ftrace infrastructure has on many an occasion (even before it
> was called ftrace and specific to -rt) helped in debugging and fixing
> futex races.
> 

Hrm, I'm not sure futex races is the key aspect of interest here.
Knowing which amount of pthread mutex lock calls ends up calling the
scheduler looks a bit more like the topic brought by this particular
use-case. Therefore, correlating the information from the nptl with the
kernel information would be useful.

Is lockdep called when a futex is taken ? Should we add instrumentation
(tracepoints) to futex.c ? If yes, was there specific instrumentation
you used with ftrace that should be added ?

Mathieu

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [ltt-dev] trace a futex

[Peter Zijlstra]

On Fri, 2008-12-05 at 07:28 -0500, Mathieu Desnoyers wrote:
> * Peter Zijlstra (peterz@infradead.org) wrote:
> > 
> > FWIW, the ftrace infrastructure has on many an occasion (even before it
> > was called ftrace and specific to -rt) helped in debugging and fixing
> > futex races.
> > 
> 
> Hrm, I'm not sure futex races is the key aspect of interest here.
> Knowing which amount of pthread mutex lock calls ends up calling the
> scheduler looks a bit more like the topic brought by this particular
> use-case. Therefore, correlating the information from the nptl with the
> kernel information would be useful.

We already have the scheduler instrucmentation, so correlating that to
known futex calls (from userspace) shouldn't be too hard. 

> Is lockdep called when a futex is taken ? 

lockdep only does kernel-internal locks - so no.

> Should we add instrumentation
> (tracepoints) to futex.c ? If yes, was there specific instrumentation
> you used with ftrace that should be added ?

Not sure we should, Thomas did the bulk of that ftrace debugging, Thomas
do you think it would be worthwhile to add some tracepoints in there?

Re: [ltt-dev] trace a futex

[Peter Zijlstra]

[ resend with a real email address for Thomas ]

On Fri, 2008-12-05 at 07:28 -0500, Mathieu Desnoyers wrote:
> * Peter Zijlstra (peterz@infradead.org) wrote:
> > 
> > FWIW, the ftrace infrastructure has on many an occasion (even before it
> > was called ftrace and specific to -rt) helped in debugging and fixing
> > futex races.
> > 
> 
> Hrm, I'm not sure futex races is the key aspect of interest here.
> Knowing which amount of pthread mutex lock calls ends up calling the
> scheduler looks a bit more like the topic brought by this particular
> use-case. Therefore, correlating the information from the nptl with the
> kernel information would be useful.

We already have the scheduler instrucmentation, so correlating that to
known futex calls (from userspace) shouldn't be too hard. 

> Is lockdep called when a futex is taken ? 

lockdep only does kernel-internal locks - so no.

> Should we add instrumentation
> (tracepoints) to futex.c ? If yes, was there specific instrumentation
> you used with ftrace that should be added ?

Not sure we should, Thomas did the bulk of that ftrace debugging, Thomas
do you think it would be worthwhile to add some tracepoints in there?