[patch 0/1] extending low-level markers

[patch 0/1] extending low-level markers

[nwatkins]

Mathieu

I have been working with your Kernel Markers infrastructure now for some
time and have run into an extendability issue.

Essentially I am failing to find a way to extend the current
__trace_mark macro with site-specific context. That is, I would like the
ability to create different 'types' of instrumentation points by bulding
upon the __trace_mark macro. A consumer of this marker could examine
the type of marker, and attach an appropriate callback function /
private data.

I have included a patch which adds a flavor field to the __trace_mark
macro. This simplified example demonstrates the functionality I'm
looking for:

#define __trace_mark(flavor, name, format, args...)
		<current macro plus the flavor field>

#define marker_flavor_XXX(name, format, args...)
		__trace_mark(XXX, name, format, args)

Here a marker of type XXX is build upon the __trace_mark macro. When a
consumer of type XXX finds markers with the XXX flavor appropriate
registration can take place.

Unless I don't fully understand all the use cases of the markers, I
don't see any other way to do this except to encode the 'type'
information in the name of the marker, and require the consumer to parse
the string to determine the type. Restricting the names of the markers
in this way seems like a bad solution.

Any help and feedback is greatly appreciated.

- Noah

-- 

Re: [patch 0/1] extending low-level markers

[Mathieu Desnoyers]

* nwatkins@ittc.ku.edu (nwatkins@ittc.ku.edu) wrote:
> Mathieu
> 
> I have been working with your Kernel Markers infrastructure now for some
> time and have run into an extendability issue.

Hi Noah,

Can you tell us a little bit more about what you are doing with the
markers ? I guess it could be useful to know so we can get a sense of
how it fits in the big picture.

> 
> Essentially I am failing to find a way to extend the current
> __trace_mark macro with site-specific context. That is, I would like the
> ability to create different 'types' of instrumentation points by bulding
> upon the __trace_mark macro. A consumer of this marker could examine
> the type of marker, and attach an appropriate callback function /
> private data.
> 

You bring a good point. Actually, we have to consider a few things:

Currently, in order to keep things simple, only one callback can be
connected to a marker at any given time. I did it this way to keep is as
simple as possible. And it is always possible to create a multiplexer
callback if ever needed that would walk on a notifier list later to
support plugging multiple probes on a given marker. But I would not like
to see the critical path of a tracer be _required_ to use a notifier
chain when the typical case is to have a single callback connected.

The current approach is to use the marker name as a way to specify
markers "group". If we go with a "flavor" enumeration instead, we would
have to add an enumeration of every markers users in marker.h, which I
am a bit reluctant to do.

Also, what makes your markers unsuitable for other probes ?

There are cases where a "generic" callback will do 95% of the job, but
there are always subtile cases where more intelligent probes are needed.
It's the case of blktrace, which must test if some arguments are NULL in
order to know if they can be dereferenced and thus if the event must be
considered. We can deal with this in two ways:
- in the kernel code, inside an immediate_if(), surrounding the marker.
  Advantage: the test is done inline, so the event can be discarded
  inline. Disadvantage: it adds a few bytes to the kernel image when
  tracing modules and probes are not loaded.
- in the probe module connected to the marker.
  +: The code is not there if the probe module is not loaded
  -: we have to do a function call before we can test some conditions
     when the probe is connected.

In my following markers version, the format strings are checked by GCC
and it emits warnings if va args doesn't match. I hope your format
strings are consistent with the arguments..

Once we know more about what you are trying to do with the markers,
we'll be in better position to bring the discussion forward.

Regards,

Mathieu


> I have included a patch which adds a flavor field to the __trace_mark
> macro. This simplified example demonstrates the functionality I'm
> looking for:
> 
> #define __trace_mark(flavor, name, format, args...)
> 		<current macro plus the flavor field>
> 
> #define marker_flavor_XXX(name, format, args...)
> 		__trace_mark(XXX, name, format, args)
> 
> Here a marker of type XXX is build upon the __trace_mark macro. When a
> consumer of type XXX finds markers with the XXX flavor appropriate
> registration can take place.
> 
> Unless I don't fully understand all the use cases of the markers, I
> don't see any other way to do this except to encode the 'type'
> information in the name of the marker, and require the consumer to parse
> the string to determine the type. Restricting the names of the markers
> in this way seems like a bad solution.
> 
> Any help and feedback is greatly appreciated.
> 
> - Noah
> 
> -- 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [patch 0/1] extending low-level markers

[Noah Watkins]

On 02/08/07 12:44 -0400, Mathieu Desnoyers wrote:
> * nwatkins@ittc.ku.edu (nwatkins@ittc.ku.edu) wrote:
> > Mathieu
> > 
> > I have been working with your Kernel Markers infrastructure now for some
> > time and have run into an extendability issue.
> 
> Hi Noah,
> 
> Can you tell us a little bit more about what you are doing with the
> markers ? I guess it could be useful to know so we can get a sense of
> how it fits in the big picture.

We have been maintaining an in-house instrumentation framework. The
instrumenation points we use differ from Kernel Markers in that the desired
callback is attached at compile time. Other than this difference, they
are implemented in much the same way. The biggest difference is that we
have been implicitly encoding the 'type' of instrumenation point by
assigning a particular callback. For example one type of instrumenation
point in our framework has the form:

ds_event(name, int, size_t, void *);

Thus, a ds_event has a definite set of arguments, and some name, which
is compatible with a trace_mark.

We would like to be able to construct a ds_event on top of the low-level
kernel markers and be able to differentiate between a standard trace_mark in
in the markers section, and a trace_mark which actually represents a
ds_event, or another event type.

Granted we could simply match on compatible argument formats, its only
that a separation between types would be nice in order to not touch
markers which are of not interest to us, hence this notion of type.
Our framework would have blindly attached some callback to all
points which had matching argument formats, perhaps many that were not
inserted using ds_event (or some other api built on top of markers).

The biggest problem we are facing now is with a set of points which
cooperate together in the sense that one references the others (or some
other connection topology). For example:

ds_point_start(NAME, params)
ds_point_end(NAME, params)

In this situation ds_point_start collects some data, stashes it in its
private data area,  and ds_point_end references the corresponding
ds_point_start when it fires. The two points are wired up by our
framework (using the marker's private data) and the NAME, in order to
avoid lookups at logging time.

So, in this case the names are the same which logically links them, but
their functionality is different. The difference in functionality could
again be encoded by the 'type' of point.

> The current approach is to use the marker name as a way to specify
> markers "group". If we go with a "flavor" enumeration instead, we would
> have to add an enumeration of every markers users in marker.h, which I
> am a bit reluctant to do.

This would have to be my biggest complaint with the 'flavor' concept as
well. However, if all points in main-line always used no concept of
flavor (or essentially the default flavor) then users wishing to use the
flavor enumeration out of main-line development could do so?

Hope this helps portray more of what we are trying to do.

Noah

Re: [patch 0/1] extending low-level markers

[Mathieu Desnoyers]

* Noah Watkins (nwatkins@ittc.ku.edu) wrote:
> On 02/08/07 12:44 -0400, Mathieu Desnoyers wrote:
> > * nwatkins@ittc.ku.edu (nwatkins@ittc.ku.edu) wrote:
> > > Mathieu
> > > 
> > > I have been working with your Kernel Markers infrastructure now for some
> > > time and have run into an extendability issue.
> > 
> > Hi Noah,
> > 
> > Can you tell us a little bit more about what you are doing with the
> > markers ? I guess it could be useful to know so we can get a sense of
> > how it fits in the big picture.
> 
> We have been maintaining an in-house instrumentation framework. The
> instrumenation points we use differ from Kernel Markers in that the desired
> callback is attached at compile time. Other than this difference, they
> are implemented in much the same way. The biggest difference is that we
> have been implicitly encoding the 'type' of instrumenation point by
> assigning a particular callback. For example one type of instrumenation
> point in our framework has the form:
> 
> ds_event(name, int, size_t, void *);
> 
> Thus, a ds_event has a definite set of arguments, and some name, which
> is compatible with a trace_mark.
> 
> We would like to be able to construct a ds_event on top of the low-level
> kernel markers and be able to differentiate between a standard trace_mark in
> in the markers section, and a trace_mark which actually represents a
> ds_event, or another event type.
> 
> Granted we could simply match on compatible argument formats, its only
> that a separation between types would be nice in order to not touch
> markers which are of not interest to us, hence this notion of type.
> Our framework would have blindly attached some callback to all
> points which had matching argument formats, perhaps many that were not
> inserted using ds_event (or some other api built on top of markers).
> 

Hrm, what is wrong with :

trace_mark(ds_myevent, "%d %zu %p", arg1, arg2, arg3);

then ?

You could then attach your probe to all markers staring with a ds_
prefix. (we should keep a list of the used prefixes somewhere) You could
match with format strings to figure out which callback should be
connected to which marker (if you don't plan to have your callback
parsing the format string dynamically).


> The biggest problem we are facing now is with a set of points which
> cooperate together in the sense that one references the others (or some
> other connection topology). For example:
> 
> ds_point_start(NAME, params)
> ds_point_end(NAME, params)
> 
> In this situation ds_point_start collects some data, stashes it in its
> private data area,  and ds_point_end references the corresponding
> ds_point_start when it fires. The two points are wired up by our
> framework (using the marker's private data) and the NAME, in order to
> avoid lookups at logging time.
> 

I wonder how you do your locking there ? I guess you are writing data
that *should* reside on the stack into the private data, which is
somewhat static. A solution to this that would be both not too intrusive
for kernel code and fast enough would be to keep a buffer of such
saved/read data, indexed with a cookie that would be passed from _start
to _end. The only issue is that this cookie would have to be placed on
the kernel stack, its space being reserved even when markers are
disabled.


> So, in this case the names are the same which logically links them, but
> their functionality is different. The difference in functionality could
> again be encoded by the 'type' of point.
> 

if you differentiate them with names like:
ds_point_start and ds_point_end

and pass the cookie as write parameter to _start and read parameter to
_end:

void somefct()
{
  long cookie;

  trace_mark(ds_point_start, "%p %d %zu %p", &cookie, arg1, arg2, arg3);

  ...

  trace_mark(ds_point_end, "%lu %d %zu %p", cookie, arg1, arg2, arg3);
}
It should do the job ?
(note: if mutual exclusion access to a static variable is insured by
proper locking in the kernel, then you don't need to do such trick)

> > The current approach is to use the marker name as a way to specify
> > markers "group". If we go with a "flavor" enumeration instead, we would
> > have to add an enumeration of every markers users in marker.h, which I
> > am a bit reluctant to do.
> 
> This would have to be my biggest complaint with the 'flavor' concept as
> well. However, if all points in main-line always used no concept of
> flavor (or essentially the default flavor) then users wishing to use the
> flavor enumeration out of main-line development could do so?
> 

Hrm, the ideal thing would be to agree on an instrumentation set for a
given subsystem/driver and to get the said instrumentation integrated in
mainline. That would sound like a better way to stop reinventing the
wheel forever. And actually, if there are some features in your tracer
that you would like to add into a mainline tracer, I'll be glad to
discuss those with you. A lot of people out there are facing the same
issues as you are anyway :)

Ideally, a marker should express what the kernel code is doing and what
information we want to extract from it more than being tied to one
particular consumer (probe).

> Hope this helps portray more of what we are trying to do.
> 

It sure helps, thanks :)

Mathieu

> Noah

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [patch 0/1] extending low-level markers

[Noah Watkins]

> Hrm, what is wrong with :
> 
> trace_mark(ds_myevent, "%d %zu %p", arg1, arg2, arg3);
> 
> then ?
> 
> You could then attach your probe to all markers staring with a ds_
> prefix. (we should keep a list of the used prefixes somewhere) You could
> match with format strings to figure out which callback should be
> connected to which marker (if you don't plan to have your callback
> parsing the format string dynamically).

This has been what I considered the only way to accomplish the 'type'
feature without adding another field to the struct, and is just fine,
especially if prefixes are listed some where, or not. I was just
interested in the possibility of adding types.

> 
> 
> > The biggest problem we are facing now is with a set of points which
> > cooperate together in the sense that one references the others (or some
> > other connection topology). For example:
> > 
> > ds_point_start(NAME, params)
> > ds_point_end(NAME, params)
> > 
> > In this situation ds_point_start collects some data, stashes it in its
> > private data area,  and ds_point_end references the corresponding
> > ds_point_start when it fires. The two points are wired up by our
> > framework (using the marker's private data) and the NAME, in order to
> > avoid lookups at logging time.
> > 
> 
> I wonder how you do your locking there ? I guess you are writing data
> that *should* reside on the stack into the private data, which is
> somewhat static. A solution to this that would be both not too intrusive
> for kernel code and fast enough would be to keep a buffer of such
> saved/read data, indexed with a cookie that would be passed from _start
> to _end. The only issue is that this cookie would have to be placed on
> the kernel stack, its space being reserved even when markers are
> disabled.
> 

The locking is internal to our framework and done on a per-type basis.
The start/end example i mentioned could represent an interval. When the
two points are wired up they both have their private data pointing at a
kmalloc'd interval structure internal to our framework, which contains
a lock.

> 
> > So, in this case the names are the same which logically links them, but
> > their functionality is different. The difference in functionality could
> > again be encoded by the 'type' of point.
> > 
> 
> if you differentiate them with names like:
> ds_point_start and ds_point_end
> 
> and pass the cookie as write parameter to _start and read parameter to
> _end:
> 
> void somefct()
> {
>   long cookie;
> 
>   trace_mark(ds_point_start, "%p %d %zu %p", &cookie, arg1, arg2, arg3);
> 
>   ...
> 
>   trace_mark(ds_point_end, "%lu %d %zu %p", cookie, arg1, arg2, arg3);
> }
> It should do the job ?
> (note: if mutual exclusion access to a static variable is insured by
> proper locking in the kernel, then you don't need to do such trick)

The cookie approach would accomplish the functionality, though it does
complicate the procedure of adding the instrumenation point. If the
start and the end events are not in the same function/file then
the linking of the cookie could be too instrusive.

In all of our cases the cookie would certainly be a void* pointing to
internal data structures. What is the objection to using the marker's
private data to point to the internal structures?

Thanks for helping out. I realize the situations I'm describing are
quite specialized to our particular situation. Many students here use
the framework for learning systems programming, and as such it evolves
often.

- Noah

> 
> > > The current approach is to use the marker name as a way to specify
> > > markers "group". If we go with a "flavor" enumeration instead, we would
> > > have to add an enumeration of every markers users in marker.h, which I
> > > am a bit reluctant to do.
> > 
> > This would have to be my biggest complaint with the 'flavor' concept as
> > well. However, if all points in main-line always used no concept of
> > flavor (or essentially the default flavor) then users wishing to use the
> > flavor enumeration out of main-line development could do so?
> > 
> 
> Hrm, the ideal thing would be to agree on an instrumentation set for a
> given subsystem/driver and to get the said instrumentation integrated in
> mainline. That would sound like a better way to stop reinventing the
> wheel forever. And actually, if there are some features in your tracer
> that you would like to add into a mainline tracer, I'll be glad to
> discuss those with you. A lot of people out there are facing the same
> issues as you are anyway :)
> 
> Ideally, a marker should express what the kernel code is doing and what
> information we want to extract from it more than being tied to one
> particular consumer (probe).
> 
> > Hope this helps portray more of what we are trying to do.
> > 
> 
> It sure helps, thanks :)
> 
> Mathieu
> 
> > Noah
> 
> -- 
> Mathieu Desnoyers
> Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [patch 0/1] extending low-level markers

[Mathieu Desnoyers]

* Noah Watkins (nwatkins@ittc.ku.edu) wrote:
> > Hrm, what is wrong with :
> > 
> > trace_mark(ds_myevent, "%d %zu %p", arg1, arg2, arg3);
> > 
> > then ?
> > 
> > You could then attach your probe to all markers staring with a ds_
> > prefix. (we should keep a list of the used prefixes somewhere) You could
> > match with format strings to figure out which callback should be
> > connected to which marker (if you don't plan to have your callback
> > parsing the format string dynamically).
> 
> This has been what I considered the only way to accomplish the 'type'
> feature without adding another field to the struct, and is just fine,
> especially if prefixes are listed some where, or not. I was just
> interested in the possibility of adding types.
> 
> > 
> > 
> > > The biggest problem we are facing now is with a set of points which
> > > cooperate together in the sense that one references the others (or some
> > > other connection topology). For example:
> > > 
> > > ds_point_start(NAME, params)
> > > ds_point_end(NAME, params)
> > > 
> > > In this situation ds_point_start collects some data, stashes it in its
> > > private data area,  and ds_point_end references the corresponding
> > > ds_point_start when it fires. The two points are wired up by our
> > > framework (using the marker's private data) and the NAME, in order to
> > > avoid lookups at logging time.
> > > 
> > 
> > I wonder how you do your locking there ? I guess you are writing data
> > that *should* reside on the stack into the private data, which is
> > somewhat static. A solution to this that would be both not too intrusive
> > for kernel code and fast enough would be to keep a buffer of such
> > saved/read data, indexed with a cookie that would be passed from _start
> > to _end. The only issue is that this cookie would have to be placed on
> > the kernel stack, its space being reserved even when markers are
> > disabled.
> > 
> 
> The locking is internal to our framework and done on a per-type basis.
> The start/end example i mentioned could represent an interval. When the
> two points are wired up they both have their private data pointing at a
> kmalloc'd interval structure internal to our framework, which contains
> a lock.
> 

Then this lock is taken in _start and unlocked in _end ? What happens if
the _start and _end markers are enabled/disabled in a non-equilibrated
way while the kernel code is being executed ? Would it deadlock the next
time the lock is taken due to a missing unlock ?

> > 
> > > So, in this case the names are the same which logically links them, but
> > > their functionality is different. The difference in functionality could
> > > again be encoded by the 'type' of point.
> > > 
> > 
> > if you differentiate them with names like:
> > ds_point_start and ds_point_end
> > 
> > and pass the cookie as write parameter to _start and read parameter to
> > _end:
> > 
> > void somefct()
> > {
> >   long cookie;
> > 
> >   trace_mark(ds_point_start, "%p %d %zu %p", &cookie, arg1, arg2, arg3);
> > 
> >   ...
> > 
> >   trace_mark(ds_point_end, "%lu %d %zu %p", cookie, arg1, arg2, arg3);
> > }
> > It should do the job ?
> > (note: if mutual exclusion access to a static variable is insured by
> > proper locking in the kernel, then you don't need to do such trick)
> 
> The cookie approach would accomplish the functionality, though it does
> complicate the procedure of adding the instrumenation point. If the
> start and the end events are not in the same function/file then
> the linking of the cookie could be too instrusive.
> 
> In all of our cases the cookie would certainly be a void* pointing to
> internal data structures. What is the objection to using the marker's
> private data to point to the internal structures?
> 

SMP, preemption, all those things while requires locking around a static
variable. And the problem of non atomicity of connexion/disconnexion of
multiple probes vs unbalanced lock/unlock.

> Thanks for helping out. I realize the situations I'm describing are
> quite specialized to our particular situation. Many students here use
> the framework for learning systems programming, and as such it evolves
> often.
> 

No problem. I think some interesting things came come out of this.

Mathieu

> - Noah
> 
> > 
> > > > The current approach is to use the marker name as a way to specify
> > > > markers "group". If we go with a "flavor" enumeration instead, we would
> > > > have to add an enumeration of every markers users in marker.h, which I
> > > > am a bit reluctant to do.
> > > 
> > > This would have to be my biggest complaint with the 'flavor' concept as
> > > well. However, if all points in main-line always used no concept of
> > > flavor (or essentially the default flavor) then users wishing to use the
> > > flavor enumeration out of main-line development could do so?
> > > 
> > 
> > Hrm, the ideal thing would be to agree on an instrumentation set for a
> > given subsystem/driver and to get the said instrumentation integrated in
> > mainline. That would sound like a better way to stop reinventing the
> > wheel forever. And actually, if there are some features in your tracer
> > that you would like to add into a mainline tracer, I'll be glad to
> > discuss those with you. A lot of people out there are facing the same
> > issues as you are anyway :)
> > 
> > Ideally, a marker should express what the kernel code is doing and what
> > information we want to extract from it more than being tied to one
> > particular consumer (probe).
> > 
> > > Hope this helps portray more of what we are trying to do.
> > > 
> > 
> > It sure helps, thanks :)
> > 
> > Mathieu
> > 
> > > Noah
> > 
> > -- 
> > Mathieu Desnoyers
> > Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
> > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [patch 0/1] extending low-level markers

[Noah Watkins]

> > > 
> > 
> > The locking is internal to our framework and done on a per-type basis.
> > The start/end example i mentioned could represent an interval. When the
> > two points are wired up they both have their private data pointing at a
> > kmalloc'd interval structure internal to our framework, which contains
> > a lock.
> > 
> 
> Then this lock is taken in _start and unlocked in _end ? What happens if
> the _start and _end markers are enabled/disabled in a non-equilibrated
> way while the kernel code is being executed ? Would it deadlock the next
> time the lock is taken due to a missing unlock ?

The locks are aquired and released in each _start and _end marker, so
the equilibrium is not a issue.

A more sane example is the insertion of values into a histogram. Instead
of the instrumenation point logging the values and having the histogram
constructed during a post-process step, data structures implemented the
histogrm are associated with the instrumenation point. The lock protects
this structure in a more intuitive way than the interval example.

We have additional instrumenation points associated with the histogram
which for example trigger the logging of the histgoram when conditions
are met (conditions are implied by the placement of the marker).

In the interval example, they provide the data consistency, and
make no attempt to insure that the markers are used in a way that
makes sense (for example an interval in a preemptable re-entrant path).

So you can see that encoding the types of markers in the name of the
marker may become cumbersome as the amount of combinations grow.

> 
> > 
> > The cookie approach would accomplish the functionality, though it does
> > complicate the procedure of adding the instrumenation point. If the
> > start and the end events are not in the same function/file then
> > the linking of the cookie could be too instrusive.
> > 
> > In all of our cases the cookie would certainly be a void* pointing to
> > internal data structures. What is the objection to using the marker's
> > private data to point to the internal structures?
> > 
> 
> SMP, preemption, all those things while requires locking around a static
> variable. And the problem of non atomicity of connexion/disconnexion of
> multiple probes vs unbalanced lock/unlock.

I think most of problems involving non atomicity of the
connection/disconnection of probes can be accomplished by good house
keeping and settings things up in phases which insure consistency. For
example wiring things up before enabling events. An issue such as
hitting a _end event before a _start event can easily be handled by
whatever framework the markers are attached to.

- Noah

> 
> > Thanks for helping out. I realize the situations I'm describing are
> > quite specialized to our particular situation. Many students here use
> > the framework for learning systems programming, and as such it evolves
> > often.
> > 
> 
> No problem. I think some interesting things came come out of this.
> 
> Mathieu
> 
> > - Noah
> > 
> > > 
> > > > > The current approach is to use the marker name as a way to specify
> > > > > markers "group". If we go with a "flavor" enumeration instead, we would
> > > > > have to add an enumeration of every markers users in marker.h, which I
> > > > > am a bit reluctant to do.
> > > > 
> > > > This would have to be my biggest complaint with the 'flavor' concept as
> > > > well. However, if all points in main-line always used no concept of
> > > > flavor (or essentially the default flavor) then users wishing to use the
> > > > flavor enumeration out of main-line development could do so?
> > > > 
> > > 
> > > Hrm, the ideal thing would be to agree on an instrumentation set for a
> > > given subsystem/driver and to get the said instrumentation integrated in
> > > mainline. That would sound like a better way to stop reinventing the
> > > wheel forever. And actually, if there are some features in your tracer
> > > that you would like to add into a mainline tracer, I'll be glad to
> > > discuss those with you. A lot of people out there are facing the same
> > > issues as you are anyway :)
> > > 
> > > Ideally, a marker should express what the kernel code is doing and what
> > > information we want to extract from it more than being tied to one
> > > particular consumer (probe).
> > > 
> > > > Hope this helps portray more of what we are trying to do.
> > > > 
> > > 
> > > It sure helps, thanks :)
> > > 
> > > Mathieu
> > > 
> > > > Noah
> > > 
> > > -- 
> > > Mathieu Desnoyers
> > > Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
> > > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> 
> -- 
> Mathieu Desnoyers
> Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [patch 0/1] extending low-level markers

[Frank Ch. Eigler]

Noah Watkins <nwatkins@ittc.ku.edu> writes:

> [...]
> The locks are aquired and released in each _start and _end marker, so
> the equilibrium is not a issue.

But it becomes an issue should preemption, or control flow upset such
as an early return or recursion, occurs between the start and end
markers.

> A more sane example is the insertion of values into a histogram. Instead
> of the instrumenation point logging the values and having the histogram
> constructed during a post-process step, data structures implemented the
> histogrm are associated with the instrumenation point. The lock protects
> this structure in a more intuitive way than the interval example.

A better way may be to do what we do in systemtap scripts: in a
interval-start type probe, store the start value in a table indexed by
thread-id and nesting-level; in the interval-end type probe, find the
corresponding value, compute the difference (as appropriate), and
store the result only then into your long-term lookup table.  This
approach requires *no locks* to survive between start & end: the
probes remain atomic.

> I think most of problems involving non atomicity of the
> connection/disconnection of probes can be accomplished by good house
> keeping and settings things up in phases which insure consistency.

A marker callback can take shortcuts, I guess, if it allows itself to
make assumptions about the ways it can be called.

I scanned over the thread, but still haven't seen a specific argument
for extra marker-type annotations that would relate to this issue.
What tool would need to scan the available markers, and how would
extra information allow it to do its job?

- FChE