collecting static data related to tracing

collecting static data related to tracing

[Johannes Berg]

Hi,

We've got some tracing in the wireless subsystem that allows us to find
out what's going on quite well. I frequently ask users to use trace-cmd
to record data and send me the resulting file. (Oh and please ... I
don't care if perf could do this as well, trace-cmd works great for me)

However, in all of this I still have to ask users for things like their
hardware and firmware version etc. I also ask for the kernel version,
but that would be easy to simply record in trace-cmd. However, device
versions are very specific to the tracer in use.

I'm thinking that we could have per subsystem "detail" files that are
provided by the respective subsystem implementation in some way, and can
then simply be included in the recorded trace file. However, I have no
idea if that is feasible to implement, or if maybe there's another way.
FWIW, all my events already contain a pointer, used as a cookie, to
identify the hardware instance, but that only allows me to read a trace
that contains data about multiple devices, it doesn't give me
information about each device. Ideally, the "detail" file would list all
the information along with the cookie, so I could connect the dots.

Thoughts?

johannes

Re: collecting static data related to tracing

[Steven Rostedt]

On Tue, 2010-05-18 at 16:09 +0200, Johannes Berg wrote:

> I'm thinking that we could have per subsystem "detail" files that are
> provided by the respective subsystem implementation in some way, and can
> then simply be included in the recorded trace file. However, I have no
> idea if that is feasible to implement, or if maybe there's another way.
> FWIW, all my events already contain a pointer, used as a cookie, to
> identify the hardware instance, but that only allows me to read a trace
> that contains data about multiple devices, it doesn't give me
> information about each device. Ideally, the "detail" file would list all
> the information along with the cookie, so I could connect the dots.
> 
> Thoughts?

The latest version of trace-cmd has an "options" section. This allows
you to add options to the file.

We could make a plugin that also can be used by trace-cmd record, that
allows you to add options. The options are written such that if a
trace-cmd does not know how to deal with them, they will be ignored.

Hmm, but the options require a unique ID. Well we could register IDs
with plugins, or add a plugin id, which uses the name of the plugin as
an identifier too.

But this would allow you to add the details you want about the system
and then have the reader be able to print it out.

How's that sound?

-- Steve

Re: collecting static data related to tracing

[Johannes Berg]

On Tue, 2010-05-18 at 10:41 -0400, Steven Rostedt wrote:

> The latest version of trace-cmd has an "options" section. This allows
> you to add options to the file.
> 
> We could make a plugin that also can be used by trace-cmd record, that
> allows you to add options. The options are written such that if a
> trace-cmd does not know how to deal with them, they will be ignored.
> 
> Hmm, but the options require a unique ID. Well we could register IDs
> with plugins, or add a plugin id, which uses the name of the plugin as
> an identifier too.
> 
> But this would allow you to add the details you want about the system
> and then have the reader be able to print it out.
> 
> How's that sound?

Sounds good, although it does require that I tell people who want to
record to also install the recording plugin, but that should be
manageable :) I can just dig out the data from the regular debugfs once
I add files containing it.

johannes

Re: collecting static data related to tracing

[Frederic Weisbecker]

On Tue, May 18, 2010 at 04:45:09PM +0200, Johannes Berg wrote:
> On Tue, 2010-05-18 at 10:41 -0400, Steven Rostedt wrote:
> 
> > The latest version of trace-cmd has an "options" section. This allows
> > you to add options to the file.
> > 
> > We could make a plugin that also can be used by trace-cmd record, that
> > allows you to add options. The options are written such that if a
> > trace-cmd does not know how to deal with them, they will be ignored.
> > 
> > Hmm, but the options require a unique ID. Well we could register IDs
> > with plugins, or add a plugin id, which uses the name of the plugin as
> > an identifier too.
> > 
> > But this would allow you to add the details you want about the system
> > and then have the reader be able to print it out.
> > 
> > How's that sound?
> 
> Sounds good, although it does require that I tell people who want to
> record to also install the recording plugin, but that should be
> manageable :) I can just dig out the data from the regular debugfs once
> I add files containing it.
> 
> johannes


This is a place where events injection might be suitable perhaps.
Either kernel or user space event injection.

kernel space injection would be a simple trace event declared that
have a callback called when it gets enabled. This callback would
inject any events it wants.

userspace injection could be a bit different, the user can inject
its own format and events content toward a debugfs or whatever file.
This would be suitable if userspace have few things to inject,
otherwise we would need to think about something else perhaps.

Re: collecting static data related to tracing

[Johannes Berg]

On Wed, 2010-05-19 at 09:28 +0200, Frederic Weisbecker wrote:

> > Sounds good, although it does require that I tell people who want to
> > record to also install the recording plugin, but that should be
> > manageable :) I can just dig out the data from the regular debugfs once
> > I add files containing it.

> This is a place where events injection might be suitable perhaps.
> Either kernel or user space event injection.
> 
> kernel space injection would be a simple trace event declared that
> have a callback called when it gets enabled. This callback would
> inject any events it wants.

Indeed, that is a good point, and it would also allow us to handle
hotplugging with the same event by sending it when a new device shows
up, and additionally when it is enabled to dump the current state.

johannes

Re: collecting static data related to tracing

[Steven Rostedt]

On Wed, 2010-05-19 at 09:28 +0200, Frederic Weisbecker wrote:
> On Tue, May 18, 2010 at 04:45:09PM +0200, Johannes Berg wrote:

> This is a place where events injection might be suitable perhaps.
> Either kernel or user space event injection.
> 
> kernel space injection would be a simple trace event declared that
> have a callback called when it gets enabled. This callback would
> inject any events it wants.
> 
> userspace injection could be a bit different, the user can inject
> its own format and events content toward a debugfs or whatever file.
> This would be suitable if userspace have few things to inject,
> otherwise we would need to think about something else perhaps.
> 

The userspace injection actually sounds like a good idea. I don't think
it needs to tell the kernel about it. Since it is running the trace.
Perhaps just have a way that the userspace can add its own events to the
recording file.

I guess this would be closer to what LTTng does with channels.

That is, you could have a separate recording stream to write events that
tell the tracer what is happening. It would be nice to have a way that
userspace can read the trace clock timer though.

-- Steve