Building a tracing userspace tool in the kernel tree

Building a tracing userspace tool in the kernel tree

[Mathieu Desnoyers]

Hi Sam,

At the kernel summit, people seemed to be interested to have the basic
userspace tools required to extract and pretty-print a trace available
within the kernel tree. Therefore, what I am trying to do is something
along the lines of

ltt/usr/
ltt/usr/tracectl/    (control tracing)
ltt/usr/tracesplice/ (splice buffers to disk)
ltt/usr/tracecat/    (merge sort and format the binary buffers into
                      human-readable text)

That would however require to create a Makefile which behaves a little
bit like what scripts/ is currently doing with hostprogs-y, only that
it's different in the sense that those tools are not required to build
the kernel and this could therefore become a more standard part of the
build process than what scripts/ is.

Is there some magic statements to put in Makefile and ltt/Makefile to
get this to build nicely with the rest of the tree ?

Thanks,

Mathieu


-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: Building a tracing userspace tool in the kernel tree

[Andrew Morton]

On Thu, 9 Oct 2008 15:16:26 -0400
Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca> wrote:

> Hi Sam,
> 
> At the kernel summit, people seemed to be interested to have the basic
> userspace tools required to extract and pretty-print a trace available
> within the kernel tree. Therefore, what I am trying to do is something
> along the lines of
> 
> ltt/usr/
> ltt/usr/tracectl/    (control tracing)
> ltt/usr/tracesplice/ (splice buffers to disk)
> ltt/usr/tracecat/    (merge sort and format the binary buffers into
>                       human-readable text)
> 
> That would however require to create a Makefile which behaves a little
> bit like what scripts/ is currently doing with hostprogs-y, only that
> it's different in the sense that those tools are not required to build
> the kernel and this could therefore become a more standard part of the
> build process than what scripts/ is.
> 
> Is there some magic statements to put in Makefile and ltt/Makefile to
> get this to build nicely with the rest of the tree ?
> 

Sam's git://git.kernel.org/pub/scm/linux/kernel/git/sam/test.git#master
tree contains a tests/ directory in which various userspace things are
compiled.  I'd suggest you build on top of that.

Sam, this tells us that what you have there is insufficiently general. 
Mathieu's tracing thing isn't a "test", but it's userspace.  We should
have

	./userspace/tests/
	./userspace/ltt/

no?

Re: Building a tracing userspace tool in the kernel tree

[Peter Zijlstra]

On Thu, 2008-10-09 at 15:16 -0400, Mathieu Desnoyers wrote:
> Hi Sam,
> 
> At the kernel summit, people seemed to be interested to have the basic
> userspace tools required to extract and pretty-print a trace available
> within the kernel tree. Therefore, what I am trying to do is something
> along the lines of
> 
> ltt/usr/
> ltt/usr/tracectl/    (control tracing)
> ltt/usr/tracesplice/ (splice buffers to disk)
> ltt/usr/tracecat/    (merge sort and format the binary buffers into
>                       human-readable text)

I'd rather have you provide that interface from the kernel much like
ftrace does. So we can do:

# cat /debug/tracing/lttng/trace

Re: Building a tracing userspace tool in the kernel tree

[Chris Friesen]

Peter Zijlstra wrote:
> On Thu, 2008-10-09 at 15:16 -0400, Mathieu Desnoyers wrote:
> 
>>Hi Sam,
>>
>>At the kernel summit, people seemed to be interested to have the basic
>>userspace tools required to extract and pretty-print a trace available
>>within the kernel tree. Therefore, what I am trying to do is something
>>along the lines of
>>
>>ltt/usr/
>>ltt/usr/tracectl/    (control tracing)
>>ltt/usr/tracesplice/ (splice buffers to disk)
>>ltt/usr/tracecat/    (merge sort and format the binary buffers into
>>                      human-readable text)
> 
> 
> I'd rather have you provide that interface from the kernel much like
> ftrace does. So we can do:
> 
> # cat /debug/tracing/lttng/trace
> 

Do we really want to reserve memory in the kernel to store all the data? 
  Assuming not, do we really want to have to deal with filesystem 
namespaces in the kernel when interpreting which file we want to log to?

Chris

Re: Building a tracing userspace tool in the kernel tree

[Peter Zijlstra]

On Thu, 2008-10-09 at 16:35 -0600, Chris Friesen wrote:
> Peter Zijlstra wrote:
> > On Thu, 2008-10-09 at 15:16 -0400, Mathieu Desnoyers wrote:
> > 
> >>Hi Sam,
> >>
> >>At the kernel summit, people seemed to be interested to have the basic
> >>userspace tools required to extract and pretty-print a trace available
> >>within the kernel tree. Therefore, what I am trying to do is something
> >>along the lines of
> >>
> >>ltt/usr/
> >>ltt/usr/tracectl/    (control tracing)
> >>ltt/usr/tracesplice/ (splice buffers to disk)
> >>ltt/usr/tracecat/    (merge sort and format the binary buffers into
> >>                      human-readable text)
> > 
> > 
> > I'd rather have you provide that interface from the kernel much like
> > ftrace does. So we can do:
> > 
> > # cat /debug/tracing/lttng/trace
> > 
> 
> Do we really want to reserve memory in the kernel to store all the data? 
>   Assuming not, do we really want to have to deal with filesystem 
> namespaces in the kernel when interpreting which file we want to log to?

Not quite sure I get what you mean here. The kernel already needs the
memory anyway, as we keep the trace buffers in memory in either case.

All this does is provide a debugfs interface that does the exact same
thing the tracecat proglet would otherwise do.

I don't know how filesystem namespaces and debugfs interact, but seeing
as non of the debugfs users seem to be bothered with that, I don't see
why we should be.

Re: Building a tracing userspace tool in the kernel tree

[Chris Friesen]

Peter Zijlstra wrote:
> On Thu, 2008-10-09 at 16:35 -0600, Chris Friesen wrote:
>>Peter Zijlstra wrote:
>>>On Thu, 2008-10-09 at 15:16 -0400, Mathieu Desnoyers wrote:

>>>>At the kernel summit, people seemed to be interested to have the basic
>>>>userspace tools required to extract and pretty-print a trace available
>>>>within the kernel tree. Therefore, what I am trying to do is something
>>>>along the lines of
>>>>
>>>>ltt/usr/
>>>>ltt/usr/tracectl/    (control tracing)
>>>>ltt/usr/tracesplice/ (splice buffers to disk)
>>>>ltt/usr/tracecat/    (merge sort and format the binary buffers into
>>>>                     human-readable text)

>>>I'd rather have you provide that interface from the kernel much like
>>>ftrace does. So we can do:
>>>
>>># cat /debug/tracing/lttng/trace

>>Do we really want to reserve memory in the kernel to store all the data? 
>>  Assuming not, do we really want to have to deal with filesystem 
>>namespaces in the kernel when interpreting which file we want to log to?
> 
> 
> Not quite sure I get what you mean here. The kernel already needs the
> memory anyway, as we keep the trace buffers in memory in either case.
> 
> All this does is provide a debugfs interface that does the exact same
> thing the tracecat proglet would otherwise do.
> 
> I don't know how filesystem namespaces and debugfs interact, but seeing
> as non of the debugfs users seem to be bothered with that, I don't see
> why we should be.

Maybe I misunderstood something.  I was under the impression that the 
standard LTT usage is to stream raw trace data to disk and then 
post-process it.  If we're writing to disk, we should probably think 
about filesystem namespaces.

Chris

Re: Building a tracing userspace tool in the kernel tree

[Mathieu Desnoyers]

* Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> On Thu, 2008-10-09 at 15:16 -0400, Mathieu Desnoyers wrote:
> > Hi Sam,
> > 
> > At the kernel summit, people seemed to be interested to have the basic
> > userspace tools required to extract and pretty-print a trace available
> > within the kernel tree. Therefore, what I am trying to do is something
> > along the lines of
> > 
> > ltt/usr/
> > ltt/usr/tracectl/    (control tracing)
> > ltt/usr/tracesplice/ (splice buffers to disk)
> > ltt/usr/tracecat/    (merge sort and format the binary buffers into
> >                       human-readable text)
> 
> I'd rather have you provide that interface from the kernel much like
> ftrace does. So we can do:
> 
> # cat /debug/tracing/lttng/trace
> 

Hi Peter,

Following our earlier design discussions, the "tracectl" program will
be easily replaceable by a debugfs interface. For now, I plan to ship my
current netlink-based tracer control version given this single kernel
module is trivially replaceable by the debugfs interface in a near
future.

On the other hand, cat won't give the sufficient performance expected
for tracing. This will be provided by "tracesplice" and "tracecat" which
will take care of exporting the binary buffer data to disk and to decode
it in a later stage (ideally off-line, after tracing has been done).

One of the next things on my todo list is to also provide an in-kernel
pretty printer which would extract the kernel buffers in text format.
But I don't expect this to be the high-speed tracer, but more a helper
facility to look at the content of "flight recorder" buffers.

Mathieu

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68