Re: [PATCH 0/11] LTTng-core (basic tracing infrastructure) 0.5.108

[Thomas Gleixner <tglx@linutronix.de>]

On Fri, 2006-09-15 at 11:16 -0700, Andrew Morton wrote:
> Me thinks our time would be best spent trying to benefit from his
> experience..

I was involved in tracer development for quite a while and I have used
them in $paying customer projects too.

> Me, I'm not particularly averse to some 50-100 static tracepoints if
> experience tells us that we need such things.  And both Karim's and Frank's
> experience does indicate that such things are needed, which carries weight.

>From my experience the tracepoints usually are not at the place where
you need them to track down a particular problem or analyse a particular
usage scenario in detail. This has been true from a kernel and from an
application programmer POV. Also many of the LTT customer I'm aware of
used their own homebrewed set of trace points.

What I always hated on static tracers is the requirement to recompile /
reboot the kernel in order to gather information. Kprobes / systemtap is
really a conveniant way to avoid this.

I completely agree that the maintenance of the "out of code" trace
scripts is a task which needs a lot of effort, but it does not offload
the maintenance effort to those modifying the code and we have not yet
another pseudo instruction/function set which is interfering with the
goal to have clear and understandable code. Hell, the code in those code
paths which are of common interest for instrumentation is already
complex enough. We really can do without adding some more obfuscated
macro constructs.

When we can maintain a basic set of tracescripts in the kernel tree and
once the necessary infrastructure is in place, I'm quite sure that quite
a lot of kernel developers would keep those fundamental trace scripts in
shape out of their own interest. It might take a while to get this going
but once it is established, distros will ship the scripts along with
dynamic tracing enabled in the kernels.

I see a major advantage over static tracing in that:

Static tracing is usually not enabled in production kernels, but the
dynamic tracing infrastructure can be enabled without costs. So you
can actually request traces (at least for the standard set of
tracepoints) from Joe User to track down complex problems.

One thing which is much more important IMHO is the availablity of
_USEFUL_ postprocessing tools to give users a real value of
instrumentation. This is a much more complex task than this whole kernel
instrumentation business. This also includes the ability to coordinate
user space _and_ kernel space instrumentation, which is necessary to
analyse complex kernel / application code interactions. 

	tglx