Re: [RFC patch 00/41] LTTng 0.105 core for Linux 2.6.27-rc9

Ingo Molnar
2009-03-06T10:12:34Z

* Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca> wrote:

> Hi,
>
> I spent the last 4-5 months working with the Fujitsu team at
> implementing the tracer elements identified as goals at Kernel
> Summit 2008 and at the following Plumber Conference. My idea
> was to incremententally adapt the LTTng tracer, currently used
> in the industry and well tested, to those requirements.
>
> I spent the last days rearranging/folding/inspecting the LTTng
> patchset to prepare it for an LKML post. The version 0.105 in
> the LTTng git tree corresponds to the patchset I am posting
> here. The said patchset will only include the core features of
> LTTng, excluding the timestamping infrastructure (trace clock)
> and excluding the instrumentation.

I'd like to merge the good bits into the tracing tree. Looking
at the patches you submitted there's a lot of avoidable overlap
with existing tracing features either present upstream already
or queued up for v2.6.30 - and we need to work more on
eliminating that overlap.

I dont think there's much fundamental disagreement just
different implementations - so we should evaluate each of those
details one by one, iteratively.

The first step would be to split the patches up into three
logical buckets:

- Unique features not present in the tracing infracture, in the
event tracer or other tracing plugins - those should be
structured as feature additions.

- Features that you consider superior to existing tracing
features of the kernel. For those, please iterate the
existing code with your enhancements - instead of a parallel
implementation.

- Items which offer nothing new and are not superior to
existing features, those should be dropped probably. This too
is a case by case thing.

Would you be interested in working with us on that? I know that
both Steve and me would be very much interested in that. If you
have time/interest to work on that then we can go through each
patch one by one and categorize them and map out the way to go.

Let me give you a few examples of existing areas of overlap:

> The corresponding git tree contains also the trace clock
> patches and the lttng instrumentation. The trace clock is
> required to use the tracer, but it can be used without the
> instrumentation : there is already a kprobes and userspace
> event support included in this patchset.

The latest tracing tree includes kernel/tracing/trace_clock.c
which offers three trace clock variants, with different
performance/precision tradeoffs:

trace_clock_local() [ for pure CPU-local tracers with no idle
events. This is the fastest but least
coherent tracing clock. ]

trace_clock() [ intermediate, scalable clock with
usable but imprecise global coherency. ]

trace_clock_global() [ globally serialized, coherent clock.
It is the slowest but most accurate variant. ]

Tracing plugins can pick their choice. (This is relatively new
code but you get the idea.)

> This tracer exports binary data through buffers using
> splice(). The resulting binary files can be parsed from
> userspace because the format string metadata is exported in
> the files. The event set can be enhanced by adding tracepoints
> to the kernel code and by creating probe modules, which
> connects callbacks to the tracepoints and contain the format
> string metainformation. Those callbacks are responsible for
> writing the data in the trace buffers. This separation between
> the trace buffer format string and the tracepoints is done on
> purpose so the core kernel instrumentation (tracepoints) is
> not exported to userspace, which will make maintainance much
> easier.

A tracepoint format specification mechanism plus working (and
fast!) zero-copy splice() support of the ring-buffer exists in
the latest tracing tree already - as you are probably aware of
because you commented on those patches a few days ago.

There are 3 good ways to go from here regarding the trace
buffering and splice code:

1- we end up switching to the lttng version in essence
2- we end up keeping the tracing tree version
3- we end up somewhere inbetween

Which point in the above spectrum we will settle down on depends
on the technical details.

Note, whichever path we choose a gradual, iterative workflow is
still needed, so that we improve the existing upstream code with
lttng enhancements gradually.

This approach works for all your other patches as well. A
direct, constructive comparison and active work on unifying them
is required.

Thanks,

Ingo