Re: [PATCH 0/11] LTTng-core (basic tracing infrastructure) 0.5.108

Ingo Molnar
2006-09-14T18:24:50Z

* Roman Zippel <zippel@linux-m68k.org> wrote:

> > for me these are all _independent_ grounds for rejection, as a generic
> > kernel infrastructure.
>
> Tracepoints of course need to be managed, but that's true for both
> dynamic and static tracepoints. [...]

that's not true, and this is the important thing that i believe you are
missing. A dynamic tracepoint is _detached_ from the normal source code
and thus is zero maintainance overhead. You dont have to maintain it
during normal development - only if you need it. You dont see the
dynamic tracepoints in the source code.

a static tracepoint, once it's in the mainline kernel, is a nonzero
maintainance overhead _until eternity_. It is a constant visual
hindrance and a constant build-correctness and boot-correctness problem
if you happen to change the code that is being traced by a static
tracepoint. Again, I am talking out of actual experience with static
tracepoints: i frequently break my kernel via static tracepoints and i
have constant maintainance cost from them. So what i do is that i try to
minimize the number of static tracepoints to _zero_. I.e. i only add
them when i need them for a given bug.

static tracepoints are inferior to dynamic tracepoints in almost every
way.

> [...] Both have their advantages and disadvantages and just hammering
> on the possible problems of static ones [...]

how about giving a line by line rebuttal to the very real problems of
static tracepoints i listed (twice already), instead of calling them
"possible problems"?

i am giving a line by line rebuttal of all arguments that come up.
Please be fair and do the same. Here are the arguments again, for a
third time. Thanks!

> > also, the other disadvantages i listed very much count too. Static
> > tracepoints are fundamentally limited because:
> >
> > - they can only be added at the source code level
> >
> > - modifying them requires a reboot which is not practical in a
> > production environment
> >
> > - there can only be a limited set of them, while many problems need
> > finegrained tracepoints tailored to the problem at hand
> >
> > - conditional tracepoints are typically either nonexistent or very
> > limited.

> > the kprobes infrastructure, despite being fairly young, is widely
> > available: powerpc, i386, x86_64, ia64 and sparc64. The other
> > architectures are free to implement them too, there's nothing
> > hardware-specific about kprobes and the "porting overhead" is in
> > essence a one-time cost - while for static tracepoints the
> > maintainance overhead goes on forever and scales linearly with the
> > number of tracepoints added.
>
> kprobes are not trivial to implement [...]

nor are smp-alternatives, which was suggested as a solution to reduce
the overhead of static tracepoints. So what's the point? It's a one-off
development overhead that has already been done for all the major
arches. If another arch needs it they can certainly implement it.

it's like arguing against ptrace on the grounds of: "application
developers can add printf if they want to debug their apps, or they can
add static tracepoints too, and besides, ptrace is hard to implement".

> I also think you highly exaggerate the maintaince overhead of static
> tracepoints, once added they hardly need any maintainance, most of the
> time you can just ignore them. [...]

hundreds (or possibly thousands) of tracepoints? Have you ever tried to
maintain that? I have and it's a nightmare.

Even assuming a rich set of hundreds of static tracepoints, it doesnt
even solve the problems at hand: people want to do much more when they
probe the kernel - and today, with DTrace under Solaris people _know_
that much better tracing _can be done_, and they _demand_ that Linux
adopts an intelligent solution. The clock is ticking for dinosaurs like
static printks and static tracepoints to debug the kernel...

> [...] The kernel is full debug prints, do you seriously suggest to
> throw them out because of their "high maintainance"?

oh yes, these days i frequently throw them out when i find them in code
i modify. (my most recent such zap was rwsemtrace()). Also, obviously
when most of them were added we didnt have good kernel debugging
infrastructure (in fact we didnt have any kernel debugging
infrastructure besides printk), so _something_ had to be used back then.
But today there's little reason to keep them. Welcome to 2006 :-)

Ingo