Re: [PATCH 0/11] LTTng-core (basic tracing infrastructure) 0.5.108

[Karim Yaghmour <karim@opersys.com>]

Ingo Molnar wrote:
> that's not true, and this is the important thing that i believe you are 
> missing. A dynamic tracepoint is _detached_ from the normal source code 
> and thus is zero maintainance overhead. You dont have to maintain it 
> during normal development - only if you need it. You dont see the 
> dynamic tracepoints in the source code.

And that's actually a problem for those who maintain such dynamic
trace points.

> a static tracepoint, once it's in the mainline kernel, is a nonzero 
> maintainance overhead _until eternity_. It is a constant visual 
> hindrance and a constant build-correctness and boot-correctness problem 
> if you happen to change the code that is being traced by a static 
> tracepoint. Again, I am talking out of actual experience with static 
> tracepoints: i frequently break my kernel via static tracepoints and i 
> have constant maintainance cost from them. So what i do is that i try to 
> minimize the number of static tracepoints to _zero_. I.e. i only add 
> them when i need them for a given bug.

Bzzt, wrong. This is your own personal experience with tracing. Marked
up code does not need to be active under all build conditions. In
fact trace points can be inactive by default at all times, except
when you choose to build them in.

And as I said elsewhere, the fact that your use of instrumentation is
solely for debugging ("i only add them when i need them for a given bug"),
I repeat that there are mortals out there that need this for their
applications.

> static tracepoints are inferior to dynamic tracepoints in almost every 
> way.

Sorry, orthogonal is the word.

> hundreds (or possibly thousands) of tracepoints? Have you ever tried to 
> maintain that? I have and it's a nightmare.

I have, and I've showed you that you're wrong. The only reason you can
make this argument is that you view these things from the point of view
of what use they are for you as a kernel developer and I will repeat
what I've said for years now: static instrumentation of the kernel
isn't meant to be useful for kernel developers. While it may indeed
be in some cases, in most cases it's likely useless, as you've been
very successfully arguing in this thread. Nevertheless there are very
legitimate uses for standardized instrumentation points.

> Even assuming a rich set of hundreds of static tracepoints, it doesnt 
> even solve the problems at hand: people want to do much more when they 
> probe the kernel - and today, with DTrace under Solaris people _know_ 
> that much better tracing _can be done_, and they _demand_ that Linux 
> adopts an intelligent solution. The clock is ticking for dinosaurs like 
> static printks and static tracepoints to debug the kernel...

Thank you, I couldn't have put it better. This paragraph, more than
any other snippet I've seen to date, clearly demonstrates why
tracing is such a contentious issue. Kernel developers use tracing
during their normal development process, and of course their gut
reaction is: why the hell would anybody need this for mainline? But
of course this misses the entire point. Kernel tracing for developers
is but a corner case of kernel tracing in general. There are very valid
and legitimate reasons for userspace to be able to obtain important
events. And of course any infrastructure developed with that in
mind should also be usable by kernel developers.

Karim