Re: [PATCH 0/11] LTTng-core (basic tracing infrastructure) 0.5.108

[Ingo Molnar <mingo@elte.hu>]

* Martin J. Bligh <mbligh@mbligh.org> wrote:

> >>Comments and reviews are very welcome.
> >
> > i have one very fundamental question: why should we do this 
> > source-intrusive method of adding tracepoints instead of the 
> > dynamic, unintrusive (and thus zero-overhead) KProbes+SystemTap 
> > method?
> 
> Because:
> 
> 1. Kprobes are more overhead when they *are* being used.

minimally so - at least on i386 and x86_64. In that sense tracing is a 
_slowpath_, and it _will_ slow things down if done excessively. I dont 
care about the tracepoint being slower by a few instructions as long as 
it has _zero effect_ on normal code, be that source code or binary code.

> 2. You can get zero overhead by CONFIG'ing things out.

but that's not how a fair chunk of people want to use tracing. People 
(enterprise customers trying to figure out performance problems, 
engineers trying to debug things on a live, production system) want to 
be able to insert a tracepoint anywhere and anytime - and also they want 
to have zero overhead from tracing if no tracepoints are used on a 
system.

> 3. (most importantly) it's a bitch to maintain tracepoints out
>    of-tree on a rapidly moving kernel

wrong: the original demo tracepoints that came with SystemTap still work 
on the current kernel, because the 'coupling' is loose: based on 
function names.

Static tracepoints on the other hand, if added via an external patch, do 
depend on the target function not moving around and the context of the 
tracepoint not being changed. (and static tracepoints if in the source 
all the time are a constant hindrance to development and code 
readability.)

and of course the big advantage of dynamic probing is its flexibility: 
you can add add-hoc tracepoints to thousands of functions, instead of 
having to maintain hundreds (or thousands) of static tracepoints all the 
time. (and if we wont end up with hundreds/thousands of static 
tracepoints then it wont be usable enough as a generic solution.)

> 4. I believe kprobes still doesn't have full access to local 
> variables.

wrong: with SystemTap you can probe local variables too (via 
jprobes/kretprobes, all in the upstream kernel already).

> Now (3) is possibly solvable by putting the points in as no-ops 
> (either insert a few nops or just a marker entry in the symbol 
> table?), but full dynamic just isn't sustainable. [...]

i'm not sure i follow. Could you explain where SystemTap has this 
difficulty?

	Ingo