Re: LTT user input

[Roger Luethi <rl@hellgate.ch>]

On Thu, 22 Jul 2004 15:47:03 -0500, zanussi@us.ibm.com wrote:
> One of the things people mentioned wanting to see during Karim's LTT
> talk at the Kernel Summit was cases where LTT had been useful to real
> users.  Here are some examples culled from the ltt/ltt-dev mailing
> lists:
[...]
> Another thing that came up was the impression that the overhead of
> tracing is too high.  I'm not sure where the number mentioned (5%)

The examples you mentioned confirm what Andrew mentioned recently:
What little public evidence there is comes from developers trying
to understand the kernel or debugging their own applications.

I'd be interested to see examples of how these tools help regular sys
admins or technically inclined users (no Aunt Tillie compatibility
required) -- IMO that would go a long way to make a case for inclusion [1].

Another concern raised at the summit (and what I am personally most
concerned about) is the overlap in all the frameworks that add logging
hooks for all kinds of purposes: auditing, performance, user level
debugging, etc.

Out of mainline examples that have been around for a while include:

- systrace http://niels.xtdnet.nl/systrace/
- syscalltrack http://syscalltrack.sourceforge.net/
- LTT http://www.opersys.com/LTT/

I wonder if a basic framework that can serve more than one purpose
makes sense.

When considering which tracing functionlity should be in mainline,
performance measurments for user-space come in pretty much at the
bottom of my list: Questions like "which process is overwriting this
config file behind my back" seem a lot more common and more likely to
be asked by people not willing or capable of compiling a patched kernel
for that purpose. And tools that are useful for kernel developers (while
unpopular with the powers that be) are nice to have in mainline because
as a kernel hacker, you often _have_ to debug the latest kernel for
which your favorite debug tool is not working yet. An argument for
adding security auditing to mainline is that it helps convince the
conservative and cautious security folks that the functionality is
accepted and here to stay.

None of these arguments apply for LTT as it presents itself: If you
are debugging or tuning a multi-threaded user space app or trying to
understand the kernel, patching some kernel supported by the respective
tool should hardly be a problem.

Please note that I just compared the relative merits of merging various
kinds of tracing functionality into mainline. I did not argue in favor
or against the inclusion of LTT-type functionality.

My point is that the best bet for tools that seem to aim at user-space
performance debugging is to demonstrate how they can be useful for a
wider audience, or to hitch a ride with a framework that does appeal
to a wider audience.

Roger

[1] You could take a page from how DTrace was introduced:
    http://www.sun.com/bigadmin/content/dtrace/
    Or take a look at:
    http://syscalltrack.sourceforge.net/when.html
    http://syscalltrack.sourceforge.net/examples.html