Re: Event logging vs enhancing printk

Michel Dagenais
2002-04-09T14:05:40Z

> People want to be able to get better debug information, with more detail
> than is currently possible with printk, hence the Event logging project.

I would like to emphasize that point; logging and tracing is of prime
importance in several areas:

- Telecom and Security. Logging and auditing is a requirement in Telecom
applications. It is for instance one of the important features of the
Distributed Security Infrastructure proposed by Ericsson Research and to
be presented in a couple of months at the Ottawa Linux Symposium.
The OSDL "Carrier Grade Linux" is certainly no different.

- Real time systems. A low overhead trace is often the only way to debug
these systems. Lineo, Monte-Vista, and FSM labs all have logging solutions,
most based on the Linux Trace Toolkit (www.opersys.com/LTT). It would be
relatively easy to merge the best features of these two systems, present
on the 2.5 status list, (high volume low overhead of LTT and event
templates and filtering of EVLOG).

- Kernel/device drivers debugging.

- Detailed performance analysis. Using the Linux Trace Toolkit, it is possible
to extract very detailed information like the time spent by a process
executing, executing on behalf of client x, waiting for file y
(read/page fault), waiting for process z...

The current printk simply does not cut it for these applications. There are
over 80000 printk statements in the kernel, using many different conventions.
A few tens of driver specific tracing facilities (SCSI, ftape, wireless...)
were implemented each with its own mechanism to compile/not the debugging
statements, trigger massive debugging output at runtime...

The EVLOG proposition strikes me as a good balance between solid features,
simplicity, and ease of integration/transition with the current printk.

Here are some of the advantages of more structured logging:

- More consistent activation mechanisms for logging points
troughout the kernel at configuration/compilation time and at runtime.

- Structured data events for which it is easier to apply filtering, querying,
analysis and detection tools.

- More compact format, when desired, where data and text descriptions are
separated. This facilitates high volume applications (lower logging
overhead, smaller files) and also enables customization/i18n of the static
text descriptions.

- In its current configuration, klogd rapidly looses events under high volumes.
The Linux Trace Toolkit with its zero-copy, kernel-daemon shared memory,
does much better under heavy debugging/tracing output.