Re: Event logging vs enhancing printk

[Michel Dagenais <michel.dagenais@polymtl.ca>]

> People want to be able to get better debug information, with more detail
> than is currently possible with printk, hence the Event logging project.

I would like to emphasize that point; logging and tracing is of prime 
importance in several areas:

- Telecom and Security. Logging and auditing is a requirement in Telecom 
  applications. It is for instance one of the important features of the 
  Distributed Security Infrastructure proposed by Ericsson Research and to 
  be presented in a couple of months at the Ottawa Linux Symposium.
  The OSDL "Carrier Grade Linux" is certainly no different.

- Real time systems. A low overhead trace is often the only way to debug
  these systems. Lineo, Monte-Vista, and FSM labs all have logging solutions,
  most based on the Linux Trace Toolkit (www.opersys.com/LTT). It would be 
  relatively easy to merge the best features of these two systems, present 
  on the 2.5 status list, (high volume low overhead of LTT and event 
  templates and filtering of EVLOG).

- Kernel/device drivers debugging.

- Detailed performance analysis. Using the Linux Trace Toolkit, it is possible
  to extract very detailed information like the time spent by a process
  executing, executing on behalf of client x, waiting for file y 
  (read/page fault), waiting for process z... 

The current printk simply does not cut it for these applications. There are
over 80000 printk statements in the kernel, using many different conventions.
A few tens of driver specific tracing facilities (SCSI, ftape, wireless...)
were implemented each with its own mechanism to compile/not the debugging
statements, trigger massive debugging output at runtime...

The EVLOG proposition strikes me as a good balance between solid features,
simplicity, and ease of integration/transition with the current printk.

Here are some of the advantages of more structured logging:

- More consistent activation mechanisms for logging points
  troughout the kernel at configuration/compilation time and at runtime.

- Structured data events for which it is easier to apply filtering, querying,
  analysis and detection tools.

- More compact format, when desired, where data and text descriptions are 
  separated. This facilitates high volume applications (lower logging 
  overhead, smaller files) and also enables customization/i18n of the static 
  text descriptions.

- In its current configuration, klogd rapidly looses events under high volumes.
  The Linux Trace Toolkit with its zero-copy, kernel-daemon shared memory,
  does much better under heavy debugging/tracing output.