[Cluster-devel] cluster/logging settings

[Steven Dake <sdake@redhat.com>]

On Tue, 2008-11-04 at 14:58 -0600, David Teigland wrote:
> On Thu, Oct 30, 2008 at 11:26:14PM -0700, Steven Dake wrote:
> > There are two types of messages.  Those intended for users/admins and
> > those intended for developers.
> > 
> > Both of these message types should always be recorded *somewhere*.  The
> > entire concept of "LOG_LEVEL_DEBUG" is dubious to me.  If you want to
> > stick with that symanetic and definition that is fine, but really a
> > LOG_LEVEL_DEBUG means "this message is for the developer".  These
> > messages should be recorded and stored when a process segfaults, aborts
> > due to assertion, or at administrative request.  Since the frequency of
> > these messages is high there is no other option for recording them since
> > they must _always_ be recorded for the purposes of debugging a field
> > failure.  Recording to disk or syslog has significant performance
> > impact.
> > 
> > The only solution for these types of messages is to record them into a
> > flight recorder buffer which can be dumped:
> > 1) at segv
> > 2) at sigabt
> > 3) at administrative request
> > 
> > This is a fundamental difference in how we have approached logging
> > debugging messages in the past but will lead to the ability to ensure we
> > _always_ have debug trace data available instead of telling the
> > user/admin "Go turn on debug and hope you can reproduce that error and
> > btw since 100000k messages are logged your disk will fill up with
> > irrelevant debug messages and your system will perform like mud".
> > 
> > Logging these in memory is the only solution that I see as suitable and
> > in all cases they should be filtered from any output source such as
> > stderr, file, or syslog.
> 
> There's a difference between high volume trace debug data stored in
> memory, and low volume informational debug data that can be easily written
> to a file.  Both kinds of data can be useful.
> 
> My programs are simple enough that low volume informational debug data is
> enough for me to identify and fix a problem.  So, low volume informational
> data is all I produce.  It can be useful to write this data to a file.
> 
> Your program is complex enough that high volume trace debug data is
> usually needed for you to identify and fix a problem.  So, high volume
> trace data is all you produce.  This is too much data to write to a file
> (by the running program).
> 
> So, we're using "DEBUG" to refer to different things.  We need to define
> two different levels (just for clarity in this discussion):
> . DEBUGLO is low volume informational data like I use
> . DEBUGHI is high volume trace data like you use
> 
> DEBUGHI messages wouldn't ever be logged to files by the program while
> running.  DEBUGLO messages could be, though, if the user configured it.
> So, circling back around, how should a user configure DEBUGLO messages to
> appear in syslog or a logfile?   In particular, what would they enter in
> the cluster.conf <logging/> section?  My suggestion is:
> 
>   syslog_level=foo
>   logfile_level=bar
> 
> where foo and bar are one of the standard priority names in syslog.h.
> So, if a user wanted DEBUGLO messages to appear in daemon.log, they'd set
> 
>   logging/<daemon>/logfile_level=debug
> 
> and if they wanted DEBUGLO messages to appear in /var/log/messages,
> 
>   logging/<daemon>/syslog_level=debug
> 
> (Note that "debug" means DEBUGLO here because DEBUGHI messages are only
> saved in memory, not to files by a running program.)
> 
> There's another separate question I have about corosync, and that's
> whether you could identify some limited number of messages that would be
> appropriate for DEBUGLO?  They would be used by non-experts to do some
> rough debugging of problems, and by experts to narrow down a problem
> before digging into the high volume trace data.  I'd suggest that a good
> starting point for DEBUGLO would be the data that openais has historically
> put in /var/log/messages.  Data that helps you quickly triage a problem
> (or verify that things are happening correctly) without stepping through
> all the trace data.
> 

I thought about this a few days and this seems to make sense.  The
priority (ie: specifically LOG_LEVEL_DEBUG messages) would be selectable
per output medium (syslog, file, or stderr) and the high performance
tracing would use the trace macros or log_rec and only ever go to
memory.  Each output medium if configured would filter based upon a
filtering priority which is LOG_LEVEL_DEBUG or others.

This does require further API changes that Fabio would have to manage
with.

Regards
-steve