[Cluster-devel] cluster/logging settings

[David Teigland <teigland@redhat.com>]

On Thu, Oct 30, 2008 at 11:26:14PM -0700, Steven Dake wrote:
> There are two types of messages.  Those intended for users/admins and
> those intended for developers.
> 
> Both of these message types should always be recorded *somewhere*.  The
> entire concept of "LOG_LEVEL_DEBUG" is dubious to me.  If you want to
> stick with that symanetic and definition that is fine, but really a
> LOG_LEVEL_DEBUG means "this message is for the developer".  These
> messages should be recorded and stored when a process segfaults, aborts
> due to assertion, or at administrative request.  Since the frequency of
> these messages is high there is no other option for recording them since
> they must _always_ be recorded for the purposes of debugging a field
> failure.  Recording to disk or syslog has significant performance
> impact.
> 
> The only solution for these types of messages is to record them into a
> flight recorder buffer which can be dumped:
> 1) at segv
> 2) at sigabt
> 3) at administrative request
> 
> This is a fundamental difference in how we have approached logging
> debugging messages in the past but will lead to the ability to ensure we
> _always_ have debug trace data available instead of telling the
> user/admin "Go turn on debug and hope you can reproduce that error and
> btw since 100000k messages are logged your disk will fill up with
> irrelevant debug messages and your system will perform like mud".
> 
> Logging these in memory is the only solution that I see as suitable and
> in all cases they should be filtered from any output source such as
> stderr, file, or syslog.

There's a difference between high volume trace debug data stored in
memory, and low volume informational debug data that can be easily written
to a file.  Both kinds of data can be useful.

My programs are simple enough that low volume informational debug data is
enough for me to identify and fix a problem.  So, low volume informational
data is all I produce.  It can be useful to write this data to a file.

Your program is complex enough that high volume trace debug data is
usually needed for you to identify and fix a problem.  So, high volume
trace data is all you produce.  This is too much data to write to a file
(by the running program).

So, we're using "DEBUG" to refer to different things.  We need to define
two different levels (just for clarity in this discussion):
. DEBUGLO is low volume informational data like I use
. DEBUGHI is high volume trace data like you use

DEBUGHI messages wouldn't ever be logged to files by the program while
running.  DEBUGLO messages could be, though, if the user configured it.
So, circling back around, how should a user configure DEBUGLO messages to
appear in syslog or a logfile?   In particular, what would they enter in
the cluster.conf <logging/> section?  My suggestion is:

  syslog_level=foo
  logfile_level=bar

where foo and bar are one of the standard priority names in syslog.h.
So, if a user wanted DEBUGLO messages to appear in daemon.log, they'd set

  logging/<daemon>/logfile_level=debug

and if they wanted DEBUGLO messages to appear in /var/log/messages,

  logging/<daemon>/syslog_level=debug

(Note that "debug" means DEBUGLO here because DEBUGHI messages are only
saved in memory, not to files by a running program.)

There's another separate question I have about corosync, and that's
whether you could identify some limited number of messages that would be
appropriate for DEBUGLO?  They would be used by non-experts to do some
rough debugging of problems, and by experts to narrow down a problem
before digging into the high volume trace data.  I'd suggest that a good
starting point for DEBUGLO would be the data that openais has historically
put in /var/log/messages.  Data that helps you quickly triage a problem
(or verify that things are happening correctly) without stepping through
all the trace data.