[Cluster-devel] logging: final call on configuration, output and implementation

[Cluster-devel] logging: final call on configuration, output and implementation

[Fabio M. Di Nitto]

Hi all,

those logging threads have been going on for way too long. It's time to
close them and make a final decision. This is a long email, so please
take time to read it all.

This is a recap of what I believe a user would like to see:

- a consistent, easy and quick way to configure logging.
- a reasonable default if nothing is specified.
- a consistent, easy to read, output.

In order to avoid misinterpretation, "debug" priority means DEBUGLO as
defined here by David:
http://www.redhat.com/archives/cluster-devel/2008-November/msg00002.html

== Configuration ==

XML version:

<logging debug="on|off" (read note below about debugging)
         to_syslog="yes|no"
         syslog_facility="local4"
         syslog_priority="debug|info|notice|etc.."
         to_file="yes|no"
         logfile="/var/log/cluster/foo.log"
         log_priority="debug|info|notice|etc..">

  <logger_subsys subsys="NAME" debug=.. />
</logging>

corosync/openais equivalent:

logging {
        debug="on|off"; (read note below about debugging)

        to_syslog="yes|no";
        syslog_facility="local4";
        syslog_priority="debug|info|notice|etc..";

        to_file="yes|no";
        logfile="/var/log/cluster/foo.log";
        log_priority="debug|info|notice|etc..";

        logger_subsys {
                subsys="NAME";
                debug=..
        }
}

(the two configuration are equivalent)

The global configuration (<logging>/logging {) contains all the keywords
required to configure logging as discussed so far and would affects
every daemon or subsystems. The common user would generally stop here.

The logger_subsys can be used to do more detailed control over logging.
This is where more advance users or developers work.

Equivalent entries in logger_subsys override configurations in the
global section.

Use of environment variables and settings from the command line can
override any configuration. This is used by some people to enable
debugging via init scripts (for example /etc/sysconfig/cman or
equivalent for other distros).


== Defaults ==

- debug = off. Switch to "on" to enable execution of debugging code
(read note below) and set log_priority to debug.

- to_syslog = yes.
- syslog_facility = local4 (default to local4 to respect old behaviour).
- syslog_priority = info (default to respect old behaviour and it seems
to be a recent compromise at this point of development, doesn't flood
logs and has enough info).

- to_file = yes
- logfile = LOGDIR/[daemon|subsystem].log. By default each
daemon/subsystem should log in its own file (mostly valid for daemons as
corosync/openais and plugins all share the same file and logging
system).
- logpriority= info (same reason as above)

to_stderr will disappear from the config options. Set to off by default
as most daemons will fork in background, we can set it "ON"
automatically when not running in background (it's the only case where
it actually makes sense).


== Output ==

to_file:

echo $(date "+%b %d %T") subsystem_or_daemon: entry_to_log
Nov 10 19:46:40 subsystem_or_daemon: entry_to_log

It is important to note 2 things. First we need to have a time stamp to
be able to compare logs if daemons are logging to different files, then
we need to know what subsystem is logging if we are logging to the same
file for everything.

Since we don't want to create N combinations that will add/remove
date/daemon_name etc, this is one format can fit them all and it is the
same that would appear in syslog (go for consistency!).

to_syslog:

openlog("subsystem_or_daemon", options, facility);
syslog(priority, "%s", entry_to_log);

(consistent with to_file).

to_stderr:

fprintf(stderr, "%s\n", entry); <- this is really free format.


== Implementations requirements ==

- (strict requirement) must support the configuration options agreed by
everybody.

- (high priority) logging should never fail. For example, if we can't
open a log file we will not fail, we will try to warn by other meanings,
but nothing should block our operations if logging is not working.

- (wish list) logging should be non-blocking. Best if we can delegate
the logging work to something else than ourselves.


== Note about debugging ==

Debugging is an art. Every developer on this planet has a different view
of what debugging is. What we want with the debug="on|off" option is an
absolute quick way to set all the different log_priority="debug" on and
have a flag that can be used in the code to follow different paths.

I often find myself setting <logging debug="on"/> and be done with
everything. No need to remember fancy keywords or understand the whole
implementation details on what overrides what and how.

At some point in future we should probably talk about debugging in
general and what means for all of us but it's outside the scope of this
email.

Fabio

[Cluster-devel] logging: final call on configuration, output and implementation

[David Teigland]

On Mon, Nov 10, 2008 at 08:09:10PM +0100, Fabio M. Di Nitto wrote:
> Hi all,
> 
> those logging threads have been going on for way too long. It's time to
> close them and make a final decision. This is a long email, so please
> take time to read it all.
> 
> This is a recap of what I believe a user would like to see:
> 
> - a consistent, easy and quick way to configure logging.
> - a reasonable default if nothing is specified.
> - a consistent, easy to read, output.

I like this.  Two minor points regarding the actual terminology; I'd like
to be a little more consistent and identify some "keywords".  Right now
the word "log" is combined with other words in a bunch of ways (logging,
logger, logfile, syslog, log_foo).  How about:

. logging, logging_subsys (common keyword "logging")
  for the config file section tags

- to_syslog, syslog_facility, syslog_priority (common keyword "syslog")
  for every parameter related to syslog

- to_logfile, logfile, logfile_priority (common keyword "logfile")
  for every parameter related to logfile

And then we have some values that are "on/off" and others that are
"yes/no"; let's pick one.

Dave

[Cluster-devel] logging: final call on configuration, output and implementation

[Fabio M. Di Nitto]

On Mon, 2008-11-10 at 13:27 -0600, David Teigland wrote:
> On Mon, Nov 10, 2008 at 08:09:10PM +0100, Fabio M. Di Nitto wrote:
> > Hi all,
> > 
> > those logging threads have been going on for way too long. It's time to
> > close them and make a final decision. This is a long email, so please
> > take time to read it all.
> > 
> > This is a recap of what I believe a user would like to see:
> > 
> > - a consistent, easy and quick way to configure logging.
> > - a reasonable default if nothing is specified.
> > - a consistent, easy to read, output.
> 
> I like this.  Two minor points regarding the actual terminology; I'd like
> to be a little more consistent and identify some "keywords".  Right now
> the word "log" is combined with other words in a bunch of ways (logging,
> logger, logfile, syslog, log_foo).  How about:
> 
> . logging, logging_subsys (common keyword "logging")
>   for the config file section tags
> 
> - to_syslog, syslog_facility, syslog_priority (common keyword "syslog")
>   for every parameter related to syslog
> 
> - to_logfile, logfile, logfile_priority (common keyword "logfile")
>   for every parameter related to logfile

+1 on all 3 of them.

The only reason I did try to avoid keyword changes was to minimize the
impact in compat layers since most of those are already in use by us.

> 
> And then we have some values that are "on/off" and others that are
> "yes/no"; let's pick one.

+1. Given the boolean value, "on/off" sounds more nerdy.

Fabio

[Linux-cluster] Re: [Cluster-devel] logging: final call on configuration, output and implementation

[Joel Becker]

On Mon, Nov 10, 2008 at 01:27:30PM -0600, David Teigland wrote:
> And then we have some values that are "on/off" and others that are
> "yes/no"; let's pick one.

	I always liked the X resource route:

/true|yes|on/i -> "True"
/false|no|off/i -> "False"

Joel

-- 

"In the beginning, the universe was created. This has made a lot 
 of people very angry, and is generally considered to have been a 
 bad move."
        - Douglas Adams

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127

[Cluster-devel] logging: final call on configuration, output and implementation

[Steven Dake]

I disagree with a global debug keyword.  At one time I thought it was a
good idea but that time has long since passed.  The idea of turning
debug to on and then having all debug output go to syslog is frightening
and will result in lost messages.  While it appears this proposal
includes the selectable log output filtering per output medium as was
discussed already, it is unclear how the debug keyword affects this.  It
would simply make sense to change the file's log priority or the
syslog's log priority if that is the behavior desired and then no need
for any extra keyword.

Regards
-steve

On Mon, 2008-11-10 at 20:09 +0100, Fabio M. Di Nitto wrote:
> Hi all,
> 
> those logging threads have been going on for way too long. It's time to
> close them and make a final decision. This is a long email, so please
> take time to read it all.
> 
> This is a recap of what I believe a user would like to see:
> 
> - a consistent, easy and quick way to configure logging.
> - a reasonable default if nothing is specified.
> - a consistent, easy to read, output.
> 
> In order to avoid misinterpretation, "debug" priority means DEBUGLO as
> defined here by David:
> http://www.redhat.com/archives/cluster-devel/2008-November/msg00002.html
> 
> == Configuration ==
> 
> XML version:
> 
> <logging debug="on|off" (read note below about debugging)
>          to_syslog="yes|no"
>          syslog_facility="local4"
>          syslog_priority="debug|info|notice|etc.."
>          to_file="yes|no"
>          logfile="/var/log/cluster/foo.log"
>          log_priority="debug|info|notice|etc..">
> 
>   <logger_subsys subsys="NAME" debug=.. />
> </logging>
> 
> corosync/openais equivalent:
> 
> logging {
>         debug="on|off"; (read note below about debugging)
> 
>         to_syslog="yes|no";
>         syslog_facility="local4";
>         syslog_priority="debug|info|notice|etc..";
> 
>         to_file="yes|no";
>         logfile="/var/log/cluster/foo.log";
>         log_priority="debug|info|notice|etc..";
> 
>         logger_subsys {
>                 subsys="NAME";
>                 debug=..
>         }
> }
> 
> (the two configuration are equivalent)
> 
> The global configuration (<logging>/logging {) contains all the keywords
> required to configure logging as discussed so far and would affects
> every daemon or subsystems. The common user would generally stop here.
> 
> The logger_subsys can be used to do more detailed control over logging.
> This is where more advance users or developers work.
> 
> Equivalent entries in logger_subsys override configurations in the
> global section.
> 
> Use of environment variables and settings from the command line can
> override any configuration. This is used by some people to enable
> debugging via init scripts (for example /etc/sysconfig/cman or
> equivalent for other distros).
> 
> 
> == Defaults ==
> 
> - debug = off. Switch to "on" to enable execution of debugging code
> (read note below) and set log_priority to debug.
> 
> - to_syslog = yes.
> - syslog_facility = local4 (default to local4 to respect old behaviour).
> - syslog_priority = info (default to respect old behaviour and it seems
> to be a recent compromise at this point of development, doesn't flood
> logs and has enough info).
> 
> - to_file = yes
> - logfile = LOGDIR/[daemon|subsystem].log. By default each
> daemon/subsystem should log in its own file (mostly valid for daemons as
> corosync/openais and plugins all share the same file and logging
> system).
> - logpriority= info (same reason as above)
> 
> to_stderr will disappear from the config options. Set to off by default
> as most daemons will fork in background, we can set it "ON"
> automatically when not running in background (it's the only case where
> it actually makes sense).
> 
> 
> == Output ==
> 
> to_file:
> 
> echo $(date "+%b %d %T") subsystem_or_daemon: entry_to_log
> Nov 10 19:46:40 subsystem_or_daemon: entry_to_log
> 
> It is important to note 2 things. First we need to have a time stamp to
> be able to compare logs if daemons are logging to different files, then
> we need to know what subsystem is logging if we are logging to the same
> file for everything.
> 
> Since we don't want to create N combinations that will add/remove
> date/daemon_name etc, this is one format can fit them all and it is the
> same that would appear in syslog (go for consistency!).
> 
> to_syslog:
> 
> openlog("subsystem_or_daemon", options, facility);
> syslog(priority, "%s", entry_to_log);
> 
> (consistent with to_file).
> 
> to_stderr:
> 
> fprintf(stderr, "%s\n", entry); <- this is really free format.
> 
> 
> == Implementations requirements ==
> 
> - (strict requirement) must support the configuration options agreed by
> everybody.
> 
> - (high priority) logging should never fail. For example, if we can't
> open a log file we will not fail, we will try to warn by other meanings,
> but nothing should block our operations if logging is not working.
> 
> - (wish list) logging should be non-blocking. Best if we can delegate
> the logging work to something else than ourselves.
> 
> 
> == Note about debugging ==
> 
> Debugging is an art. Every developer on this planet has a different view
> of what debugging is. What we want with the debug="on|off" option is an
> absolute quick way to set all the different log_priority="debug" on and
> have a flag that can be used in the code to follow different paths.
> 
> I often find myself setting <logging debug="on"/> and be done with
> everything. No need to remember fancy keywords or understand the whole
> implementation details on what overrides what and how.
> 
> At some point in future we should probably talk about debugging in
> general and what means for all of us but it's outside the scope of this
> email.
> 
> Fabio
> 

[Cluster-devel] logging: final call on configuration, output and implementation

[Fabio M. Di Nitto]

On Mon, 2008-11-10 at 17:46 -0700, Steven Dake wrote:
> I disagree with a global debug keyword. 
>  At one time I thought it was a
> good idea but that time has long since passed.  The idea of turning
> debug to on and then having all debug output go to syslog is frightening
> and will result in lost messages.  While it appears this proposal
> includes the selectable log output filtering per output medium as was
> discussed already, it is unclear how the debug keyword affects this.  It
> would simply make sense to change the file's log priority or the
> syslog's log priority if that is the behavior desired and then no need
> for any extra keyword.

You have these two situations:

print_log(LOG_DEBUG, "doing this and that....\n");

if (debug) { /*
 gather_some_data_that_is_very_expensive_operation_to_do_all_the_time();
 print_log(LOG_DEBUG, "print those extra data\n");
}

as it is now, it would basically be an alias to set logpriority to DEBUG
but enables people to execute debugging code conditionally and as I
wrote it is an easy keyword to remember compared to
syslog_priority/logpriority.

Fabio

[Cluster-devel] logging: final call on configuration, output and implementation

[Steven Dake]

On Tue, 2008-11-11 at 05:55 +0100, Fabio M. Di Nitto wrote:
> On Mon, 2008-11-10 at 17:46 -0700, Steven Dake wrote:
> > I disagree with a global debug keyword. 
> >  At one time I thought it was a
> > good idea but that time has long since passed.  The idea of turning
> > debug to on and then having all debug output go to syslog is frightening
> > and will result in lost messages.  While it appears this proposal
> > includes the selectable log output filtering per output medium as was
> > discussed already, it is unclear how the debug keyword affects this.  It
> > would simply make sense to change the file's log priority or the
> > syslog's log priority if that is the behavior desired and then no need
> > for any extra keyword.
> 
> You have these two situations:
> 
> print_log(LOG_DEBUG, "doing this and that....\n");
> 
> if (debug) { /*
>  gather_some_data_that_is_very_expensive_operation_to_do_all_the_time();
>  print_log(LOG_DEBUG, "print those extra data\n");
> }
> 
> as it is now, it would basically be an alias to set logpriority to DEBUG
> but enables people to execute debugging code conditionally and as I
> wrote it is an easy keyword to remember compared to
> syslog_priority/logpriority.
> 
> Fabio
> 

The second situation doesn't exist in any code I have written and never
would.  Having any conditional debug output is asking for trouble.  Been
down that road, done that, and discarded that idea...  The "debughi" or
high volume debug messages do not go through log_printf nor would they
be committed to any persistent log (only memory).  The output of the
logging message is significantly more expensive then that of gathering
logging data.

Turning debug on for all of the entire stack to be output to syslog is
not satisfactory because messages would be lost in overload conditions.
Logging to file is only a slight bit better solution but if you really
must have debug output in a persistent store that doesn't occur as a
result of a failure, logging to file is the only suitable answer.

A global debug option without selecting log output is not a workable
solution because of overload of syslog, even overload of the filesystem,
or other issues.

What makes sense is to have a mechanism to set the priority for each
specific log output mechanism and forget about any global debug option
nonsense.

Regards
-steve

[Cluster-devel] logging: final call on configuration, output and implementation

[Fabio M. Di Nitto]

On Mon, 2008-11-10 at 22:47 -0700, Steven Dake wrote:
> On Tue, 2008-11-11 at 05:55 +0100, Fabio M. Di Nitto wrote:
> > On Mon, 2008-11-10 at 17:46 -0700, Steven Dake wrote:
> > > I disagree with a global debug keyword. 
> > >  At one time I thought it was a
> > > good idea but that time has long since passed.  The idea of turning
> > > debug to on and then having all debug output go to syslog is frightening
> > > and will result in lost messages.  While it appears this proposal
> > > includes the selectable log output filtering per output medium as was
> > > discussed already, it is unclear how the debug keyword affects this.  It
> > > would simply make sense to change the file's log priority or the
> > > syslog's log priority if that is the behavior desired and then no need
> > > for any extra keyword.
> > 
> > You have these two situations:
> > 
> > print_log(LOG_DEBUG, "doing this and that....\n");
> > 
> > if (debug) { /*
> >  gather_some_data_that_is_very_expensive_operation_to_do_all_the_time();
> >  print_log(LOG_DEBUG, "print those extra data\n");
> > }
> > 
> > as it is now, it would basically be an alias to set logpriority to DEBUG
> > but enables people to execute debugging code conditionally and as I
> > wrote it is an easy keyword to remember compared to
> > syslog_priority/logpriority.
> > 
> > Fabio
> > 
> 
> The second situation doesn't exist in any code I have written and never
> would.

Clearly you haven't read what I wrote in the debugging note.

> Turning debug on for all of the entire stack to be output to syslog is
> not satisfactory because messages would be lost in overload conditions.
> Logging to file is only a slight bit better solution but if you really
> must have debug output in a persistent store that doesn't occur as a
> result of a failure, logging to file is the only suitable answer.

Please point me to where I wrote that it should go to syslog as I only
mentioned logfile_priority so far.

Fabio

[Cluster-devel] logging: final call on configuration, output and implementation

[Steven Dake]

On Tue, 2008-11-11 at 06:54 +0100, Fabio M. Di Nitto wrote:
> On Mon, 2008-11-10 at 22:47 -0700, Steven Dake wrote:
> > On Tue, 2008-11-11 at 05:55 +0100, Fabio M. Di Nitto wrote:
> > > On Mon, 2008-11-10 at 17:46 -0700, Steven Dake wrote:
> > > > I disagree with a global debug keyword. 
> > > >  At one time I thought it was a
> > > > good idea but that time has long since passed.  The idea of turning
> > > > debug to on and then having all debug output go to syslog is frightening
> > > > and will result in lost messages.  While it appears this proposal
> > > > includes the selectable log output filtering per output medium as was
> > > > discussed already, it is unclear how the debug keyword affects this.  It
> > > > would simply make sense to change the file's log priority or the
> > > > syslog's log priority if that is the behavior desired and then no need
> > > > for any extra keyword.
> > > 
> > > You have these two situations:
> > > 
> > > print_log(LOG_DEBUG, "doing this and that....\n");
> > > 
> > > if (debug) { /*
> > >  gather_some_data_that_is_very_expensive_operation_to_do_all_the_time();
> > >  print_log(LOG_DEBUG, "print those extra data\n");
> > > }
> > > 
> > > as it is now, it would basically be an alias to set logpriority to DEBUG
> > > but enables people to execute debugging code conditionally and as I
> > > wrote it is an easy keyword to remember compared to
> > > syslog_priority/logpriority.
> > > 
> > > Fabio
> > > 
> > 
> > The second situation doesn't exist in any code I have written and never
> > would.
> 
> Clearly you haven't read what I wrote in the debugging note.
> 

I read it but don't agree you can have a discussion about logging and
flight recording without discussing how debugging fits into the log
system.


> > Turning debug on for all of the entire stack to be output to syslog is
> > not satisfactory because messages would be lost in overload conditions.
> > Logging to file is only a slight bit better solution but if you really
> > must have debug output in a persistent store that doesn't occur as a
> > result of a failure, logging to file is the only suitable answer.
> 
> Please point me to where I wrote that it should go to syslog as I only
> mentioned logfile_priority so far.
> 

If syslog is configured it will go to syslog by default in your scheme.

Regards
-steve

> Fabio
> 

[Cluster-devel] logging: final call on configuration, output and implementation

[Fabio M. Di Nitto]

Just for the record, Steven and I had a chat on IRC. This is the
transcript:

07:51 < riley_dt> yes i read your original message
07:52 < riley_dt> i was reswponding to your email not dave's
07:52 < riley_dt> your right i had not read dave's message at all
07:53 < fabbione> riley_dt: well.. nothing I can do about that, but the
thread is evolving in a positive direction 
                  IMHO. There are only few bits that needs smoothing
07:53 < riley_dt> i see the correction in dave's email
07:53 < fabbione> and I am simply talking from the end of the thread. I
can't possible know what have you read or not
07:54 < riley_dt> well you can put debug in the top level config
07:54 < riley_dt> but i dont intend to do anything about it :)
07:55 < fabbione> riley_dt: and that _IS_ fine :)
07:55 < fabbione> riley_dt: allow others to use it if they want
07:55 < riley_dt> also the oring has to go
07:55 < fabbione> oring?
07:55 < riley_dt> DEBUG|WARN
07:55 < riley_dt> should be DEBUG then debug+ are logged
07:55 < fabbione> there is no oring...
07:55 < fabbione> it's an example to show what values can be there
07:55 < fabbione> you can't ORING priorities
07:56 < riley_dt> you could or select priorities
07:56 < riley_dt> i thought that is what you proposed
07:56 < riley_dt> then we are good to go
07:56 < fabbione> no.. simply a list of options 
07:56 < fabbione> i received some comments from other people in my inbox
that I need to sort out
07:56 < fabbione> they didn't go to mailing lists
07:56 < riley_dt> well for trace there is oring
07:56 < riley_dt> ie: TRACE1|TRACE2
07:57 < riley_dt> it isn't "trace 1 or trace2" it is an oring
07:57 < fabbione> some are really good points.. so before everybody
dives into implementing, let's wait for the "final 
                  email"
07:57 < fabbione> gotcha...
07:57 < riley_dt> when I see | I think "or" 
07:57 < riley_dt> overloaded term i guess
08:49 < fabbione> riley_dt: just for the record, can you please reply to
thread and tell that you understood what we 
                  have been talking about? I strongly want to avoid the
situation where in 2 months from now we will be 
                  fighting again:" I said this, not this, understood
that etc."
10:11 < riley_dt> sure


On Mon, 2008-11-10 at 23:00 -0700, Steven Dake wrote:
> On Tue, 2008-11-11 at 06:54 +0100, Fabio M. Di Nitto wrote:
> > On Mon, 2008-11-10 at 22:47 -0700, Steven Dake wrote:
> > > On Tue, 2008-11-11 at 05:55 +0100, Fabio M. Di Nitto wrote:
> > > > On Mon, 2008-11-10 at 17:46 -0700, Steven Dake wrote:
> > > > > I disagree with a global debug keyword. 
> > > > >  At one time I thought it was a
> > > > > good idea but that time has long since passed.  The idea of turning
> > > > > debug to on and then having all debug output go to syslog is frightening
> > > > > and will result in lost messages.  While it appears this proposal
> > > > > includes the selectable log output filtering per output medium as was
> > > > > discussed already, it is unclear how the debug keyword affects this.  It
> > > > > would simply make sense to change the file's log priority or the
> > > > > syslog's log priority if that is the behavior desired and then no need
> > > > > for any extra keyword.
> > > > 
> > > > You have these two situations:
> > > > 
> > > > print_log(LOG_DEBUG, "doing this and that....\n");
> > > > 
> > > > if (debug) { /*
> > > >  gather_some_data_that_is_very_expensive_operation_to_do_all_the_time();
> > > >  print_log(LOG_DEBUG, "print those extra data\n");
> > > > }
> > > > 
> > > > as it is now, it would basically be an alias to set logpriority to DEBUG
> > > > but enables people to execute debugging code conditionally and as I
> > > > wrote it is an easy keyword to remember compared to
> > > > syslog_priority/logpriority.
> > > > 
> > > > Fabio
> > > > 
> > > 
> > > The second situation doesn't exist in any code I have written and never
> > > would.
> > 
> > Clearly you haven't read what I wrote in the debugging note.
> > 
> 
> I read it but don't agree you can have a discussion about logging and
> flight recording without discussing how debugging fits into the log
> system.
> 
> 
> > > Turning debug on for all of the entire stack to be output to syslog is
> > > not satisfactory because messages would be lost in overload conditions.
> > > Logging to file is only a slight bit better solution but if you really
> > > must have debug output in a persistent store that doesn't occur as a
> > > result of a failure, logging to file is the only suitable answer.
> > 
> > Please point me to where I wrote that it should go to syslog as I only
> > mentioned logfile_priority so far.
> > 
> 
> If syslog is configured it will go to syslog by default in your scheme.
> 
> Regards
> -steve
> 
> > Fabio
> > 
> 

[Cluster-devel] Re: logging: more input

[Fabio M. Di Nitto]

Hi guys,

I have received a bunch of comments and suggestions in my personal inbox
from people that didn't want to participate in the thread directly.

Some of them are absolutely valid points IMHO.

So here they are:

On Mon, 2008-11-10 at 20:09 +0100, Fabio M. Di Nitto wrote:

> == Output ==
> 
> to_file:
> 
> echo $(date "+%b %d %T") subsystem_or_daemon: entry_to_log
> Nov 10 19:46:40 subsystem_or_daemon: entry_to_log

Quoting:

--
I've not read any of the threads but did you consider a date format that
sorts easily i.e. YYYY MM DD-based?

And for investigating races across files/nodes using sub-second
timestamps (e.g. 19:46:40.123 ) and adding the node name.

So you can just extract relevant sections from several files, cat them
all together, sort them and then review the sequence of happenings.

If you're generalising, make the log format string a customisable option
similar to apache.
--

I agree that a more precise time and sortable format is a major winner.
He also has a good point to add the node name to the log.

My reply to the "customisable format" request was based on the
discussion we had at the Summit (that we want to avoid parsing lots of
different logging etc.) and his reply was:

Quoting:

--

1. Most people will use the default or one of the example lines you
supply.

2. If it is a problem, you've got the corresponding config line so it
should be straightforward to have a utility to convert it back to your
preferred canonical output format.
(And yes, "Nov 14" format can also be parsed and converted back into a
sortable format without much difficulty, so it's no big deal.)

3. If lots of people change the log format, then you probably chose a
poor default.

4. There's no reason why log-to-file should exactly match syslog format.
If you want syslog format, use syslog.  You want the best format to
assist you in debugging problems etc.

--

Fabio