[Cluster-devel] fenced logsys/cman/ccs setup

[Fabio M. Di Nitto <fdinitto@redhat.com>]

On Wed, 2 Jul 2008, David Teigland wrote:

> On Wed, Jul 02, 2008 at 06:31:10AM +0200, Fabio M. Di Nitto wrote:
>>> @@ -9,18 +11,62 @@ static int open_ccs(void)
>>> 		sleep(1);
>>> 		if (++i > 9 && !(i % 10))
>>> 			log_error("connect to ccs error %d, "
>>> -				  "check ccsd or cluster status", cd);
>>> +				  "check cluster status", cd);
>>> +
>>> +		/* FIXME: do we want this infinite? */
>>> +		if (i > 10)
>>> +			break;
>>
>> I think we want this to be infinite (and consistent across the board) or
>> configurable the same way other daemons/tools do so that users can decide
>> how long to wait.
>
> OK, I think the following logic supports that:
>
> - During startup, a program first connects to cman and waits for it to be
>  ready (this should be a consistent and finite retry period).

Aggreed. Some daemons get a timeout setting from the tools and stuff like 
that. We can review that after we start standardizing around.

> - Then the program connects to ccs.  I think we can assume that since cman
>  is running, the ccs connection will work within a short time.  So,
>  it's safe to just put an infinite loop around the ccs_connect.

If we loop in cman and wait for is_available, i doubt there is any need to 
loop in ccs at all. ccs plugin is initialized before cman within the 
aisexec execution path.

>>> +#define DEFAULT_FACILITY	LOG_DAEMON
>>
>> You can either do:
>>
>> #define DEFAULT_FACILITY SYSLOGFACILITY
>>
>> or just use SYSLOGFACILITY instead.
>>
>> SYSLOGFACILITY is defined by the build system as default build option and
>> can be set by packagers according to distro rules/best practice/etc.
>
> OK, that works for fenced.  What about things that aren't a daemon,
> though?  Perhaps they'd like a different default?

This is a build time setting that's usually set by the distro. I don't see 
the use case to have them logging at different facility.

>
>>> +#define DEFAULT_PRIORITY	LOG_LEVEL_ERROR
>>
>> I don't have a DEFAULT_PRIORITY in the build system (maybe i can add it)
>> but it should be LOG_LEVEL_INFO. At least this is the best practise we
>> used so far.
>
> OK, just to be clear, this setting is used:
> - during startup before reading the user preferences in cluster.conf
> - during running if the user has set no preference in cluster.conf

This is right.

>
> and the defaults should assume that the majority of people will never add
> logging preferences to cluster.conf.  (Both because we picked good
> defaults, and because they're lazy to look up and write the xml.)

Indeed. So for now let's set it to LOG_LEVEL_INFO and I will make it a 
build option for the user to decide what is the built-in default.

>
>>> +#define DEFAULT_FILE		NULL
>>
>> #define DEFAULT_FILE LOGDIR "/fenced.log"
>>
>> LOGDIR is set by the build system (same reasons as SYSLOGFACILITY). We
>> want files by default consistently across the board.
>
> I think that by default we should probably have all logging go to
> /var/log/messages like it has in the past.  Or, if we're daring, maybe
> have it all go to /var/log/cluster.log.  But, I really doubt that people
> want all programs to log to different files.

I will answer this in the other email.

>>> +#define LEVEL_PATH
>>> "/cluster/logging/logger_subsys[@subsys=\"FENCED\"]/@syslog_level"
>>> +#define DEBUG_PATH
>>> "/cluster/logging/logger_subsys[@subsys=\"FENCED\"]/@debug"
>>
>> I am kind of curious.. why do you add defines for some queries but not for
>> others?
>
> I'm still in limbo about what coding style I like there; in this case it's
> if the line goes over 80 chars.

ehe ok :)

>> You need to check for:
>>
>> /cluster/logging/@debug - global debug on/off switch.
>>
>> the other daemons respects globla debug setting if no subsystem debug
>> setting is available.
>
> OK, so subsystem setting has priority over global setting.

Exactly.

>>
>> - cmdline debug:
>>   allows the user to enable debugging manually only on that specific
>>   daemon on that specific node.
>>
>> - envvar debug on (see for example CMAN_DEBUG or CCSD_DEBUG or QDISK_...):
>>   these are easy to set within init scripts and /etc/defaults/... without
>>   changing the daemon invokation in the init script itself that is not
>>   always trivial and it is not a recommended practise.
>>   the use case is similar (but more flexible) to the cmdline debug.
>>   It allows the users to enable debugging for one specific node.
>>
>> cmdline and envvar debugging overrides whatever is in the config.
>
> This is taking it too far for me (overengineering).  You'll need a new
> debugging system to debug your complicated debugging system.

It is much simpler than you think, really. If debug is set by cmdline (the 
way you usually do it as developer) or envvar (very useful for init 
scripts that we never use directly, but production users do), you don't 
even need to query ccs for those info :)

> There two debug schemes:
>
> 1. for users/customers:  uses logsys, configured in cluster.conf

^^^ this is always cluster wide. Doesn't allow you to do debugging on a 
single node.

> 2. for me:  does not use logsys, controlled via command line

2 doesn't exist really. It's for "us" all developers ;) and some of us 
would like to use logsys.

Fabio

--
I'm going to make him an offer he can't refuse.