[Cluster-devel] spectator setting in cluster.conf

[Cluster-devel] spectator setting in cluster.conf

[David Teigland]

I believe that all cluster.conf settings currently map to a specific
setting in one specific subsystem.  We now have a proposal to add an
abstract "spectator" setting that would be translated into multiple
lower-level, subsystem-specific settings.  It's not clear what the
cluster.conf syntax would look like, but it would need to be per-node.

Initially, there could be three specific results of this setting:

1. gfs_controld would read this and append the "spectator" mount option
   to all gfs mounts made by the node.  This causes gfs to not use a
   journal, and to never write to the disk (includes never doing
   recovery).

   The standard, lower-level mechanism for doing this (*without* the
   new setting in cluster.conf), is to add "spectator" to /etc/fstab
   entries.  We need to be careful when inventing specialized ways of
   doing things outside the standard practice.

2. /etc/init.d/cman would skip the 'fence_tool join' step for nodes
   with this setting.  Because gfs is guaranteed to never write to the
   fs, we can allow the node to mount without joining the fence domain.
   If the spectator node fails, it would not be fenced.

   The standard, low-level mechanism for doing this (without the new
   setting in cluster.conf) is to add a new option to the init-script's
   config file /etc/sysconfig/cman.

3. service_cman.lcrso (the cman openais plugin) would read this and
   assign the node 0 votes, so its membership wouldn't affect quorum.

   The standard way to do this would be to set votes="0" in the
   <clusternode/> entry.

In each case, the existence of the lower-level setting would override the
effect of the abstract spectator setting.

So, setting spectator for a node would be a shortcut for configuring the
three items above.  We may also decide to add other effects in other
subsystems.  Typically, this kind of abstraction is translated by a higher
layer of software, like conga.  But, this may be a case where the
abstraction is useful enough that it should be available to non-conga
users.  I don't have any preference in the matter and would be happy to
see it implemented in either layer.

Dave

[Cluster-devel] spectator setting in cluster.conf

[Fabio Massimo Di Nitto]

Hi David,

David Teigland wrote:
> I believe that all cluster.conf settings currently map to a specific
> setting in one specific subsystem.  We now have a proposal to add an
> abstract "spectator" setting that would be translated into multiple
> lower-level, subsystem-specific settings.  It's not clear what the
> cluster.conf syntax would look like, but it would need to be per-node.

I did a couple of tests and:

<clusternode name="node1" nodeid="1" votes="1">
 <spectator/>
</clusternode>

seems to be the sanest one and the query would look like:

/cluster/clusternodes/clusternode[@name=\"%s\"]/spectator"

no ccs_get error and empty str (looking at general ccs_get sintax from cmanccs.c).

If we want to use this sintax:

<clusternode name="node1" nodeid="1" votes="1" spectator="..">

then we need to add a value and parse it. At that point you might as well
set votes to 0 directly.

So I would personally prefer the first one.

> 2. /etc/init.d/cman would skip the 'fence_tool join' step for nodes
>    with this setting.  Because gfs is guaranteed to never write to the
>    fs, we can allow the node to mount without joining the fence domain.
>    If the spectator node fails, it would not be fenced.
> 
>    The standard, low-level mechanism for doing this (without the new
>    setting in cluster.conf) is to add a new option to the init-script's
>    config file /etc/sysconfig/cman.

This one will be tricky because we cannot ask ccsd for this setting via
ccs_tools before ccs is talking to cman and that won't happen until cman is
started. There is a little risk of a race condition where we will need to wait
for cman/ccsd to talk before we can decide if we need to execute
"fence_tool join" or not. It is a small window but still there.
This problem will not exists once we kill ccsd.

> 3. service_cman.lcrso (the cman openais plugin) would read this and
>    assign the node 0 votes, so its membership wouldn't affect quorum.
> 
>    The standard way to do this would be to set votes="0" in the
>    <clusternode/> entry.

I have a test patch for it if the assumption is that:

- spectator is more important than votes (spectator will override votes settings)
- spectator setting cannot be changed by anything other than the cluster.conf
(read below)

> In each case, the existence of the lower-level setting would override the
> effect of the abstract spectator setting.

What kind of lower-level settings are you thinking about? env vars? In cman,
votes are checked at the very beginning of the startup process. There isn't much
other than env vars to override.

Also note that the same env vars should be set across the whole cluster to have
an real effect. At that point I believe it would be easier to just update the
config to remove/add spectator and recalculate quorum and votes.

Cheers
Fabio

-- 
I'm going to make him an offer he can't refuse.

[Cluster-devel] spectator setting in cluster.conf

[David Teigland]

On Tue, Oct 23, 2007 at 07:25:07AM +0200, Fabio Massimo Di Nitto wrote:
> <clusternode name="node1" nodeid="1" votes="1">
>  <spectator/>
> </clusternode>
> 
> seems to be the sanest one and the query would look like:

agree

> I have a test patch for it if the assumption is that:
> 
> - spectator is more important than votes (spectator will override votes
> settings)
> - spectator setting cannot be changed by anything other than the cluster.conf
> (read below)

great

> > In each case, the existence of the lower-level setting would override the
> > effect of the abstract spectator setting.
> 
> What kind of lower-level settings are you thinking about? env vars?

A "ro" or "rw" (or "nospectator"?) option in /etc/fstab would override the
addition of "spectator" to the mount options implied by <spectator/>.

We'd add a variable to /etc/sysconfig/cman that would tell the init script
to run fence_tool join.  If this was set, it would override the skipping
of that step implied by <spectator/>.

An explicit votes= setting for a node would override the 0 votes implied
by <spectator/>.

Dave

[Cluster-devel] spectator setting in cluster.conf

[Fabio Massimo Di Nitto]

David Teigland wrote:
> On Tue, Oct 23, 2007 at 07:25:07AM +0200, Fabio Massimo Di Nitto wrote:
>> <clusternode name="node1" nodeid="1" votes="1">
>>  <spectator/>
>> </clusternode>
>>
>> seems to be the sanest one and the query would look like:
> 
> agree

Ok I will prepare a cman test patch based on this sintax.

> 
>> I have a test patch for it if the assumption is that:
>>
>> - spectator is more important than votes (spectator will override votes
>> settings)
>> - spectator setting cannot be changed by anything other than the cluster.conf
>> (read below)
> 
> great
> 
>>> In each case, the existence of the lower-level setting would override the
>>> effect of the abstract spectator setting.
>> What kind of lower-level settings are you thinking about? env vars?
> 
> A "ro" or "rw" (or "nospectator"?) option in /etc/fstab would override the
> addition of "spectator" to the mount options implied by <spectator/>.

This is part of gfs_controld/gfs*mount.

> We'd add a variable to /etc/sysconfig/cman that would tell the init script
> to run fence_tool join.  If this was set, it would override the skipping
> of that step implied by <spectator/>.

sounds reasonable..

> An explicit votes= setting for a node would override the 0 votes implied
> by <spectator/>.

So ok. I need to understand you better because I think what I wrote before is in
contradiction with this override.

In my patch spectator overrides votes="" no matter if they are specified or not.

Here you say that spectator overrides automatic setting of votes="" when
votes="" is not specified in the config. So in my head this implies two config
changes to set a node to spectator. Remove the votes="" entry (if any and IME is
quite common in the configs) and add spectator.

I think it makes more sense (to me) to override votes="" in full when spectator
is set. One config change, one flag and it's done. Override could be done via
env vars that can be set the same way as for "fence_tool join" in
etc/sysconfig/cman.

Make sense?

Fabio

-- 
I'm going to make him an offer he can't refuse.

[Cluster-devel] spectator setting in cluster.conf

[David Teigland]

On Tue, Oct 23, 2007 at 06:58:32PM +0200, Fabio Massimo Di Nitto wrote:
> > An explicit votes= setting for a node would override the 0 votes implied
> > by <spectator/>.
> 
> So ok. I need to understand you better because I think what I wrote
> before is in contradiction with this override.
> 
> In my patch spectator overrides votes="" no matter if they are specified
> or not.
> 
> Here you say that spectator overrides automatic setting of votes="" when
> votes="" is not specified in the config. So in my head this implies two
> config changes to set a node to spectator. Remove the votes="" entry (if
> any and IME is quite common in the configs) and add spectator.
> 
> I think it makes more sense (to me) to override votes="" in full when
> spectator is set. 

If no votes are specified, the default is 1.  If no votes are specified
and <spectator/> exists, then the default is 0.  If you want to override
either of these defaults, then you include votes="N".  A specified value
must always override a default value.

If an existing cluster.conf has <clusternode ... votes="1">, the votes
setting is obviously extraneous.  And making this node a full spectator
would require either:

- changing to an explicit votes="0" and adding <spectator/>, or
- removing the explicit votes setting altogether and adding <spectator/>

It needs to be possible to have a spectator node with 1 vote (or more),
and if you're saying that this config:

  <clusternode name="node1" nodeid="1" votes="1">
    <spectator/>
  </clusternode>

should *not* work to do that, then it's madness :-)

Dave

[Cluster-devel] spectator setting in cluster.conf

[Fabio Massimo Di Nitto]

David Teigland wrote:
> On Tue, Oct 23, 2007 at 06:58:32PM +0200, Fabio Massimo Di Nitto wrote:
>>> An explicit votes= setting for a node would override the 0 votes implied
>>> by <spectator/>.
>> So ok. I need to understand you better because I think what I wrote
>> before is in contradiction with this override.
>>
>> In my patch spectator overrides votes="" no matter if they are specified
>> or not.
>>
>> Here you say that spectator overrides automatic setting of votes="" when
>> votes="" is not specified in the config. So in my head this implies two
>> config changes to set a node to spectator. Remove the votes="" entry (if
>> any and IME is quite common in the configs) and add spectator.
>>
>> I think it makes more sense (to me) to override votes="" in full when
>> spectator is set. 
> 
> If no votes are specified, the default is 1.  If no votes are specified
> and <spectator/> exists, then the default is 0.  If you want to override
> either of these defaults, then you include votes="N".  A specified value
> must always override a default value.

agreed :)

> If an existing cluster.conf has <clusternode ... votes="1">, the votes
> setting is obviously extraneous.  And making this node a full spectator
> would require either:
> 
> - changing to an explicit votes="0" and adding <spectator/>, or
> - removing the explicit votes setting altogether and adding <spectator/>

Ok perfect. That's all I wanted to agree on.

> It needs to be possible to have a spectator node with 1 vote (or more),
> and if you're saying that this config:
> 
>   <clusternode name="node1" nodeid="1" votes="1">
>     <spectator/>
>   </clusternode>
> 
> should *not* work to do that, then it's madness :-)

In my original proposal this snippet would have turned the node into a spectator
with <spectator/> having a higher priority over votes. That was all the origin
of my misunderstanding and I had to make sure we were on the same page.

Thanks for explaining over and over :)

Fabio

-- 
I'm going to make him an offer he can't refuse.

[Cluster-devel] spectator setting in cluster.conf

[Fabio Massimo Di Nitto]

David Teigland wrote:

> 
>> I have a test patch for it if the assumption is that:

> great
> 

The patch in attachment should take care of cman according to what we discussed.

Fabio

-- 
I'm going to make him an offer he can't refuse.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: spectator.diff
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20071024/831c8c08/attachment.ksh>

[Cluster-devel] spectator setting in cluster.conf

[Fabio Massimo Di Nitto]

Fabio Massimo Di Nitto wrote:
> David Teigland wrote:
> 
>>> I have a test patch for it if the assumption is that:
> 
>> great
>>
> 
> The patch in attachment should take care of cman according to what we discussed.
> 
> Fabio
> 
> 

Updated patch on top of CVS HEAD after Ryan fixes to cman.

Fabio

-- 
I'm going to make him an offer he can't refuse.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: spectator2.diff
URL: <http://listman.redhat.com/archives/cluster-devel/attachments/20071024/e08a83c2/attachment.ksh>