[Drbd-dev] changing the fencing policy on the fly

[Drbd-dev] changing the fencing policy on the fly

[Petrakis, Peter]

Hi All,
 
I have a challenge with the use of fencing and the state of flux in our
cluster. The system 
will setup drbd devices in advance and use them independent of it's
peer, which will mirror
it's config once it joins the cluster. Since the fencing helper makes
use of drbdadm and thus
the config file we have a problem. If fencing were to be triggered, it
would immediately fail
with a bad exit status because the config file is inconsistent. We use a
naming convention
for the disks in our cluster so the only piece of information I don't
know at this time is my
peers ip address.

To work around this bug I set the fencing policy to 'dont-care' and then
once the resource
is connected and uptodate, detach the backing store and then reattach it
with the desired
fencing policy. This leaves us vulnerable however, should the link go
down for any reason we're
in big trouble.

I'm trying to eliminate this hole without detaching the backing store. I
can't play tricks with
aliasing the eth interface to some address because I'm not physically in
control of the box and don't
want to ruin someone's network in case they hook up the wires
incorrectly. DRBD doesn't seem to support
IPV6 link local addressing which would help in this case because the
nodes are directly attached.

What I'd like to do is to change the fencing policy on demand and the
mechanism I wish to use is sysfs.
Do you guys see any pitfalls with this approach, locking especially? The
reason I want to use sysfs is two
fold. I want to get away from using drbdadm for the userspace helper and
it'll make writing software to
manage drbd more straight forward. Thanks in advance.

Peter

Re: [Drbd-dev] changing the fencing policy on the fly

[Lars Ellenberg]

On Mon, Nov 30, 2009 at 01:51:02PM -0500, Petrakis, Peter wrote:
> Hi All,
>  
> I have a challenge with the use of fencing and the state of flux in our
> cluster. The system 
> will setup drbd devices in advance and use them independent of it's
> peer, which will mirror
> it's config once it joins the cluster. Since the fencing helper makes
> use of drbdadm and thus
> the config file we have a problem. If fencing were to be triggered, it
> would immediately fail
> with a bad exit status because the config file is inconsistent. We use a
> naming convention
> for the disks in our cluster so the only piece of information I don't
> know at this time is my
> peers ip address.
> 
> To work around this bug I set the fencing policy to 'dont-care' and then
> once the resource
> is connected and uptodate, detach the backing store and then reattach it
> with the desired
> fencing policy. This leaves us vulnerable however, should the link go
> down for any reason we're
> in big trouble.
> 
> I'm trying to eliminate this hole without detaching the backing store. I
> can't play tricks with
> aliasing the eth interface to some address because I'm not physically in
> control of the box and don't
> want to ruin someone's network in case they hook up the wires
> incorrectly. DRBD doesn't seem to support
> IPV6 link local addressing which would help in this case because the
> nodes are directly attached.

IPv6 is supposed to work.

> What I'd like to do is to change the fencing policy on demand and the
> mechanism I wish to use is sysfs.
> Do you guys see any pitfalls with this approach, locking especially? The
> reason I want to use sysfs is two
> fold. I want to get away from using drbdadm for the userspace helper and
> it'll make writing software to
> manage drbd more straight forward. Thanks in advance.

modinfo drbd
usermode_helper:string
	(default: /sbin/drbdadm)
	also available via /sys/modules/drbd/parameters/usermode_helper

You can add your own wrapper around drbdadm as "hanlder" helper.

But feel free to write a patch to replace the DRBD config interface
(currently netlink/connector resp. genetlink based) to
a sysfs based one, and we might consider it ;)

Use a "DRBD" bus, each sub dir being a single drbd,
expose all parameters we currently have,
see that you don't step on each others toes when configuring those.
you probably need a "live" and a "pending" config set,
so you can change paramters when configuring in sequence,
then "commit" them in one go.

Post on drbd-dev and possibly on lkml.

Converting single parameters to sysfs in some ad-hoc fashion,
just because you feel _this_ one should be there,
is not useful und unmaintainable.

Thanks,

	Lars

Re: [Drbd-dev] changing the fencing policy on the fly

[Petrakis, Peter]

 

> -----Original Message-----
> > IPV6 link local addressing which would help in this case 
> because the 
> > nodes are directly attached.
> 
> IPv6 is supposed to work.

and it does, it's this particular construct doesn't seem supported,
fe80::a00:5aff:fe38:6f86%eth0 .

It's not an issue anyways, I was misunderstood how it worked, I
assumed I needed only my MAC address and not my peers. 

> modinfo drbd
> usermode_helper:string
> 	(default: /sbin/drbdadm)
> 	also available via /sys/modules/drbd/parameters/usermode_helper
> 
> You can add your own wrapper around drbdadm as "hanlder" helper.

That's definitely a possibility. 
 
> But feel free to write a patch to replace the DRBD config 
> interface (currently netlink/connector resp. genetlink based) 
> to a sysfs based one, and we might consider it ;)

ok :)
 
> Use a "DRBD" bus, each sub dir being a single drbd, expose 
> all parameters we currently have, see that you don't step on 
> each others toes when configuring those.
> you probably need a "live" and a "pending" config set, so you 
> can change paramters when configuring in sequence, then 
> "commit" them in one go.
> 
> Post on drbd-dev and possibly on lkml.
> 
> Converting single parameters to sysfs in some ad-hoc fashion, 
> just because you feel _this_ one should be there, is not 
> useful und unmaintainable.

Agreed. I scanned the code again regarding the use of the fencing
policy and noticed something that might solve this particular problem.

enum drbd_disk_state drbd_try_outdate_peer(struct drbd_conf *mdev)
{
    char *ex_to_string;
    int r;
    enum drbd_disk_state nps;
    enum fencing_policy fp;

    D_ASSERT(mdev->state.pdsk == DUnknown);

    if (inc_local_if_state(mdev, Consistent)) {
        fp = mdev->bc->dc.fencing;
        dec_local(mdev);
    } else {
        WARN("Not outdating peer, I'm not even Consistent myself.\n");
        return mdev->state.pdsk;
    }

    if (fp == Stonith)
        _drbd_request_state(mdev, NS(susp,1), ChgWaitComplete);

    r = drbd_khelper(mdev, "outdate-peer");

    switch ((r>>8) & 0xff) {
    case 3: /* peer is inconsistent */
        ex_to_string = "peer is inconsistent or worse";
        nps = Inconsistent;
        break;
    case 4:
        ex_to_string = "peer is outdated";
        nps = Outdated;
        break;
    case 5: /* peer was down, we will(have) create(d) a new UUID
anyways... */
        /* If we would be more strict, we would return DUnknown here. */
        ex_to_string = "peer is unreachable, assumed to be dead";
        nps = Outdated;
        break;
...

Nowhere does the code check that a valid network config ever existed. It
presuppposes that a peer must exist if a fencing policy is configured.
Could
I safely argue that if mdev->net_conf == NULL then we could return 5?
Thanks.

Peter

Re: [Drbd-dev] changing the fencing policy on the fly

[Petrakis, Peter]

Implementing a usermode helper seems to have solved all my problems.
I'm a little dissapointed though, it would've been cool to have the 
green light to do the sysfs work :). It doesn't seem like any of the
kernel drivers that require usermode configuration have fully solved
this issue, even mdadm. There are arguments to use sysfs for reading and
tweaking and then configfs for the first time create stuff.

http://lkml.indiana.edu/hypermail/linux/kernel/0602.0/0462.html

Though the only real user of configfs to date is ocfs2!

Peter

> -----Original Message-----
> From: drbd-dev-bounces@lists.linbit.com 
> [mailto:drbd-dev-bounces@lists.linbit.com] On Behalf Of Lars Ellenberg
> Sent: Monday, November 30, 2009 2:11 PM
> To: drbd-dev@lists.linbit.com
> Subject: Re: [Drbd-dev] changing the fencing policy on the fly
> 
> On Mon, Nov 30, 2009 at 01:51:02PM -0500, Petrakis, Peter wrote:
> > Hi All,
> >  
> > I have a challenge with the use of fencing and the state of flux in 
> > our cluster. The system will setup drbd devices in advance and use 
> > them independent of it's peer, which will mirror it's 
> config once it 
> > joins the cluster. Since the fencing helper makes use of 
> drbdadm and 
> > thus the config file we have a problem. If fencing were to be 
> > triggered, it would immediately fail with a bad exit status because 
> > the config file is inconsistent. We use a naming convention for the 
> > disks in our cluster so the only piece of information I 
> don't know at 
> > this time is my peers ip address.
> > 
> > To work around this bug I set the fencing policy to 'dont-care' and 
> > then once the resource is connected and uptodate, detach 
> the backing 
> > store and then reattach it with the desired fencing policy. This 
> > leaves us vulnerable however, should the link go down for 
> any reason 
> > we're in big trouble.
> > 
> > I'm trying to eliminate this hole without detaching the 
> backing store. 
> > I can't play tricks with aliasing the eth interface to some address 
> > because I'm not physically in control of the box and don't want to 
> > ruin someone's network in case they hook up the wires incorrectly. 
> > DRBD doesn't seem to support
> > IPV6 link local addressing which would help in this case 
> because the 
> > nodes are directly attached.
> 
> IPv6 is supposed to work.
> 
> > What I'd like to do is to change the fencing policy on 
> demand and the 
> > mechanism I wish to use is sysfs.
> > Do you guys see any pitfalls with this approach, locking 
> especially? 
> > The reason I want to use sysfs is two fold. I want to get away from 
> > using drbdadm for the userspace helper and it'll make 
> writing software 
> > to manage drbd more straight forward. Thanks in advance.
> 
> modinfo drbd
> usermode_helper:string
> 	(default: /sbin/drbdadm)
> 	also available via /sys/modules/drbd/parameters/usermode_helper
> 
> You can add your own wrapper around drbdadm as "hanlder" helper.
> 
> But feel free to write a patch to replace the DRBD config 
> interface (currently netlink/connector resp. genetlink based) 
> to a sysfs based one, and we might consider it ;)
> 
> Use a "DRBD" bus, each sub dir being a single drbd, expose 
> all parameters we currently have, see that you don't step on 
> each others toes when configuring those.
> you probably need a "live" and a "pending" config set, so you 
> can change paramters when configuring in sequence, then 
> "commit" them in one go.
> 
> Post on drbd-dev and possibly on lkml.
> 
> Converting single parameters to sysfs in some ad-hoc fashion, 
> just because you feel _this_ one should be there, is not 
> useful und unmaintainable.
> 
> Thanks,
> 
> 	Lars
> 
> _______________________________________________
> drbd-dev mailing list
> drbd-dev@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-dev
>