[Drbd-dev] drbdsetup events patch to announce Split-brain

[Drbd-dev] drbdsetup events patch to announce Split-brain

[Montrose, Ernest]

[-- Attachment #1.1: Type: text/plain, Size: 382 bytes --]

Hi,
 
Included here is a small patch that I find useful when monitoring with
drbdsetup().  It adds the ability to generate an netlink event to user
space when drbd Split-brain and disconnect.  Based on that event we can
then take action. I could not find a more
convenient way to know that a split-brain occurred other then looking in
/var/log/messages.
 
Thanks
 
EM--

[-- Attachment #1.2: Type: text/html, Size: 4866 bytes --]

[-- Attachment #2: sb_event.patch --]
[-- Type: application/octet-stream, Size: 3208 bytes --]

Index: drbd/drbd_receiver.c
===================================================================
--- drbd/drbd_receiver.c	(revision 19016)
+++ drbd/drbd_receiver.c	(working copy)
@@ -2011,6 +2011,7 @@
 		drbd_uuid_dump(mdev,"self",mdev->bc->md.uuid);
 		drbd_uuid_dump(mdev,"peer",mdev->p_uuid);
 		drbd_force_state(mdev,NS(conn,Disconnecting));
+              drbd_bcast_split_brain(mdev,1);//Tell user we split brained 
 		return conn_mask;
 	}
 
Index: drbd/drbd_nl.c
===================================================================
--- drbd/drbd_nl.c	(revision 19016)
+++ drbd/drbd_nl.c	(working copy)
@@ -1759,6 +1759,41 @@
 	cn_netlink_send(cn_reply, CN_IDX_DRBD, GFP_KERNEL);
 }
 
+void drbd_bcast_split_brain(drbd_dev *mdev,int flag)
+{
+	char buffer[sizeof(struct cn_msg)+
+		    sizeof(struct drbd_nl_cfg_reply)+
+		    sizeof(struct split_brain_tag_len_struct)+
+		    sizeof(short int)];
+	struct cn_msg *cn_reply = (struct cn_msg *) buffer;
+	struct drbd_nl_cfg_reply* reply = (struct drbd_nl_cfg_reply*)cn_reply->data;
+	unsigned short *tl = reply->tag_list;
+	int val = flag;
+       
+	*tl++ = T_split_brain;
+	*tl++ = sizeof(int);
+       memcpy(tl, &val, sizeof(int));
+	tl=(unsigned short*)((char*)tl + sizeof(int));
+	*tl++ = TT_END; /* Close the tag list */
+
+	cn_reply->id.idx = CN_IDX_DRBD;
+	cn_reply->id.val = CN_VAL_DRBD;
+
+	cn_reply->seq = atomic_add_return(1,&drbd_nl_seq);
+	cn_reply->ack = 0; // not used here.
+	cn_reply->len = sizeof(struct drbd_nl_cfg_reply) +
+		(int)((char*)tl - (char*)reply->tag_list);
+	cn_reply->flags = 0;
+
+	reply->packet_type = P_split_brain;
+	reply->minor = mdev_to_minor(mdev);
+	reply->ret_code = NoError;
+
+	TRACE(TraceTypeNl, TraceLvlSummary, nl_trace_reply(cn_reply););
+
+	cn_netlink_send(cn_reply, CN_IDX_DRBD, GFP_KERNEL);
+}
+
 void drbd_bcast_sync_progress(drbd_dev *mdev)
 {
 	char buffer[sizeof(struct cn_msg)+
Index: drbd/linux/drbd_nl.h
===================================================================
--- drbd/linux/drbd_nl.h	(revision 19016)
+++ drbd/linux/drbd_nl.h	(working copy)
@@ -97,6 +97,10 @@
 	INTEGER(	43,	T_MAY_IGNORE,	sync_progress)
 )
 
+PACKET(split_brain, 24,
+	INTEGER(	44,	T_MAY_IGNORE,	split_brain)
+)
+
 #undef PACKET
 #undef INTEGER
 #undef INT64
Index: drbd/drbd_int.h
===================================================================
--- drbd/drbd_int.h	(revision 19016)
+++ drbd/drbd_int.h	(working copy)
@@ -1417,6 +1417,7 @@
 int __init drbd_nl_init(void);
 void drbd_bcast_state(drbd_dev *mdev);
 void drbd_bcast_sync_progress(drbd_dev *mdev);
+void drbd_bcast_split_brain(drbd_dev *mdev,int flag);
 
 /*
  * inline helper functions
Index: user/drbdsetup.c
===================================================================
--- user/drbdsetup.c	(revision 19016)
+++ user/drbdsetup.c	(working copy)
@@ -1231,6 +1231,14 @@
 				synced % 10);
 		} else fprintf(stderr,"Missing tag !?\n");
 		break;
+	case P_split_brain:
+		if (consume_tag_int(T_split_brain, reply->tag_list, &synced)) {
+			printf("%u SB %d %d\n", 
+				seq,
+				reply->minor,
+				synced);
+		} else fprintf(stderr,"Missing tag !?\n");
+		break;
 	default:
 		printf("%u ?? %d <other message>\n",seq, reply->minor);
 		break;

Re: [Drbd-dev] drbdsetup events patch to announce Split-brain

[Philipp Reisner]

On Monday 01 October 2007 23:24:02 Montrose, Ernest wrote:
> Hi,
>
> Included here is a small patch that I find useful when monitoring with
> drbdsetup().  It adds the ability to generate an netlink event to user
> space when drbd Split-brain and disconnect.  Based on that event we can
> then take action. I could not find a more
> convenient way to know that a split-brain occurred other then looking in
> /var/log/messages.
>

Hi Ernest,

I have nearly overlooked this patch... 
You are right, it is definitely a good idea to have that. 

I just need to change the tag number for the split_brain flag,
since that number is already allocated in drbd-8.2.0. 

The question I have, what is the flag good for ? Do you envision
an future use of it ?  otherwise I would drop it, it is possible
to have such an cn packet, that does not carry any futher data...

-phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :

RE: [Drbd-dev] drbdsetup events patch to announce Split-brain

[Montrose, Ernest]

Phil,
Currently we do not use the payload for anything.  I was just being
paranoid thinking that we might need it in the future.  But you can drop
it if you wish.  If we have a pressing need for it in the future, we'll
deal with it then.

Thanks.
EM--

-----Original Message-----
From: Philipp Reisner [mailto:philipp.reisner@linbit.com] 
Sent: Thursday, October 25, 2007 6:16 AM
To: drbd-dev@linbit.com
Cc: Montrose, Ernest
Subject: Re: [Drbd-dev] drbdsetup events patch to announce Split-brain

On Monday 01 October 2007 23:24:02 Montrose, Ernest wrote:
> Hi,
>
> Included here is a small patch that I find useful when monitoring with
> drbdsetup().  It adds the ability to generate an netlink event to user
> space when drbd Split-brain and disconnect.  Based on that event we
can
> then take action. I could not find a more
> convenient way to know that a split-brain occurred other then looking
in
> /var/log/messages.
>

Hi Ernest,

I have nearly overlooked this patch... 
You are right, it is definitely a good idea to have that. 

I just need to change the tag number for the split_brain flag,
since that number is already allocated in drbd-8.2.0. 

The question I have, what is the flag good for ? Do you envision
an future use of it ?  otherwise I would drop it, it is possible
to have such an cn packet, that does not carry any futher data...

-phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :

Re: [Drbd-dev] drbdsetup events patch to announce Split-brain

[Philipp Reisner]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

On Thursday 25 October 2007 14:24:55 Montrose, Ernest wrote:
> Phil,
> Currently we do not use the payload for anything.  I was just being
> paranoid thinking that we might need it in the future.  But you can drop
> it if you wish.  If we have a pressing need for it in the future, we'll
> deal with it then.
>

Ernest,

Over the weekend I realized that the split brain event is quite
similar to a number of events we already signal back to userspace.

The invocation of handlers is also visible through the "drbdsetup events"
interface.

The attached patch creates an handler called "split-brain". If you
can live with that interface as well, I will commit that...

In the future I want to go this way:

  * Allow one to set the usermode_helper to the empty string. Then it 
    is still possible to get all events through the "drbdsetup events"
    interface.
  * The "outdate-peer" usermode_helper sends an exit code back
    to kernel space. This is not yet possible when one only uses
    the "drbdsetup events" interface. This should get fixed.

-Phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :

[-- Attachment #2: sb_as_handler.diff --]
[-- Type: text/x-diff, Size: 3402 bytes --]

diff --git a/documentation/drbd.conf.sgml b/documentation/drbd.conf.sgml
index b8c1282..c4b59af 100644
--- a/documentation/drbd.conf.sgml
+++ b/documentation/drbd.conf.sgml
@@ -247,7 +247,7 @@ resource r0 {
     Optional parameters:
     <option>pri-on-incon-degr</option>, <option>pri-lost-after-sb</option>,
     <option>pri-lost</option>, <option>outdate-peer</option>,
-    <option>local-io-error</option>.
+    <option>local-io-error</option>, <option>split-brain</option>.
   </para>
   </listitem>
 </varlistentry>
@@ -934,6 +934,13 @@ the unit is 1 second.
   <indexterm><primary>drbd.conf</primary><secondary>local-io-error </secondary></indexterm>
   </para></listitem> </varlistentry>
 
+<varlistentry>
+  <term><option>split-brain <replaceable>cmd</replaceable></option></term>
+  <listitem><para> DRBD detected a split brain situation. Manual recovery is necessary. 
+      This handler should alert someone on duty.
+  <indexterm><primary>drbd.conf</primary><secondary>split-brain </secondary></indexterm>
+  </para></listitem> </varlistentry>
+
 </variablelist>
 </refsect2>
 </refsect1>
diff --git a/drbd/drbd_receiver.c b/drbd/drbd_receiver.c
index 6b5d180..e8aab71 100644
--- a/drbd/drbd_receiver.c
+++ b/drbd/drbd_receiver.c
@@ -2015,6 +2015,7 @@ STATIC drbd_conns_t drbd_sync_handshake(drbd_dev *mdev, drbd_role_t peer_role,
 		drbd_uuid_dump(mdev,"self",mdev->bc->md.uuid);
 		drbd_uuid_dump(mdev,"peer",mdev->p_uuid);
 		drbd_force_state(mdev,NS(conn,Disconnecting));
+		drbd_khelper(mdev,"split-brain");
 		return conn_mask;
 	}
 
diff --git a/scripts/drbd.conf b/scripts/drbd.conf
index a0f9d59..c5ecc54 100644
--- a/scripts/drbd.conf
+++ b/scripts/drbd.conf
@@ -148,6 +148,9 @@ resource r0 {
     # The node is currently primary, but should become sync target
     # after the negotiating phase. Alert someone about this incident.
     #pri-lost "echo pri-lost. Have a look at the log files. | mail -s 'DRBD Alert' root";
+
+    # Notify someone in case DRBD split brained. 
+    #split-brain "echo split-brain. drbdadm -- --discard-my-data connect $DRBD_RESOURCE ? | mail -s 'DRBD Alert' root";
   }
 
   startup {
diff --git a/user/drbdadm_main.c b/user/drbdadm_main.c
index 174ef5e..02cc8bf 100644
--- a/user/drbdadm_main.c
+++ b/user/drbdadm_main.c
@@ -239,6 +239,7 @@ struct adm_cmd cmds[] = {
   { "outdate-peer",      adm_khelper,   3,1,0 },
   { "local-io-error",    adm_khelper,   3,1,0 },
   { "pri-lost",          adm_khelper,   3,1,0 },
+  { "split-brain",       adm_khelper,   3,1,0 },
   { "set-gi",            admm_generic,  4,1,0 },
   { "suspend-io",        adm_generic_s, 4,1,0 },
   { "resume-io",         adm_generic_s, 4,1,0 },
diff --git a/user/drbdadm_scanner.fl b/user/drbdadm_scanner.fl
index d023446..7bec521 100644
--- a/user/drbdadm_scanner.fl
+++ b/user/drbdadm_scanner.fl
@@ -92,7 +92,8 @@ become-primary-on	{ DP; CP; return TK_STARTUP_OPTION;     }
 wait-after-sb		{ DP; CP; return TK_STARTUP_SWITCH;     }
 pri-on-incon-degr	{ DP; CP; return TK_HANDLER_OPTION;	}
 pri-lost-after-sb	{ DP; CP; return TK_HANDLER_OPTION;	}
-pri-lost	        { DP; CP; return TK_HANDLER_OPTION;     }
+pri-lost		{ DP; CP; return TK_HANDLER_OPTION;     }
+split-brain		{ DP; CP; return TK_HANDLER_OPTION;     }
 outdate-peer		{ DP; CP; return TK_HANDLER_OPTION;	}
 local-io-error		{ DP; CP; return TK_HANDLER_OPTION;     }
 {IPV4ADDR}		{ DP; CP; return TK_IPADDR;		}

RE: [Drbd-dev] drbdsetup events patch to announce Split-brain

[Montrose, Ernest]

Phil,
I think this will work for us actually. So feel free to check it in.

Thanks,

EM--
-----Original Message-----
From: Philipp Reisner [mailto:philipp.reisner@linbit.com] 
Sent: Monday, October 29, 2007 6:20 AM
To: Montrose, Ernest
Cc: drbd-dev@linbit.com
Subject: Re: [Drbd-dev] drbdsetup events patch to announce Split-brain

On Thursday 25 October 2007 14:24:55 Montrose, Ernest wrote:
> Phil,
> Currently we do not use the payload for anything.  I was just being
> paranoid thinking that we might need it in the future.  But you can
drop
> it if you wish.  If we have a pressing need for it in the future,
we'll
> deal with it then.
>

Ernest,

Over the weekend I realized that the split brain event is quite
similar to a number of events we already signal back to userspace.

The invocation of handlers is also visible through the "drbdsetup
events"
interface.

The attached patch creates an handler called "split-brain". If you
can live with that interface as well, I will commit that...

In the future I want to go this way:

  * Allow one to set the usermode_helper to the empty string. Then it 
    is still possible to get all events through the "drbdsetup events"
    interface.
  * The "outdate-peer" usermode_helper sends an exit code back
    to kernel space. This is not yet possible when one only uses
    the "drbdsetup events" interface. This should get fixed.

-Phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :