[Drbd-dev] DRBD8: Split-brain if primary and syncTarget

[Drbd-dev] DRBD8: Split-brain if primary and syncTarget

[Montrose, Ernest]

[-- Attachment #1: Type: text/plain, Size: 1387 bytes --]

Hi all,
 
We are seeing an issue with split brain if one node is syncing as
syncTarget while being Primary.
two node A and B.
* make B primary and the syncTarget
* Start a sync.
* ifdown eth1 to break communication
* ifup eth1.
* then on the node in standalone "drbdadm connect"
We get a split-brain.
 
I think the  problem is that if we are primary and we lose contact from
the other side
we generate a new current UUID which causes a Split-Brain next time we
connect.
This only happens if we are the sync target and we are primary. Perhaps
we 
should not generate a UUID if we were syncing when the disconnect
happen.
Below is a possible patch for this in after_state_ch():

Index: drbd/drbd_main.c
===================================================================
--- drbd/drbd_main.c    (revision 11592)
+++ drbd/drbd_main.c    (working copy)
@@ -901,7 +901,8 @@
                        mdev->p_uuid = NULL;
                }
                if (inc_local(mdev)) {
-                      if (ns.role == Primary &&
mdev->bc->md.uuid[Bitmap] == 0 ) {
+                      if (ns.role == Primary &&
mdev->bc->md.uuid[Bitmap] == 0 &&
+            os.conn < SyncSource ) {
                                /* Only do it if we have not yet done
it... */
                                drbd_uuid_new_current(mdev);
                        }
 
 

[-- Attachment #2: Type: text/html, Size: 7831 bytes --]

Re: [Drbd-dev] DRBD8: Split-brain if primary and syncTarget

[Goswin von Brederlow]

"Montrose, Ernest" <Ernest.Montrose@stratus.com> writes:

> :o="urn:schemas-microsoft-com:office:office"
> xmlns:w="urn:schemas-microsoft-com:office:word"
> xmlns="http://www.w3.org/TR/REC-html40">
>
> Hi all,:p>
>
> :p> 
>
> We are seeing an issue with split brain if one node is syncing as syncTarget
> while being Primary.:p>
>
> two node A and B.
> * make B primary and the syncTarget
> * Start a sync.
> * ifdown eth1 to break communication
> * ifup eth1.
> * then on the node in standalone "drbdadm connect":p>
>
> We get a split-brain.:p>
>
> :p> 
>
> I think the  problem is that if we are primary and we lose contact from the
> other side
> we generate a new current UUID which causes a Split-Brain next time we
> connect.
> This only happens if we are the sync target and we are primary. Perhaps we
> should not generate a UUID if we were syncing when the disconnect happen.
> Below is a possible patch for this in after_state_ch()::p>

I think the new UUID should be generated the first time there is a
write but not before.

MfG
        Goswin

Re: [Drbd-dev] DRBD8: Split-brain if primary and syncTarget

[Philipp Reisner]

Am Donnerstag, 8. März 2007 23:21 schrieb Montrose, Ernest:
> Hi all,
>
> We are seeing an issue with split brain if one node is syncing as
> syncTarget while being Primary.
> two node A and B.
> * make B primary and the syncTarget
> * Start a sync.
> * ifdown eth1 to break communication
> * ifup eth1.
> * then on the node in standalone "drbdadm connect"
> We get a split-brain.
>
> I think the  problem is that if we are primary and we lose contact from
> the other side we generate a new current UUID which causes a Split-Brain
> next time we connect.
> This only happens if we are the sync target and we are primary. Perhaps
> we should not generate a UUID if we were syncing when the disconnect
> happen. Below is a possible patch for this in after_state_ch():

Hi Ernest,

I think the current behaviour is correct.

* When a node is SyncTarget it actually exposes the data of the sync
  source node to its applications. (And the applications can potentially 
  see the data when the SyncTarget node is primary.)

* When you disconnect such a node, it has to fall back to its local
  data set. == suddenly the applications see a different data set,
  and of course the apps might continue to modify this data set...

* Wen you reconnect this, you have a split brain situation. But you 
  might let the automatic-split-brain resolving handler solve the
  situation. Use some after-sb-?pri settings, and an rr-conflict of
  "violently" E.g.:

  after-sb-0pri discard-least-changes
  after-sb-1pri violently-as0p
  after-sb-2pri violently-as0p
  rr-conflict   violently

  Then the resync should continue. Since the "violently" allows DRBD
  to change the data set again, that is seen on the Primary node.

-Phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :

RE: [Drbd-dev] DRBD8: Split-brain if primary and syncTarget

[Montrose, Ernest]

Phil,
Our config is close to what you suggested. But we have after_sb_0pri set to 
"discard_zero_changes". Hmmm...I have to test and think about this some more.

Thanks,

EM--
-----Original Message-----
From: Philipp Reisner [mailto:philipp.reisner@linbit.com] 
Sent: Monday, March 12, 2007 10:28 AM
To: drbd-dev@linbit.com
Cc: Montrose, Ernest
Subject: Re: [Drbd-dev] DRBD8: Split-brain if primary and syncTarget

Am Donnerstag, 8. März 2007 23:21 schrieb Montrose, Ernest:
> Hi all,
>
> We are seeing an issue with split brain if one node is syncing as
> syncTarget while being Primary.
> two node A and B.
> * make B primary and the syncTarget
> * Start a sync.
> * ifdown eth1 to break communication
> * ifup eth1.
> * then on the node in standalone "drbdadm connect"
> We get a split-brain.
>
> I think the  problem is that if we are primary and we lose contact from
> the other side we generate a new current UUID which causes a Split-Brain
> next time we connect.
> This only happens if we are the sync target and we are primary. Perhaps
> we should not generate a UUID if we were syncing when the disconnect
> happen. Below is a possible patch for this in after_state_ch():

Hi Ernest,

I think the current behaviour is correct.

* When a node is SyncTarget it actually exposes the data of the sync
  source node to its applications. (And the applications can potentially 
  see the data when the SyncTarget node is primary.)

* When you disconnect such a node, it has to fall back to its local
  data set. == suddenly the applications see a different data set,
  and of course the apps might continue to modify this data set...

* Wen you reconnect this, you have a split brain situation. But you 
  might let the automatic-split-brain resolving handler solve the
  situation. Use some after-sb-?pri settings, and an rr-conflict of
  "violently" E.g.:

  after-sb-0pri discard-least-changes
  after-sb-1pri violently-as0p
  after-sb-2pri violently-as0p
  rr-conflict   violently

  Then the resync should continue. Since the "violently" allows DRBD
  to change the data set again, that is seen on the Primary node.

-Phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :

Re: [Drbd-dev] DRBD8: Split-brain if primary and syncTarget

[Philipp Reisner]

[-- Attachment #1: Type: text/plain, Size: 2512 bytes --]

Am Montag, 12. März 2007 15:28 schrieb Philipp Reisner:
> Am Donnerstag, 8. März 2007 23:21 schrieb Montrose, Ernest:
> > Hi all,
> >
> > We are seeing an issue with split brain if one node is syncing as
> > syncTarget while being Primary.
> > two node A and B.
> > * make B primary and the syncTarget
> > * Start a sync.
> > * ifdown eth1 to break communication
> > * ifup eth1.
> > * then on the node in standalone "drbdadm connect"
> > We get a split-brain.
> >
> > I think the  problem is that if we are primary and we lose contact from
> > the other side we generate a new current UUID which causes a Split-Brain
> > next time we connect.
> > This only happens if we are the sync target and we are primary. Perhaps
> > we should not generate a UUID if we were syncing when the disconnect
> > happen. Below is a possible patch for this in after_state_ch():
>
> Hi Ernest,
>
> I think the current behaviour is correct.
>
> * When a node is SyncTarget it actually exposes the data of the sync
>   source node to its applications. (And the applications can potentially
>   see the data when the SyncTarget node is primary.)
>
> * When you disconnect such a node, it has to fall back to its local
>   data set. == suddenly the applications see a different data set,
>   and of course the apps might continue to modify this data set...
>
> * Wen you reconnect this, you have a split brain situation. But you
>   might let the automatic-split-brain resolving handler solve the
>   situation. Use some after-sb-?pri settings, and an rr-conflict of
>   "violently" E.g.:
>
>   after-sb-0pri discard-least-changes
>   after-sb-1pri violently-as0p
>   after-sb-2pri violently-as0p
>   rr-conflict   violently
>
>   Then the resync should continue. Since the "violently" allows DRBD
>   to change the data set again, that is seen on the Primary node.

Hmmm. I just had a look at the code in drbd_sync_handshake(), and came
to the conclusion that the handling of the inconsistent disk state was
a bit obscure.

With the attached patch the after-sb-?pri settings are of no impact
this such an situation any longer. Only the "rr-conflict" setting
should influence the outcome...

If it works for you with that patch, I will commit it...

-phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :

[-- Attachment #2: look_at_inconsistent_first.diff --]
[-- Type: text/x-diff, Size: 1599 bytes --]

Index: drbd_receiver.c
===================================================================
--- drbd_receiver.c	(revision 2786)
+++ drbd_receiver.c	(working copy)
@@ -1932,20 +1932,24 @@
 STATIC drbd_conns_t drbd_sync_handshake(drbd_dev *mdev, drbd_role_t peer_role,
 					drbd_disks_t peer_disk)
 {
-	int hg,rule_nr;
+	int hg,rule_nr=0;
 	drbd_conns_t rv = conn_mask;
 	drbd_disks_t mydisk;
 
 	mydisk = mdev->state.disk;
 	if( mydisk == Negotiating ) mydisk = mdev->new_state_tmp.disk;
 
-	hg = drbd_uuid_compare(mdev,&rule_nr);
+	// Look if a disk is inconsistent. Only if this does not find 
+	// a decission look at the UUIDs.
+	if(mydisk==Inconsistent && peer_disk>Inconsistent) hg=-1;
+	else if(mydisk>Inconsistent && peer_disk==Inconsistent) hg= 1;
+	else hg = drbd_uuid_compare(mdev,&rule_nr);
 
 	MTRACE(TraceTypeUuid,TraceLvlSummary,
 	       INFO("drbd_sync_handshake:\n");
 	       drbd_uuid_dump(mdev,"self",mdev->bc->md.uuid);
 	       drbd_uuid_dump(mdev,"peer",mdev->p_uuid);
-	       INFO("uuid_compare()=%d by rule %d\n",hg,rule_nr);
+	       INFO("have_good=%d by rule %d\n",hg,rule_nr);
 	    );
 
 	if (hg == 100 || (hg == -100 && mdev->net_conf->always_asbp) ) {
@@ -1990,13 +1994,6 @@
 		}
 	}
 
-	if (abs(hg) < 100) {
-		// This is needed in case someone does an invalidate on an
-		// disconnected node. This has priority.
-		if(mydisk==Inconsistent && peer_disk>Inconsistent) hg=-1;
-		if(mydisk>Inconsistent && peer_disk==Inconsistent) hg= 1;
-	}
-
 	if (hg == -1000) {
 		ALERT("Unrelated data, dropping connection!\n");
 		drbd_force_state(mdev,NS(conn,Disconnecting));

RE: [Drbd-dev] DRBD8: Split-brain if primary and syncTarget

[Montrose, Ernest]

Phil,
That patch worked just fine.  You can commit it!

Thanks

EM--

-----Original Message-----
From: Philipp Reisner [mailto:philipp.reisner@linbit.com] 
Sent: Monday, March 12, 2007 10:52 AM
To: drbd-dev@linbit.com
Cc: Montrose, Ernest
Subject: Re: [Drbd-dev] DRBD8: Split-brain if primary and syncTarget

Am Montag, 12. März 2007 15:28 schrieb Philipp Reisner:
> Am Donnerstag, 8. März 2007 23:21 schrieb Montrose, Ernest:
> > Hi all,
> >
> > We are seeing an issue with split brain if one node is syncing as
> > syncTarget while being Primary.
> > two node A and B.
> > * make B primary and the syncTarget
> > * Start a sync.
> > * ifdown eth1 to break communication
> > * ifup eth1.
> > * then on the node in standalone "drbdadm connect"
> > We get a split-brain.
> >
> > I think the  problem is that if we are primary and we lose contact from
> > the other side we generate a new current UUID which causes a Split-Brain
> > next time we connect.
> > This only happens if we are the sync target and we are primary. Perhaps
> > we should not generate a UUID if we were syncing when the disconnect
> > happen. Below is a possible patch for this in after_state_ch():
>
> Hi Ernest,
>
> I think the current behaviour is correct.
>
> * When a node is SyncTarget it actually exposes the data of the sync
>   source node to its applications. (And the applications can potentially
>   see the data when the SyncTarget node is primary.)
>
> * When you disconnect such a node, it has to fall back to its local
>   data set. == suddenly the applications see a different data set,
>   and of course the apps might continue to modify this data set...
>
> * Wen you reconnect this, you have a split brain situation. But you
>   might let the automatic-split-brain resolving handler solve the
>   situation. Use some after-sb-?pri settings, and an rr-conflict of
>   "violently" E.g.:
>
>   after-sb-0pri discard-least-changes
>   after-sb-1pri violently-as0p
>   after-sb-2pri violently-as0p
>   rr-conflict   violently
>
>   Then the resync should continue. Since the "violently" allows DRBD
>   to change the data set again, that is seen on the Primary node.

Hmmm. I just had a look at the code in drbd_sync_handshake(), and came
to the conclusion that the handling of the inconsistent disk state was
a bit obscure.

With the attached patch the after-sb-?pri settings are of no impact
this such an situation any longer. Only the "rr-conflict" setting
should influence the outcome...

If it works for you with that patch, I will commit it...

-phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :

Re: [Drbd-dev] DRBD8: Split-brain if primary and syncTarget

[Philipp Reisner]

Hi Ernest,

Suddenty I realized that with the last change in the drbd_sync_handshake()
code we could resync unrelated data-sets. I changed it once more there.

I am pretty convinced that it will continue to work for your issue, 
I just wanted to let you know, so that you can re-test it it you 
have the time...

-Phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :

RE: [Drbd-dev] DRBD8: Split-brain if primary and syncTarget

[Montrose, Ernest]

Phil,
I will retest and let you know if any issues arise..

Thanks!

EM--

-----Original Message-----
From: Philipp Reisner [mailto:philipp.reisner@linbit.com] 
Sent: Tuesday, March 13, 2007 6:23 AM
To: drbd-dev@linbit.com
Cc: Montrose, Ernest
Subject: Re: [Drbd-dev] DRBD8: Split-brain if primary and syncTarget

Hi Ernest,

Suddenty I realized that with the last change in the
drbd_sync_handshake()
code we could resync unrelated data-sets. I changed it once more there.

I am pretty convinced that it will continue to work for your issue, 
I just wanted to let you know, so that you can re-test it it you 
have the time...

-Phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :