Re: [Drbd-dev] [CASE 12] 2 primary split-brain is not detected

[Philipp Reisner <philipp.reisner@linbit.com>]

[-- Attachment #1: Type: text/plain, Size: 4065 bytes --]

Hi,

thanks for the detailed description.
Please find attached a proposed fix. I want to think about it for one more
day before I push it onto the master branch.

best regards,
  phil

> Hello,
> 
> thank you for your reply. appreciated that
> 
> I tested again with current GIT-head version(Commit: [9c711b4]), reproduce
> this issue, and same result is happened.
> 
> more detail descriptions are below :
> 
> 1) Env:
> 
> - CentOS 7 64bit
> - i7 4790K CPU 4Ghz , memory :1G
> - 2 node VM
> 
> 2) drbd version (self build)
> 
> [root@drbd9-02 ]# ls -l
> /lib/modules/3.10.0-229.7.2.el7.x86_64/updates/drbd.ko
> -rw-r--r-- 1 root root 10657876 * Jan 21* 11:01
> /lib/modules/3.10.0-229.7.2.el7.x86_64/updates/drbd.ko
> 
> 3) scenario:
> - basically, progressed to 2-primary split-brain scenario.
> - first, 2 node status is secondary-secondary uptodate
> node 1 node 2
> drbdadm disconnect r0
> drbdadm primary r0
> copy files on node1 volume
>   drbdadm primary r0
>   copy files on node2 volume
> drbdadm connect r0
> 
> 4) result
> 
> - 2 primary Split-brain did not occurred.
> - according to drbd kernel log, I guess that connecting phase is failed and
> retry to connect between 2 nodes, repeatedly.
> 
> 
> you will see my drbd configuration and kernel log files, attached.
> 
> I hope that help for you.
> 
> Best Regards
> Sung-eun kim
> from Seoul
> 
> 
> [image: 설명: logo]
> 
> 기술연구소 / DRBD팀        Mobile : 010-2813-4843
> 
> *김 성 은 **차장               sekim@mantech.co.kr <sekim@mantech.co.kr>*
> 
> 서울특별시 성동구 성수일로4길 25 서울숲코오롱디지털타워 12층
> 
> Tel : 02-2136-6913 / Fax : 02-575-4858 / 콜센터 : 1833-7790
> 
> http://www.mantech.co.kr
> 
> [image: 본문 이미지 1]
> 
> 2016-01-20 18:26 GMT+09:00 Philipp Reisner <philipp.reisner@linbit.com>:
> > Hi,
> > 
> > I am following up on CASE 12 now.
> > 
> > I was not able to reproduce what you describe in this report.
> > Please check that you can still reproduce it with current GIT-head.
> > 
> > If yes, please re-send it with a more detailed description, including:
> >  Configuration used.
> >  Logfiles from begin to end of the test.
> > 
> > Thanks!
> > 
> > Best regards,
> > 
> >  Phil
> >  
> > > Hello,
> > > 
> > > I am Software Manager in Mantech, and testing drbd 9.
> > > 
> > > some problems are occurred. CASE 12 is one of them.
> > > 
> > > [CASE 12] 2 primary split-brain is not detected.
> > > 
> > > Env:
> > > - Linux VM 2 nodes
> > > - Linux drbd version 9.0.0
> > > 
> > > Test scenario:
> > > 
> > > - 2 node's status is uptodate
> > > 
> > > node 1node 2drbdadm disconnect r0 drbdadm primary r0 copy files on
> > > node1   drbdadm
> > > primary r0 copy files on node2
> > > drbdadm connect r0
> > > 
> > > - at this point, we predicted that 2 primary split-brain is occurred and
> > > drbd9 detect it.
> > > 
> > > But split-brain is not occurred, not detected. no debug log.
> > > Just retry connection log is showing between two nodes.
> > > 
> > > Question is :
> > > 
> > > 1) How can I test 2 primary split-brain situation ?
> > > 
> > > 2) Does support 2 primary split-brain in drbd9 ?
> > > 
> > > 
> > > PS)
> > 
> > http://drbd.linbit.com/users-guide-9.0/s-configure-split-brain-behavior.ht
> > ml> 
> > > #s-split-brain-notification
> > > 
> > > drbd9's manual describe 'after-sb-2pri' option.
> > > 
> > > Thanks,
> > > 
> > > 
> > > Best Regards
> > > Sungeun Kim
> > > 
> > > 
> > > 
> > > 
> > > [image: 설명: logo]
> > > 
> > > 기술연구소 / DRBD팀        Mobile : 010-2813-4843
> > > 
> > > *김 성 은 **차장               sekim@mantech.co.kr <sekim@mantech.co.kr>*
> > > 
> > > 서울특별시 성동구 성수일로4길 25 서울숲코오롱디지털타워 12층
> > > 
> > > Tel : 02-2136-6913 / Fax : 02-575-4858 / 콜센터 : 1833-7790
> > > 
> > > http://www.mantech.co.kr
> > > 
> > > [image: 본문 이미지 1]

[-- Attachment #2: case12_fix.patch --]
[-- Type: text/x-patch, Size: 1854 bytes --]

commit b775310ecb7549122d6e39b88c84b48ea65a5dc5
Author: Philipp Reisner <philipp.reisner@linbit.com>
Date:   Sat Jan 23 23:49:13 2016 +0100

    drbd: Do not outdate myself if I am primary

diff --git a/drbd/drbd_state.c b/drbd/drbd_state.c
index ee63141..8a3ca9f 100644
--- a/drbd/drbd_state.c
+++ b/drbd/drbd_state.c
@@ -3574,6 +3574,9 @@ void __outdate_myself(struct drbd_resource *resource)
 	struct drbd_device *device;
 	int vnr;
 
+	if (resource->role[NOW] == R_PRIMARY)
+		return;
+
 	idr_for_each_entry(&resource->devices, device, vnr) {
 		if (device->disk_state[NOW] > D_OUTDATED)
 			__change_disk_state(device, D_OUTDATED);

commit c6963c44c48c99e7248265ee4318914e69b4a6fd
Author: Philipp Reisner <philipp.reisner@linbit.com>
Date:   Sat Jan 23 22:56:06 2016 +0100

    drbd: When connect fails because too many primaries assume split-brain
    
    If there are more primaries than allowed, and connect fails therefore
    we do not get a chance to look at the UUIDs.
    In order to match expectations (and drbd-8 behaviour) call the
    split-brain handler in this case as well.

diff --git a/drbd/drbd_receiver.c b/drbd/drbd_receiver.c
index 904d444..bd59827 100644
--- a/drbd/drbd_receiver.c
+++ b/drbd/drbd_receiver.c
@@ -667,6 +667,10 @@ int connect_work(struct drbd_work *work, int cancel)
 		connection->connect_timer.expires = jiffies + HZ/20;
 		add_timer(&connection->connect_timer);
 		return 0; /* Return early. Keep the reference on the connection! */
+	} else if (rv == SS_TWO_PRIMARIES) {
+		change_cstate(connection, C_DISCONNECTING, CS_HARD);
+		drbd_alert(connection, "Split-Brain since more primaries than allowed; dropping connection!\n");
+		drbd_khelper(NULL, connection, "split-brain");
 	} else {
 		drbd_info(connection, "Failure to connect; retrying\n");
 		change_cstate(connection, C_NETWORK_FAILURE, CS_HARD);