[Drbd-dev] DRBD8: An attach after a disk fault on the sync target causes a deadlock in WFBitMapT

[Drbd-dev] DRBD8: An attach after a disk fault on the sync target causes a deadlock in WFBitMapT

[Montrose, Ernest]

[-- Attachment #1: Type: text/plain, Size: 530 bytes --]


Hi all,
We are seeing a problem where while syncing if the sync target detaches
then attempts to attach, both nodes end up in cs:WABitMapT state.  This
happens if both nodes where in Secondary/Secondary state and the dettach
happened on the SyncTarget.

I think what's happening is that the syncsource is losing the fact that
the synctarget is out of date.  After_state_ch() clears that fact out. I
am not  sure that the included patch is the proper fix for this but it
will illustrates the issue. 

Thanks,

EM--

[-- Attachment #2: wbitmat.patch --]
[-- Type: application/octet-stream, Size: 1207 bytes --]

Index: drbd/drbd_receiver.c
===================================================================
--- drbd/drbd_receiver.c	(revision 15995)
+++ drbd/drbd_receiver.c	(working copy)
@@ -1956,7 +1956,8 @@
 	}
 
 	if( (mydisk==Inconsistent && peer_disk>Inconsistent) ||
-	    (peer_disk==Inconsistent && mydisk>Inconsistent) )  {
+	    (peer_disk==Inconsistent && mydisk>Inconsistent) ||
+           (drbd_md_test_flag(mdev->bc,MDF_PeerOutDated) && mydisk>Inconsistent))  {
 		int f = (hg == -100) || abs(hg) == 2;
 		hg = mydisk > Inconsistent ? 1 : -1;
 		if(f) hg=hg*2;
Index: drbd/drbd_main.c
===================================================================
--- drbd/drbd_main.c	(revision 15995)
+++ drbd/drbd_main.c	(working copy)
@@ -863,7 +863,8 @@
 		if (mdev->state.disk > Inconsistent)   mdf |= MDF_Consistent;
 		if (mdev->state.disk > Outdated)       mdf |= MDF_WasUpToDate;
 		if (mdev->state.pdsk <= Outdated &&
-		    mdev->state.pdsk >= Inconsistent)  mdf |= MDF_PeerOutDated;
+		    (mdev->state.pdsk >= Inconsistent ||
+                   mdev->state.pdsk == Diskless))  mdf |= MDF_PeerOutDated;
 		if( mdf != mdev->bc->md.flags) {
 			mdev->bc->md.flags = mdf;
 			drbd_md_mark_dirty(mdev);

Re: [Drbd-dev] DRBD8: An attach after a disk fault on the sync target causes a deadlock in WFBitMapT

[Philipp Reisner]

On Monday 09 July 2007 17:25:08 Montrose, Ernest wrote:
> Hi all,
> We are seeing a problem where while syncing if the sync target detaches
> then attempts to attach, both nodes end up in cs:WABitMapT state.  This
> happens if both nodes where in Secondary/Secondary state and the dettach
> happened on the SyncTarget.
>

Hi Ernest,

Thanks for pointing this out.

The cause for this was that the UUIDs...

Node1:       C:0:H:H
Node2:       C:0:H:H

upon start of resync an "S" gets created.

sync target: S:0:C:H
sync source: C:S:H:H

but it should be

sync target: S:0:H:H
sync source: C:S:H:H

On the sync target side we got S left of C while on the sync 
source side we had C left of S. This was bad. The should not
appear in different order!

C should be left of S on sync source, and nowhere on the sync
target. (= even further left = in the future )

Ok, I guess the explanation is sufficient incomprehensible ;)


Index: drbd_receiver.c
===================================================================
--- drbd_receiver.c     (revision 2965)
+++ drbd_receiver.c     (revision 2966)
@@ -2474,7 +2474,9 @@
        if (drbd_recv(mdev, h->payload, h->length) != h->length)
                return FALSE;

-       drbd_uuid_set(mdev,Current,be64_to_cpu(p->uuid));
+       /* Here the _drbd_uuid_ functions are right, current should
+          _not_ be rotated into the history */
+       _drbd_uuid_set(mdev,Current,be64_to_cpu(p->uuid));
        _drbd_uuid_set(mdev,Bitmap,0UL);

        drbd_start_resync(mdev,SyncTarget);

-Phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :

RE: [Drbd-dev] DRBD8: An attach after a disk fault on the sync target causes a deadlock in WFBitMapT

[Montrose, Ernest]

Phil,
OK..Interesting..I'll have to fully understand the explanation later :)
Let me test the fix first. I'll let you know if anything bad happens.  

Thanks!

EM--

-----Original Message-----
From: Philipp Reisner [mailto:philipp.reisner@linbit.com] 
Sent: Tuesday, July 10, 2007 9:48 AM
To: drbd-dev@linbit.com
Cc: Montrose, Ernest
Subject: Re: [Drbd-dev] DRBD8: An attach after a disk fault on the sync
target causes a deadlock in WFBitMapT

On Monday 09 July 2007 17:25:08 Montrose, Ernest wrote:
> Hi all,
> We are seeing a problem where while syncing if the sync target
detaches
> then attempts to attach, both nodes end up in cs:WABitMapT state.
This
> happens if both nodes where in Secondary/Secondary state and the
dettach
> happened on the SyncTarget.
>

Hi Ernest,

Thanks for pointing this out.

The cause for this was that the UUIDs...

Node1:       C:0:H:H
Node2:       C:0:H:H

upon start of resync an "S" gets created.

sync target: S:0:C:H
sync source: C:S:H:H

but it should be

sync target: S:0:H:H
sync source: C:S:H:H

On the sync target side we got S left of C while on the sync 
source side we had C left of S. This was bad. The should not
appear in different order!

C should be left of S on sync source, and nowhere on the sync
target. (= even further left = in the future )

Ok, I guess the explanation is sufficient incomprehensible ;)


Index: drbd_receiver.c
===================================================================
--- drbd_receiver.c     (revision 2965)
+++ drbd_receiver.c     (revision 2966)
@@ -2474,7 +2474,9 @@
        if (drbd_recv(mdev, h->payload, h->length) != h->length)
                return FALSE;

-       drbd_uuid_set(mdev,Current,be64_to_cpu(p->uuid));
+       /* Here the _drbd_uuid_ functions are right, current should
+          _not_ be rotated into the history */
+       _drbd_uuid_set(mdev,Current,be64_to_cpu(p->uuid));
        _drbd_uuid_set(mdev,Bitmap,0UL);

        drbd_start_resync(mdev,SyncTarget);

-Phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :