[Drbd-dev] Resync Stalls at 100% patch problem

[Drbd-dev] Resync Stalls at 100% patch problem

[Montrose, Ernest]

Phil,
Unfortunately it seems like the patch for receive_state() in
drbd_receive.c has a problem.  Acquiring the req_lock at the top of this
routine causes the cstate machine to get confused.  To reproduce this
just do:
Drbdadm detach d0;sleep 3;drbdadm attach d0 for instance. One side ends
up WBTmpT and the other WbitmapS and deadlock.  I am researching more
but wanted to let you.  If I just remove the early lock then things are
OK..:(

EM--

RE: [Drbd-dev] Resync Stalls at 100% patch problem

[Montrose, Ernest]

BTW, I forgot to say that this happens only if one of the nodes is st:
Primary/Secondary.

EM--

-----Original Message-----
From: drbd-dev-bounces@linbit.com [mailto:drbd-dev-bounces@linbit.com]
On Behalf Of Montrose, Ernest
Sent: Tuesday, June 05, 2007 6:50 PM
To: Philipp Reisner; drbd-dev@linbit.com
Subject: [Drbd-dev] Resync Stalls at 100% patch problem

Phil,
Unfortunately it seems like the patch for receive_state() in
drbd_receive.c has a problem.  Acquiring the req_lock at the top of this
routine causes the cstate machine to get confused.  To reproduce this
just do:
Drbdadm detach d0;sleep 3;drbdadm attach d0 for instance. One side ends
up WBTmpT and the other WbitmapS and deadlock.  I am researching more
but wanted to let you.  If I just remove the early lock then things are
OK..:(

EM--
_______________________________________________
drbd-dev mailing list
drbd-dev@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-dev

RE: [Drbd-dev] Resync Stalls at 100% patch problem

[Montrose, Ernest]

[-- Attachment #1: Type: text/plain, Size: 1700 bytes --]

Phil,
I looked into the patch issue a bit more. 
This problem happens if we acquire the req_lock early.  What I think is
going on is this:

Primary host ---->attach---> send_state()
Peer -----> receive_state()-->after_state_ch()-->send_bitmap()
Primary Host ---->receive_bitmap()***Unexpected cstate "Connected"
Peer ----> send_state()
Primary Host --->receive_state() and we are deadlock.

The primary host is receiving the bitmap too early.  Essentially the
peer should call send_state() before calling after_state_ch()

I noticed the patch had moved that logic after calling after_state_ch()
so moving it back before after_state_ch() should be OK.  Any reason why
it was moved down?

I tested the included patch that moved just before we call
after_state_ch().
Let me know.

Thanks.

EM--


-----Original Message-----
From: drbd-dev-bounces@linbit.com [mailto:drbd-dev-bounces@linbit.com]
On Behalf Of Montrose, Ernest
Sent: Tuesday, June 05, 2007 6:50 PM
To: Philipp Reisner; drbd-dev@linbit.com
Subject: [Drbd-dev] Resync Stalls at 100% patch problem

Phil,
Unfortunately it seems like the patch for receive_state() in
drbd_receive.c has a problem.  Acquiring the req_lock at the top of this
routine causes the cstate machine to get confused.  To reproduce this
just do:
Drbdadm detach d0;sleep 3;drbdadm attach d0 for instance. One side ends
up WBTmpT and the other WbitmapS and deadlock.  I am researching more
but wanted to let you.  If I just remove the early lock then things are
OK..:(

EM--
_______________________________________________
drbd-dev mailing list
drbd-dev@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-dev


[-- Attachment #2: sync_100.patch --]
[-- Type: application/octet-stream, Size: 1455 bytes --]

Index: drbd_receiver.c
===================================================================
--- drbd_receiver.c	(revision 14923)
+++ drbd_receiver.c	(working copy)
@@ -2431,7 +2431,18 @@
 	rv = _drbd_set_state(mdev,ns,ChgStateVerbose | ChgStateHard);
 	spin_unlock_irq(&mdev->req_lock);
 	if (rv==SS_Success) {
-		after_state_ch(mdev,os,ns,ChgStateVerbose | ChgStateHard);
+          if (oconn > WFReportParams ) {
+             if( nconn > Connected && peer_state.conn <= Connected) {
+                // we want resync, peer has not yet decided to sync...
+                drbd_send_uuids(mdev);
+                drbd_send_state(mdev);
+             }
+             else if (nconn == Connected && peer_state.disk == Negotiating) {
+                // peer is waiting for us to respond...
+                drbd_send_state(mdev);
+             }
+          }
+          after_state_ch(mdev,os,ns,ChgStateVerbose | ChgStateHard);
 	}
 
 	if(rv < SS_Success) {
@@ -2439,18 +2450,6 @@
 		return FALSE;
 	}
 
-	if (oconn > WFReportParams ) {
-		if( nconn > Connected && peer_state.conn <= Connected) {
-			// we want resync, peer has not yet decided to sync...
-			drbd_send_uuids(mdev);
-			drbd_send_state(mdev);
-		}
-		else if (nconn == Connected && peer_state.disk == Negotiating) {
-			// peer is waiting for us to respond...
-			drbd_send_state(mdev);
-		}
-	}
-
 	mdev->net_conf->want_lose = 0;
 
 	/* FIXME assertion for (gencounts do not diverge) */

Re: [Drbd-dev] Resync Stalls at 100% patch problem

[Philipp Reisner]

On Thursday 07 June 2007 00:14:13 Montrose, Ernest wrote:
> Phil,
> I looked into the patch issue a bit more.
> This problem happens if we acquire the req_lock early.  What I think is
> going on is this:
>
> Primary host ---->attach---> send_state()
> Peer -----> receive_state()-->after_state_ch()-->send_bitmap()
> Primary Host ---->receive_bitmap()***Unexpected cstate "Connected"
> Peer ----> send_state()
> Primary Host --->receive_state() and we are deadlock.
>
> The primary host is receiving the bitmap too early.  Essentially the
> peer should call send_state() before calling after_state_ch()
>
> I noticed the patch had moved that logic after calling after_state_ch()
> so moving it back before after_state_ch() should be OK.  Any reason why
> it was moved down?
>
> I tested the included patch that moved just before we call
> after_state_ch().
> Let me know.
>

Hi Ernest,

Sorry for the slow responses recently. We had a public holiday here,
and I was out in the mountains...

I have now committed a patch very similar to your suggestion.

-phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :

RE: [Drbd-dev] Resync Stalls at 100% patch problem

[Montrose, Ernest]

Phil,
Ah! Vacation.. I will test it out and let you know if I find any issues.

Thanks!

EM--

-----Original Message-----
From: Philipp Reisner [mailto:philipp.reisner@linbit.com] 
Sent: Friday, June 08, 2007 5:51 AM
To: drbd-dev@linbit.com
Cc: Montrose, Ernest
Subject: Re: [Drbd-dev] Resync Stalls at 100% patch problem

On Thursday 07 June 2007 00:14:13 Montrose, Ernest wrote:
> Phil,
> I looked into the patch issue a bit more.
> This problem happens if we acquire the req_lock early.  What I think
is
> going on is this:
>
> Primary host ---->attach---> send_state()
> Peer -----> receive_state()-->after_state_ch()-->send_bitmap()
> Primary Host ---->receive_bitmap()***Unexpected cstate "Connected"
> Peer ----> send_state()
> Primary Host --->receive_state() and we are deadlock.
>
> The primary host is receiving the bitmap too early.  Essentially the
> peer should call send_state() before calling after_state_ch()
>
> I noticed the patch had moved that logic after calling
after_state_ch()
> so moving it back before after_state_ch() should be OK.  Any reason
why
> it was moved down?
>
> I tested the included patch that moved just before we call
> after_state_ch().
> Let me know.
>

Hi Ernest,

Sorry for the slow responses recently. We had a public holiday here,
and I was out in the mountains...

I have now committed a patch very similar to your suggestion.

-phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :