[Drbd-dev] DRBD8: Receive_state() won't dec_local after a disk failure on peer.

[Drbd-dev] DRBD8: Receive_state() won't dec_local after a disk failure on peer.

[Montrose, Ernest]

[-- Attachment #1: Type: text/plain, Size: 751 bytes --]


Hi all,
We have been seeing a problem where a cluster of two systems, X and Y.
X is Primary and gets a disk fault.  X goes Diskless.
Y now is forced to be Primary. 
X recovers from the fault. 
But now Y gets a disk fault and goes Diskless but Stay Primary.
At this point I/O from r0 hangs on Y!  

A check on /proc/<ip>/wchan for the worker thread reveals that we are
waiting forever for local_cnt to become 0 in after_state_ch(). So the
worker thread will process the Net_read.  What happened is that after
the first failure on X, receive_state() on Y failed to call dec_local().
The pdisk received state is Diskless therefore we won't dec_local(). The
included patch illustrates the problem and attempts to fix it. 

Thanks.
EM--

[-- Attachment #2: drbd_recv.patch --]
[-- Type: application/octet-stream, Size: 710 bytes --]

Index: drbd/drbd_receiver.c
===================================================================
--- drbd/drbd_receiver.c	(revision 15901)
+++ drbd/drbd_receiver.c	(working copy)
@@ -2408,11 +2408,12 @@
 	if (nconn == WFReportParams ) nconn = Connected;
 
 	if (mdev->p_uuid && oconn <= Connected &&
-	    inc_local_if_state(mdev,Negotiating) &&
 	    peer_state.disk >= Negotiating) {
-		nconn=drbd_sync_handshake(mdev,peer_state.role,peer_state.disk);
-		dec_local(mdev);
-
+          if(inc_local_if_state(mdev,Negotiating)) {
+             nconn=drbd_sync_handshake(mdev,peer_state.role,peer_state.disk);
+             dec_local(mdev);
+          }
+          
 		if(nconn == conn_mask) return FALSE;
 	}
 

Re: [Drbd-dev] DRBD8: Receive_state() won't dec_local after a disk failure on peer.

[Philipp Reisner]

On Friday 29 June 2007 23:40:45 Montrose, Ernest wrote:
> Hi all,
> We have been seeing a problem where a cluster of two systems, X and Y.
> X is Primary and gets a disk fault.  X goes Diskless.
> Y now is forced to be Primary.
> X recovers from the fault.
> But now Y gets a disk fault and goes Diskless but Stay Primary.
> At this point I/O from r0 hangs on Y!
>
> A check on /proc/<ip>/wchan for the worker thread reveals that we are
> waiting forever for local_cnt to become 0 in after_state_ch(). So the
> worker thread will process the Net_read.  What happened is that after
> the first failure on X, receive_state() on Y failed to call dec_local().
> The pdisk received state is Diskless therefore we won't dec_local(). The
> included patch illustrates the problem and attempts to fix it.
>
> Thanks.
> EM--


Right.

I changed the patch to:

--- branches/drbd-8.0/drbd/drbd_receiver.c      2007-07-02 08:44:22 UTC (rev 2962)
+++ branches/drbd-8.0/drbd/drbd_receiver.c      2007-07-02 09:54:58 UTC (rev 2963)
@@ -2408,8 +2408,8 @@
        if (nconn == WFReportParams ) nconn = Connected;
 
        if (mdev->p_uuid && oconn <= Connected &&
-           inc_local_if_state(mdev,Negotiating) &&
-           peer_state.disk >= Negotiating) {
+           peer_state.disk >= Negotiating &&
+           inc_local_if_state(mdev,Negotiating) ) {
                nconn=drbd_sync_handshake(mdev,peer_state.role,peer_state.disk);
                dec_local(mdev);
 
since C guarantees us the evaluate the right argument of && only if
the left argument is true.

-Phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :

RE: [Drbd-dev] DRBD8: Receive_state() won't dec_local after a disk failure on peer.

[Montrose, Ernest]

Phil,
Though this approach is really cute, it is error prone though.
Actually, the next guy that adds to the conditional might break us
again.  Maybe add a comment to say that the inc_local() must be last.
Hopefully the next guy will read it :)

Thanks!
EM--
-----Original Message-----
From: Philipp Reisner [mailto:philipp.reisner@linbit.com] 
Sent: Monday, July 02, 2007 5:58 AM
To: drbd-dev@linbit.com
Cc: Montrose, Ernest
Subject: Re: [Drbd-dev] DRBD8: Receive_state() won't dec_local after a
disk failure on peer.

On Friday 29 June 2007 23:40:45 Montrose, Ernest wrote:
> Hi all,
> We have been seeing a problem where a cluster of two systems, X and Y.
> X is Primary and gets a disk fault.  X goes Diskless.
> Y now is forced to be Primary.
> X recovers from the fault.
> But now Y gets a disk fault and goes Diskless but Stay Primary.
> At this point I/O from r0 hangs on Y!
>
> A check on /proc/<ip>/wchan for the worker thread reveals that we are
> waiting forever for local_cnt to become 0 in after_state_ch(). So the
> worker thread will process the Net_read.  What happened is that after
> the first failure on X, receive_state() on Y failed to call
dec_local().
> The pdisk received state is Diskless therefore we won't dec_local().
The
> included patch illustrates the problem and attempts to fix it.
>
> Thanks.
> EM--


Right.

I changed the patch to:

--- branches/drbd-8.0/drbd/drbd_receiver.c      2007-07-02 08:44:22 UTC
(rev 2962)
+++ branches/drbd-8.0/drbd/drbd_receiver.c      2007-07-02 09:54:58 UTC
(rev 2963)
@@ -2408,8 +2408,8 @@
        if (nconn == WFReportParams ) nconn = Connected;
 
        if (mdev->p_uuid && oconn <= Connected &&
-           inc_local_if_state(mdev,Negotiating) &&
-           peer_state.disk >= Negotiating) {
+           peer_state.disk >= Negotiating &&
+           inc_local_if_state(mdev,Negotiating) ) {
 
nconn=drbd_sync_handshake(mdev,peer_state.role,peer_state.disk);
                dec_local(mdev);
 
since C guarantees us the evaluate the right argument of && only if
the left argument is true.

-Phil
-- 
: Dipl-Ing Philipp Reisner                      Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH          Fax +43-1-8178292-82 :
: Vivenotgasse 48, 1120 Vienna, Austria        http://www.linbit.com :