[Drbd-dev] [PATCH 0/3] Implement {begin|after}-resync-source handlers

[Drbd-dev] [PATCH 0/3] Implement {begin|after}-resync-source handlers

[Iustin Pop]

These small patches duplicate the resync-target handlers for the other
end of the connection.

Some notes:

- I'm not familiar with drbdadm itself, so I'm not sure if the changes
  to it are sane; I did check and test that the kernel changes are fine
  and fire the proper callbacks, but was not able to test the drbdadm
  configuration
- Aborting if the userspace helper exits with non-zero code is done to
  mirror the before-resync-target helper; but ignoring its result is
  another choice, if tearing down the connections is deemed to harsh

Iustin Pop (3):
  Implement before|after-resync-source in drbdadm
  Make the drbd driver generate resync-source callbacks
  Update documentation for resync-source handlers

 documentation/drbd.conf.xml |   27 ++++++++++++++++++++++++++-
 drbd/drbd_worker.c          |   14 ++++++++++++++
 scripts/drbd.conf.example   |    4 ++++
 scripts/global_common.conf  |    2 ++
 user/drbdadm_main.c         |    2 ++
 user/drbdadm_scanner.fl     |    2 ++
 6 files changed, 50 insertions(+), 1 deletions(-)

[Drbd-dev] [PATCH 1/3] Implement before|after-resync-source in drbdadm

[Iustin Pop]

---
 scripts/global_common.conf |    2 ++
 user/drbdadm_main.c        |    2 ++
 user/drbdadm_scanner.fl    |    2 ++
 3 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/scripts/global_common.conf b/scripts/global_common.conf
index 5269515..ad07ce2 100644
--- a/scripts/global_common.conf
+++ b/scripts/global_common.conf
@@ -15,6 +15,8 @@ common {
 		# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
 		# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
 		# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
+		# before-resync-source /bin/true;
+		# after-resync-target /bin/true;
 	}
 
 	startup {
diff --git a/user/drbdadm_main.c b/user/drbdadm_main.c
index b697fd6..38d427a 100644
--- a/user/drbdadm_main.c
+++ b/user/drbdadm_main.c
@@ -327,6 +327,8 @@ struct adm_cmd cmds[] = {
 
 	{"before-resync-target", adm_khelper, DRBD_acf3_handler},
 	{"after-resync-target", adm_khelper, DRBD_acf3_handler},
+	{"before-resync-source", adm_khelper, DRBD_acf3_handler},
+	{"after-resync-source", adm_khelper, DRBD_acf3_handler},
 	{"pri-on-incon-degr", adm_khelper, DRBD_acf3_handler},
 	{"pri-lost-after-sb", adm_khelper, DRBD_acf3_handler},
 	{"fence-peer", adm_khelper, DRBD_acf3_handler},
diff --git a/user/drbdadm_scanner.fl b/user/drbdadm_scanner.fl
index d4cad32..a4a9441 100644
--- a/user/drbdadm_scanner.fl
+++ b/user/drbdadm_scanner.fl
@@ -141,6 +141,8 @@ fence-peer		{ DP; CP; return TK_HANDLER_OPTION;	}
 local-io-error		{ DP; CP; return TK_HANDLER_OPTION;     }
 before-resync-target	{ DP; CP; return TK_HANDLER_OPTION;	}
 after-resync-target	{ DP; CP; return TK_HANDLER_OPTION;	}
+before-resync-source	{ DP; CP; return TK_HANDLER_OPTION;	}
+after-resync-source	{ DP; CP; return TK_HANDLER_OPTION;	}
 memlimit		{ DP; CP; return TK_PROXY_OPTION;       }
 read-loops		{ DP; CP; return TK_PROXY_OPTION;       }
 compression		{ DP; CP; return TK_PROXY_OPTION;       }
-- 
1.7.0.1

[Drbd-dev] [PATCH 2/3] Make the drbd driver generate resync-source callbacks

[Iustin Pop]

---
 drbd/drbd_worker.c |   14 ++++++++++++++
 1 files changed, 14 insertions(+), 0 deletions(-)

diff --git a/drbd/drbd_worker.c b/drbd/drbd_worker.c
index df50bef..bee4ce2 100644
--- a/drbd/drbd_worker.c
+++ b/drbd/drbd_worker.c
@@ -761,6 +761,9 @@ int drbd_resync_finished(struct drbd_conf *mdev)
 		if (os.conn == C_SYNC_TARGET || os.conn == C_PAUSED_SYNC_T)
 			khelper_cmd = "after-resync-target";
 
+		if (os.conn == C_SYNC_SOURCE || os.conn == C_PAUSED_SYNC_S)
+			khelper_cmd = "after-resync-source";
+
 		if (mdev->csums_tfm && mdev->rs_total) {
 			const unsigned long s = mdev->rs_same_csum;
 			const unsigned long t = mdev->rs_total;
@@ -1365,6 +1368,17 @@ void drbd_start_resync(struct drbd_conf *mdev, enum drbd_conns side)
 			drbd_force_state(mdev, NS(conn, C_DISCONNECTING));
 			return;
 		}
+	} else {
+		/* Notify the user-space helper that we're becoming
+		   the source of a resync */
+		r = drbd_khelper(mdev, "before-resync-source");
+		r = (r >> 8) & 0xff;
+		if (r > 0) {
+			dev_info(DEV, "before-resync-source handler returned %d, "
+			     "dropping connection.\n", r);
+			drbd_force_state(mdev, NS(conn, C_DISCONNECTING));
+			return;
+		}
 	}
 
 	drbd_state_lock(mdev);
-- 
1.7.0.1

[Drbd-dev] [PATCH 3/3] Update documentation for resync-source handlers

[Iustin Pop]

---
 documentation/drbd.conf.xml |   27 ++++++++++++++++++++++++++-
 scripts/drbd.conf.example   |    4 ++++
 2 files changed, 30 insertions(+), 1 deletions(-)

diff --git a/documentation/drbd.conf.xml b/documentation/drbd.conf.xml
index 8e4fdfd..656d2ce 100644
--- a/documentation/drbd.conf.xml
+++ b/documentation/drbd.conf.xml
@@ -330,7 +330,8 @@ resource r0 {
     <option>pri-on-incon-degr</option>, <option>pri-lost-after-sb</option>,
     <option>pri-lost</option>, <option>fence-peer</option> (formerly oudate-peer),
     <option>local-io-error</option>, <option>initial-split-brain</option>, <option>split-brain</option>,
-    <option>before-resync-target</option>, <option>after-resync-target</option>.
+    <option>before-resync-target</option>, <option>after-resync-target</option>,
+    <option>before-resync-source</option>, <option>after-resync-source</option>.
   </para>
           </listitem>
         </varlistentry>
@@ -1509,6 +1510,30 @@ the unit is 1 second.
   </para>
           </listitem>
         </varlistentry>
+        <varlistentry>
+          <term>
+            <option>before-resync-source <replaceable>cmd</replaceable></option>
+          </term>
+          <listitem>
+            <para><indexterm significance="normal"><primary>drbd.conf</primary><secondary>before-resync-source </secondary></indexterm>
+  DRBD calls this handler just before a resync beginns on the node
+  that becomes resync source. It might be used to tweak the resync parameters
+  or the backing block device.
+  </para>
+          </listitem>
+        </varlistentry>
+        <varlistentry>
+          <term>
+            <option>after-resync-source <replaceable>cmd</replaceable></option>
+          </term>
+          <listitem>
+            <para><indexterm significance="normal"><primary>drbd.conf</primary><secondary>after-resync-source </secondary></indexterm>
+  DRBD calls this handler just after a resync operation finished on
+  the node which served as the resync source. It might be used to
+  unpause any other minors that were suspended, log the resync end, etc.
+  </para>
+          </listitem>
+        </varlistentry>
       </variablelist>
     </refsect2>
     <refsect2>
diff --git a/scripts/drbd.conf.example b/scripts/drbd.conf.example
index cc58814..6ae8816 100644
--- a/scripts/drbd.conf.example
+++ b/scripts/drbd.conf.example
@@ -174,6 +174,10 @@ resource r0 {
     # -- <additional lvcreate options>
     #before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
     #after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
+
+    # Example setting for the resync-source-handlers.
+    #before-resync-source "/usr/bin/logger Begun resync";
+    #after-resync-source "/usr/bin/logger Finished resync";
   }
 
   startup {
-- 
1.7.0.1

Re: [Drbd-dev] [PATCH 0/3] Implement {begin|after}-resync-source handlers

[Philipp Reisner]

Am Montag 17 Mai 2010 14:37:57 schrieb Iustin Pop:
> These small patches duplicate the resync-target handlers for the other
> end of the connection.
>

Hi Iustin,

The patches look okay so far, I am considering to apply them
to our tree.

* Have you ever considered to use the "drbdsetup events" 
  interface instead?

* Are you going to use them in your organisation? 

Best,
 Phil

Re: [Drbd-dev] [PATCH 0/3] Implement {begin|after}-resync-source handlers

[Iustin Pop]

On Mon, May 17, 2010 at 11:44:40PM +0200, Philipp Reisner wrote:
> Am Montag 17 Mai 2010 14:37:57 schrieb Iustin Pop:
> > These small patches duplicate the resync-target handlers for the other
> > end of the connection.
> >
> 
> Hi Iustin,
> 
> The patches look okay so far, I am considering to apply them
> to our tree.
> 
> * Have you ever considered to use the "drbdsetup events" 
>   interface instead?

Yep, I thought about it, and might use it if the handler method is not
good enough (for whatever reason). But I'd rather like to use the
handlers, since they allow reaction before the actual sync starts, and
thus they eliminate a window of "uncontrolled" syncs (see below why we
want to control them).

> * Are you going to use them in your organisation? 

That too, but better said we're going to use them in Ganeti (opensource,
cluster-based virtualization manager). Background: Ganeti uses only the
low-level drbdsetup interface, and not the drbdadm one, since we manage
the minor numbers assignments dynamically. This dynamic part hits us,
because we cannot make easy 'resync-after' rules (especially as on
bigger Ganeti clusters, like 20-40 nodes, you could get easily into
circular dependencies if you're not careful.

So the plan is to write a custom usermode_helper (right now Ganeti
documentations says to use /bin/true :), so we're not losing any
functionality by moving to a custom helper), that will simply restrict
DRBD to a single sync (either incoming or outgoing) per machine, in
order to not overload any machine's I/O resources. When a begin-sync
request comes in, either source or target, the helper will look if any
other minors are undergoing syncs, and if so will drbdsetup pause-sync
the minor - I hope that this is safe to do from the callback, my few
tests show it so. And when a after-resync-* requests comes, we look if
there are any paused syncs (on our end), and if so then we'll unpause
one.

Now, if drbd (the kernel driver) would have a setting like 'max_syncs',
then we wouldn't have to implement anything in Ganeti, but that is
probably more work in the kernel - which is harder, compared with a
simple userspace shell script.

Hope my explanation makes sense…
iustin

Re: [Drbd-dev] [PATCH 0/3] Implement {begin|after}-resync-source handlers

[Philipp Reisner]

[..]
> > * Are you going to use them in your organisation?
> 
> That too, but better said we're going to use them in Ganeti (opensource,
> cluster-based virtualization manager). Background: Ganeti uses only the
> low-level drbdsetup interface, and not the drbdadm one, since we manage
> the minor numbers assignments dynamically. This dynamic part hits us,
> because we cannot make easy 'resync-after' rules (especially as on
> bigger Ganeti clusters, like 20-40 nodes, you could get easily into
> circular dependencies if you're not careful.
> 
> So the plan is to write a custom usermode_helper (right now Ganeti
> documentations says to use /bin/true :), so we're not losing any
> functionality by moving to a custom helper), that will simply restrict
> DRBD to a single sync (either incoming or outgoing) per machine, in
> order to not overload any machine's I/O resources. When a begin-sync
> request comes in, either source or target, the helper will look if any
> other minors are undergoing syncs, and if so will drbdsetup pause-sync
> the minor - I hope that this is safe to do from the callback, my few
> tests show it so. And when a after-resync-* requests comes, we look if
> there are any paused syncs (on our end), and if so then we'll unpause
> one.
> 
> Now, if drbd (the kernel driver) would have a setting like 'max_syncs',
> then we wouldn't have to implement anything in Ganeti, but that is
> probably more work in the kernel - which is harder, compared with a
> simple userspace shell script.
> 
> Hope my explanation makes sense…

Hi Iustin,

Yes, makes sense. It will go in, after the 8.3.8 release dust settled.

Best,
 Phil
-- 
: Dipl-Ing Philipp Reisner
: LINBIT | Your Way to High Availability
: Tel: +43-1-8178292-50, Fax: +43-1-8178292-82
: http://www.linbit.com

DRBD(R) and LINBIT(R) are registered trademarks of LINBIT, Austria.