[Drbd-dev] [PATCH 00/10] Zero out devices instead of initial full sync

[Drbd-dev] [PATCH 00/10] Zero out devices instead of initial full sync

[Nick Wang]

Full sync for drbd initial usually take a long time, especically
when network become the bottleneck the syncing. Simply skip the
full sync with "--clear-bitmap" may not the perfect solution
for all the cases. So this patches can be used to zero out 
devices locally instead of a full sync,two make consistent block 
device. This approach can be useful when lack of network bandwidth 
to sync.

The patches add one new option "--zap-devices" to "new-current-uuid" 
to zero out devices. Besides the change of drbd, also need to modify 
drbd-utils for the flag.

All patches are compiled/tested against SLES12.

Signed-off-by: Nick Wang <nwang@suse.com>
CC: Philipp Reisner <philipp.reisner@linbit.com>
CC: Lars Ellenberg <lars.ellenberg@linbit.com>
CC: drbd-dev@lists.linbit.com
CC: linux-kernel@vger.kernel.org

 drbd/drbd_int.h        |  9 +++++
 drbd/drbd_main.c       | 42 ++++++++++++++++++++-
 drbd/drbd_nl.c         | 42 ++++++++++++++++++++-
 drbd/drbd_protocol.h   |  1 +
 drbd/drbd_receiver.c   | 99 ++++++++++++++++++++++++++++++++++++++++++++++++++
 drbd/linux/drbd_genl.h |  1 +
 6 files changed, 191 insertions(+), 3 deletions(-)

-- 
1.8.4.5

[Drbd-dev] [PATCH 01/10] drbd: Fix potenial risk of overlap the old history when move history.

[Nick Wang]

Fix the potential history overlap when using more than two history uuids.

Signed-off-by: Nick Wang <nwang@suse.com>
CC: Philipp Reisner <philipp.reisner@linbit.com>
CC: Lars Ellenberg <lars.ellenberg@linbit.com>
CC: drbd-dev@lists.linbit.com
CC: linux-kernel@vger.kernel.org

---
 drbd/drbd_main.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drbd/drbd_main.c b/drbd/drbd_main.c
index f41aa8a..2ffb569 100644
--- a/drbd/drbd_main.c
+++ b/drbd/drbd_main.c
@@ -3466,8 +3466,8 @@ void drbd_uuid_move_history(struct drbd_device *device) __must_hold(local)
 {
 	int i;
 
-	for (i = UI_HISTORY_START; i < UI_HISTORY_END; i++)
-		device->ldev->md.uuid[i+1] = device->ldev->md.uuid[i];
+	for (i = UI_HISTORY_END; i > UI_HISTORY_START; i--)
+		device->ldev->md.uuid[i] = device->ldev->md.uuid[i-1];
 }
 
 void __drbd_uuid_set(struct drbd_device *device, int idx, u64 val) __must_hold(local)
-- 
1.8.4.5

[Drbd-dev] [PATCH 02/10] Add options zap_devices to new-current-uuid for zeroing out device before initial sync.

[Nick Wang]

Add zap-devices options for new-current-uuid.
Need to change .h file in drbd-utils for user space.

Signed-off-by: Nick Wang <nwang@suse.com>
CC: Philipp Reisner <philipp.reisner@linbit.com>
CC: Lars Ellenberg <lars.ellenberg@linbit.com>
CC: drbd-dev@lists.linbit.com
CC: linux-kernel@vger.kernel.org

---
 drbd/drbd_nl.c         | 9 +++++++++
 drbd/linux/drbd_genl.h | 1 +
 2 files changed, 10 insertions(+)

diff --git a/drbd/drbd_nl.c b/drbd/drbd_nl.c
index 9c14cf3..1ef4551 100644
--- a/drbd/drbd_nl.c
+++ b/drbd/drbd_nl.c
@@ -4000,6 +4000,7 @@ int drbd_adm_new_c_uuid(struct sk_buff *skb, struct genl_info *info)
 	struct drbd_device *device;
 	enum drbd_ret_code retcode;
 	int skip_initial_sync = 0;
+	int zero_out_devices = 0;
 	int err;
 	struct new_c_uuid_parms args;
 
@@ -4034,6 +4035,14 @@ int drbd_adm_new_c_uuid(struct sk_buff *skb, struct genl_info *info)
 	    device->ldev->md.uuid[UI_CURRENT] == UUID_JUST_CREATED && args.clear_bm) {
 		drbd_info(device, "Preparing to skip initial sync\n");
 		skip_initial_sync = 1;
+	/* this is "zero out" devices to make it all zero.
+	* ignore "zero out" if both "clear_bm" and "zap_devices" set. */
+	} else if (device->state.conn == C_CONNECTED &&
+	    first_peer_device(device)->connection->agreed_pro_version >= 90 &&
+	    device->ldev->md.uuid[UI_CURRENT] == UUID_JUST_CREATED &&
+		args.zap_devices) {
+		drbd_info(device, "Preparing to zero out devices, will take a long time\n");
+		zero_out_devices = 1;
 	} else if (device->state.conn != C_STANDALONE) {
 		retcode = ERR_CONNECTED;
 		goto out_dec;
diff --git a/drbd/linux/drbd_genl.h b/drbd/linux/drbd_genl.h
index 5db53f5..eef8d8c 100644
--- a/drbd/linux/drbd_genl.h
+++ b/drbd/linux/drbd_genl.h
@@ -240,6 +240,7 @@ GENL_struct(DRBD_NLA_START_OV_PARMS, 9, start_ov_parms,
 
 GENL_struct(DRBD_NLA_NEW_C_UUID_PARMS, 10, new_c_uuid_parms,
 	__flg_field(1, DRBD_GENLA_F_MANDATORY, clear_bm)
+	__flg_field(2, DRBD_GENLA_F_MANDATORY, zap_devices)
 )
 
 GENL_struct(DRBD_NLA_TIMEOUT_PARMS, 11, timeout_parms,
-- 
1.8.4.5

[Drbd-dev] [PATCH 03/10] A function to zero out drbd backing device.

[Nick Wang]

A function to zero out local backing device.

Signed-off-by: Nick Wang <nwang@suse.com>
CC: Philipp Reisner <philipp.reisner@linbit.com>
CC: Lars Ellenberg <lars.ellenberg@linbit.com>
CC: drbd-dev@lists.linbit.com
CC: linux-kernel@vger.kernel.org

---
 drbd/drbd_int.h      |  1 +
 drbd/drbd_receiver.c | 21 +++++++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/drbd/drbd_int.h b/drbd/drbd_int.h
index 08d6648..9cc0ef9 100644
--- a/drbd/drbd_int.h
+++ b/drbd/drbd_int.h
@@ -1662,6 +1662,7 @@ extern void drbd_send_acks_wf(struct work_struct *ws);
 extern bool drbd_rs_c_min_rate_throttle(struct drbd_device *device);
 extern bool drbd_rs_should_slow_down(struct drbd_device *device, sector_t sector,
 		bool throttle_if_app_is_waiting);
+extern int zero_out_local_device(struct drbd_device *device);
 extern int drbd_submit_peer_request(struct drbd_device *,
 				    struct drbd_peer_request *, const unsigned,
 				    const int);
diff --git a/drbd/drbd_receiver.c b/drbd/drbd_receiver.c
index 5e6b149..47b8fe5 100644
--- a/drbd/drbd_receiver.c
+++ b/drbd/drbd_receiver.c
@@ -1594,6 +1594,27 @@ void drbd_bump_write_ordering(struct drbd_resource *resource, struct drbd_backin
 void conn_wait_active_ee_empty(struct drbd_connection *connection);
 
 /**
+ * zero_out_local_device()
+ * @device: DRBD device.
+ *
+ * Description:
+ * Zero out drbd backing device when creating new uuid.
+ *
+**/
+int zero_out_local_device(struct drbd_device *device)
+{
+	struct block_device *bdev;
+
+	bdev = device->ldev->backing_bdev;
+	if (device->ldev->known_size != drbd_get_capacity(bdev))
+		device->ldev->known_size = drbd_get_capacity(bdev);
+
+	/* zero out the backing device */
+	return blkdev_issue_zeroout(bdev, 0,
+		device->ldev->known_size , GFP_NOIO, false);
+}
+
+/**
  * drbd_submit_peer_request()
  * @device:	DRBD device.
  * @peer_req:	peer request
-- 
1.8.4.5

[Drbd-dev] [PATCH 04/10] A function to send peer node about starting zero out, using UI_FLAGS 16.

[Nick Wang]

Using UI_FLAGS(16) of P_UUIDS to start zeroing device of peer node.

Signed-off-by: Nick Wang <nwang@suse.com>
CC: Philipp Reisner <philipp.reisner@linbit.com>
CC: Lars Ellenberg <lars.ellenberg@linbit.com>
CC: drbd-dev@lists.linbit.com
CC: linux-kernel@vger.kernel.org

---
 drbd/drbd_int.h  | 1 +
 drbd/drbd_main.c | 5 +++++
 2 files changed, 6 insertions(+)

diff --git a/drbd/drbd_int.h b/drbd/drbd_int.h
index 9cc0ef9..d43adf2 100644
--- a/drbd/drbd_int.h
+++ b/drbd/drbd_int.h
@@ -1205,6 +1205,7 @@ extern int __drbd_send_protocol(struct drbd_connection *connection, enum drbd_pa
 extern int drbd_send_protocol(struct drbd_connection *connection);
 extern int drbd_send_uuids(struct drbd_peer_device *);
 extern int drbd_send_uuids_skip_initial_sync(struct drbd_peer_device *);
+extern int drbd_send_zero_out_start(struct drbd_peer_device *);
 extern void drbd_gen_and_send_sync_uuid(struct drbd_peer_device *);
 extern int drbd_send_sizes(struct drbd_peer_device *peer_device, int trigger_reply, enum dds_flags flags);
 extern int drbd_send_state(struct drbd_peer_device *, union drbd_state);
diff --git a/drbd/drbd_main.c b/drbd/drbd_main.c
index 2ffb569..841ed30 100644
--- a/drbd/drbd_main.c
+++ b/drbd/drbd_main.c
@@ -908,6 +908,11 @@ int drbd_send_uuids_skip_initial_sync(struct drbd_peer_device *peer_device)
 	return _drbd_send_uuids(peer_device, 8);
 }
 
+int drbd_send_zero_out_start(struct drbd_peer_device *peer_device)
+{
+	return _drbd_send_uuids(peer_device, 16);
+}
+
 void drbd_print_uuids(struct drbd_device *device, const char *text)
 {
 	if (get_ldev_if_state(device, D_NEGOTIATING)) {
-- 
1.8.4.5

[Drbd-dev] [PATCH 05/10] Start zero out device when drbdadm new-current-uuid --zap-device

[Nick Wang]

Two new device flags to represent peer zero out status.
Start zero out local device. 

Signed-off-by: Nick Wang <nwang@suse.com>
CC: Philipp Reisner <philipp.reisner@linbit.com>
CC: Lars Ellenberg <lars.ellenberg@linbit.com>
CC: drbd-dev@lists.linbit.com
CC: linux-kernel@vger.kernel.org

---
 drbd/drbd_int.h |  4 ++++
 drbd/drbd_nl.c  | 33 ++++++++++++++++++++++++++++++++-
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/drbd/drbd_int.h b/drbd/drbd_int.h
index d43adf2..b49095e 100644
--- a/drbd/drbd_int.h
+++ b/drbd/drbd_int.h
@@ -623,6 +623,10 @@ enum {
 	RS_START,		/* tell worker to start resync/OV */
 	RS_PROGRESS,		/* tell worker that resync made significant progress */
 	RS_DONE,		/* tell worker that resync is done */
+
+	/* used for zero out device */
+	ZERO_DONE,		/* succeed on zero out a device */
+	ZERO_FAIL,		/* fail to zero out a device */
 };
 
 struct drbd_bitmap; /* opaque for drbd_device */
diff --git a/drbd/drbd_nl.c b/drbd/drbd_nl.c
index 1ef4551..ad499a6 100644
--- a/drbd/drbd_nl.c
+++ b/drbd/drbd_nl.c
@@ -4001,6 +4001,7 @@ int drbd_adm_new_c_uuid(struct sk_buff *skb, struct genl_info *info)
 	enum drbd_ret_code retcode;
 	int skip_initial_sync = 0;
 	int zero_out_devices = 0;
+	int zero_out_err = 0;
 	int err;
 	struct new_c_uuid_parms args;
 
@@ -4051,7 +4052,7 @@ int drbd_adm_new_c_uuid(struct sk_buff *skb, struct genl_info *info)
 	drbd_uuid_set(device, UI_BITMAP, 0); /* Rotate UI_BITMAP to History 1, etc... */
 	drbd_uuid_new_current(device); /* New current, previous to UI_BITMAP */
 
-	if (args.clear_bm) {
+	if (args.clear_bm || args.zap_devices) {
 		err = drbd_bitmap_io(device, &drbd_bmio_clear_n_write,
 			"clear_n_write from new_c_uuid", BM_LOCKED_MASK);
 		if (err) {
@@ -4067,6 +4068,36 @@ int drbd_adm_new_c_uuid(struct sk_buff *skb, struct genl_info *info)
 					CS_VERBOSE, NULL);
 			spin_unlock_irq(&device->resource->req_lock);
 		}
+		if (zero_out_devices) {
+			drbd_send_zero_out_start(first_peer_device(device));
+			_drbd_uuid_set(device, UI_BITMAP, 0);
+			drbd_print_uuids(device, "cleared bitmap UUID for zeroing device");
+
+			/* CLear bit flag of zero out */
+			clear_bit(ZERO_DONE, &device->flags);
+			clear_bit(ZERO_FAIL, &device->flags);
+			zero_out_err = zero_out_local_device(device);
+
+			if (zero_out_err) {
+				drbd_err(device, "Failed to zero out local device\n");
+				set_bit(ZERO_FAIL, &device->flags);
+				drbd_chk_io_error(device, 1, DRBD_WRITE_ERROR);
+			} else {
+				drbd_info(device, "Finish zero out device\n");
+
+				if (test_and_clear_bit(ZERO_DONE, &device->flags)) {
+					spin_lock_irq(&device->resource->req_lock);
+					_drbd_set_state(_NS2(device, disk, D_UP_TO_DATE,
+						pdsk, D_UP_TO_DATE), CS_VERBOSE, NULL);
+					spin_unlock_irq(&device->resource->req_lock);
+				} else if (test_and_clear_bit(ZERO_FAIL, &device->flags)) {
+					drbd_info(device, "Peer device has already failed on zero out\n");
+				} else {
+					/* waiting for peer device finished */
+					set_bit(ZERO_DONE, &device->flags);
+				}
+			}
+		}
 	}
 
 	drbd_md_sync(device);
-- 
1.8.4.5

[Drbd-dev] [PATCH 06/10] New packet P_ZERO_OUT to get the status of zeroing out device from peer node.

[Nick Wang]

New packet P_ZERO_OUT for zero out device.

Signed-off-by: Nick Wang <nwang@suse.com>
CC: Philipp Reisner <philipp.reisner@linbit.com>
CC: Lars Ellenberg <lars.ellenberg@linbit.com>
CC: drbd-dev@lists.linbit.com
CC: linux-kernel@vger.kernel.org

---
 drbd/drbd_protocol.h |  1 +
 drbd/drbd_receiver.c | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+)

diff --git a/drbd/drbd_protocol.h b/drbd/drbd_protocol.h
index 405b181..3a82442 100644
--- a/drbd/drbd_protocol.h
+++ b/drbd/drbd_protocol.h
@@ -59,6 +59,7 @@ enum drbd_packet {
 	/* REQ_DISCARD. We used "discard" in different contexts before,
 	 * which is why I chose TRIM here, to disambiguate. */
 	P_TRIM                = 0x31,
+	P_ZERO_OUT            = 0x32,
 
 	P_MAY_IGNORE	      = 0x100, /* Flag to test if (cmd > P_MAY_IGNORE) ... */
 	P_MAX_OPT_CMD	      = 0x101,
diff --git a/drbd/drbd_receiver.c b/drbd/drbd_receiver.c
index 47b8fe5..41c958a 100644
--- a/drbd/drbd_receiver.c
+++ b/drbd/drbd_receiver.c
@@ -4404,6 +4404,39 @@ static union drbd_state convert_state(union drbd_state ps)
 	return ms;
 }
 
+static int receive_zero_out_state(struct drbd_connection *connection, struct packet_info *pi)
+{
+	struct drbd_peer_device *peer_device;
+	struct drbd_device *device;
+	struct p_state *p = pi->data;
+	unsigned int isfail;
+
+	peer_device = conn_peer_device(connection, pi->vnr);
+	if (!peer_device)
+		return -EIO;
+	device = peer_device->device;
+
+	isfail = be32_to_cpu(p->state);
+
+	if (isfail) {
+		drbd_info(device, "Failed to zero out peer device\n");
+		set_bit(ZERO_FAIL, &device->flags);
+	} else {
+		drbd_info(device, "Finished zero out peer device\n");
+		if (test_and_clear_bit(ZERO_DONE, &device->flags)) {
+			spin_lock_irq(&device->resource->req_lock);
+			_drbd_set_state(_NS2(device, disk, D_UP_TO_DATE,
+				pdsk, D_UP_TO_DATE), CS_VERBOSE, NULL);
+			spin_unlock_irq(&device->resource->req_lock);
+		} else {
+			/* waiting for local device finished */
+			set_bit(ZERO_DONE, &device->flags);
+		}
+	}
+
+	return 0;
+}
+
 static int receive_req_state(struct drbd_connection *connection, struct packet_info *pi)
 {
 	struct drbd_peer_device *peer_device;
@@ -5029,6 +5062,7 @@ static struct data_cmd drbd_cmd_handler[] = {
 	[P_CONN_ST_CHG_REQ] = { 0, sizeof(struct p_req_state), receive_req_conn_state },
 	[P_PROTOCOL_UPDATE] = { 1, sizeof(struct p_protocol), receive_protocol },
 	[P_TRIM]	    = { 0, sizeof(struct p_trim), receive_Data },
+	[P_ZERO_OUT]	    = { 0, sizeof(struct p_state), receive_zero_out_state },
 };
 
 static void drbdd(struct drbd_connection *connection)
-- 
1.8.4.5

[Drbd-dev] [PATCH 07/10] Using P_ZERO_OUT to send back device zero out status.

[Nick Wang]

Send back status of zero out device to peer node.

Signed-off-by: Nick Wang <nwang@suse.com>
CC: Philipp Reisner <philipp.reisner@linbit.com>
CC: Lars Ellenberg <lars.ellenberg@linbit.com>
CC: drbd-dev@lists.linbit.com
CC: linux-kernel@vger.kernel.org

---
 drbd/drbd_int.h  |  2 ++
 drbd/drbd_main.c | 28 ++++++++++++++++++++++++++++
 2 files changed, 30 insertions(+)

diff --git a/drbd/drbd_int.h b/drbd/drbd_int.h
index b49095e..e956eb4 100644
--- a/drbd/drbd_int.h
+++ b/drbd/drbd_int.h
@@ -1210,6 +1210,8 @@ extern int drbd_send_protocol(struct drbd_connection *connection);
 extern int drbd_send_uuids(struct drbd_peer_device *);
 extern int drbd_send_uuids_skip_initial_sync(struct drbd_peer_device *);
 extern int drbd_send_zero_out_start(struct drbd_peer_device *);
+extern int drbd_send_zero_out_ok(struct drbd_peer_device *);
+extern int drbd_send_zero_out_fail(struct drbd_peer_device *);
 extern void drbd_gen_and_send_sync_uuid(struct drbd_peer_device *);
 extern int drbd_send_sizes(struct drbd_peer_device *peer_device, int trigger_reply, enum dds_flags flags);
 extern int drbd_send_state(struct drbd_peer_device *, union drbd_state);
diff --git a/drbd/drbd_main.c b/drbd/drbd_main.c
index 841ed30..5892ff3 100644
--- a/drbd/drbd_main.c
+++ b/drbd/drbd_main.c
@@ -913,6 +913,34 @@ int drbd_send_zero_out_start(struct drbd_peer_device *peer_device)
 	return _drbd_send_uuids(peer_device, 16);
 }
 
+/**
+ * _drbd_send_zero_out_state() - Sends the drbd state to the peer
+ * @peer_device:	DRBD peer device.
+ * @state:	Device zero out status.
+ */
+static int _drbd_send_zero_out_state(struct drbd_peer_device *peer_device, unsigned int status)
+{
+	struct drbd_socket *sock;
+	struct p_state *p;
+
+	sock = &peer_device->connection->data;
+	p = drbd_prepare_command(peer_device, sock);
+	if (!p)
+		return -EIO;
+	p->state = cpu_to_be32(status);
+	return drbd_send_command(peer_device, sock, P_ZERO_OUT, sizeof(*p), NULL, 0);
+}
+
+int drbd_send_zero_out_ok(struct drbd_peer_device *peer_device)
+{
+	return _drbd_send_zero_out_state(peer_device, 0);
+}
+
+int drbd_send_zero_out_fail(struct drbd_peer_device *peer_device)
+{
+	return _drbd_send_zero_out_state(peer_device, 1);
+}
+
 void drbd_print_uuids(struct drbd_device *device, const char *text)
 {
 	if (get_ldev_if_state(device, D_NEGOTIATING)) {
-- 
1.8.4.5

[Drbd-dev] [PATCH 08/10] Receive UI_FLAGS 16 of P_UUIDS to start zero out devices.

[Nick Wang]

Receive P_UUIDS protocol to start zero out device.

Signed-off-by: Nick Wang <nwang@suse.com>
CC: Philipp Reisner <philipp.reisner@linbit.com>
CC: Lars Ellenberg <lars.ellenberg@linbit.com>
CC: drbd-dev@lists.linbit.com
CC: linux-kernel@vger.kernel.org

---
 drbd/drbd_receiver.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/drbd/drbd_receiver.c b/drbd/drbd_receiver.c
index 41c958a..cbb66e6 100644
--- a/drbd/drbd_receiver.c
+++ b/drbd/drbd_receiver.c
@@ -4304,6 +4304,7 @@ static int receive_uuids(struct drbd_connection *connection, struct packet_info
 	struct p_uuids *p = pi->data;
 	u64 *p_uuid;
 	int i, updated_uuids = 0;
+	int zero_out_err = 0;
 
 	peer_device = conn_peer_device(connection, pi->vnr);
 	if (!peer_device)
@@ -4338,6 +4339,11 @@ static int receive_uuids(struct drbd_connection *connection, struct packet_info
 			peer_device->connection->agreed_pro_version >= 90 &&
 			device->ldev->md.uuid[UI_CURRENT] == UUID_JUST_CREATED &&
 			(p_uuid[UI_FLAGS] & 8);
+		int zero_out_devices =
+			device->state.conn == C_CONNECTED &&
+			peer_device->connection->agreed_pro_version >= 90 &&
+			device->ldev->md.uuid[UI_CURRENT] == UUID_JUST_CREATED &&
+			(p_uuid[UI_FLAGS] & 16);
 		if (skip_initial_sync) {
 			drbd_info(device, "Accepted new current UUID, preparing to skip initial sync\n");
 			drbd_bitmap_io(device, &drbd_bmio_clear_n_write,
@@ -4350,6 +4356,23 @@ static int receive_uuids(struct drbd_connection *connection, struct packet_info
 			drbd_md_sync(device);
 			updated_uuids = 1;
 		}
+
+		if (zero_out_devices) {
+			drbd_info(device, "Accepted to zero out devices, will take a long time\n");
+			_drbd_uuid_set(device, UI_CURRENT, p_uuid[UI_CURRENT]);
+			_drbd_uuid_set(device, UI_BITMAP, 0);
+			drbd_print_uuids(device, "cleared bitmap UUID for zeroing device");
+			zero_out_err = zero_out_local_device(device);
+			if (zero_out_err) {
+				drbd_err(device, "Failed to zero out local device\n");
+				drbd_send_zero_out_fail(peer_device);
+				drbd_chk_io_error(device, 1, DRBD_WRITE_ERROR);
+			} else {
+				drbd_info(device, "Finish zero out device\n");
+				drbd_send_zero_out_ok(peer_device);
+			}
+		}
+
 		put_ldev(device);
 	} else if (device->state.disk < D_INCONSISTENT &&
 		   device->state.role == R_PRIMARY) {
-- 
1.8.4.5

[Drbd-dev] [PATCH 09/10] Notify peer to change status after both side finished zero out.

[Nick Wang]

Update peer node state after both finished.

Signed-off-by: Nick Wang <nwang@suse.com>
CC: Philipp Reisner <philipp.reisner@linbit.com>
CC: Lars Ellenberg <lars.ellenberg@linbit.com>
CC: drbd-dev@lists.linbit.com
CC: linux-kernel@vger.kernel.org

---
 drbd/drbd_int.h      |  1 +
 drbd/drbd_main.c     |  5 +++++
 drbd/drbd_receiver.c | 13 +++++++++++++
 3 files changed, 19 insertions(+)

diff --git a/drbd/drbd_int.h b/drbd/drbd_int.h
index e956eb4..0a92f7d 100644
--- a/drbd/drbd_int.h
+++ b/drbd/drbd_int.h
@@ -1210,6 +1210,7 @@ extern int drbd_send_protocol(struct drbd_connection *connection);
 extern int drbd_send_uuids(struct drbd_peer_device *);
 extern int drbd_send_uuids_skip_initial_sync(struct drbd_peer_device *);
 extern int drbd_send_zero_out_start(struct drbd_peer_device *);
+extern int drbd_send_zero_out_finish(struct drbd_peer_device *);
 extern int drbd_send_zero_out_ok(struct drbd_peer_device *);
 extern int drbd_send_zero_out_fail(struct drbd_peer_device *);
 extern void drbd_gen_and_send_sync_uuid(struct drbd_peer_device *);
diff --git a/drbd/drbd_main.c b/drbd/drbd_main.c
index 5892ff3..f9f1a2e 100644
--- a/drbd/drbd_main.c
+++ b/drbd/drbd_main.c
@@ -913,6 +913,11 @@ int drbd_send_zero_out_start(struct drbd_peer_device *peer_device)
 	return _drbd_send_uuids(peer_device, 16);
 }
 
+int drbd_send_zero_out_finish(struct drbd_peer_device *peer_device)
+{
+	return _drbd_send_uuids(peer_device, 32);
+}
+
 /**
  * _drbd_send_zero_out_state() - Sends the drbd state to the peer
  * @peer_device:	DRBD peer device.
diff --git a/drbd/drbd_receiver.c b/drbd/drbd_receiver.c
index cbb66e6..240a557 100644
--- a/drbd/drbd_receiver.c
+++ b/drbd/drbd_receiver.c
@@ -4344,6 +4344,10 @@ static int receive_uuids(struct drbd_connection *connection, struct packet_info
 			peer_device->connection->agreed_pro_version >= 90 &&
 			device->ldev->md.uuid[UI_CURRENT] == UUID_JUST_CREATED &&
 			(p_uuid[UI_FLAGS] & 16);
+		int zero_out_finish =
+			device->state.conn == C_CONNECTED &&
+			peer_device->connection->agreed_pro_version >= 90 &&
+			(p_uuid[UI_FLAGS] & 32);
 		if (skip_initial_sync) {
 			drbd_info(device, "Accepted new current UUID, preparing to skip initial sync\n");
 			drbd_bitmap_io(device, &drbd_bmio_clear_n_write,
@@ -4373,6 +4377,14 @@ static int receive_uuids(struct drbd_connection *connection, struct packet_info
 			}
 		}
 
+		if (zero_out_finish) {
+			drbd_info(device, "Both side finished zero out devices.\n");
+			spin_lock_irq(&device->resource->req_lock);
+			_drbd_set_state(_NS2(device, disk, D_UP_TO_DATE, pdsk, D_UP_TO_DATE),
+					CS_VERBOSE, NULL);
+			spin_unlock_irq(&device->resource->req_lock);
+		}
+
 		put_ldev(device);
 	} else if (device->state.disk < D_INCONSISTENT &&
 		   device->state.role == R_PRIMARY) {
@@ -4451,6 +4463,7 @@ static int receive_zero_out_state(struct drbd_connection *connection, struct pac
 			_drbd_set_state(_NS2(device, disk, D_UP_TO_DATE,
 				pdsk, D_UP_TO_DATE), CS_VERBOSE, NULL);
 			spin_unlock_irq(&device->resource->req_lock);
+			drbd_send_zero_out_finish(peer_device);
 		} else {
 			/* waiting for local device finished */
 			set_bit(ZERO_DONE, &device->flags);
-- 
1.8.4.5

[Drbd-dev] [PATCH 10/10] Update flag when receive uuid.

[Nick Wang]

Clear bitmap write and add lock when change state.

Signed-off-by: Nick Wang <nwang@suse.com>
CC: Philipp Reisner <philipp.reisner@linbit.com>
CC: Lars Ellenberg <lars.ellenberg@linbit.com>
CC: drbd-dev@lists.linbit.com
CC: linux-kernel@vger.kernel.org

---
 drbd/drbd_receiver.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drbd/drbd_receiver.c b/drbd/drbd_receiver.c
index 240a557..944db4c 100644
--- a/drbd/drbd_receiver.c
+++ b/drbd/drbd_receiver.c
@@ -4355,17 +4355,23 @@ static int receive_uuids(struct drbd_connection *connection, struct packet_info
 					BM_LOCKED_TEST_ALLOWED);
 			_drbd_uuid_set(device, UI_CURRENT, p_uuid[UI_CURRENT]);
 			_drbd_uuid_set(device, UI_BITMAP, 0);
+			spin_lock_irq(&device->resource->req_lock);
 			_drbd_set_state(_NS2(device, disk, D_UP_TO_DATE, pdsk, D_UP_TO_DATE),
 					CS_VERBOSE, NULL);
+			spin_unlock_irq(&device->resource->req_lock);
 			drbd_md_sync(device);
 			updated_uuids = 1;
 		}
 
 		if (zero_out_devices) {
 			drbd_info(device, "Accepted to zero out devices, will take a long time\n");
+			drbd_bitmap_io(device, &drbd_bmio_clear_n_write,
+					"clear_n_write from receive_uuids",
+					BM_LOCKED_TEST_ALLOWED);
 			_drbd_uuid_set(device, UI_CURRENT, p_uuid[UI_CURRENT]);
 			_drbd_uuid_set(device, UI_BITMAP, 0);
 			drbd_print_uuids(device, "cleared bitmap UUID for zeroing device");
+
 			zero_out_err = zero_out_local_device(device);
 			if (zero_out_err) {
 				drbd_err(device, "Failed to zero out local device\n");
@@ -4375,6 +4381,8 @@ static int receive_uuids(struct drbd_connection *connection, struct packet_info
 				drbd_info(device, "Finish zero out device\n");
 				drbd_send_zero_out_ok(peer_device);
 			}
+			drbd_md_sync(device);
+			updated_uuids = 1;
 		}
 
 		if (zero_out_finish) {
-- 
1.8.4.5

Re: [Drbd-dev] [PATCH 00/10] Zero out devices instead of initial full sync

[Lars Ellenberg]

On Wed, Jun 10, 2015 at 03:48:19PM +0800, Nick Wang wrote:
> Full sync for drbd initial usually take a long time, especically
> when network become the bottleneck the syncing. Simply skip the
> full sync with "--clear-bitmap" may not the perfect solution
> for all the cases. So this patches can be used to zero out 
> devices locally instead of a full sync,two make consistent block 
> device. This approach can be useful when lack of network bandwidth 
> to sync.
> 
> The patches add one new option "--zap-devices" to "new-current-uuid" 
> to zero out devices. Besides the change of drbd, also need to modify 
> drbd-utils for the flag.
> 
> All patches are compiled/tested against SLES12.

Completely unnecessary.
Zero out the devices in userspace, then set up DRBD,
and skip the initial sync.

-- 
: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA  and  Pacemaker support and consulting

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

Re: [Drbd-dev] [PATCH 00/10] Zero out devices instead of initial full sync

[Lars Ellenberg]

resent, accidentally truncated the Cc list.
also added one paragraph.

On Wed, Jun 10, 2015 at 03:48:19PM +0800, Nick Wang wrote:
> Full sync for drbd initial usually take a long time, especically
> when network become the bottleneck the syncing. Simply skip the
> full sync with "--clear-bitmap" may not the perfect solution
> for all the cases. So this patches can be used to zero out 
> devices locally instead of a full sync,two make consistent block 
> device. This approach can be useful when lack of network bandwidth 
> to sync.
> 
> The patches add one new option "--zap-devices" to "new-current-uuid" 
> to zero out devices. Besides the change of drbd, also need to modify 
> drbd-utils for the flag.
> 
> All patches are compiled/tested against SLES12.

Completely unnecessary.
Zero out the devices in userspace, then set up DRBD,
and skip the initial sync.

On backends that support it,
you can already simply blkdiscard /dev/drbdX
(or use the implicit discard of some mkfs)
once you set it up for the first time,
even when the initial resync just began,
and that discard will finish the "initial resync",
without taking up any real network bandwidth.

In short: I don't see the point.

	Lars Ellenberg