[Drbd-dev] DRBD does not always send UnplugRemote when it should

[Drbd-dev] DRBD does not always send UnplugRemote when it should

[Graham, Simon]

[-- Attachment #1: Type: text/plain, Size: 928 bytes --]

In looking at DRBD performance, I noticed that DRBD does not always send
an UnplugRemote command when (I think) it should - I believe this is
because it uses the mdev UNPLUG_REMOTE flag to control whether or not
the command would be helpful and this flag is set in the worker thread
as part of actually sending the data packet.

This means that an unplug call can be made from above between the time
the bio is submitted and the time the worker thread starts processing it
- this call will be ignored because the flag it not set yet.
Subsequently, the flag gets set but it's too late by then and you can
see a very long delay on the secondary until the unplug timeout expires
there.

I think the fix is to set the flag in the queue_for_net_write action in
_req_mod - this is called inline with the bio so the flag will be set
before the unplug routine is called.

Proposed patch against 8.2 attached.
Simon


[-- Attachment #2: unplug-remote.patch --]
[-- Type: application/octet-stream, Size: 1057 bytes --]

diff --git a/drbd/drbd_main.c b/drbd/drbd_main.c
index 943ef0b..ccf2e17 100644
--- a/drbd/drbd_main.c
+++ b/drbd/drbd_main.c
@@ -2023,7 +2023,6 @@ int drbd_send_dblock(struct drbd_conf *mdev, struct drbd_request *req)
 	p.dp_flags = cpu_to_be32(dp_flags);
 	dump_packet(mdev, mdev->data.socket, 0, (void *)&p, __FILE__, __LINE__);
 	blk_add_trace_bio(mdev->rq_queue, req->master_bio, BLK_TA_GETRQ);
-	set_bit(UNPLUG_REMOTE, &mdev->flags);
 	ok = (sizeof(p) ==
 		drbd_send(mdev, mdev->data.socket, &p, sizeof(p), MSG_MORE));
 	if (ok && dgs) {
diff --git a/drbd/drbd_req.c b/drbd/drbd_req.c
index 745713e..74fe905 100644
--- a/drbd/drbd_req.c
+++ b/drbd/drbd_req.c
@@ -661,6 +661,8 @@ void _req_mod(struct drbd_request *req, enum drbd_req_event what, int error)
 		 *
 		 * Add req to the (now) current epoch (barrier). */
 
+		set_bit(UNPLUG_REMOTE, &mdev->flags); /* Worth sending unplug when higher layer unplugs */
+
 		/* see drbd_make_request_common,
 		 * just after it grabs the req_lock */
 		D_ASSERT(test_bit(CREATE_BARRIER, &mdev->flags) == 0);

Re: [Drbd-dev] DRBD does not always send UnplugRemote when it should

[Lars Ellenberg]

On Sun, Nov 01, 2009 at 05:16:07PM -0500, Graham, Simon wrote:
> In looking at DRBD performance, I noticed that DRBD does not always send
> an UnplugRemote command when (I think) it should - I believe this is
> because it uses the mdev UNPLUG_REMOTE flag to control whether or not
> the command would be helpful and this flag is set in the worker thread
> as part of actually sending the data packet.
> 
> This means that an unplug call can be made from above between the time
> the bio is submitted and the time the worker thread starts processing it
> - this call will be ignored because the flag it not set yet.
> Subsequently, the flag gets set but it's too late by then and you can
> see a very long delay on the secondary until the unplug timeout expires
> there.
> 
> I think the fix is to set the flag in the queue_for_net_write action in
> _req_mod - this is called inline with the bio so the flag will be set
> before the unplug routine is called.
> 
> Proposed patch against 8.2 attached.

Thanks, that looks reasonable.
I *think* we had it like that once sometime, and then changed it again.
After all, the queue_for_net_read does it, too...

Will apply to 8.3.


-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.