Distributed Replicated Block Device (DRBD) development
 help / color / mirror / Atom feed
From: Kitae Yoo <ryugitae777@gmail.com>
To: drbd-dev@lists.linux.dev
Cc: Kitae Yoo <ryugitae777@gmail.com>
Subject: [PATCH] drbd: fix resync stall when P_RS_CANCEL_AHEAD overtakes the state change
Date: Mon,  2 Nov 2026 16:01:13 +0900	[thread overview]
Message-ID: <20261102070113.52210-1-ryugitae777@gmail.com> (raw)

When the sync source pulls ahead (on-congestion pull-ahead) during a
resync, it answers pending resync requests with P_RS_CANCEL_AHEAD on the
control socket and sends its Ahead state change on the data socket. The
two sockets are not ordered, so the cancel can arrive first.
got_NegRSDReply() then sets SYNC_TARGET_TO_BEHIND while we are still
L_SYNC_TARGET, and make_resync_request() returns early without re-arming
resync_timer. Nothing else re-arms it.

This ends in a permanent stall in two ways:

1) The Ahead state arrives and we become Behind. With resync_timer dead,
   w_resync_timer() never runs the branch that clears resync_active.
   When the peer returns to SyncSource, the "resync-after-behind" start
   is postponed forever by do_start_resync() because resync_active is
   still set ("postponing start_resync" repeats).

2) A previously postponed resync start fires while the peer is Ahead
   again, so we become SyncTarget and the peer cancels our requests. The
   flag is set while we are SyncTarget. When the peer returns to
   SyncSource we never left SyncTarget, so neither the
   non-sync -> sync transition that clears the flag nor the
   "resync-after-behind" path (old peer state L_BEHIND) is taken.

In both cases the connection stays up, out-of-sync stays constant and no
resync traffic flows. Only a disconnect/connect recovers.

Fix both:
- Keep re-arming resync_timer on the early return, so that once we are
  Behind, w_resync_timer() clears resync_active as intended.
- In receive_state(), when the peer reports SyncSource while we are
  SyncTarget with SYNC_TARGET_TO_BEHIND set, clear the flag and kick
  resync_timer.

This does not remove the reordering itself, which would need a protocol
change; it makes the target recover from it.

Signed-off-by: Kitae Yoo <ryugitae777@gmail.com>
---

Notes:
    Environment:
      DRBD 9.3.4 (Ubuntu PPA 9.3.4-1ppa1~noble1), kernel 6.8.0-124-generic,
      2 nodes, protocol A, on-congestion pull-ahead, congestion-fill 8M,
      sndbuf-size 10M, max-buffers 20000, c-max-rate 200M.
      Link: WireGuard tunnel (MTU 1280) between two sites.
      Resources managed by LINSTOR.
    
    Reproduction:
      On the primary, repeated bursts of 64k random writes (fio, iodepth 16,
      20s on / 5s off) on a 5G volume, so the source repeatedly pulls ahead
      while a resync from the previous burst is still running.
    
      Unpatched (both nodes): 2 stalls in one run; each needed a
      disconnect/connect to recover. Earlier runs showed the same, with
      "postponing start_resync" repeating on the target.
    
      Patched module on the sync target only (source unpatched), two runs of
      40 bursts each:
        - source went Ahead 15 times, 13 of them during a resync
        - "peer pulled ahead during resync" logged 5 times on the target:
          2 with the case 1 order (pulled ahead -> SyncTarget->Behind ~1s
          later -> Behind->SyncTarget the next second), 3 with the case 2
          order, each followed within 1s by "peer back from ahead, resuming
          resync requests"
        - 0 stalls; resync finished each time shortly after the last burst
        - no warnings or oopses in the kernel log
    
      Not covered: the patched node acting as sync source, and >2 nodes.
    
    Side note, not addressed by this patch: while we are SyncTarget, a stale
    queued start_resync keeps logging "postponing start_resync" about every
    100ms (~1900 lines in 3 minutes in one run). An alternative/additional
    approach would be to drop such a stale start once we are already
    SyncTarget, or to not start the resync while the peer is still Ahead.

 drbd/drbd_receiver.c | 12 ++++++++++++
 drbd/drbd_sender.c   |  6 +++++-
 2 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/drbd/drbd_receiver.c b/drbd/drbd_receiver.c
index 51a61fbc3..357c3aafa 100644
--- a/drbd/drbd_receiver.c
+++ b/drbd/drbd_receiver.c
@@ -10569,6 +10569,18 @@ static int receive_state(struct drbd_connection *connection, struct packet_info
 		return 0;
 	}
 
+	/* The peer pulled ahead and came back while we stayed SyncTarget
+	 * (its P_RS_CANCEL_AHEAD outran the Ahead state change), so the
+	 * SyncTarget -> Behind -> SyncTarget path that would clear the flag
+	 * is never taken.  Clear it here and resume issuing requests.
+	 */
+	if (peer_state.conn == L_SYNC_SOURCE &&
+	    peer_device->repl_state[NOW] == L_SYNC_TARGET &&
+	    test_and_clear_bit(SYNC_TARGET_TO_BEHIND, peer_device->flags)) {
+		drbd_info(peer_device, "peer back from ahead, resuming resync requests\n");
+		mod_timer(&peer_device->resync_timer, jiffies + RS_MAKE_REQS_INTV);
+	}
+
 	/* Start resync after AHEAD/BEHIND */
 	if (connection->agreed_pro_version >= 110 &&
 	    peer_state.conn == L_SYNC_SOURCE && old_peer_state.conn == L_BEHIND) {
diff --git a/drbd/drbd_sender.c b/drbd/drbd_sender.c
index 5c27980d8..a6d1be578 100644
--- a/drbd/drbd_sender.c
+++ b/drbd/drbd_sender.c
@@ -1390,8 +1390,12 @@ static int make_resync_request(struct drbd_peer_device *peer_device, int cancel)
 	if (test_bit(SYNC_TARGET_TO_BEHIND, peer_device->flags)) {
 		/* If a P_RS_CANCEL_AHEAD on control socket overtook the
 		 * already queued data and state change to Ahead/Behind,
-		 * don't add more resync requests, just wait it out. */
+		 * don't add more resync requests, just wait it out.
+		 * Keep polling: once we are Behind, w_resync_timer() clears
+		 * resync_active; nothing else re-arms this timer.
+		 */
 		drbd_info_ratelimit(peer_device, "peer pulled ahead during resync\n");
+		mod_timer(&peer_device->resync_timer, jiffies + RS_MAKE_REQS_INTV);
 		return 0;
 	}
 	if (!get_ldev(device)) {
-- 
2.50.1 (Apple Git-155)


                 reply	other threads:[~2026-11-02  7:01 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261102070113.52210-1-ryugitae777@gmail.com \
    --to=ryugitae777@gmail.com \
    --cc=drbd-dev@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox