[BUG] 9.3.4: resync between two Secondaries stuck at 100% - flush request never acked (2 volumes, new-peer before attach)

[BUG] 9.3.4: resync between two Secondaries stuck at 100% - flush request never acked (2 volumes, new-peer before attach)

[Dominic Ferard]

Hello,

On a two-volume resource, a resync between two Secondaries reaches
100% (out-of-sync 0) and never completes: the target stays
Inconsistent/SyncTarget on both volumes indefinitely. It happens when
the source was brought up with a plain "drbdadm up" and its second
volume's peer bitmap_dagtag is higher than its first volume's. We hit
it in production (two standby hosts, both Secondary while the Primary
OS is down) and have reproduced it in two VMs. Details, debugfs output
and our reading of the source are below. Please correct the analysis
if we have it wrong.


Environment
-----------

- drbd-dkms 9.3.4-1ppa1~resolute1 (GIT-hash
  e59b287b199f7224468af85b68a10e83c68213f5), drbd-utils
  9.35.0-1ppa1~resolute1, LINBIT PPA
- Ubuntu 26.04, kernel 7.0.0-38-generic
- 2 nodes, protocol C, 1 resource with 2 volumes, external metadata,
  auto-promote no

    resource r0 {
        options { auto-promote no; }
        net { protocol C; }
        on pm1 {
            node-id 0;
            address 10.10.10.1:7799;
            volume 0 { device /dev/drbd0; disk /dev/disk/by-partlabel/drbd-r0; meta-disk /dev/disk/by-partlabel/drbd-r0-md; }
            volume 1 { device /dev/drbd1; disk /dev/disk/by-partlabel/drbd-r1; meta-disk /dev/disk/by-partlabel/drbd-r1-md; }
        }
        on pm2 {
            node-id 1;
            address 10.10.10.2:7799;
            volume 0 { device /dev/drbd0; disk /dev/disk/by-partlabel/drbd-r0; meta-disk /dev/disk/by-partlabel/drbd-r0-md; }
            volume 1 { device /dev/drbd1; disk /dev/disk/by-partlabel/drbd-r1; meta-disk /dev/disk/by-partlabel/drbd-r1-md; }
        }
    }

(Production also has al-extents 6433 and resync-controller settings;
they are not needed to reproduce.)


Reproduction
------------

Start with both nodes connected and UpToDate.

    pm1: make the resource dagtag non-zero
      pm1# drbdadm primary r0
      pm1# dd if=/dev/urandom of=/dev/drbd0 bs=1M count=10 oflag=direct
      pm1# dd if=/dev/urandom of=/dev/drbd1 bs=1M count=10 oflag=direct

    pm2 goes away
      pm2# drbdadm down r0

    pm1 writes vol0 FIRST, then vol1, so vol1's peer bitmap_dagtag
    ends up higher
      pm1# dd if=/dev/urandom of=/dev/drbd0 bs=1M count=20 oflag=direct
      pm1# dd if=/dev/urandom of=/dev/drbd1 bs=1M count=20 oflag=direct
      pm1# drbdadm secondary r0 && drbdadm down r0

    both come back as Secondary with a plain "up" (new-peer before
    attach)
      pm1# drbdadm up r0
      pm2# drbdadm up r0

Peer slot 1 in pm1's metadata before the "up" (after apply-al):

    vol0 peer[1]  bitmap-dagtag 0x000000000002D000
    vol1 peer[1]  bitmap-dagtag 0x0000000000037000

Result: pm2 is SyncTarget on both volumes with out-of-sync 0 and stays
there (we watched for several minutes; in production it stayed for
about 15 minutes until we intervened).

debugfs, /sys/kernel/debug/drbd/resources/r0/connections/<peer>/debug:

    pm1 (SyncSource), connection pm2:
     resource->dagtag_sector: 225280      (0x37000, from vol1's metadata)
    last_peer_ack_dagtag_seen: 0
       flush_requests_dagtag: 184320      (0x2D000 = vol0's dagtag = dagtag_before_attach)
              flush_sequence: 1
     flush_forward_sent_mask:             (empty: never acked)

    pm2 (SyncTarget), connection pm1:
          pending_flush_mask: pm1,

"drbdadm disconnect r0; drbdadm connect r0" on pm2 starts a 0 KB
resync that also never finishes. pm1 afterwards still shows
last_peer_ack_dagtag_seen: 0 and flush_requests_dagtag: 184320, now
with flush_sequence: 2.

"drbdadm primary r0" on pm1 plus a single 4 KiB write moves
last_peer_ack_dagtag_seen to 225288, and the resync finishes at once.

Controls (same VMs):

- Vol0 written last (so vol0's peer bitmap_dagtag is higher), plain
  "up": the resync finishes. flush_requests_dagtag = 0, while
  last_peer_ack_dagtag_seen is also 0.
- Equal or zero bitmap dagtags (the peer left before any writes):
  finishes.
- Original order, but pm1 brought up with "drbdadm adjust --skip-net
  r0" and then "drbdadm adjust r0" (attach before new-peer): finishes
  in 1 s. last_peer_ack_dagtag_seen = 307208 >= flush_requests_dagtag
  = 266248.


Analysis (from reading the 9.3.4 source)
----------------------------------------

1. "drbdadm up" issues new-resource, new-minor, new-peer, new-path,
   peer-device-options, attach, connect (per "drbdadm -d up"), so
   new-peer runs before any attach.

2. new-peer (drbd_nl.c ~5118) initialises
   connection->last_peer_ack_dagtag_seen = resource->dagtag_sector,
   "so that flushes can be acked even if no further writes occur". At
   that point the dagtag is still 0.

3. Each attach calls update_resource_dagtag() (drbd_nl.c ~2843). The
   first volume raises dagtag_sector from 0 to its max peer
   bitmap_dagtag (dagtag_before_attach = 0). If the second volume's
   bitmap_dagtag is higher, it sets dagtag_before_attach to the first
   volume's (non-zero) value and dagtag_from_backing_dev to its own.

4. When the SyncTarget sends P_FLUSH_REQUESTS, receive_flush_requests()
   (drbd_receiver.c ~5176) sees dagtag_sector ==
   dagtag_from_backing_dev, so it uses dagtag_before_attach (the
   non-zero vol0 value) as flush_requests_dagtag.

5. drbd_check_flush_dagtag_reached() (drbd_receiver.c ~12408) acks
   only when last_peer_ack_dagtag_seen >= flush_requests_dagtag. With
   no Primary anywhere, last_peer_ack_dagtag_seen never moves from 0,
   so the ack never comes.

6. On the target, drbd_resync_check_finished() (drbd_sender.c ~1078)
   waits on drbd_any_flush_pending(), which is resource-wide, so both
   volumes stay SyncTarget. (In production, one volume had already
   enabled peer replication and the other had not.)

With one volume, dagtag_before_attach stays 0 and the ack always
passes, which is why we only started seeing this after moving from
one volume to two.

Possible fixes (your call): when an attach raises dagtag_sector, also
bring last_peer_ack_dagtag_seen up on existing connections that have no
Primary peer; or clamp the flush target in receive_flush_requests() so
it cannot exceed what this node could ever see acked when it has no
Primary peers.


Workaround
----------

On the standby nodes we attach before creating the peer: an
ExecStartPre= for drbd@r0.service runs "drbdadm adjust --skip-net r0"
ahead of its normal "adjust".

Happy to test patches or gather more data; the VM reproduction takes
about two minutes.

Thanks,
Dominic Ferard

Re: [BUG] 9.3.4: resync between two Secondaries stuck at 100% - flush request never acked (2 volumes, new-peer before attach)

[Philipp Reisner]

Hi Dominic,

thank you for the excellent report. Your analysis is correct. We
found the same bug independently, and the fix is already in git:

  drbd-9.2: 24374f123a95 ("drbd: keep the dagtag of the last request
            across several attaches")
  https://github.com/LINBIT/drbd/commit/24374f123a95877956c3c95a5080b6afae75b28c

  master:   eec2d913473b (merge of the above into drbd-9.3)
  https://github.com/LINBIT/drbd/commit/eec2d913473bf00370b1049e549f10ac0e6181cd

Only 9.3.4 is affected in practice. It is the first release that
also sends the flush to Secondaries. Earlier releases flush only
Primary peers, and a Primary's own writes move the peer acks forward.

The fix will be in 9.3.5, which we plan to release on November 9th.

If you want to confirm the fix in your VMs before then, the commit
above applies on top of 9.3.4.

Best regards,
Philipp

Am Sat, Oct 24, 2026 at 05:11:08PM +0100 schrieb Dominic Ferard:
> Hello,
> 
> On a two-volume resource, a resync between two Secondaries reaches
> 100% (out-of-sync 0) and never completes: the target stays
> Inconsistent/SyncTarget on both volumes indefinitely. It happens when
> the source was brought up with a plain "drbdadm up" and its second
> volume's peer bitmap_dagtag is higher than its first volume's. We hit
> it in production (two standby hosts, both Secondary while the Primary
> OS is down) and have reproduced it in two VMs. Details, debugfs output
> and our reading of the source are below. Please correct the analysis
> if we have it wrong.
[...]