9.3.4: resync after a crashed Primary misses blocks (silent divergence) if that node was promoted in place after being Secondary under another Primary

9.3.4: resync after a crashed Primary misses blocks (silent divergence) if that node was promoted in place after being Secondary under another Primary

[Dominic]

Hello,

Last week I reported the two-volume flush stall (thank you for the quick
confirmation). While testing our failover design, we found another
problem on 9.3.4. This one is a silent divergence, so I wanted you to
know about it.

SUMMARY

Node A has been Secondary while node B was Primary. A is then promoted
in place, with no down/attach in between. A writes after a disconnect
and goes down without demoting (as a root-on-DRBD Primary does at
shutdown). When A comes back, it is SyncSource and the resync completes
normally. But some blocks that A wrote after the disconnect are not
resynced, so the two disks then differ while both report UpToDate.

If A is taken down and up (detach/attach) before the in-place
promotion, the resync is correct.

ENVIRONMENT

- drbd-dkms 9.3.4-1ppa1~resolute1 (GIT-hash e59b287b199f), drbd-utils
  9.35.0-1ppa1~resolute1 (LINBIT PPA)
- Ubuntu 26.04, kernel 7.0.0-38-generic, two VirtualBox VMs
- 2 nodes, protocol C, one resource with 2 volumes, external metadata,
  auto-promote no, no fencing. al-extents 6433. Same r0.res as in my
  24 Oct report.
- The Ubuntu drbd-utils units drbd-graceful-disconnect.service (active)
  and drbd-graceful-down.service (masked on A, to mimic a root-on-DRBD
  Primary that can't demote) are in use.
- Every "up" is attach first (adjust --skip-net, then adjust), to avoid
  the bug from my last report.
- Not tried on git master.

REPRODUCTION (A = pm1, node-id 0; B = pm2, node-id 1; ext4 on volume 0)

1. Connected, both UpToDate, both Secondary.
2. On B: drbdadm primary r0; mount /dev/drbd0; write a file; umount;
   drbdadm secondary r0. (Still connected throughout; A stays up.)
3. On A, with no down/up: drbdadm primary r0; mount /dev/drbd0; write
   10 MiB to a file; sync.
4. On A: systemctl reboot. drbd-graceful-disconnect disconnects first
   (B goes Outdated). The ext4 unmount then writes the superblock etc.
   after the disconnect. A never demotes (graceful-down is masked).
5. A boots; drbdadm adjust --skip-net r0; drbdadm adjust r0.
   A reports "Marked additional 0 KB as out-of-sync based on AL", is
   SyncSource and resyncs; both volumes end UpToDate.
6. Compare the backing devices: volume 0 differs.

Reproduced 3 times out of 3 (two scripted runs and one first sighting).

RESULTS

Failing run (with step 2), volume 0:

  pm1 kernel: drbd r0/0 drbd0 pm2: self B83C73CDF184048C:74B0ADF14B06F564:DA5CAA724336D98E:A923FA7E7C56E246 bits:3072 flags:2022
  pm1 kernel: drbd r0/0 drbd0 pm2: peer 74B0ADF14B06F564:0000000000000000:A923FA7E7C56E246:D8B36CA284BD34EA bits:0 flags:120
  pm1 kernel: drbd r0/0 drbd0 pm2: uuid_compare()=source-use-bitmap by rule=bitmap-self
  pm1 kernel: drbd r0/0 drbd0 pm2: Began resync as SyncSource (will sync 12288 KB [3072 bits set]).
  after resync: md5 of the volume 0 backing device differs between the
  nodes; the differing 4 KiB blocks are block 0 (4 MiB extent 0, the ext4
  superblock) and block 65536 (offset 256 MiB, extent 64). Volume 1
  matches.

Control without step 2 (A was never Secondary under a Primary B):
same sequence otherwise, "will sync 24576 KB [6144 bits set]", and the
checksums match afterwards.

Control with step 2 plus "drbdadm down r0; adjust --skip-net; adjust"
on A before step 3: 6144 bits, checksums match.

So the bitmap A brings back covers 3072 bits instead of 6144, and the
missing part includes blocks A wrote after the disconnect.

OUR GUESS AT THE MECHANISM (not verified in the code)

While B was Primary, we see "Disabling local AL-updates (optimization)"
and later "Enabling local AL-updates" in the logs. We suspect that
extents made active in A's in-memory activity log while it was
Secondary (from B's replicated writes) stay "hot" after A is promoted.
A's own later writes to those extents then cause no on-disk AL
transaction. A never demotes, so its in-memory bitmap is never written
out. After the crash, applying the on-disk AL doesn't cover those
extents, so their blocks are never resynced. The affected extents
(0 and 64) are exactly the ones B wrote while it was Primary, which
fits this. The detach/attach control also fits it.

IMPACT AND OUR WORKAROUND

The disks diverge silently: both sides report UpToDate after a
"successful" resync. Our production design promotes only from a fresh
initramfs (attach, then primary), so we are not exposed. Anyone who
promotes a long-running Secondary in place and later loses that Primary
without a clean demote could be. We worked around it by always doing
down/up before a promotion. We repaired the test VMs with
"drbdadm invalidate-remote" and confirmed with checksums.

We have the full kernel logs from both nodes, the scripts, and the VMs
(powered off), and can re-run anything you'd like, including a build
from git.

Best regards,
Dominic

Re: 9.3.4: resync after a crashed Primary misses blocks (silent divergence) if that node was promoted in place after being Secondary under another Primary

[Philipp Reisner]

Hi Dominic,

thank you for the detailed report. Your analysis of the mechanism is
correct, and the bug is already fixed in git. The fix is not in a
release yet.

In a two-node setup, a Secondary whose peer is Primary stops writing
activity log transactions ("Disabling local AL-updates
(optimization)"). The Primary has those extents in its own on-disk AL,
so a crash of the Primary is covered. The Secondary still tracks the
active extents in its in-memory AL.

The fix is on drbd-9.2 and on master (9.3):

  drbd-9.2: 2529ec433ac0 drbd: write the in-core activity log when AL...
            afd879ce60e7 drbd: write the superblock before AL updates go off
  master:   39d0a42c51a1, cabad431fc02 (merges of the two commits)

Your workaround (down/up before an in-place promotion) avoids the bug.
Setting max-peers to 2 or more in the resource configuration also
avoids it, because the two-node optimization then never turns AL
updates off. This setting takes effect only with newly created
metadata.

You offered to re-run your scripts. A run with current master on your
VMs would confirm the fix for your exact sequence, which differs from
the one in our test.

Best regards,
Phil


On Tue, Oct 27, 2026 at 7:38 AM Dominic <dominic@let2.com> wrote:
>
> Hello,
>
> Last week I reported the two-volume flush stall (thank you for the quick
> confirmation). While testing our failover design, we found another
> problem on 9.3.4. This one is a silent divergence, so I wanted you to
> know about it.
>
> SUMMARY
>
> Node A has been Secondary while node B was Primary. A is then promoted
> in place, with no down/attach in between. A writes after a disconnect
> and goes down without demoting (as a root-on-DRBD Primary does at
> shutdown). When A comes back, it is SyncSource and the resync completes
> normally. But some blocks that A wrote after the disconnect are not
> resynced, so the two disks then differ while both report UpToDate.
>
> If A is taken down and up (detach/attach) before the in-place
> promotion, the resync is correct.
>
> ENVIRONMENT
>
> - drbd-dkms 9.3.4-1ppa1~resolute1 (GIT-hash e59b287b199f), drbd-utils
>   9.35.0-1ppa1~resolute1 (LINBIT PPA)
> - Ubuntu 26.04, kernel 7.0.0-38-generic, two VirtualBox VMs
> - 2 nodes, protocol C, one resource with 2 volumes, external metadata,
>   auto-promote no, no fencing. al-extents 6433. Same r0.res as in my
>   24 Oct report.
> - The Ubuntu drbd-utils units drbd-graceful-disconnect.service (active)
>   and drbd-graceful-down.service (masked on A, to mimic a root-on-DRBD
>   Primary that can't demote) are in use.
> - Every "up" is attach first (adjust --skip-net, then adjust), to avoid
>   the bug from my last report.
> - Not tried on git master.
>
> REPRODUCTION (A = pm1, node-id 0; B = pm2, node-id 1; ext4 on volume 0)
>
> 1. Connected, both UpToDate, both Secondary.
> 2. On B: drbdadm primary r0; mount /dev/drbd0; write a file; umount;
>    drbdadm secondary r0. (Still connected throughout; A stays up.)
> 3. On A, with no down/up: drbdadm primary r0; mount /dev/drbd0; write
>    10 MiB to a file; sync.
> 4. On A: systemctl reboot. drbd-graceful-disconnect disconnects first
>    (B goes Outdated). The ext4 unmount then writes the superblock etc.
>    after the disconnect. A never demotes (graceful-down is masked).
> 5. A boots; drbdadm adjust --skip-net r0; drbdadm adjust r0.
>    A reports "Marked additional 0 KB as out-of-sync based on AL", is
>    SyncSource and resyncs; both volumes end UpToDate.
> 6. Compare the backing devices: volume 0 differs.
>
> Reproduced 3 times out of 3 (two scripted runs and one first sighting).
>
> RESULTS
>
> Failing run (with step 2), volume 0:
>
>   pm1 kernel: drbd r0/0 drbd0 pm2: self B83C73CDF184048C:74B0ADF14B06F564:DA5CAA724336D98E:A923FA7E7C56E246 bits:3072 flags:2022
>   pm1 kernel: drbd r0/0 drbd0 pm2: peer 74B0ADF14B06F564:0000000000000000:A923FA7E7C56E246:D8B36CA284BD34EA bits:0 flags:120
>   pm1 kernel: drbd r0/0 drbd0 pm2: uuid_compare()=source-use-bitmap by rule=bitmap-self
>   pm1 kernel: drbd r0/0 drbd0 pm2: Began resync as SyncSource (will sync 12288 KB [3072 bits set]).
>   after resync: md5 of the volume 0 backing device differs between the
>   nodes; the differing 4 KiB blocks are block 0 (4 MiB extent 0, the ext4
>   superblock) and block 65536 (offset 256 MiB, extent 64). Volume 1
>   matches.
>
> Control without step 2 (A was never Secondary under a Primary B):
> same sequence otherwise, "will sync 24576 KB [6144 bits set]", and the
> checksums match afterwards.
>
> Control with step 2 plus "drbdadm down r0; adjust --skip-net; adjust"
> on A before step 3: 6144 bits, checksums match.
>
> So the bitmap A brings back covers 3072 bits instead of 6144, and the
> missing part includes blocks A wrote after the disconnect.
>
> OUR GUESS AT THE MECHANISM (not verified in the code)
>
> While B was Primary, we see "Disabling local AL-updates (optimization)"
> and later "Enabling local AL-updates" in the logs. We suspect that
> extents made active in A's in-memory activity log while it was
> Secondary (from B's replicated writes) stay "hot" after A is promoted.
> A's own later writes to those extents then cause no on-disk AL
> transaction. A never demotes, so its in-memory bitmap is never written
> out. After the crash, applying the on-disk AL doesn't cover those
> extents, so their blocks are never resynced. The affected extents
> (0 and 64) are exactly the ones B wrote while it was Primary, which
> fits this. The detach/attach control also fits it.
>
> IMPACT AND OUR WORKAROUND
>
> The disks diverge silently: both sides report UpToDate after a
> "successful" resync. Our production design promotes only from a fresh
> initramfs (attach, then primary), so we are not exposed. Anyone who
> promotes a long-running Secondary in place and later loses that Primary
> without a clean demote could be. We worked around it by always doing
> down/up before a promotion. We repaired the test VMs with
> "drbdadm invalidate-remote" and confirmed with checksums.
>
> We have the full kernel logs from both nodes, the scripts, and the VMs
> (powered off), and can re-run anything you'd like, including a build
> from git.
>
> Best regards,
> Dominic

Re: 9.3.4: resync after a crashed Primary misses blocks (silent divergence) if that node was promoted in place after being Secondary under another Primary

[Dominic]

Hi Phil,

Thank you for the quick and clear answer.

I re-ran our sequence on current master in the same two VMs:
git 0192e7af3 ("Merge drbd-9.2/793f90dff53c ..."), which contains
39d0a42c51a1 and cabad431fc02, built with "make" against
7.0.0-38-generic and installed in place of the 9.3.4 dkms module on
both nodes. drbd-utils is unchanged (9.35.0), and so is the
configuration (2 volumes, max-peers left at the default).

Results, with the same steps as in my report:

- Failing case (B Primary, write, demote, then in-place promotion of
  A), twice: the log still shows "Disabling local AL-updates
  (optimization)" while B is Primary. After A's crash and attach, A
  resyncs 28672 KB [7168 bits set] on volume 0 (9.3.4: 3072 bits), and
  the backing devices match on both volumes. Both runs passed.
- Control without the B Primary step: 6144 bits, match.

So the fix covers our exact sequence. On 9.3.4 it failed 3 times out
of 3; on master it passed 2 out of 2, plus the control.

For production we will keep promoting only after a fresh attach, and
move to the first release that carries the fix.

Best regards,
Dominic