[PATCH 2/2] drbd: send state before bitmap to avoid receive_bitmap race

[PATCH 2/2] drbd: send state before bitmap to avoid receive_bitmap race

[zhengbing.huang]

When a node becomes SyncSource (L_WF_BITMAP_S), after_state_chg() on the
worker thread queues the bitmap send while receive_state() on the receiver
thread sends the P_STATE that lets the peer reach L_WF_BITMAP_T. These race,
so the bitmap can arrive while the peer is still L_ESTABLISHED and get
dropped in receive_bitmap(), logging "unexpected repl_state (Established) in
receive_bitmap".

Send the current state synchronously before queueing the bitmap so the peer
transitions to L_WF_BITMAP_T first.

Signed-off-by: zhengbing.huang <zhengbing.huang@easystack.cn>
---
 drbd/drbd_state.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drbd/drbd_state.c b/drbd/drbd_state.c
index 4ddb31cb8..65414a774 100644
--- a/drbd/drbd_state.c
+++ b/drbd/drbd_state.c
@@ -4884,6 +4884,12 @@ static int w_after_state_change(struct drbd_work *w, int unused)
 				 * it does no harm to resync a small amount of
 				 * additional data. */
 				drbd_set_pending_out_of_sync(peer_device);
+				/* Tell the peer about our new state before sending the
+				 * bitmap, so it can reach L_WF_BITMAP_T first. Otherwise
+				 * the bitmap may arrive while the peer is still
+				 * L_ESTABLISHED and get dropped in receive_bitmap().
+				 */
+				drbd_send_state(peer_device, new_state);
 				/* ldev_safe: ref from extra_ldev_ref_for_after_state_chg() */
 				drbd_queue_bitmap_io(device, &drbd_send_bitmap, NULL,
 						"send_bitmap (WFBitMapS)",
-- 
2.43.0

Re: [PATCH 2/2] drbd: send state before bitmap to avoid receive_bitmap race

[Philipp Reisner]

Hi Zhengbing,

Thanks for sending this.

I can reproduce the reordering itself in our simulator: when the
forced node handshakes on a P_STATE from its peer that still says
L_ESTABLISHED, its worker sends the bitmap before the receiver sends
the uuids and the P_STATE from that branch. But the peer does not drop
it. The forced node's P_STATE from drbd_set_role() is ahead of the
bitmap on the data socket, and the peer has CONSIDER_RESYNC set from
the promotion's commit. So it runs its own handshake on that P_STATE
and is L_WF_BITMAP_T before the bitmap arrives. I could not find an
interleaving of a 2-node forced promotion where the bitmap is dropped,
including with the peer diskless during the promotion and with a
concurrent pause-sync on either node.

So I suspect your case has a peer that does not run that handshake
itself. Can you tell me how you hit it?

- the sequence of commands, the node count, and which nodes were
  Inconsistent/UpToDate/diskless
- the DRBD version, and the agreed protocol version if it is older
  than 118
- the kernel log from both nodes around the "unexpected repl_state"
  message

The concern with the patch as it stands:

The uuids are deliberately sent before the state. Your new P_STATE
goes out from the worker before those uuids, so the peer may run its
handshake with stale uuids for us.

Best regads,
 Phil

Am Wed, Sep 23, 2026 at 02:01:09PM +0800 schrieb zhengbing.huang:
> When a node becomes SyncSource (L_WF_BITMAP_S), after_state_chg() on the
> worker thread queues the bitmap send while receive_state() on the receiver
> thread sends the P_STATE that lets the peer reach L_WF_BITMAP_T. These race,
> so the bitmap can arrive while the peer is still L_ESTABLISHED and get
> dropped in receive_bitmap(), logging "unexpected repl_state (Established) in
> receive_bitmap".
> 
> Send the current state synchronously before queueing the bitmap so the peer
> transitions to L_WF_BITMAP_T first.
> 
> Signed-off-by: zhengbing.huang <zhengbing.huang@easystack.cn>
> ---
>  drbd/drbd_state.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/drbd/drbd_state.c b/drbd/drbd_state.c
> index 4ddb31cb8..65414a774 100644
> --- a/drbd/drbd_state.c
> +++ b/drbd/drbd_state.c
> @@ -4884,6 +4884,12 @@ static int w_after_state_change(struct drbd_work *w, int unused)
>  				 * it does no harm to resync a small amount of
>  				 * additional data. */
>  				drbd_set_pending_out_of_sync(peer_device);
> +				/* Tell the peer about our new state before sending the
> +				 * bitmap, so it can reach L_WF_BITMAP_T first. Otherwise
> +				 * the bitmap may arrive while the peer is still
> +				 * L_ESTABLISHED and get dropped in receive_bitmap().
> +				 */
> +				drbd_send_state(peer_device, new_state);
>  				/* ldev_safe: ref from extra_ldev_ref_for_after_state_chg() */
>  				drbd_queue_bitmap_io(device, &drbd_send_bitmap, NULL,
>  						"send_bitmap (WFBitMapS)",
> -- 
> 2.43.0
> 
>