[RFC PATCH v1 0/2] migration/rdma: Allow multiple writes per chunk

[RFC PATCH v1 0/2] migration/rdma: Allow multiple writes per chunk

[Yanfei Xu]

My understanding is that within each iteration round, the HVAs carried by
different WRs never overlap, so serializing WRs within the same chunk isn't
necessary. This patchset builds on that assumption — please correct me if
I've missed or misunderstood anything.

This series improves RDMA migration throughput during the iteration phase
and iterable stop-copy phase by allowing multiple writes to the same
registration chunk.

RDMA tracks RAM in registration chunks.  During the final iterable
transfer, workloads always leave many scattered 4 KiB dirty pages, with
several pages falling into the same chunk.  The current code waits for
an earlier write to a chunk to complete before posting the next one.
These serial completion waits can leave send queue capacity unused and
limit RDMA bandwidth, increasing downtime.

The first patch replaces the per-chunk transit bitmap with reference
counts.  The second patch removes the per-chunk wait.

rdma_registration_start/stop functions called in each round synchronously
drain all in-flight WRs with cq, which can serve as a barrier between
rounds.

The test configuration is:

  RDMA chunk size:    32 MiB
  Guest:              32 vCPUs, 128 GiB RAM
  Test runs:          10
  Max HCA bandwidth:  100 Gbs
  Workload:           idle

Average results:              Before       After
  pin-all is true
  Downtime                    396.7 ms     345.7 ms        ~12.9% improve
  Final iterable bandwidth    9952.0 MiB/s 11639.10 MiB/s  ~17.0% improve


Average results:              Before       After
  pin-all is false
  Downtime                    248.7 ms     191.1 ms        ~23.2% improve
  Final iterable bandwidth    5605.1 MiB/s 9826.0 MiB/s    ~75.3% improve

Note: Final iterable bandwidth actually means the bandwidth during
qemu_savevm_state_complete_precopy_iterable(). non-iterable data doesn't
use RDMA Write.

Yanfei Xu (2):
  migration/rdma: Track in-flight writes with refcounts
  migration/rdma: Allow multiple in-flight writes per chunk

 migration/rdma.c       | 65 +++++++++++++++++++++++-------------------
 migration/trace-events |  5 ++--
 2 files changed, 37 insertions(+), 33 deletions(-)

-- 
2.20.1

[RFC PATCH v1 1/2] migration/rdma: Track in-flight writes with refcounts

[Yanfei Xu]

The transit bitmap can only represent one outstanding write per
registration chunk.  Replace it with a per-chunk reference count and
assert the count on each update.

Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
---
 migration/rdma.c       | 48 ++++++++++++++++++++++++++++++------------
 migration/trace-events |  4 ++--
 2 files changed, 37 insertions(+), 15 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index d1f44a5f55..973a7a745a 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -199,7 +199,7 @@ typedef struct RDMALocalBlock {
     unsigned int   src_index;       /* (Only used on dest) */
     bool           is_ram_block;
     int            nb_chunks;
-    unsigned long *transit_bitmap;
+    unsigned int  *transit_refcnt;
     unsigned long *unregister_bitmap;
 } RDMALocalBlock;
 
@@ -552,6 +552,29 @@ static inline uint8_t *ram_chunk_end(const RDMALocalBlock *rdma_ram_block,
     return result;
 }
 
+static inline bool qemu_rdma_chunk_in_transit(const RDMALocalBlock *block,
+                                              uint64_t chunk)
+{
+    assert(chunk < block->nb_chunks);
+    return block->transit_refcnt[chunk] > 0;
+}
+
+static inline void qemu_rdma_chunk_transit_inc(RDMALocalBlock *block,
+                                               uint64_t chunk)
+{
+    assert(chunk < block->nb_chunks);
+    assert(block->transit_refcnt[chunk] < UINT_MAX);
+    block->transit_refcnt[chunk]++;
+}
+
+static inline void qemu_rdma_chunk_transit_dec(RDMALocalBlock *block,
+                                               uint64_t chunk)
+{
+    assert(chunk < block->nb_chunks);
+    assert(block->transit_refcnt[chunk] > 0);
+    block->transit_refcnt[chunk]--;
+}
+
 static void rdma_add_block(RDMAContext *rdma, const char *block_name,
                            void *host_addr,
                            ram_addr_t block_offset, uint64_t length)
@@ -585,8 +608,7 @@ static void rdma_add_block(RDMAContext *rdma, const char *block_name,
     block->index = local->nb_blocks;
     block->src_index = ~0U; /* Filled in by the receipt of the block list */
     block->nb_chunks = ram_chunk_index(host_addr, host_addr + length) + 1UL;
-    block->transit_bitmap = bitmap_new(block->nb_chunks);
-    bitmap_clear(block->transit_bitmap, 0, block->nb_chunks);
+    block->transit_refcnt = g_new0(unsigned int, block->nb_chunks);
     block->unregister_bitmap = bitmap_new(block->nb_chunks);
     bitmap_clear(block->unregister_bitmap, 0, block->nb_chunks);
     block->remote_keys = g_new0(uint32_t, block->nb_chunks);
@@ -601,8 +623,7 @@ static void rdma_add_block(RDMAContext *rdma, const char *block_name,
                          (uintptr_t) block->local_host_addr,
                          block->offset, block->length,
                          (uintptr_t) (block->local_host_addr + block->length),
-                         BITS_TO_LONGS(block->nb_chunks) *
-                             sizeof(unsigned long) * 8,
+                         block->nb_chunks * sizeof(*block->transit_refcnt),
                          block->nb_chunks);
 
     local->nb_blocks++;
@@ -673,8 +694,8 @@ static void rdma_delete_block(RDMAContext *rdma, RDMALocalBlock *block)
         block->mr = NULL;
     }
 
-    g_free(block->transit_bitmap);
-    block->transit_bitmap = NULL;
+    g_free(block->transit_refcnt);
+    block->transit_refcnt = NULL;
 
     g_free(block->unregister_bitmap);
     block->unregister_bitmap = NULL;
@@ -716,8 +737,9 @@ static void rdma_delete_block(RDMAContext *rdma, RDMALocalBlock *block)
     trace_rdma_delete_block(block, (uintptr_t)block->local_host_addr,
                            block->offset, block->length,
                             (uintptr_t)(block->local_host_addr + block->length),
-                           BITS_TO_LONGS(block->nb_chunks) *
-                               sizeof(unsigned long) * 8, block->nb_chunks);
+                           block->nb_chunks *
+                               sizeof(*block->transit_refcnt),
+                           block->nb_chunks);
 
     g_free(old);
 
@@ -1237,7 +1259,7 @@ static int qemu_rdma_unregister_waiting(RDMAContext *rdma)
          */
         clear_bit(chunk, block->unregister_bitmap);
 
-        if (test_bit(chunk, block->transit_bitmap)) {
+        if (qemu_rdma_chunk_in_transit(block, chunk)) {
             trace_qemu_rdma_unregister_waiting_inflight(chunk);
             continue;
         }
@@ -1328,7 +1350,7 @@ static int qemu_rdma_poll(RDMAContext *rdma, struct ibv_cq *cq,
                                    index, chunk, block->local_host_addr,
                                    (void *)(uintptr_t)block->remote_host_addr);
 
-        clear_bit(chunk, block->transit_bitmap);
+        qemu_rdma_chunk_transit_dec(block, chunk);
 
         if (rdma->nb_sent > 0) {
             rdma->nb_sent--;
@@ -1889,7 +1911,7 @@ retry:
     chunk_end = ram_chunk_end(block, chunk + chunks);
 
 
-    while (test_bit(chunk, block->transit_bitmap)) {
+    while (qemu_rdma_chunk_in_transit(block, chunk)) {
         (void)count;
         trace_qemu_rdma_write_one_block(count++, current_index, chunk,
                 sge.addr, length, rdma->nb_sent, block->nb_chunks);
@@ -2053,7 +2075,7 @@ retry:
         return -1;
     }
 
-    set_bit(chunk, block->transit_bitmap);
+    qemu_rdma_chunk_transit_inc(block, chunk);
     qatomic_add(&mig_stats.normal_pages, sge.length / qemu_target_page_size());
     /*
      * We are adding to transferred the amount of data written, but no
diff --git a/migration/trace-events b/migration/trace-events
index af0e784535..172761be78 100644
--- a/migration/trace-events
+++ b/migration/trace-events
@@ -256,9 +256,9 @@ qemu_rdma_write_one_recvregres(int mykey, int theirkey, uint64_t chunk) "Receive
 qemu_rdma_write_one_sendreg(uint64_t chunk, int len, int index, int64_t offset) "Sending registration request chunk %" PRIu64 " for %d bytes, index: %d, offset: %" PRId64
 qemu_rdma_write_one_top(uint64_t chunks, uint64_t size) "Writing %" PRIu64 " chunks, (%" PRIu64 " MB)"
 qemu_rdma_write_one_zero(uint64_t chunk, int len, int index, int64_t offset) "Entire chunk is zero, sending compress: %" PRIu64 " for %d bytes, index: %d, offset: %" PRId64
-rdma_add_block(const char *block_name, int block, uint64_t addr, uint64_t offset, uint64_t len, uint64_t end, uint64_t bits, int chunks) "Added Block: '%s':%d, addr: %" PRIu64 ", offset: %" PRIu64 " length: %" PRIu64 " end: %" PRIu64 " bits %" PRIu64 " chunks %d"
+rdma_add_block(const char *block_name, int block, uint64_t addr, uint64_t offset, uint64_t len, uint64_t end, uint64_t refcnt_bytes, int chunks) "Added Block: '%s':%d, addr: %" PRIu64 ", offset: %" PRIu64 " length: %" PRIu64 " end: %" PRIu64 " refcnt-bytes %" PRIu64 " chunks %d"
 rdma_block_notification_handle(const char *name, int index) "%s at %d"
-rdma_delete_block(void *block, uint64_t addr, uint64_t offset, uint64_t len, uint64_t end, uint64_t bits, int chunks) "Deleted Block: %p, addr: %" PRIu64 ", offset: %" PRIu64 " length: %" PRIu64 " end: %" PRIu64 " bits %" PRIu64 " chunks %d"
+rdma_delete_block(void *block, uint64_t addr, uint64_t offset, uint64_t len, uint64_t end, uint64_t refcnt_bytes, int chunks) "Deleted Block: %p, addr: %" PRIu64 ", offset: %" PRIu64 " length: %" PRIu64 " end: %" PRIu64 " refcnt-bytes %" PRIu64 " chunks %d"
 rdma_registration_handle_compress(int64_t length, int index, int64_t offset) "Zapping zero chunk: %" PRId64 " bytes, index %d, offset %" PRId64
 rdma_registration_handle_finished(void) ""
 rdma_registration_handle_ram_blocks(void) ""
-- 
2.20.1

[RFC PATCH v1 2/2] migration/rdma: Allow multiple in-flight writes per chunk

[Yanfei Xu]

qemu_rdma_write_one() waits for an earlier write to the same
registration chunk to complete.  This serializes disjoint dirty ranges
in a chunk and leaves send queue capacity unused.

Remove the per-chunk wait and use the reference counts to track all
outstanding writes.  The existing per-iteration drain remains the
completion barrier.

Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
---
 migration/rdma.c       | 19 +------------------
 migration/trace-events |  1 -
 2 files changed, 1 insertion(+), 19 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 973a7a745a..63bc357657 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -1869,7 +1869,7 @@ static int qemu_rdma_write_one(RDMAContext *rdma,
     struct ibv_sge sge;
     struct ibv_send_wr send_wr = { 0 };
     struct ibv_send_wr *bad_wr;
-    int reg_result_idx, ret, count = 0;
+    int reg_result_idx, ret;
     uint64_t chunk, chunks;
     uint64_t chunk_size = migrate_rdma_chunk_size();
     uint8_t *chunk_start, *chunk_end;
@@ -1910,23 +1910,6 @@ retry:
 
     chunk_end = ram_chunk_end(block, chunk + chunks);
 
-
-    while (qemu_rdma_chunk_in_transit(block, chunk)) {
-        (void)count;
-        trace_qemu_rdma_write_one_block(count++, current_index, chunk,
-                sge.addr, length, rdma->nb_sent, block->nb_chunks);
-
-        ret = qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, NULL);
-
-        if (ret < 0) {
-            error_setg(errp, "Failed to Wait for previous write to complete "
-                    "block %d chunk %" PRIu64
-                    " current %" PRIu64 " len %" PRIu64 " %d",
-                    current_index, chunk, sge.addr, length, rdma->nb_sent);
-            return -1;
-        }
-    }
-
     if (!rdma->pin_all || !block->is_ram_block) {
         if (!block->remote_keys[chunk]) {
             /*
diff --git a/migration/trace-events b/migration/trace-events
index 172761be78..253ff71891 100644
--- a/migration/trace-events
+++ b/migration/trace-events
@@ -249,7 +249,6 @@ qemu_rdma_unregister_waiting_proc(uint64_t chunk, int pos) "Processing unregiste
 qemu_rdma_unregister_waiting_send(uint64_t chunk) "Sending unregister for chunk: %" PRIu64
 qemu_rdma_unregister_waiting_complete(uint64_t chunk) "Unregister for chunk: %" PRIu64 " complete."
 qemu_rdma_write_flush(int sent) "sent total: %d"
-qemu_rdma_write_one_block(int count, int block, uint64_t chunk, uint64_t current, uint64_t len, int nb_sent, int nb_chunks) "(%d) Not clobbering: block: %d chunk %" PRIu64 " current %" PRIu64 " len %" PRIu64 " %d %d"
 qemu_rdma_write_one_post(uint64_t chunk, long addr, long remote, uint32_t len) "Posting chunk: %" PRIu64 ", addr: 0x%lx remote: 0x%lx, bytes %" PRIu32
 qemu_rdma_write_one_queue_full(void) ""
 qemu_rdma_write_one_recvregres(int mykey, int theirkey, uint64_t chunk) "Received registration result: my key: 0x%x their key 0x%x, chunk %" PRIu64
-- 
2.20.1

Re: [RFC PATCH v1 2/2] migration/rdma: Allow multiple in-flight writes per chunk

[Peter Xu]

On Thu, Aug 20, 2026 at 08:58:33PM +0800, Yanfei Xu wrote:
> qemu_rdma_write_one() waits for an earlier write to the same
> registration chunk to complete.  This serializes disjoint dirty ranges
> in a chunk and leaves send queue capacity unused.
> 
> Remove the per-chunk wait and use the reference counts to track all
> outstanding writes.  The existing per-iteration drain remains the
> completion barrier.

I think this should work (by accident; will explain below), but I want to
raise the definition of iteration here, and it may or may not be what was
expected.

Migration core has this problem likely since 10+ years ago when it start to
have two definitions..

- Each time ram_save_iterate() is invoked: this is the "iteration" that
  RDMA is tracking, it does qemu_rdma_drain_cq() when finishing for each
  call (applies to complete() too)

- Each time migration RAM core syncs dirty info and re-scans the whole
  guest memories (all ramblocks)

For RDMA (and non-RDMA too), what matters is for each same page its new
version always lands *after* its old version.  IIUC, what it really needs
is defintion 2), not 1)..  See the call of multifd_ram_sync_per_round() of
find_dirty_block(), so it was called "a round" there, but I believe we
report such iteration count (in reality, "dirty-sync-count") in QMP
query-migrate with this concept.

I still think relying on the qemu_rdma_drain_cq() should be fine for now,
it's because currently we hold bitmap_mutex across the whole
ram_save_iterate() (NOTE: postcopy preempt may release it.. another thing
to discuss..), so bmap at least for precopy shouldn't be able to change, it
also means for each ram_save_iterate() we shouldn't be sending the same
page twice.  But I think it's risky relying on that fact, e.g. we already
have concurrent sync dirty bitmap, like cpu_throttle_dirty_sync_timer_tick,
so maybe it's better RDMA also flush its pages at definition 2) not 1).
IIUC, it'll also improve on performance because RDMA needs to flush less.

Maybe we should make it a generic API in find_dirty_block(), like
notifiers, so that multifd (and maybe RDMA too?) doesn't need to hard code
things like multifd_ram_flush_and_sync().

Thanks,

> 
> Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
> ---
>  migration/rdma.c       | 19 +------------------
>  migration/trace-events |  1 -
>  2 files changed, 1 insertion(+), 19 deletions(-)
> 
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 973a7a745a..63bc357657 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -1869,7 +1869,7 @@ static int qemu_rdma_write_one(RDMAContext *rdma,
>      struct ibv_sge sge;
>      struct ibv_send_wr send_wr = { 0 };
>      struct ibv_send_wr *bad_wr;
> -    int reg_result_idx, ret, count = 0;
> +    int reg_result_idx, ret;
>      uint64_t chunk, chunks;
>      uint64_t chunk_size = migrate_rdma_chunk_size();
>      uint8_t *chunk_start, *chunk_end;
> @@ -1910,23 +1910,6 @@ retry:
>  
>      chunk_end = ram_chunk_end(block, chunk + chunks);
>  
> -
> -    while (qemu_rdma_chunk_in_transit(block, chunk)) {
> -        (void)count;
> -        trace_qemu_rdma_write_one_block(count++, current_index, chunk,
> -                sge.addr, length, rdma->nb_sent, block->nb_chunks);
> -
> -        ret = qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, NULL);
> -
> -        if (ret < 0) {
> -            error_setg(errp, "Failed to Wait for previous write to complete "
> -                    "block %d chunk %" PRIu64
> -                    " current %" PRIu64 " len %" PRIu64 " %d",
> -                    current_index, chunk, sge.addr, length, rdma->nb_sent);
> -            return -1;
> -        }
> -    }
> -
>      if (!rdma->pin_all || !block->is_ram_block) {
>          if (!block->remote_keys[chunk]) {
>              /*
> diff --git a/migration/trace-events b/migration/trace-events
> index 172761be78..253ff71891 100644
> --- a/migration/trace-events
> +++ b/migration/trace-events
> @@ -249,7 +249,6 @@ qemu_rdma_unregister_waiting_proc(uint64_t chunk, int pos) "Processing unregiste
>  qemu_rdma_unregister_waiting_send(uint64_t chunk) "Sending unregister for chunk: %" PRIu64
>  qemu_rdma_unregister_waiting_complete(uint64_t chunk) "Unregister for chunk: %" PRIu64 " complete."
>  qemu_rdma_write_flush(int sent) "sent total: %d"
> -qemu_rdma_write_one_block(int count, int block, uint64_t chunk, uint64_t current, uint64_t len, int nb_sent, int nb_chunks) "(%d) Not clobbering: block: %d chunk %" PRIu64 " current %" PRIu64 " len %" PRIu64 " %d %d"
>  qemu_rdma_write_one_post(uint64_t chunk, long addr, long remote, uint32_t len) "Posting chunk: %" PRIu64 ", addr: 0x%lx remote: 0x%lx, bytes %" PRIu32
>  qemu_rdma_write_one_queue_full(void) ""
>  qemu_rdma_write_one_recvregres(int mykey, int theirkey, uint64_t chunk) "Received registration result: my key: 0x%x their key 0x%x, chunk %" PRIu64
> -- 
> 2.20.1
> 

-- 
Peter Xu

Re: [RFC PATCH v1 0/2] migration/rdma: Allow multiple writes per chunk

[Peter Xu]

On Thu, Aug 20, 2026 at 08:58:31PM +0800, Yanfei Xu wrote:
> My understanding is that within each iteration round, the HVAs carried by
> different WRs never overlap, so serializing WRs within the same chunk isn't
> necessary. This patchset builds on that assumption — please correct me if
> I've missed or misunderstood anything.
> 
> This series improves RDMA migration throughput during the iteration phase
> and iterable stop-copy phase by allowing multiple writes to the same
> registration chunk.
> 
> RDMA tracks RAM in registration chunks.  During the final iterable
> transfer, workloads always leave many scattered 4 KiB dirty pages, with
> several pages falling into the same chunk.  The current code waits for
> an earlier write to a chunk to complete before posting the next one.
> These serial completion waits can leave send queue capacity unused and
> limit RDMA bandwidth, increasing downtime.
> 
> The first patch replaces the per-chunk transit bitmap with reference
> counts.  The second patch removes the per-chunk wait.
> 
> rdma_registration_start/stop functions called in each round synchronously
> drain all in-flight WRs with cq, which can serve as a barrier between
> rounds.
> 
> The test configuration is:
> 
>   RDMA chunk size:    32 MiB
>   Guest:              32 vCPUs, 128 GiB RAM
>   Test runs:          10
>   Max HCA bandwidth:  100 Gbs
>   Workload:           idle
> 
> Average results:              Before       After
>   pin-all is true
>   Downtime                    396.7 ms     345.7 ms        ~12.9% improve
>   Final iterable bandwidth    9952.0 MiB/s 11639.10 MiB/s  ~17.0% improve
> 
> 
> Average results:              Before       After
>   pin-all is false
>   Downtime                    248.7 ms     191.1 ms        ~23.2% improve
>   Final iterable bandwidth    5605.1 MiB/s 9826.0 MiB/s    ~75.3% improve
> 
> Note: Final iterable bandwidth actually means the bandwidth during
> qemu_savevm_state_complete_precopy_iterable(). non-iterable data doesn't
> use RDMA Write.

So I left some comment in patch 2, but maybe I should just comment directly
here.. please check that first.

If you would agree with what I said there (and you should have noticed
another patch I sent days ago removing UNREGISTER path), I think what we
really need might be:

- Move RDMA draining from ram_save_iterate()/complete() into each time we
  finish scanning a round

- Remove transit_bitmap completely (if you see after applying your this
  series, the UNREGISTER should be the only user..), because with the
  correct draining IIUC we don't need this anymore

Thanks,

-- 
Peter Xu