Re: [RFC PATCH v1 0/2] migration/rdma: Allow multiple writes per chunk

[Peter Xu <peterx@redhat.com>]

On Thu, Aug 20, 2026 at 08:58:31PM +0800, Yanfei Xu wrote:
> My understanding is that within each iteration round, the HVAs carried by
> different WRs never overlap, so serializing WRs within the same chunk isn't
> necessary. This patchset builds on that assumption — please correct me if
> I've missed or misunderstood anything.
> 
> This series improves RDMA migration throughput during the iteration phase
> and iterable stop-copy phase by allowing multiple writes to the same
> registration chunk.
> 
> RDMA tracks RAM in registration chunks.  During the final iterable
> transfer, workloads always leave many scattered 4 KiB dirty pages, with
> several pages falling into the same chunk.  The current code waits for
> an earlier write to a chunk to complete before posting the next one.
> These serial completion waits can leave send queue capacity unused and
> limit RDMA bandwidth, increasing downtime.
> 
> The first patch replaces the per-chunk transit bitmap with reference
> counts.  The second patch removes the per-chunk wait.
> 
> rdma_registration_start/stop functions called in each round synchronously
> drain all in-flight WRs with cq, which can serve as a barrier between
> rounds.
> 
> The test configuration is:
> 
>   RDMA chunk size:    32 MiB
>   Guest:              32 vCPUs, 128 GiB RAM
>   Test runs:          10
>   Max HCA bandwidth:  100 Gbs
>   Workload:           idle
> 
> Average results:              Before       After
>   pin-all is true
>   Downtime                    396.7 ms     345.7 ms        ~12.9% improve
>   Final iterable bandwidth    9952.0 MiB/s 11639.10 MiB/s  ~17.0% improve
> 
> 
> Average results:              Before       After
>   pin-all is false
>   Downtime                    248.7 ms     191.1 ms        ~23.2% improve
>   Final iterable bandwidth    5605.1 MiB/s 9826.0 MiB/s    ~75.3% improve
> 
> Note: Final iterable bandwidth actually means the bandwidth during
> qemu_savevm_state_complete_precopy_iterable(). non-iterable data doesn't
> use RDMA Write.

So I left some comment in patch 2, but maybe I should just comment directly
here.. please check that first.

If you would agree with what I said there (and you should have noticed
another patch I sent days ago removing UNREGISTER path), I think what we
really need might be:

- Move RDMA draining from ram_save_iterate()/complete() into each time we
  finish scanning a round

- Remove transit_bitmap completely (if you see after applying your this
  series, the UNREGISTER should be the only user..), because with the
  correct draining IIUC we don't need this anymore

Thanks,

-- 
Peter Xu