Re: [PATCH v3] migration/rdma: add x-rdma-chunk-size parameter

[Peter Xu <peterx@redhat.com>]

On Tue, Mar 31, 2026 at 06:33:23PM +0800, Zhang, GuoQing (Sam) wrote:
> 
> On 2026/3/31 11:30, Zhijian Li (Fujitsu) wrote:
> > [Some people who received this message don't often get email from lizhijian@fujitsu.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> > 
> > On 31/03/2026 00:10, Peter Xu wrote:
> > > Hi, Samuel,
> > > 
> > > On Fri, Mar 27, 2026 at 02:50:06PM +0800, Samuel Zhang wrote:
> > > > The default 1MB RDMA chunk size causes slow live migration because
> > > > each chunk triggers a write_flush (ibv_post_send). For 8GB RAM,
> > > > 1MB chunk size produces ~15000 flushes vs ~3700 with 1024MB chunk size.
> > > > 
> > > > Add x-rdma-chunk-size parameter to configure the RDMA chunk size for
> > > > faster migration.
> > > > Usage: `migrate_set_parameter x-rdma-chunk-size 1024M`
> > > > 
> > > > Performance with RDMA live migration of 8GB RAM VM:
> > > > 
> > > > | x-rdma-chunk-size (B) | time (s) | throughput (MB/s) |
> > > > |-----------------------|----------|-------------------|
> > > > | 1M (default)          | 37.915   |  1,007            |
> > > This is the default. It surprised me a bit knowing it can only reach 1GB/s
> > > throughput with the current code base.  Do you know why?  I thought RDMA
> > > should be much faster than this on throughput with whatever hardware setup.
> > 
> > Regarding the baseline performance, Samuel's numbers look reasonable. I checked
> > some of my old test data on a ConnectX-4 Lx card years ago, and the throughput
> > was around 10 Gbps (~1.25 GB/s), which is consistent with the 1 GB/s he reported.
> > 
> > > > | 32M                   | 17.880   |  2,260            |
> > > > | 1024M                 |  4.368   | 17,529            |
> > My guess for the dramatic performance improvement is that a larger chunk size
> > allows qemu_rdma_write() to batch more *contiguous dirty pages* into a single,
> > more efficient RDMA send operation.
> 
> The `throughput` data is collected from `info migrate` qemu monitor command
> after live-migration.
> 
> Yes, Zhijian is right. As each chunk triggers a write_flush and each flush
> involves posting an RDMA WRITE and WAITING for completion, there's software
> overhead here.
> 
> For 8GB RAM VM migration, 1MB chunk size produces ~15000 flushes. The
> software overhead adds up and prevents the RDMA hardware from sustaining
> high throughput.
> 
> When chunk size is 1GB, there are ~3700 flushes. Reduced flush count means
> reduced software overhead and improved overall throughput.
> 

OK, thanks both.

> 
> > 
> > Is there any workloads running on the guest during the migration, or just an idle guest? @Samuel
> 
> 
> The guest is idle when I test the migration and collect the data.
> 
> 
> > 
> > Given the significant benefit and the fact that the patch itself is straightforward,
> > I think it's a worthwhile addition.
> > 
> > Acked-by: Li Zhijian <lizhijian@fujitsu.com>
> 
> 
> Thank you for the ack, Zhijian!
> 
> 
> > 
> > 
> > 
> > > > Signed-off-by: Samuel Zhang <guoqing.zhang@amd.com>
> > > One thing to mention is RDMA migration is in odd-fixes stage, actually it
> > > doesn't have a real maintainer so it is kind of "orphaned".  In this case,
> > > I actually won't suggest we add any new knobs for performance reasons.
> > > 
> > > Do you have a strong reason to propose this patch to land upstream?  Is it
> > > used in production systems and it solves some real problems for you?
> 
> 
> We have VMs with large RAM and find TCP live-migration is not fast enough
> and expect RDMA migration can be faster.
> 
> But we found the rdma mode migration speed is slower than tcp mode. See
> following data.
> 
> 
> 8GB RAM idle VM live-migration performance:
> | transport mode       | time (s) | throughput (MB/s) |
> |----------------------|----------|-------------------|
> | TCP                  | 36.89    |  1,081            |

What is the NIC setup?  Did you try to enable multifd to offload zeropage
detections?  Or is that not feasible due to some reason?

> | RDMA, 1MB chunk size | 37.915   |  1,007            |
> | RDMA, 1GB chunk size |  4.368   | 17,529            |
> 
> This patch allows us to use larger chunk size for faster RDMA migration.

Sure, Zhijian's point is reasonable.  If he's fine, I'm OK.

Thanks,

> 
> 
> Regards
> Sam
> 
> 
> > > 
> > > I also wonder what Zhijian would say on this.
> > > 
> > > Thanks,
> > > 
> 

-- 
Peter Xu