[PATCH 0/2] migration/rdma: fix blocking readv and zero-page accounting

[PATCH 0/2] migration/rdma: fix blocking readv and zero-page accounting

[Bin Guo]

This small series fixes two independent issues in the RDMA migration
code path that were noticed while reviewing the QIOChannelRDMA behavior
and migration statistics during RDMA live migration.

Patch 1 makes QIOChannelRDMA's readv honor the blocking flag set via
io_set_blocking().  Previously, in blocking mode it returned immediately
when the receive buffer could not satisfy the request, diverging from
other QIOChannel implementations.  The fix loops on
qemu_rdma_exchange_recv() while the channel is in blocking mode, and
removes the now-stale XXX comments about unimplemented blocking
support.

Patch 2 fixes transfer accounting for the RDMA zero page compression
path.  When a zero page is detected, an RDMA_CONTROL_COMPRESS message
is sent instead of an RDMA Write, but mig_stats.rdma_bytes and
ram_transferred_add() were never updated, so migration progress and
bandwidth were undercounted.  The patch accounts the wire cost
(RDMAControlHeader + RDMACompress payload), matching how the non-RDMA
path accounts for zero page markers.

Both patches are small and self-contained; no behavior change is
expected outside the RDMA migration path.

Bin Guo (2):
  migration/rdma: honor blocking mode in QIOChannelRDMA readv
  migration/rdma: account transferred bytes for zero page compression

 migration/rdma.c | 62 ++++++++++++++++++++++--------------------------
 1 file changed, 29 insertions(+), 33 deletions(-)

-- 
2.50.1 (Apple Git-155)

[PATCH 1/2] migration/rdma: honor blocking mode in QIOChannelRDMA readv

[Bin Guo]

QIOChannelRDMA's readv ignored the blocking flag set via
io_set_blocking.  In blocking mode it returned immediately instead
of waiting for data, unlike other QIOChannel implementations.

Loop on qemu_rdma_exchange_recv() when the channel is blocking and
the receive buffer cannot satisfy the request.  Also remove the
stale XXX comments about unimplemented blocking support.

Signed-off-by: Bin Guo <guobin@linux.alibaba.com>
---
 migration/rdma.c | 46 +++++++++++++++++++++-------------------------
 1 file changed, 21 insertions(+), 25 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 3e37a1d440..201cb9eb12 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -388,7 +388,7 @@ struct QIOChannelRDMA {
     QIOChannel parent;
     RDMAContext *rdmain;
     RDMAContext *rdmaout;
-    bool blocking; /* XXX we don't actually honour this yet */
+    bool blocking;
 };
 
 /*
@@ -2710,32 +2710,29 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc,
             break;
         }
 
-
-        /* We've got nothing at all, so lets wait for
-         * more to arrive
-         */
-        ret = qemu_rdma_exchange_recv(rdma, &head, RDMA_CONTROL_QEMU_FILE,
-                                      errp);
-
-        if (ret < 0) {
-            rdma->errored = true;
-            return -1;
-        }
-
         /*
-         * SEND was received with new bytes, now try again.
+         * We've got nothing at all, so lets wait for
+         * more to arrive.
          */
-        len = qemu_rdma_fill(rdma, data, want, 0);
-        done += len;
-        want -= len;
-
-        /* Still didn't get enough, so lets just return */
-        if (want) {
-            if (done == 0) {
-                return QIO_CHANNEL_ERR_BLOCK;
-            } else {
-                break;
+        do {
+            ret = qemu_rdma_exchange_recv(rdma, &head,
+                                          RDMA_CONTROL_QEMU_FILE, errp);
+            if (ret < 0) {
+                rdma->errored = true;
+                return -1;
             }
+
+            /*
+             * SEND was received with new bytes, now try again.
+             */
+            len = qemu_rdma_fill(rdma, data, want, 0);
+            done += len;
+            want -= len;
+            data += len;
+        } while (want && rioc->blocking);
+
+        if (want && done == 0) {
+            return QIO_CHANNEL_ERR_BLOCK;
         }
     }
     return done;
@@ -2771,7 +2768,6 @@ static int qio_channel_rdma_set_blocking(QIOChannel *ioc,
                                          Error **errp)
 {
     QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc);
-    /* XXX we should make readv/writev actually honour this :-) */
     rioc->blocking = blocking;
     return 0;
 }
-- 
2.50.1 (Apple Git-155)

Re: [PATCH 1/2] migration/rdma: honor blocking mode in QIOChannelRDMA readv

[Peter Xu]

On Fri, May 29, 2026 at 03:04:07PM +0800, Bin Guo wrote:
> QIOChannelRDMA's readv ignored the blocking flag set via
> io_set_blocking.  In blocking mode it returned immediately instead
> of waiting for data, unlike other QIOChannel implementations.
> 
> Loop on qemu_rdma_exchange_recv() when the channel is blocking and
> the receive buffer cannot satisfy the request.  Also remove the
> stale XXX comments about unimplemented blocking support.

Zhijian would be the best to comment on this, but based on my limited
understanding, this patch made it the other way round: the channel didn't
respect blocking mode because it always blocks, not always non-block.

> 
> Signed-off-by: Bin Guo <guobin@linux.alibaba.com>
> ---
>  migration/rdma.c | 46 +++++++++++++++++++++-------------------------
>  1 file changed, 21 insertions(+), 25 deletions(-)
> 
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 3e37a1d440..201cb9eb12 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -388,7 +388,7 @@ struct QIOChannelRDMA {
>      QIOChannel parent;
>      RDMAContext *rdmain;
>      RDMAContext *rdmaout;
> -    bool blocking; /* XXX we don't actually honour this yet */
> +    bool blocking;
>  };
>  
>  /*
> @@ -2710,32 +2710,29 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc,
>              break;
>          }
>  
> -
> -        /* We've got nothing at all, so lets wait for
> -         * more to arrive
> -         */
> -        ret = qemu_rdma_exchange_recv(rdma, &head, RDMA_CONTROL_QEMU_FILE,
> -                                      errp);
> -
> -        if (ret < 0) {
> -            rdma->errored = true;
> -            return -1;
> -        }
> -
>          /*
> -         * SEND was received with new bytes, now try again.
> +         * We've got nothing at all, so lets wait for
> +         * more to arrive.
>           */
> -        len = qemu_rdma_fill(rdma, data, want, 0);
> -        done += len;
> -        want -= len;
> -
> -        /* Still didn't get enough, so lets just return */
> -        if (want) {
> -            if (done == 0) {
> -                return QIO_CHANNEL_ERR_BLOCK;
> -            } else {
> -                break;
> +        do {
> +            ret = qemu_rdma_exchange_recv(rdma, &head,
> +                                          RDMA_CONTROL_QEMU_FILE, errp);

IIUC this is the function that always blocks.  If to support non-blocking,
my understanding is we need to make this return immediately when no data.

Also, this is only the read path; there's also write.

I'm not sure if supporting non-blocking is trivial to RDMA..

Thanks,

> +            if (ret < 0) {
> +                rdma->errored = true;
> +                return -1;
>              }
> +
> +            /*
> +             * SEND was received with new bytes, now try again.
> +             */
> +            len = qemu_rdma_fill(rdma, data, want, 0);
> +            done += len;
> +            want -= len;
> +            data += len;
> +        } while (want && rioc->blocking);
> +
> +        if (want && done == 0) {
> +            return QIO_CHANNEL_ERR_BLOCK;
>          }
>      }
>      return done;
> @@ -2771,7 +2768,6 @@ static int qio_channel_rdma_set_blocking(QIOChannel *ioc,
>                                           Error **errp)
>  {
>      QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc);
> -    /* XXX we should make readv/writev actually honour this :-) */
>      rioc->blocking = blocking;
>      return 0;
>  }
> -- 
> 2.50.1 (Apple Git-155)
> 

-- 
Peter Xu

Re: [PATCH 1/2] migration/rdma: honor blocking mode in QIOChannelRDMA readv

[Bin Guo]

On Tue, 16 Jun 2026, Peter Xu wrote:
> this patch made it the other way round: the channel didn't respect
> blocking mode because it always blocks, not always non-block.

You're right, the commit message was misleading.

The function already blocks inside qemu_rdma_exchange_recv() when
waiting for the next RDMA SEND message.  What it didn't do was keep
blocking when the bytes from a *single* receive were not enough to
satisfy the full request -- it would return partial data instead of
waiting for more.

In blocking mode a QIOChannel readv should keep trying until the
request is satisfied (or an error occurs), matching the behaviour of
other QIOChannel implementations.  That's what the loop is for.

> Also, this is only the read path; there's also write.

The write path already loops over the full iov in
qio_channel_rdma_writev(), so it handles the blocking case
correctly.  Only read needed fixing.

I'll send a v2 with a corrected commit message.  Thanks for the
review.

Bin Guo

Re: [PATCH 1/2] migration/rdma: honor blocking mode in QIOChannelRDMA readv

[laiqi.gb via qemu development]

From: Bin Guo <guobin@linux.alibaba.com>

On Tue, 16 Jun 2026, Peter Xu wrote:
> this patch made it the other way round: the channel didn't respect
> blocking mode because it always blocks, not always non-block.

You're right, the commit message was misleading.

The function already blocks inside qemu_rdma_exchange_recv() when
waiting for the next RDMA SEND message.  What it didn't do was keep
blocking when the bytes from a *single* receive were not enough to
satisfy the full request -- it would return partial data instead of
waiting for more.

In blocking mode a QIOChannel readv should keep trying until the
request is satisfied (or an error occurs), matching the behaviour of
other QIOChannel implementations.  That's what the loop is for.

> Also, this is only the read path; there's also write.

The write path already loops over the full iov in
qio_channel_rdma_writev(), so it handles the blocking case
correctly.  Only read needed fixing.

I send a v2 with a corrected commit message.  Thanks for the
review.

Bin Guo

[PATCH 2/2] migration/rdma: account transferred bytes for zero page compression

[Bin Guo]

RDMA zero page compression sends an RDMA_CONTROL_COMPRESS message
instead of an RDMA Write, but the transfer accounting
(mig_stats.rdma_bytes and ram_transferred_add) was not updated.
This caused migration progress and bandwidth to be undercounted.

Account the wire cost (RDMAControlHeader + RDMACompress payload),
matching how the non-RDMA path accounts for zero page markers.

Signed-off-by: Bin Guo <guobin@linux.alibaba.com>
---
 migration/rdma.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 201cb9eb12..f08468881f 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -1930,15 +1930,15 @@ retry:
                 }
 
                 /*
-                 * TODO: Here we are sending something, but we are not
-                 * accounting for anything transferred.  The following is wrong:
-                 *
-                 * stat64_add(&mig_stats.rdma_bytes, sge.length);
-                 *
-                 * because we are using some kind of compression.  I
-                 * would think that head.len would be the more similar
-                 * thing to a correct value.
+                 * Account for the control message we just sent.
+                 * The actual bytes on the wire are the control header
+                 * plus the RDMACompress payload, not the original page
+                 * size (which was never transferred).
                  */
+                qatomic_add(&mig_stats.rdma_bytes,
+                            sizeof(RDMAControlHeader) + sizeof(RDMACompress));
+                ram_transferred_add(sizeof(RDMAControlHeader) +
+                                    sizeof(RDMACompress));
                 qatomic_add(&mig_stats.zero_pages,
                             sge.length / qemu_target_page_size());
                 return 1;
-- 
2.50.1 (Apple Git-155)

Re: [PATCH 0/2] migration/rdma: fix blocking readv and zero-page accounting

[Zhijian Li (Fujitsu)]

On 29/05/2026 15:04, Bin Guo wrote:
> This small series fixes two independent issues in the RDMA migration
> code path that were noticed while reviewing the QIOChannelRDMA behavior
> and migration statistics during RDMA live migration.
> 
> Patch 1 makes QIOChannelRDMA's readv honor the blocking flag set via
> io_set_blocking().  Previously, in blocking mode it returned immediately
> when the receive buffer could not satisfy the request, diverging from
> other QIOChannel implementations.  The fix loops on
> qemu_rdma_exchange_recv() while the channel is in blocking mode, and
> removes the now-stale XXX comments about unimplemented blocking
> support.
> 
> Patch 2 fixes transfer accounting for the RDMA zero page compression
> path.  When a zero page is detected, an RDMA_CONTROL_COMPRESS message
> is sent instead of an RDMA Write, but mig_stats.rdma_bytes and
> ram_transferred_add() were never updated, so migration progress and
> bandwidth were undercounted.  The patch accounts the wire cost
> (RDMAControlHeader + RDMACompress payload), matching how the non-RDMA
> path accounts for zero page markers.
> 
> Both patches are small and self-contained; no behavior change is
> expected outside the RDMA migration path.
> 
> Bin Guo (2):
>    migration/rdma: honor blocking mode in QIOChannelRDMA readv
>    migration/rdma: account transferred bytes for zero page compression

For the series

Reviewed-by: Li Zhijian <lizhijian@fujitsu.com>


> 
>   migration/rdma.c | 62 ++++++++++++++++++++++--------------------------
>   1 file changed, 29 insertions(+), 33 deletions(-)
>