[PATCH 0/2] migration/rdma: fix blocking readv and zero-page accounting

[PATCH 0/2] migration/rdma: fix blocking readv and zero-page accounting

[Bin Guo]

This small series fixes two independent issues in the RDMA migration
code path that were noticed while reviewing the QIOChannelRDMA behavior
and migration statistics during RDMA live migration.

Patch 1 makes QIOChannelRDMA's readv honor the blocking flag set via
io_set_blocking().  Previously, in blocking mode it returned immediately
when the receive buffer could not satisfy the request, diverging from
other QIOChannel implementations.  The fix loops on
qemu_rdma_exchange_recv() while the channel is in blocking mode, and
removes the now-stale XXX comments about unimplemented blocking
support.

Patch 2 fixes transfer accounting for the RDMA zero page compression
path.  When a zero page is detected, an RDMA_CONTROL_COMPRESS message
is sent instead of an RDMA Write, but mig_stats.rdma_bytes and
ram_transferred_add() were never updated, so migration progress and
bandwidth were undercounted.  The patch accounts the wire cost
(RDMAControlHeader + RDMACompress payload), matching how the non-RDMA
path accounts for zero page markers.

Both patches are small and self-contained; no behavior change is
expected outside the RDMA migration path.

Bin Guo (2):
  migration/rdma: honor blocking mode in QIOChannelRDMA readv
  migration/rdma: account transferred bytes for zero page compression

 migration/rdma.c | 62 ++++++++++++++++++++++--------------------------
 1 file changed, 29 insertions(+), 33 deletions(-)

-- 
2.50.1 (Apple Git-155)

[PATCH 1/2] migration/rdma: honor blocking mode in QIOChannelRDMA readv

[Bin Guo]

QIOChannelRDMA's readv ignored the blocking flag set via
io_set_blocking.  In blocking mode it returned immediately instead
of waiting for data, unlike other QIOChannel implementations.

Loop on qemu_rdma_exchange_recv() when the channel is blocking and
the receive buffer cannot satisfy the request.  Also remove the
stale XXX comments about unimplemented blocking support.

Signed-off-by: Bin Guo <guobin@linux.alibaba.com>
---
 migration/rdma.c | 46 +++++++++++++++++++++-------------------------
 1 file changed, 21 insertions(+), 25 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 3e37a1d440..201cb9eb12 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -388,7 +388,7 @@ struct QIOChannelRDMA {
     QIOChannel parent;
     RDMAContext *rdmain;
     RDMAContext *rdmaout;
-    bool blocking; /* XXX we don't actually honour this yet */
+    bool blocking;
 };
 
 /*
@@ -2710,32 +2710,29 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc,
             break;
         }
 
-
-        /* We've got nothing at all, so lets wait for
-         * more to arrive
-         */
-        ret = qemu_rdma_exchange_recv(rdma, &head, RDMA_CONTROL_QEMU_FILE,
-                                      errp);
-
-        if (ret < 0) {
-            rdma->errored = true;
-            return -1;
-        }
-
         /*
-         * SEND was received with new bytes, now try again.
+         * We've got nothing at all, so lets wait for
+         * more to arrive.
          */
-        len = qemu_rdma_fill(rdma, data, want, 0);
-        done += len;
-        want -= len;
-
-        /* Still didn't get enough, so lets just return */
-        if (want) {
-            if (done == 0) {
-                return QIO_CHANNEL_ERR_BLOCK;
-            } else {
-                break;
+        do {
+            ret = qemu_rdma_exchange_recv(rdma, &head,
+                                          RDMA_CONTROL_QEMU_FILE, errp);
+            if (ret < 0) {
+                rdma->errored = true;
+                return -1;
             }
+
+            /*
+             * SEND was received with new bytes, now try again.
+             */
+            len = qemu_rdma_fill(rdma, data, want, 0);
+            done += len;
+            want -= len;
+            data += len;
+        } while (want && rioc->blocking);
+
+        if (want && done == 0) {
+            return QIO_CHANNEL_ERR_BLOCK;
         }
     }
     return done;
@@ -2771,7 +2768,6 @@ static int qio_channel_rdma_set_blocking(QIOChannel *ioc,
                                          Error **errp)
 {
     QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc);
-    /* XXX we should make readv/writev actually honour this :-) */
     rioc->blocking = blocking;
     return 0;
 }
-- 
2.50.1 (Apple Git-155)

Re: [PATCH 1/2] migration/rdma: honor blocking mode in QIOChannelRDMA readv

[Peter Xu]

On Fri, May 29, 2026 at 03:04:07PM +0800, Bin Guo wrote:
> QIOChannelRDMA's readv ignored the blocking flag set via
> io_set_blocking.  In blocking mode it returned immediately instead
> of waiting for data, unlike other QIOChannel implementations.
> 
> Loop on qemu_rdma_exchange_recv() when the channel is blocking and
> the receive buffer cannot satisfy the request.  Also remove the
> stale XXX comments about unimplemented blocking support.

Zhijian would be the best to comment on this, but based on my limited
understanding, this patch made it the other way round: the channel didn't
respect blocking mode because it always blocks, not always non-block.

> 
> Signed-off-by: Bin Guo <guobin@linux.alibaba.com>
> ---
>  migration/rdma.c | 46 +++++++++++++++++++++-------------------------
>  1 file changed, 21 insertions(+), 25 deletions(-)
> 
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 3e37a1d440..201cb9eb12 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -388,7 +388,7 @@ struct QIOChannelRDMA {
>      QIOChannel parent;
>      RDMAContext *rdmain;
>      RDMAContext *rdmaout;
> -    bool blocking; /* XXX we don't actually honour this yet */
> +    bool blocking;
>  };
>  
>  /*
> @@ -2710,32 +2710,29 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc,
>              break;
>          }
>  
> -
> -        /* We've got nothing at all, so lets wait for
> -         * more to arrive
> -         */
> -        ret = qemu_rdma_exchange_recv(rdma, &head, RDMA_CONTROL_QEMU_FILE,
> -                                      errp);
> -
> -        if (ret < 0) {
> -            rdma->errored = true;
> -            return -1;
> -        }
> -
>          /*
> -         * SEND was received with new bytes, now try again.
> +         * We've got nothing at all, so lets wait for
> +         * more to arrive.
>           */
> -        len = qemu_rdma_fill(rdma, data, want, 0);
> -        done += len;
> -        want -= len;
> -
> -        /* Still didn't get enough, so lets just return */
> -        if (want) {
> -            if (done == 0) {
> -                return QIO_CHANNEL_ERR_BLOCK;
> -            } else {
> -                break;
> +        do {
> +            ret = qemu_rdma_exchange_recv(rdma, &head,
> +                                          RDMA_CONTROL_QEMU_FILE, errp);

IIUC this is the function that always blocks.  If to support non-blocking,
my understanding is we need to make this return immediately when no data.

Also, this is only the read path; there's also write.

I'm not sure if supporting non-blocking is trivial to RDMA..

Thanks,

> +            if (ret < 0) {
> +                rdma->errored = true;
> +                return -1;
>              }
> +
> +            /*
> +             * SEND was received with new bytes, now try again.
> +             */
> +            len = qemu_rdma_fill(rdma, data, want, 0);
> +            done += len;
> +            want -= len;
> +            data += len;
> +        } while (want && rioc->blocking);
> +
> +        if (want && done == 0) {
> +            return QIO_CHANNEL_ERR_BLOCK;
>          }
>      }
>      return done;
> @@ -2771,7 +2768,6 @@ static int qio_channel_rdma_set_blocking(QIOChannel *ioc,
>                                           Error **errp)
>  {
>      QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc);
> -    /* XXX we should make readv/writev actually honour this :-) */
>      rioc->blocking = blocking;
>      return 0;
>  }
> -- 
> 2.50.1 (Apple Git-155)
> 

-- 
Peter Xu

Re: [PATCH 1/2] migration/rdma: honor blocking mode in QIOChannelRDMA readv

[Bin Guo]

On Tue, 16 Jun 2026, Peter Xu wrote:
> this patch made it the other way round: the channel didn't respect
> blocking mode because it always blocks, not always non-block.

You're right, the commit message was misleading.

The function already blocks inside qemu_rdma_exchange_recv() when
waiting for the next RDMA SEND message.  What it didn't do was keep
blocking when the bytes from a *single* receive were not enough to
satisfy the full request -- it would return partial data instead of
waiting for more.

In blocking mode a QIOChannel readv should keep trying until the
request is satisfied (or an error occurs), matching the behaviour of
other QIOChannel implementations.  That's what the loop is for.

> Also, this is only the read path; there's also write.

The write path already loops over the full iov in
qio_channel_rdma_writev(), so it handles the blocking case
correctly.  Only read needed fixing.

I'll send a v2 with a corrected commit message.  Thanks for the
review.

Bin Guo

Re: [PATCH 1/2] migration/rdma: honor blocking mode in QIOChannelRDMA readv

[laiqi.gb via qemu development]

From: Bin Guo <guobin@linux.alibaba.com>

On Tue, 16 Jun 2026, Peter Xu wrote:
> this patch made it the other way round: the channel didn't respect
> blocking mode because it always blocks, not always non-block.

You're right, the commit message was misleading.

The function already blocks inside qemu_rdma_exchange_recv() when
waiting for the next RDMA SEND message.  What it didn't do was keep
blocking when the bytes from a *single* receive were not enough to
satisfy the full request -- it would return partial data instead of
waiting for more.

In blocking mode a QIOChannel readv should keep trying until the
request is satisfied (or an error occurs), matching the behaviour of
other QIOChannel implementations.  That's what the loop is for.

> Also, this is only the read path; there's also write.

The write path already loops over the full iov in
qio_channel_rdma_writev(), so it handles the blocking case
correctly.  Only read needed fixing.

I send a v2 with a corrected commit message.  Thanks for the
review.

Bin Guo

[PATCH 2/2] migration/rdma: account transferred bytes for zero page compression

[Bin Guo]

RDMA zero page compression sends an RDMA_CONTROL_COMPRESS message
instead of an RDMA Write, but the transfer accounting
(mig_stats.rdma_bytes and ram_transferred_add) was not updated.
This caused migration progress and bandwidth to be undercounted.

Account the wire cost (RDMAControlHeader + RDMACompress payload),
matching how the non-RDMA path accounts for zero page markers.

Signed-off-by: Bin Guo <guobin@linux.alibaba.com>
---
 migration/rdma.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 201cb9eb12..f08468881f 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -1930,15 +1930,15 @@ retry:
                 }
 
                 /*
-                 * TODO: Here we are sending something, but we are not
-                 * accounting for anything transferred.  The following is wrong:
-                 *
-                 * stat64_add(&mig_stats.rdma_bytes, sge.length);
-                 *
-                 * because we are using some kind of compression.  I
-                 * would think that head.len would be the more similar
-                 * thing to a correct value.
+                 * Account for the control message we just sent.
+                 * The actual bytes on the wire are the control header
+                 * plus the RDMACompress payload, not the original page
+                 * size (which was never transferred).
                  */
+                qatomic_add(&mig_stats.rdma_bytes,
+                            sizeof(RDMAControlHeader) + sizeof(RDMACompress));
+                ram_transferred_add(sizeof(RDMAControlHeader) +
+                                    sizeof(RDMACompress));
                 qatomic_add(&mig_stats.zero_pages,
                             sge.length / qemu_target_page_size());
                 return 1;
-- 
2.50.1 (Apple Git-155)

Re: [PATCH 0/2] migration/rdma: fix blocking readv and zero-page accounting

[Zhijian Li (Fujitsu)]

On 29/05/2026 15:04, Bin Guo wrote:
> This small series fixes two independent issues in the RDMA migration
> code path that were noticed while reviewing the QIOChannelRDMA behavior
> and migration statistics during RDMA live migration.
> 
> Patch 1 makes QIOChannelRDMA's readv honor the blocking flag set via
> io_set_blocking().  Previously, in blocking mode it returned immediately
> when the receive buffer could not satisfy the request, diverging from
> other QIOChannel implementations.  The fix loops on
> qemu_rdma_exchange_recv() while the channel is in blocking mode, and
> removes the now-stale XXX comments about unimplemented blocking
> support.
> 
> Patch 2 fixes transfer accounting for the RDMA zero page compression
> path.  When a zero page is detected, an RDMA_CONTROL_COMPRESS message
> is sent instead of an RDMA Write, but mig_stats.rdma_bytes and
> ram_transferred_add() were never updated, so migration progress and
> bandwidth were undercounted.  The patch accounts the wire cost
> (RDMAControlHeader + RDMACompress payload), matching how the non-RDMA
> path accounts for zero page markers.
> 
> Both patches are small and self-contained; no behavior change is
> expected outside the RDMA migration path.
> 
> Bin Guo (2):
>    migration/rdma: honor blocking mode in QIOChannelRDMA readv
>    migration/rdma: account transferred bytes for zero page compression

For the series

Reviewed-by: Li Zhijian <lizhijian@fujitsu.com>


> 
>   migration/rdma.c | 62 ++++++++++++++++++++++--------------------------
>   1 file changed, 29 insertions(+), 33 deletions(-)
> 

[PATCH v2 1/2] migration/rdma: honor blocking mode in QIOChannelRDMA readv

[Bin Guo]

QIOChannelRDMA's readv already blocks inside qemu_rdma_exchange_recv()
when waiting for the next RDMA SEND message.  What it did not do was
keep blocking when the bytes from a single receive were insufficient
to satisfy the full request -- it returned partial data (or EAGAIN)
instead of waiting for more.

Loop on qemu_rdma_exchange_recv() when the channel is blocking and
the receive buffer cannot satisfy the request.  This matches the
behaviour of other QIOChannel implementations, which block until
the full request is satisfied (or an error occurs).

Also remove the stale XXX comments about unimplemented blocking support.

Reviewed-by: Li Zhijian <lizhijian@fujitsu.com>
Signed-off-by: Bin Guo <guobin@linux.alibaba.com>
---
 migration/rdma.c | 46 +++++++++++++++++++++-------------------------
 1 file changed, 21 insertions(+), 25 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 3e37a1d440..201cb9eb12 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -388,7 +388,7 @@ struct QIOChannelRDMA {
     QIOChannel parent;
     RDMAContext *rdmain;
     RDMAContext *rdmaout;
-    bool blocking; /* XXX we don't actually honour this yet */
+    bool blocking;
 };
 
 /*
@@ -2710,32 +2710,29 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc,
             break;
         }
 
-
-        /* We've got nothing at all, so lets wait for
-         * more to arrive
-         */
-        ret = qemu_rdma_exchange_recv(rdma, &head, RDMA_CONTROL_QEMU_FILE,
-                                      errp);
-
-        if (ret < 0) {
-            rdma->errored = true;
-            return -1;
-        }
-
         /*
-         * SEND was received with new bytes, now try again.
+         * We've got nothing at all, so lets wait for
+         * more to arrive.
          */
-        len = qemu_rdma_fill(rdma, data, want, 0);
-        done += len;
-        want -= len;
-
-        /* Still didn't get enough, so lets just return */
-        if (want) {
-            if (done == 0) {
-                return QIO_CHANNEL_ERR_BLOCK;
-            } else {
-                break;
+        do {
+            ret = qemu_rdma_exchange_recv(rdma, &head,
+                                          RDMA_CONTROL_QEMU_FILE, errp);
+            if (ret < 0) {
+                rdma->errored = true;
+                return -1;
             }
+
+            /*
+             * SEND was received with new bytes, now try again.
+             */
+            len = qemu_rdma_fill(rdma, data, want, 0);
+            done += len;
+            want -= len;
+            data += len;
+        } while (want && rioc->blocking);
+
+        if (want && done == 0) {
+            return QIO_CHANNEL_ERR_BLOCK;
         }
     }
     return done;
@@ -2771,7 +2768,6 @@ static int qio_channel_rdma_set_blocking(QIOChannel *ioc,
                                          Error **errp)
 {
     QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc);
-    /* XXX we should make readv/writev actually honour this :-) */
     rioc->blocking = blocking;
     return 0;
 }
-- 
2.50.1 (Apple Git-155)

[PATCH v2 2/2] migration/rdma: account transferred bytes for zero page compression

[Bin Guo]

RDMA zero page compression sends an RDMA_CONTROL_COMPRESS message
instead of an RDMA Write, but the transfer accounting
(mig_stats.rdma_bytes and ram_transferred_add) was not updated.
This caused migration progress and bandwidth to be undercounted.

Account the wire cost (RDMAControlHeader + RDMACompress payload),
matching how the non-RDMA path accounts for zero page markers.

Reviewed-by: Li Zhijian <lizhijian@fujitsu.com>
Signed-off-by: Bin Guo <guobin@linux.alibaba.com>
---
 migration/rdma.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 201cb9eb12..f08468881f 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -1930,15 +1930,15 @@ retry:
                 }
 
                 /*
-                 * TODO: Here we are sending something, but we are not
-                 * accounting for anything transferred.  The following is wrong:
-                 *
-                 * stat64_add(&mig_stats.rdma_bytes, sge.length);
-                 *
-                 * because we are using some kind of compression.  I
-                 * would think that head.len would be the more similar
-                 * thing to a correct value.
+                 * Account for the control message we just sent.
+                 * The actual bytes on the wire are the control header
+                 * plus the RDMACompress payload, not the original page
+                 * size (which was never transferred).
                  */
+                qatomic_add(&mig_stats.rdma_bytes,
+                            sizeof(RDMAControlHeader) + sizeof(RDMACompress));
+                ram_transferred_add(sizeof(RDMAControlHeader) +
+                                    sizeof(RDMACompress));
                 qatomic_add(&mig_stats.zero_pages,
                             sge.length / qemu_target_page_size());
                 return 1;
-- 
2.50.1 (Apple Git-155)

Re: [PATCH v2 1/2] migration/rdma: honor blocking mode in QIOChannelRDMA readv

[Peter Xu]

On Wed, Jun 17, 2026 at 02:43:31PM +0800, Bin Guo wrote:
> QIOChannelRDMA's readv already blocks inside qemu_rdma_exchange_recv()
> when waiting for the next RDMA SEND message.  What it did not do was
> keep blocking when the bytes from a single receive were insufficient
> to satisfy the full request -- it returned partial data (or EAGAIN)
> instead of waiting for more.

Sorry if I wasn't clear when commenting, but what I meant is, this is
correct behavior for blocking/nonblocking.

If io_readv() reads partial, IIUC returning how much it reads is the
correct behavior.  You can refer to qio_channel_socket_readv().

Here, RDMA doesn't respect blocking is because it _always_ blocks.  That
is, qemu_rdma_exchange_recv() always will block even if blocking=false.

I believe it will normally stop working when it's used in a coroutine,
because normally we rely on non-blocking to allow it fallback to caller
then things like qio_channel_wait_cond() will yield if it's coroutine.

I recall RDMA still works only because it has some internal hack to yield,
maybe that's qemu_rdma_wait_comp_channel(), but I really don't know RDMA
well, at least so far.. maybe I should try to improve at some point.
Before that, it would be good Zhijian can have another look on this.

Let me loop in Dan too.

Thanks,

> 
> Loop on qemu_rdma_exchange_recv() when the channel is blocking and
> the receive buffer cannot satisfy the request.  This matches the
> behaviour of other QIOChannel implementations, which block until
> the full request is satisfied (or an error occurs).
> 
> Also remove the stale XXX comments about unimplemented blocking support.
> 
> Reviewed-by: Li Zhijian <lizhijian@fujitsu.com>
> Signed-off-by: Bin Guo <guobin@linux.alibaba.com>
> ---
>  migration/rdma.c | 46 +++++++++++++++++++++-------------------------
>  1 file changed, 21 insertions(+), 25 deletions(-)
> 
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 3e37a1d440..201cb9eb12 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -388,7 +388,7 @@ struct QIOChannelRDMA {
>      QIOChannel parent;
>      RDMAContext *rdmain;
>      RDMAContext *rdmaout;
> -    bool blocking; /* XXX we don't actually honour this yet */
> +    bool blocking;
>  };
>  
>  /*
> @@ -2710,32 +2710,29 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc,
>              break;
>          }
>  
> -
> -        /* We've got nothing at all, so lets wait for
> -         * more to arrive
> -         */
> -        ret = qemu_rdma_exchange_recv(rdma, &head, RDMA_CONTROL_QEMU_FILE,
> -                                      errp);
> -
> -        if (ret < 0) {
> -            rdma->errored = true;
> -            return -1;
> -        }
> -
>          /*
> -         * SEND was received with new bytes, now try again.
> +         * We've got nothing at all, so lets wait for
> +         * more to arrive.
>           */
> -        len = qemu_rdma_fill(rdma, data, want, 0);
> -        done += len;
> -        want -= len;
> -
> -        /* Still didn't get enough, so lets just return */
> -        if (want) {
> -            if (done == 0) {
> -                return QIO_CHANNEL_ERR_BLOCK;
> -            } else {
> -                break;
> +        do {
> +            ret = qemu_rdma_exchange_recv(rdma, &head,
> +                                          RDMA_CONTROL_QEMU_FILE, errp);
> +            if (ret < 0) {
> +                rdma->errored = true;
> +                return -1;
>              }
> +
> +            /*
> +             * SEND was received with new bytes, now try again.
> +             */
> +            len = qemu_rdma_fill(rdma, data, want, 0);
> +            done += len;
> +            want -= len;
> +            data += len;
> +        } while (want && rioc->blocking);
> +
> +        if (want && done == 0) {
> +            return QIO_CHANNEL_ERR_BLOCK;
>          }
>      }
>      return done;
> @@ -2771,7 +2768,6 @@ static int qio_channel_rdma_set_blocking(QIOChannel *ioc,
>                                           Error **errp)
>  {
>      QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc);
> -    /* XXX we should make readv/writev actually honour this :-) */
>      rioc->blocking = blocking;
>      return 0;
>  }
> -- 
> 2.50.1 (Apple Git-155)
> 

-- 
Peter Xu

Re: [PATCH v2 1/2] migration/rdma: honor blocking mode in QIOChannelRDMA readv

[laiqi.gb via qemu development]

From: Bin Guo <guobin@linux.alibaba.com>

On Wed, 18 Jun 2026, Peter Xu wrote:
> If io_readv() reads partial, IIUC returning how much it reads is the
> correct behavior. You can refer to qio_channel_socket_readv().

You're right.  I misread the QIOChannel contract.  readv() should
return whatever is available, not loop until the full request is
satisfied.  The socket implementation confirms that.

> RDMA doesn't respect blocking is because it _always_ blocks. That is,
> qemu_rdma_exchange_recv() always will block even if blocking=false.

Yes, that's the real issue.  My v2 patch only changed the blocking
path and left qemu_rdma_exchange_recv() blocking unconditionally in
non-blocking mode, which is wrong and can break coroutine use cases.

I'll drop [PATCH 1/2].  The blocking/non-blocking handling in RDMA
needs a more fundamental rework than this series attempted.

> Also, I'd like Zhijian to take another look, and cc Dan too.

Understood.

For [PATCH 2/2] "migration/rdma: account transferred bytes for zero
page compression", it is an independent fix for RDMA migration
accounting and unrelated to the blocking-mode discussion.  I'd like
to keep that one separate and submit it on its own if there are no
further comments.

Thanks,
Bin Guo