[PATCH 0/3] migration/rdma: Misc RDMA migration fixes

[PATCH 0/3] migration/rdma: Misc RDMA migration fixes

[Yanfei Xu]

This series fixes three small RDMA migration issues.

Yanfei Xu (3):
  migration/rdma: Fix write-side shutdown
  migration/rdma: Post initial receive before accepting
  migration/rdma: Retry control sends on full queue

 migration/rdma.c | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)

[PATCH 1/3] migration/rdma: Fix write-side shutdown

[Yanfei Xu]

qio_channel_rdma_shutdown() reads rdmain for both the input and
output contexts.  As a result, a write-side shutdown marks the input
context as errored while leaving the output context active.  A shutdown
of both directions also fails to stop the output context.

Read rdmaout from the output context so each shutdown direction marks
the corresponding RDMA context as errored.

Fixes: 54db882f073a ("migration: implement the shutdown for RDMA QIOChannel")
Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
---
 migration/rdma.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 438419d189..e17ace62b0 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -2982,7 +2982,7 @@ qio_channel_rdma_shutdown(QIOChannel *ioc,
     RCU_READ_LOCK_GUARD();
 
     rdmain = qatomic_rcu_read(&rioc->rdmain);
-    rdmaout = qatomic_rcu_read(&rioc->rdmain);
+    rdmaout = qatomic_rcu_read(&rioc->rdmaout);
 
     switch (how) {
     case QIO_CHANNEL_SHUTDOWN_READ:
-- 
2.20.1

Re: [PATCH 1/3] migration/rdma: Fix write-side shutdown

[Fabiano Rosas]

"Yanfei Xu" <yanfei.xu@bytedance.com> writes:

> qio_channel_rdma_shutdown() reads rdmain for both the input and
> output contexts.  As a result, a write-side shutdown marks the input
> context as errored while leaving the output context active.  A shutdown
> of both directions also fails to stop the output context.
>
> Read rdmaout from the output context so each shutdown direction marks
> the corresponding RDMA context as errored.
>
> Fixes: 54db882f073a ("migration: implement the shutdown for RDMA QIOChannel")
> Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
> ---
>  migration/rdma.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 438419d189..e17ace62b0 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -2982,7 +2982,7 @@ qio_channel_rdma_shutdown(QIOChannel *ioc,
>      RCU_READ_LOCK_GUARD();
>  
>      rdmain = qatomic_rcu_read(&rioc->rdmain);
> -    rdmaout = qatomic_rcu_read(&rioc->rdmain);
> +    rdmaout = qatomic_rcu_read(&rioc->rdmaout);
>  
>      switch (how) {
>      case QIO_CHANNEL_SHUTDOWN_READ:

Reviewed-by: Fabiano Rosas <farosas@suse.de>

Re: [PATCH 1/3] migration/rdma: Fix write-side shutdown

[Peter Xu]

On Mon, Aug 17, 2026 at 06:51:15PM +0800, Yanfei Xu wrote:
> qio_channel_rdma_shutdown() reads rdmain for both the input and
> output contexts.  As a result, a write-side shutdown marks the input
> context as errored while leaving the output context active.  A shutdown
> of both directions also fails to stop the output context.
> 
> Read rdmaout from the output context so each shutdown direction marks
> the corresponding RDMA context as errored.
> 
> Fixes: 54db882f073a ("migration: implement the shutdown for RDMA QIOChannel")
> Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
> ---
>  migration/rdma.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 438419d189..e17ace62b0 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -2982,7 +2982,7 @@ qio_channel_rdma_shutdown(QIOChannel *ioc,
>      RCU_READ_LOCK_GUARD();
>  
>      rdmain = qatomic_rcu_read(&rioc->rdmain);
> -    rdmaout = qatomic_rcu_read(&rioc->rdmain);
> +    rdmaout = qatomic_rcu_read(&rioc->rdmaout);
>  
>      switch (how) {
>      case QIO_CHANNEL_SHUTDOWN_READ:
> -- 
> 2.20.1
> 

I recall there was a patch like this somewhere, looks like we missed that..

Reviewed-by: Peter Xu <peterx@redhat.com>

-- 
Peter Xu

[PATCH 2/3] migration/rdma: Post initial receive before accepting

[Yanfei Xu]

The source return-path thread can send READY before the destination has
posted its initial receive.  Since RNR retries default to zero, the send
can fail and abort migration.

Post the initial receive before rdma_accept() to close this window.

Fixes: 55cc1b5937a8 ("migration: create a dedicated connection for rdma return path")
Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
---
 migration/rdma.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index e17ace62b0..6e8436ccc1 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3263,6 +3263,12 @@ static int qemu_rdma_accept(RDMAContext *rdma)
         }
     }
 
+    ret = qemu_rdma_post_recv_control(rdma, RDMA_WRID_READY, &err);
+    if (ret < 0) {
+        error_report_err(err);
+        goto err_rdma_dest_wait;
+    }
+
     /* Accept the second connection request for return path */
     if ((migrate_postcopy() || migrate_return_path())
         && !rdma->is_return_path) {
@@ -3295,12 +3301,6 @@ static int qemu_rdma_accept(RDMAContext *rdma)
     rdma_ack_cm_event(cm_event);
     rdma->connected = true;
 
-    ret = qemu_rdma_post_recv_control(rdma, RDMA_WRID_READY, &err);
-    if (ret < 0) {
-        error_report_err(err);
-        goto err_rdma_dest_wait;
-    }
-
     qemu_rdma_dump_gid("dest_connect", rdma->cm_id);
 
     return 0;
-- 
2.20.1

Re: [PATCH 2/3] migration/rdma: Post initial receive before accepting

[Peter Xu]

On Mon, Aug 17, 2026 at 06:51:16PM +0800, Yanfei Xu wrote:
> The source return-path thread can send READY before the destination has
> posted its initial receive.  Since RNR retries default to zero, the send
> can fail and abort migration.
> 
> Post the initial receive before rdma_accept() to close this window.
> 
> Fixes: 55cc1b5937a8 ("migration: create a dedicated connection for rdma return path")
> Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>

This one looks correct to me,

Reviewed-by: Peter Xu <peterx@redhat.com>

Do you have way to reproduce it, or is this theory (or can be reproduced by
adding delays)?

> ---
>  migration/rdma.c | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/migration/rdma.c b/migration/rdma.c
> index e17ace62b0..6e8436ccc1 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -3263,6 +3263,12 @@ static int qemu_rdma_accept(RDMAContext *rdma)
>          }
>      }
>  
> +    ret = qemu_rdma_post_recv_control(rdma, RDMA_WRID_READY, &err);
> +    if (ret < 0) {
> +        error_report_err(err);
> +        goto err_rdma_dest_wait;
> +    }
> +
>      /* Accept the second connection request for return path */
>      if ((migrate_postcopy() || migrate_return_path())
>          && !rdma->is_return_path) {
> @@ -3295,12 +3301,6 @@ static int qemu_rdma_accept(RDMAContext *rdma)
>      rdma_ack_cm_event(cm_event);
>      rdma->connected = true;
>  
> -    ret = qemu_rdma_post_recv_control(rdma, RDMA_WRID_READY, &err);
> -    if (ret < 0) {
> -        error_report_err(err);
> -        goto err_rdma_dest_wait;
> -    }
> -
>      qemu_rdma_dump_gid("dest_connect", rdma->cm_id);
>  
>      return 0;
> -- 
> 2.20.1
> 

-- 
Peter Xu

[PATCH 3/3] migration/rdma: Retry control sends on full queue

[Yanfei Xu]

RAM writes and control messages share the send queue.  If
outstanding writes fill it, RDMA writes drain a completion and retry,
but control sends fail the migration.

Drain one outstanding write and retry the control send on ENOMEM.

Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
---
 migration/rdma.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 6e8436ccc1..d1f44a5f55 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -1572,9 +1572,19 @@ static int qemu_rdma_post_send_control(RDMAContext *rdma, uint8_t *buf,
         memcpy(wr->control + sizeof(RDMAControlHeader), buf, head->len);
     }
 
-
+retry:
     ret = ibv_post_send(rdma->qp, &send_wr, &bad_wr);
 
+    if (ret == ENOMEM && rdma->nb_sent) {
+        ret = qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, NULL);
+        if (ret < 0) {
+            error_setg(errp, "rdma migration: failed to make room for "
+                       "control send");
+            return -1;
+        }
+        goto retry;
+    }
+
     if (ret > 0) {
         error_setg(errp, "Failed to use post IB SEND for control");
         return -1;
-- 
2.20.1

Re: [PATCH 3/3] migration/rdma: Retry control sends on full queue

[Peter Xu]

On Mon, Aug 17, 2026 at 06:51:17PM +0800, Yanfei Xu wrote:
> RAM writes and control messages share the send queue.  If
> outstanding writes fill it, RDMA writes drain a completion and retry,
> but control sends fail the migration.
> 
> Drain one outstanding write and retry the control send on ENOMEM.
> 
> Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
> ---
>  migration/rdma.c | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 6e8436ccc1..d1f44a5f55 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -1572,9 +1572,19 @@ static int qemu_rdma_post_send_control(RDMAContext *rdma, uint8_t *buf,
>          memcpy(wr->control + sizeof(RDMAControlHeader), buf, head->len);
>      }
>  
> -
> +retry:
>      ret = ibv_post_send(rdma->qp, &send_wr, &bad_wr);
>  
> +    if (ret == ENOMEM && rdma->nb_sent) {
> +        ret = qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, NULL);
> +        if (ret < 0) {
> +            error_setg(errp, "rdma migration: failed to make room for "
> +                       "control send");
> +            return -1;
> +        }
> +        goto retry;
> +    }

Looks also correct, but two questions:

- Should we provide a helper instead of duplicating the WRITE op handling?
  I believe only WRITE wrids can be on the fly.

- Could ENOMEM be returned when nb_sent==0?  If that check applies to WRITE
  path too?

Thanks,

> +
>      if (ret > 0) {
>          error_setg(errp, "Failed to use post IB SEND for control");
>          return -1;
> -- 
> 2.20.1
> 

-- 
Peter Xu