[PATCH 0/3] migration/rdma: Misc RDMA migration fixes

[PATCH 0/3] migration/rdma: Misc RDMA migration fixes

[Yanfei Xu]

This series fixes three small RDMA migration issues.

Yanfei Xu (3):
  migration/rdma: Fix write-side shutdown
  migration/rdma: Post initial receive before accepting
  migration/rdma: Retry control sends on full queue

 migration/rdma.c | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)

[PATCH 1/3] migration/rdma: Fix write-side shutdown

[Yanfei Xu]

qio_channel_rdma_shutdown() reads rdmain for both the input and
output contexts.  As a result, a write-side shutdown marks the input
context as errored while leaving the output context active.  A shutdown
of both directions also fails to stop the output context.

Read rdmaout from the output context so each shutdown direction marks
the corresponding RDMA context as errored.

Fixes: 54db882f073a ("migration: implement the shutdown for RDMA QIOChannel")
Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
---
 migration/rdma.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 438419d189..e17ace62b0 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -2982,7 +2982,7 @@ qio_channel_rdma_shutdown(QIOChannel *ioc,
     RCU_READ_LOCK_GUARD();
 
     rdmain = qatomic_rcu_read(&rioc->rdmain);
-    rdmaout = qatomic_rcu_read(&rioc->rdmain);
+    rdmaout = qatomic_rcu_read(&rioc->rdmaout);
 
     switch (how) {
     case QIO_CHANNEL_SHUTDOWN_READ:
-- 
2.20.1

Re: [PATCH 1/3] migration/rdma: Fix write-side shutdown

[Fabiano Rosas]

"Yanfei Xu" <yanfei.xu@bytedance.com> writes:

> qio_channel_rdma_shutdown() reads rdmain for both the input and
> output contexts.  As a result, a write-side shutdown marks the input
> context as errored while leaving the output context active.  A shutdown
> of both directions also fails to stop the output context.
>
> Read rdmaout from the output context so each shutdown direction marks
> the corresponding RDMA context as errored.
>
> Fixes: 54db882f073a ("migration: implement the shutdown for RDMA QIOChannel")
> Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
> ---
>  migration/rdma.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 438419d189..e17ace62b0 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -2982,7 +2982,7 @@ qio_channel_rdma_shutdown(QIOChannel *ioc,
>      RCU_READ_LOCK_GUARD();
>  
>      rdmain = qatomic_rcu_read(&rioc->rdmain);
> -    rdmaout = qatomic_rcu_read(&rioc->rdmain);
> +    rdmaout = qatomic_rcu_read(&rioc->rdmaout);
>  
>      switch (how) {
>      case QIO_CHANNEL_SHUTDOWN_READ:

Reviewed-by: Fabiano Rosas <farosas@suse.de>

Re: [PATCH 1/3] migration/rdma: Fix write-side shutdown

[Peter Xu]

On Mon, Aug 17, 2026 at 06:51:15PM +0800, Yanfei Xu wrote:
> qio_channel_rdma_shutdown() reads rdmain for both the input and
> output contexts.  As a result, a write-side shutdown marks the input
> context as errored while leaving the output context active.  A shutdown
> of both directions also fails to stop the output context.
> 
> Read rdmaout from the output context so each shutdown direction marks
> the corresponding RDMA context as errored.
> 
> Fixes: 54db882f073a ("migration: implement the shutdown for RDMA QIOChannel")
> Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
> ---
>  migration/rdma.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 438419d189..e17ace62b0 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -2982,7 +2982,7 @@ qio_channel_rdma_shutdown(QIOChannel *ioc,
>      RCU_READ_LOCK_GUARD();
>  
>      rdmain = qatomic_rcu_read(&rioc->rdmain);
> -    rdmaout = qatomic_rcu_read(&rioc->rdmain);
> +    rdmaout = qatomic_rcu_read(&rioc->rdmaout);
>  
>      switch (how) {
>      case QIO_CHANNEL_SHUTDOWN_READ:
> -- 
> 2.20.1
> 

I recall there was a patch like this somewhere, looks like we missed that..

Reviewed-by: Peter Xu <peterx@redhat.com>

-- 
Peter Xu

[PATCH 2/3] migration/rdma: Post initial receive before accepting

[Yanfei Xu]

The source return-path thread can send READY before the destination has
posted its initial receive.  Since RNR retries default to zero, the send
can fail and abort migration.

Post the initial receive before rdma_accept() to close this window.

Fixes: 55cc1b5937a8 ("migration: create a dedicated connection for rdma return path")
Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
---
 migration/rdma.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index e17ace62b0..6e8436ccc1 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3263,6 +3263,12 @@ static int qemu_rdma_accept(RDMAContext *rdma)
         }
     }
 
+    ret = qemu_rdma_post_recv_control(rdma, RDMA_WRID_READY, &err);
+    if (ret < 0) {
+        error_report_err(err);
+        goto err_rdma_dest_wait;
+    }
+
     /* Accept the second connection request for return path */
     if ((migrate_postcopy() || migrate_return_path())
         && !rdma->is_return_path) {
@@ -3295,12 +3301,6 @@ static int qemu_rdma_accept(RDMAContext *rdma)
     rdma_ack_cm_event(cm_event);
     rdma->connected = true;
 
-    ret = qemu_rdma_post_recv_control(rdma, RDMA_WRID_READY, &err);
-    if (ret < 0) {
-        error_report_err(err);
-        goto err_rdma_dest_wait;
-    }
-
     qemu_rdma_dump_gid("dest_connect", rdma->cm_id);
 
     return 0;
-- 
2.20.1

Re: [PATCH 2/3] migration/rdma: Post initial receive before accepting

[Peter Xu]

On Mon, Aug 17, 2026 at 06:51:16PM +0800, Yanfei Xu wrote:
> The source return-path thread can send READY before the destination has
> posted its initial receive.  Since RNR retries default to zero, the send
> can fail and abort migration.
> 
> Post the initial receive before rdma_accept() to close this window.
> 
> Fixes: 55cc1b5937a8 ("migration: create a dedicated connection for rdma return path")
> Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>

This one looks correct to me,

Reviewed-by: Peter Xu <peterx@redhat.com>

Do you have way to reproduce it, or is this theory (or can be reproduced by
adding delays)?

> ---
>  migration/rdma.c | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/migration/rdma.c b/migration/rdma.c
> index e17ace62b0..6e8436ccc1 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -3263,6 +3263,12 @@ static int qemu_rdma_accept(RDMAContext *rdma)
>          }
>      }
>  
> +    ret = qemu_rdma_post_recv_control(rdma, RDMA_WRID_READY, &err);
> +    if (ret < 0) {
> +        error_report_err(err);
> +        goto err_rdma_dest_wait;
> +    }
> +
>      /* Accept the second connection request for return path */
>      if ((migrate_postcopy() || migrate_return_path())
>          && !rdma->is_return_path) {
> @@ -3295,12 +3301,6 @@ static int qemu_rdma_accept(RDMAContext *rdma)
>      rdma_ack_cm_event(cm_event);
>      rdma->connected = true;
>  
> -    ret = qemu_rdma_post_recv_control(rdma, RDMA_WRID_READY, &err);
> -    if (ret < 0) {
> -        error_report_err(err);
> -        goto err_rdma_dest_wait;
> -    }
> -
>      qemu_rdma_dump_gid("dest_connect", rdma->cm_id);
>  
>      return 0;
> -- 
> 2.20.1
> 

-- 
Peter Xu

Re: [PATCH 2/3] migration/rdma: Post initial receive before accepting

[Jinpu Wang]

On Wed, Aug 19, 2026 at 9:17 PM Peter Xu <peterx@redhat.com> wrote:
>
> On Mon, Aug 17, 2026 at 06:51:16PM +0800, Yanfei Xu wrote:
> > The source return-path thread can send READY before the destination has
> > posted its initial receive.  Since RNR retries default to zero, the send
> > can fail and abort migration.
> >
> > Post the initial receive before rdma_accept() to close this window.
> >
> > Fixes: 55cc1b5937a8 ("migration: create a dedicated connection for rdma return path")
> > Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
>
> This one looks correct to me,
>
> Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Jack Wang <jinpu.wang@cloud.ionos.com>
>
> Do you have way to reproduce it, or is this theory (or can be reproduced by
> adding delays)?
>
> > ---
> >  migration/rdma.c | 12 ++++++------
> >  1 file changed, 6 insertions(+), 6 deletions(-)
> >
> > diff --git a/migration/rdma.c b/migration/rdma.c
> > index e17ace62b0..6e8436ccc1 100644
> > --- a/migration/rdma.c
> > +++ b/migration/rdma.c
> > @@ -3263,6 +3263,12 @@ static int qemu_rdma_accept(RDMAContext *rdma)
> >          }
> >      }
> >
> > +    ret = qemu_rdma_post_recv_control(rdma, RDMA_WRID_READY, &err);
> > +    if (ret < 0) {
> > +        error_report_err(err);
> > +        goto err_rdma_dest_wait;
> > +    }
> > +
> >      /* Accept the second connection request for return path */
> >      if ((migrate_postcopy() || migrate_return_path())
> >          && !rdma->is_return_path) {
> > @@ -3295,12 +3301,6 @@ static int qemu_rdma_accept(RDMAContext *rdma)
> >      rdma_ack_cm_event(cm_event);
> >      rdma->connected = true;
> >
> > -    ret = qemu_rdma_post_recv_control(rdma, RDMA_WRID_READY, &err);
> > -    if (ret < 0) {
> > -        error_report_err(err);
> > -        goto err_rdma_dest_wait;
> > -    }
> > -
> >      qemu_rdma_dump_gid("dest_connect", rdma->cm_id);
> >
> >      return 0;
> > --
> > 2.20.1
> >
>
> --
> Peter Xu
>

Re: [PATCH 2/3] migration/rdma: Post initial receive before accepting

[Yanfei Xu]

On 2026/8/20 03:16, Peter Xu wrote:
> On Mon, Aug 17, 2026 at 06:51:16PM +0800, Yanfei Xu wrote:
>> The source return-path thread can send READY before the destination has
>> posted its initial receive.  Since RNR retries default to zero, the send
>> can fail and abort migration.
>>
>> Post the initial receive before rdma_accept() to close this window.
>>
>> Fixes: 55cc1b5937a8 ("migration: create a dedicated connection for rdma return path")
>> Signed-off-by: Yanfei Xu<yanfei.xu@bytedance.com>
> This one looks correct to me,
>
> Reviewed-by: Peter Xu<peterx@redhat.com>
>
> Do you have way to reproduce it, or is this theory (or can be reproduced by
> adding delays)?

Yes, adding delays can reproduce it.

For example with sleep(1) and related traces, log at source side:

2026-08-20T05:55:11.471421Z qemu_rdma_exchange_get_response_start 
CONTROL: READY receiving...
2026-08-20T05:55:11.471422Z qemu_rdma_post_send_control CONTROL: sending 
QEMU FILE..
2026-08-20T05:55:11.471447Z qemu_rdma_poll_other other completion 2000 
received left 0
2026-08-20T05:55:11.471589Z qemu_rdma_poll_recv completion 0 received 
(4000) left 0
2026-08-20T05:55:11.471592Z qemu_rdma_poll_other other completion 4000 
received left 0
2026-08-20T05:55:11.471594Z qemu_rdma_exchange_get_response_start 
CONTROL: READY receiving...
2026-08-20T05:55:11.471596Z qemu_rdma_post_send_control CONTROL: sending 
QEMU FILE..
2026-08-20T05:55:11.471617Z qemu_rdma_poll_other other completion 2000 
received left 0
2026-08-20T05:55:11.705561Z qemu_rdma_close
2026-08-20T05:55:11.705587Z qemu_rdma_close
2026-08-20T05:55:11.705626Z qemu-system-x86_64: rdma migration: send 
polling control error
2026-08-20T05:55:11.707001Z qemu-system-x86_64: warning: Early error. 
Sending error.
2026-08-20T05:55:11.707031Z qemu_rdma_post_send_control CONTROL: sending 
ERROR..
2026-08-20T05:55:11.707038Z qemu-system-x86_64: warning: rdma migration: 
send polling control error
2026-08-20T05:55:11.707301Z qemu_rdma_cleanup_disconnect
2026-08-20T05:55:11.710569Z qemu_rdma_cleanup_disconnect

Regards,
Yanfei

Re: [PATCH 2/3] migration/rdma: Post initial receive before accepting

[Peter Xu]

On Thu, Aug 20, 2026 at 02:22:26PM +0800, Yanfei Xu wrote:
> 
> On 2026/8/20 03:16, Peter Xu wrote:
> > On Mon, Aug 17, 2026 at 06:51:16PM +0800, Yanfei Xu wrote:
> > > The source return-path thread can send READY before the destination has
> > > posted its initial receive.  Since RNR retries default to zero, the send
> > > can fail and abort migration.
> > > 
> > > Post the initial receive before rdma_accept() to close this window.
> > > 
> > > Fixes: 55cc1b5937a8 ("migration: create a dedicated connection for rdma return path")
> > > Signed-off-by: Yanfei Xu<yanfei.xu@bytedance.com>
> > This one looks correct to me,
> > 
> > Reviewed-by: Peter Xu<peterx@redhat.com>
> > 
> > Do you have way to reproduce it, or is this theory (or can be reproduced by
> > adding delays)?
> 
> Yes, adding delays can reproduce it.
> 
> For example with sleep(1) and related traces, log at source side:

Thanks for the info, then let's add some of such info into the commit log,
to be clear on how it would trigger (and how likely).

-- 
Peter Xu

[PATCH 3/3] migration/rdma: Retry control sends on full queue

[Yanfei Xu]

RAM writes and control messages share the send queue.  If
outstanding writes fill it, RDMA writes drain a completion and retry,
but control sends fail the migration.

Drain one outstanding write and retry the control send on ENOMEM.

Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
---
 migration/rdma.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 6e8436ccc1..d1f44a5f55 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -1572,9 +1572,19 @@ static int qemu_rdma_post_send_control(RDMAContext *rdma, uint8_t *buf,
         memcpy(wr->control + sizeof(RDMAControlHeader), buf, head->len);
     }
 
-
+retry:
     ret = ibv_post_send(rdma->qp, &send_wr, &bad_wr);
 
+    if (ret == ENOMEM && rdma->nb_sent) {
+        ret = qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, NULL);
+        if (ret < 0) {
+            error_setg(errp, "rdma migration: failed to make room for "
+                       "control send");
+            return -1;
+        }
+        goto retry;
+    }
+
     if (ret > 0) {
         error_setg(errp, "Failed to use post IB SEND for control");
         return -1;
-- 
2.20.1

Re: [PATCH 3/3] migration/rdma: Retry control sends on full queue

[Peter Xu]

On Mon, Aug 17, 2026 at 06:51:17PM +0800, Yanfei Xu wrote:
> RAM writes and control messages share the send queue.  If
> outstanding writes fill it, RDMA writes drain a completion and retry,
> but control sends fail the migration.
> 
> Drain one outstanding write and retry the control send on ENOMEM.
> 
> Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
> ---
>  migration/rdma.c | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 6e8436ccc1..d1f44a5f55 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -1572,9 +1572,19 @@ static int qemu_rdma_post_send_control(RDMAContext *rdma, uint8_t *buf,
>          memcpy(wr->control + sizeof(RDMAControlHeader), buf, head->len);
>      }
>  
> -
> +retry:
>      ret = ibv_post_send(rdma->qp, &send_wr, &bad_wr);
>  
> +    if (ret == ENOMEM && rdma->nb_sent) {
> +        ret = qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, NULL);
> +        if (ret < 0) {
> +            error_setg(errp, "rdma migration: failed to make room for "
> +                       "control send");
> +            return -1;
> +        }
> +        goto retry;
> +    }

Looks also correct, but two questions:

- Should we provide a helper instead of duplicating the WRITE op handling?
  I believe only WRITE wrids can be on the fly.

- Could ENOMEM be returned when nb_sent==0?  If that check applies to WRITE
  path too?

Thanks,

> +
>      if (ret > 0) {
>          error_setg(errp, "Failed to use post IB SEND for control");
>          return -1;
> -- 
> 2.20.1
> 

-- 
Peter Xu

Re: [PATCH 3/3] migration/rdma: Retry control sends on full queue

[Yanfei Xu]

Hi peter,

On 2026/8/20 03:18, Peter Xu wrote:
> On Mon, Aug 17, 2026 at 06:51:17PM +0800, Yanfei Xu wrote:
>> RAM writes and control messages share the send queue.  If
>> outstanding writes fill it, RDMA writes drain a completion and retry,
>> but control sends fail the migration.
>>
>> Drain one outstanding write and retry the control send on ENOMEM.
>>
>> Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
>> ---
>>   migration/rdma.c | 12 +++++++++++-
>>   1 file changed, 11 insertions(+), 1 deletion(-)
>>
>> diff --git a/migration/rdma.c b/migration/rdma.c
>> index 6e8436ccc1..d1f44a5f55 100644
>> --- a/migration/rdma.c
>> +++ b/migration/rdma.c
>> @@ -1572,9 +1572,19 @@ static int qemu_rdma_post_send_control(RDMAContext *rdma, uint8_t *buf,
>>           memcpy(wr->control + sizeof(RDMAControlHeader), buf, head->len);
>>       }
>>   
>> -
>> +retry:
>>       ret = ibv_post_send(rdma->qp, &send_wr, &bad_wr);
>>   
>> +    if (ret == ENOMEM && rdma->nb_sent) {
>> +        ret = qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, NULL);
>> +        if (ret < 0) {
>> +            error_setg(errp, "rdma migration: failed to make room for "
>> +                       "control send");
>> +            return -1;
>> +        }
>> +        goto retry;
>> +    }
> Looks also correct, but two questions:
>
> - Should we provide a helper instead of duplicating the WRITE op handling?
>    I believe only WRITE wrids can be on the fly.

Yes, control message is syncronized, only WRITE can be on the fly. A 
helper is a good suggestion. Will do.

>
> - Could ENOMEM be returned when nb_sent==0?  If that check applies to WRITE
>    path too?

ENOMEM can be regarded as SQ is full to RDMA usage in qemu. Actually ENOMEM
is determined by provider and could have other meaning like
inline_data > qp->max_inline_data in mlx5. Based on qemu codes, I think it's
fine without nb_sent==0

In addition, I encountered this bug when I attempt to send dirty pages 
belongs
to same chunk in parallel. That could more efficiently utilizes 
throughput when
many scattered page in one chunk, and quickly exhausts SQ's WRs. Will post a
RFC with more data.

Regards,
Yanfei

>
> Thanks,
>
>> +
>>       if (ret > 0) {
>>           error_setg(errp, "Failed to use post IB SEND for control");
>>           return -1;
>> -- 
>> 2.20.1
>>

Re: [PATCH 3/3] migration/rdma: Retry control sends on full queue

[Peter Xu]

On Thu, Aug 20, 2026 at 06:27:12PM +0800, Yanfei Xu wrote:
> Hi peter,
> 
> On 2026/8/20 03:18, Peter Xu wrote:
> > On Mon, Aug 17, 2026 at 06:51:17PM +0800, Yanfei Xu wrote:
> > > RAM writes and control messages share the send queue.  If
> > > outstanding writes fill it, RDMA writes drain a completion and retry,
> > > but control sends fail the migration.
> > > 
> > > Drain one outstanding write and retry the control send on ENOMEM.
> > > 
> > > Signed-off-by: Yanfei Xu <yanfei.xu@bytedance.com>
> > > ---
> > >   migration/rdma.c | 12 +++++++++++-
> > >   1 file changed, 11 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/migration/rdma.c b/migration/rdma.c
> > > index 6e8436ccc1..d1f44a5f55 100644
> > > --- a/migration/rdma.c
> > > +++ b/migration/rdma.c
> > > @@ -1572,9 +1572,19 @@ static int qemu_rdma_post_send_control(RDMAContext *rdma, uint8_t *buf,
> > >           memcpy(wr->control + sizeof(RDMAControlHeader), buf, head->len);
> > >       }
> > > -
> > > +retry:
> > >       ret = ibv_post_send(rdma->qp, &send_wr, &bad_wr);
> > > +    if (ret == ENOMEM && rdma->nb_sent) {
> > > +        ret = qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, NULL);
> > > +        if (ret < 0) {
> > > +            error_setg(errp, "rdma migration: failed to make room for "
> > > +                       "control send");
> > > +            return -1;
> > > +        }
> > > +        goto retry;
> > > +    }
> > Looks also correct, but two questions:
> > 
> > - Should we provide a helper instead of duplicating the WRITE op handling?
> >    I believe only WRITE wrids can be on the fly.
> 
> Yes, control message is syncronized, only WRITE can be on the fly. A helper
> is a good suggestion. Will do.
> 
> > 
> > - Could ENOMEM be returned when nb_sent==0?  If that check applies to WRITE
> >    path too?
> 
> ENOMEM can be regarded as SQ is full to RDMA usage in qemu. Actually ENOMEM
> is determined by provider and could have other meaning like
> inline_data > qp->max_inline_data in mlx5. Based on qemu codes, I think it's
> fine without nb_sent==0

I'm not familiar with mlx5 impl that you're discussing here, but IIUC the
point is we should be able to capture all recoverable faults and retry,
meanwhile we should fail immediately on non-recoverable faults.

From the name of the errno (ENOMEM), I expect non-recoverable faults can
happen with it.. unless this is something special to libibverbs to
explicitly imply "queue full"..

I wonder if it means this nb_sent!=0 check should indeed make sense, but I
also wonder if we should add a number of retry so as to capture real ENOMEM
errors otherwise that is not recoverable?  As long as it won't keep
spinning with the same error then we should be good.

> 
> In addition, I encountered this bug when I attempt to send dirty pages
> belongs
> to same chunk in parallel. That could more efficiently utilizes throughput
> when
> many scattered page in one chunk, and quickly exhausts SQ's WRs. Will post a
> RFC with more data.

Sure, I hope that still makes sure different versions of a same page will
be still ordered.

Thanks,

-- 
Peter Xu