[PATCH 0/2] migration/postcopy: shared-fault accounting and multifd docs

[PATCH 0/2] migration/postcopy: shared-fault accounting and multifd docs

[Bin Guo]

Two small patches from the same investigation of the postcopy
shared-page-fault path:

- Patch 1 documents that multifd and postcopy-ram can be enabled
  together since the 10.1 release, and how the two features interact.

- Patch 2 resolves the "TODO: support blocktime tracking" in
  postcopy_request_shared_page() by reporting faults taken by memory
  sharing processes (e.g. a vhost-user backend) with a foreign tid, so
  they show up in the latency reports without polluting the vCPU
  blocktime ones.

The patches are independent of each other; patch 2 was verified with a
vhost-user postcopy migration, where the shared faults now show up in
the latency reports and the page request accounting stays balanced.

Bin Guo (2):
  docs/migration: document that postcopy can be used with multifd
  migration/postcopy: account faults taken by memory sharing processes

 docs/devel/migration/postcopy.rst | 26 ++++++++++++++++++++++++++
 migration/postcopy-ram.c          | 26 ++++++++++++++++----------
 migration/postcopy-ram.h          | 10 ++++++++++
 3 files changed, 52 insertions(+), 10 deletions(-)

-- 
2.50.1 (Apple Git-155)

[PATCH 1/2] docs/migration: document that postcopy can be used with multifd

[Bin Guo]

Since commit e27418861288 ("migration: enable multifd and postcopy
together") in the 10.1 release, the multifd and postcopy-ram capabilities
are no longer mutually exclusive, but the postcopy documentation does not
mention multifd at all, leaving users unaware that the two features can be
combined.

Add a short section describing how they interact: multifd carries the
pages of the precopy phase, the channels are flushed and synced before the
switchover (because loading a page in a multifd receive thread is not
atomic with regard to a running vCPU), and the postcopy phase itself does
not use the multifd channels.

Signed-off-by: Bin Guo <guobin@linux.alibaba.com>
---
 docs/devel/migration/postcopy.rst | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/docs/devel/migration/postcopy.rst b/docs/devel/migration/postcopy.rst
index e319388d8f..570a3a1ce6 100644
--- a/docs/devel/migration/postcopy.rst
+++ b/docs/devel/migration/postcopy.rst
@@ -294,6 +294,32 @@ the background migration channel.  Anyone who cares about latencies of page
 faults during a postcopy migration should enable this feature.  By default,
 it's not enabled.
 
+Postcopy with multifd
+---------------------
+
+The ``multifd`` capability can be enabled together with ``postcopy-ram``
+since the 10.1 QEMU release.  The two features apply to different phases of
+the migration:
+
+  - During the precopy phase, guest pages are sent over the multifd channels
+    as usual, so the initial RAM transfer can use all of them.
+
+  - Just before switching to postcopy, the source flushes and syncs the
+    multifd channels.  This guarantees that all the pages already queued on
+    those channels are loaded on the destination *before* the destination
+    CPUs are started, which is required because loading a page in a multifd
+    receive thread is not atomic with regard to a running vCPU.
+
+  - During the postcopy phase the multifd channels are no longer used for
+    guest pages.  Both the background stream and the pages requested by the
+    destination go through the background migration channel instead, or
+    through the preempt channel for the requested pages when postcopy
+    preemption is enabled.
+
+Consequently, multifd only speeds up the precopy phase of a postcopy
+migration.  The bandwidth available once postcopy has started is the same as
+without multifd.
+
 Postcopy blocktime statistics
 -----------------------------
 
-- 
2.50.1 (Apple Git-155)

Re: [PATCH 1/2] docs/migration: document that postcopy can be used with multifd

[Fabiano Rosas]

Bin Guo <guobin@linux.alibaba.com> writes:

> Since commit e27418861288 ("migration: enable multifd and postcopy
> together") in the 10.1 release, the multifd and postcopy-ram capabilities
> are no longer mutually exclusive, but the postcopy documentation does not
> mention multifd at all, leaving users unaware that the two features can be
> combined.
>
> Add a short section describing how they interact: multifd carries the
> pages of the precopy phase, the channels are flushed and synced before the
> switchover (because loading a page in a multifd receive thread is not
> atomic with regard to a running vCPU), and the postcopy phase itself does
> not use the multifd channels.
>
> Signed-off-by: Bin Guo <guobin@linux.alibaba.com>
> ---
>  docs/devel/migration/postcopy.rst | 26 ++++++++++++++++++++++++++
>  1 file changed, 26 insertions(+)
>
> diff --git a/docs/devel/migration/postcopy.rst b/docs/devel/migration/postcopy.rst
> index e319388d8f..570a3a1ce6 100644
> --- a/docs/devel/migration/postcopy.rst
> +++ b/docs/devel/migration/postcopy.rst
> @@ -294,6 +294,32 @@ the background migration channel.  Anyone who cares about latencies of page
>  faults during a postcopy migration should enable this feature.  By default,
>  it's not enabled.
>  
> +Postcopy with multifd
> +---------------------
> +
> +The ``multifd`` capability can be enabled together with ``postcopy-ram``
> +since the 10.1 QEMU release.  The two features apply to different phases of
> +the migration:
> +
> +  - During the precopy phase, guest pages are sent over the multifd channels
> +    as usual, so the initial RAM transfer can use all of them.
> +
> +  - Just before switching to postcopy, the source flushes and syncs the
> +    multifd channels.  This guarantees that all the pages already queued on
> +    those channels are loaded on the destination *before* the destination
> +    CPUs are started, which is required because loading a page in a multifd
> +    receive thread is not atomic with regard to a running vCPU.
> +
> +  - During the postcopy phase the multifd channels are no longer used for
> +    guest pages.  Both the background stream and the pages requested by the
> +    destination go through the background migration channel instead, or
> +    through the preempt channel for the requested pages when postcopy
> +    preemption is enabled.
> +
> +Consequently, multifd only speeds up the precopy phase of a postcopy
> +migration.  The bandwidth available once postcopy has started is the same as
> +without multifd.
> +
>  Postcopy blocktime statistics
>  -----------------------------

Looks good to me, I'll wait a bit to allow Peter to comment.

Acked-by: Fabiano Rosas <farosas@suse.de>

Re: [PATCH 1/2] docs/migration: document that postcopy can be used with multifd

[Peter Xu]

On Tue, Sep 22, 2026 at 11:44:03AM -0300, Fabiano Rosas wrote:
> Bin Guo <guobin@linux.alibaba.com> writes:
> 
> > Since commit e27418861288 ("migration: enable multifd and postcopy
> > together") in the 10.1 release, the multifd and postcopy-ram capabilities
> > are no longer mutually exclusive, but the postcopy documentation does not
> > mention multifd at all, leaving users unaware that the two features can be
> > combined.
> >
> > Add a short section describing how they interact: multifd carries the
> > pages of the precopy phase, the channels are flushed and synced before the
> > switchover (because loading a page in a multifd receive thread is not
> > atomic with regard to a running vCPU), and the postcopy phase itself does
> > not use the multifd channels.
> >
> > Signed-off-by: Bin Guo <guobin@linux.alibaba.com>
> > ---
> >  docs/devel/migration/postcopy.rst | 26 ++++++++++++++++++++++++++
> >  1 file changed, 26 insertions(+)
> >
> > diff --git a/docs/devel/migration/postcopy.rst b/docs/devel/migration/postcopy.rst
> > index e319388d8f..570a3a1ce6 100644
> > --- a/docs/devel/migration/postcopy.rst
> > +++ b/docs/devel/migration/postcopy.rst
> > @@ -294,6 +294,32 @@ the background migration channel.  Anyone who cares about latencies of page
> >  faults during a postcopy migration should enable this feature.  By default,
> >  it's not enabled.
> >  
> > +Postcopy with multifd
> > +---------------------
> > +
> > +The ``multifd`` capability can be enabled together with ``postcopy-ram``
> > +since the 10.1 QEMU release.  The two features apply to different phases of
> > +the migration:
> > +
> > +  - During the precopy phase, guest pages are sent over the multifd channels
> > +    as usual, so the initial RAM transfer can use all of them.
> > +
> > +  - Just before switching to postcopy, the source flushes and syncs the
> > +    multifd channels.  This guarantees that all the pages already queued on
> > +    those channels are loaded on the destination *before* the destination
> > +    CPUs are started, which is required because loading a page in a multifd
> > +    receive thread is not atomic with regard to a running vCPU.
> > +
> > +  - During the postcopy phase the multifd channels are no longer used for
> > +    guest pages.  Both the background stream and the pages requested by the
> > +    destination go through the background migration channel instead, or
> > +    through the preempt channel for the requested pages when postcopy
> > +    preemption is enabled.
> > +
> > +Consequently, multifd only speeds up the precopy phase of a postcopy
> > +migration.  The bandwidth available once postcopy has started is the same as
> > +without multifd.
> > +
> >  Postcopy blocktime statistics
> >  -----------------------------
> 
> Looks good to me, I'll wait a bit to allow Peter to comment.
> 
> Acked-by: Fabiano Rosas <farosas@suse.de>

The updates look all reasonable,

Reviewed-by: Peter Xu <peterx@redhat.com>

For Bin, just FYI in case it matters: there's a known bug for old machine
types when both features enabled (with preempt, which libvirt enables by
default), it's still during review upstream:

https://lore.kernel.org/all/20260916213558.587304-1-peterx@redhat.com/#t

Thanks,

-- 
Peter Xu

[PATCH 2/2] migration/postcopy: account faults taken by memory sharing processes

[Bin Guo]

Page faults taken by a process that shares guest memory with us (e.g. a
vhost-user backend) are resolved over the migration stream just like our
own, but they are not accounted for, so they are missing from the latency
reports.  There is no thread id to report them with: the client's
userfaultfd is not required to enable UFFD_FEATURE_THREAD_ID, and
libvhost-user does not, so the ptid in the fault message stays zero.  Even
when the feature is enabled, the reported ids belong to another process
and can never match one of our vCPUs.  Passing zero would not work either,
as mark_postcopy_blocktime_begin() reads it as "no thread id" and returns
early.

Report those faults with POSTCOPY_TID_FOREIGN instead, which takes the
same path as the faults of our own non-vCPU threads: no vCPU is involved,
so the vCPU blocktime reports are left alone and only the latency of the
fault is accounted.  This also resolves the "TODO: support blocktime
tracking" left in postcopy_request_shared_page().

Signed-off-by: Bin Guo <guobin@linux.alibaba.com>
---
 migration/postcopy-ram.c | 26 ++++++++++++++++----------
 migration/postcopy-ram.h | 10 ++++++++++
 2 files changed, 26 insertions(+), 10 deletions(-)

diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index 5509cbdcb8..00e2a6fc62 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -1122,8 +1122,6 @@ int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb,
                                         qemu_ram_get_idstr(rb), rb_offset);
         return postcopy_wake_shared(pcfd, client_addr, rb);
     }
-    /* TODO: support blocktime tracking */
-
     /*
      * The page will be placed by qemu_ufd_copy_ioctl(), which removes the
      * matching entry from mis->page_requested (and drops
@@ -1133,10 +1131,16 @@ int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb,
      * backend's address space and can never equal that host address, so the
      * removal would miss forever, leaking page_requested_count and hanging
      * postcopy teardown.
+     *
+     * We have no thread id to report for the client: its userfaultfd is not
+     * required to enable UFFD_FEATURE_THREAD_ID, and even when it does the
+     * thread ids it reports belong to another process, so they could never
+     * match one of our vCPUs.  Report the fault as a foreign one, so that it
+     * still shows up in the latency reports.
      */
     postcopy_request_page(mis, rb, aligned_rbo,
                           (uint64_t)(uintptr_t)qemu_ram_get_host_addr(rb) +
-                          aligned_rbo, 0);
+                          aligned_rbo, POSTCOPY_TID_FOREIGN);
     return 0;
 }
 
@@ -1234,7 +1238,8 @@ bool try_mark_postcopy_blocktime_begin(MigrationIncomingState *mis,
  * blocking time.  It's protected by @page_request_mutex.
  *
  * @addr: faulted host virtual address
- * @ptid: faulted process thread id
+ * @ptid: faulted process thread id, or POSTCOPY_TID_FOREIGN when the fault
+ *        was taken by a thread of another process
  * @rb: ramblock appropriate to addr
  */
 void mark_postcopy_blocktime_begin(uintptr_t addr, uint32_t ptid,
@@ -1256,7 +1261,7 @@ void mark_postcopy_blocktime_begin(uintptr_t addr, uint32_t ptid,
     assert(!ramblock_recv_bitmap_test(rb, (void *)addr));
 
     current = get_current_ns();
-    cpu = blocktime_get_vcpu(dc, ptid);
+    cpu = ptid == POSTCOPY_TID_FOREIGN ? -1 : blocktime_get_vcpu(dc, ptid);
 
     if (cpu >= 0) {
         /* How many faults on this vCPU in total? */
@@ -1281,11 +1286,12 @@ void mark_postcopy_blocktime_begin(uintptr_t addr, uint32_t ptid,
         }
     } else {
         /*
-         * For non-vCPU thread faults, we don't care about tid or cpu index
-         * or time the thread is blocked (e.g., a kworker trying to help
-         * KVM when async_pf=on is OK to be blocked and not affect guest
-         * responsiveness), but we care about latency.  Track it with
-         * cpu=-1.
+         * For faults that did not come from a vCPU thread, we don't care
+         * about tid or cpu index or time the thread is blocked (e.g., a
+         * kworker trying to help KVM when async_pf=on is OK to be blocked
+         * and not affect guest responsiveness; likewise for a thread of a
+         * process that shares guest memory with us), but we care about
+         * latency.  Track it with cpu=-1.
          *
          * Note that this will NOT affect blocktime reports on vCPU being
          * blocked, but only about system-wide latency reports.
diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h
index edba7b0240..028f774651 100644
--- a/migration/postcopy-ram.h
+++ b/migration/postcopy-ram.h
@@ -200,6 +200,16 @@ bool postcopy_is_paused(MigrationStatus status);
 bool try_mark_postcopy_blocktime_begin(MigrationIncomingState *mis,
                                        RAMBlock *rb, ram_addr_t start,
                                        uint64_t haddr, uint32_t tid);
+
+/*
+ * Thread id to report for a fault that did not come from one of our own
+ * threads, e.g. one taken by a process that shares guest memory with us.  It
+ * can never collide with a real thread id, as Linux pid values are far below
+ * this.  See mark_postcopy_blocktime_begin() for how such faults are
+ * accounted.
+ */
+#define POSTCOPY_TID_FOREIGN  ((uint32_t)-1)
+
 void mark_postcopy_blocktime_begin(uintptr_t addr, uint32_t ptid,
                                    RAMBlock *rb);
 
-- 
2.50.1 (Apple Git-155)

Re: [PATCH 2/2] migration/postcopy: account faults taken by memory sharing processes

[Peter Xu]

On Tue, Sep 15, 2026 at 02:37:19PM +0800, Bin Guo wrote:
> Page faults taken by a process that shares guest memory with us (e.g. a
> vhost-user backend) are resolved over the migration stream just like our
> own, but they are not accounted for, so they are missing from the latency
> reports.  There is no thread id to report them with: the client's
> userfaultfd is not required to enable UFFD_FEATURE_THREAD_ID, and
> libvhost-user does not, so the ptid in the fault message stays zero.  Even
> when the feature is enabled, the reported ids belong to another process
> and can never match one of our vCPUs.  Passing zero would not work either,
> as mark_postcopy_blocktime_begin() reads it as "no thread id" and returns
> early.
> 
> Report those faults with POSTCOPY_TID_FOREIGN instead, which takes the
> same path as the faults of our own non-vCPU threads: no vCPU is involved,
> so the vCPU blocktime reports are left alone and only the latency of the
> fault is accounted.  This also resolves the "TODO: support blocktime
> tracking" left in postcopy_request_shared_page().
> 
> Signed-off-by: Bin Guo <guobin@linux.alibaba.com>

This one looks fine in general, but I feel like POSTCOPY_TID_FOREIGN as a
new flag didn't bring us anything, since the FOREIGN is treated the same as
the main thread then.

Logically, the vhost-user process will have its TID too, so it means if the
vhost-user process open the userfaultfd with TID cap it'll work.  But I
agree with your approach here that unfortunately we didn't have control
over it, the pre-existing TID support was pretty much hard-coded to QEMU so
far only.  Maybe it's not yet as helpful to make it formal to have
vhost-user servers supporting it.

But then, I think we can simply enable tid=0 rather than introducing
POSTCOPY_TID_FOREIGN.  It means only vCPU threads are special, all the rest
accounting will fall into non-vcpu category, and with tid=0,
blocktime_get_vcpu() should already return -1, or we can move the tid=0
check from mark_postcopy_blocktime_begin() to blocktime_get_vcpu() instead:
returning -1 if tid=0, and it's safe as vcpu threads shouldn't have tid=0.

Enabling tid=0 brings one more benefit: IIUC it'll also make blocktime work
naturally on old kernels without UFFD_FEATURE_THREAD_ID defined: currently,
ufd_check_and_apply() will simply bypass that feature, then tid will be
reported as 0 on old kernels.  It means then blocktime will work on those
kernels too now, except that all faults accounted into non-vcpu category,
which makes sense, because kernel is too old to identify where it came
from.

Thanks,

-- 
Peter Xu