[PATCH 0/2] migration: include timing and RAM stats on destination when query-migrate

[PATCH 0/2] migration: include timing and RAM stats on destination when query-migrate

[Trieu Huynh]

From: Trieu Huynh <vikingtc4@gmail.com>

When query-migrate is called on the *destination* QEMU after a precopy
migration completes it returns only {"status": "completed"} — no timing,
no RAM statistics.  The source correctly returns total-time, downtime,
setup-time, and full ram stats.

This series fixes the gap in two commits:

* commit 1: Track start/end timestamps and per-page-type counters in
  MigrationIncomingState, recorded in the receive path.

* commit 2: Make source-only MigrationStats fields optional in the QAPI
  schema so the destination can populate info->ram with only the fields
  that are meaningful on the receive side.  Fields computed from tracked
  counters (transferred, normal, duplicate, mbps, pages-per-second) are
  shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are
  absent.

The QAPI change (commit 2) is ABI-compatible: optional fields that are
not set are simply absent from the output.  Source-side callers are
unaffected because populate_ram_info() sets all optional fields via
has_* setters, so the source output is identical to before.

On dst, {"execute":"query-migrate"}
* As-is:
{
    "return": {
        "status": "completed"
}
* To-be:
{
    "return": {
        "status": "completed",
        "total-time": 94,
        "ram": {
            "total": 554508288,
            "pages-per-second": 1440234,
            "page-size": 4096,
            "remaining": 0,
            "mbps": 97.955404255319152,
            "transferred": 1150976,
            "duplicate": 135101,
            "normal-bytes": 1150976,
            "normal": 281
        }
    }
}

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151

Trieu Huynh (2):
  migration: track timing and received pages in MigrationIncomingState
  migration: expose RAM stats and timing on destination via
    query-migrate

 migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
 migration/migration.h | 17 +++++++++++++++++
 migration/ram.c       |  3 +++
 qapi/migration.json   | 14 +++++++-------
 4 files changed, 64 insertions(+), 7 deletions(-)

-- 
2.43.0

[PATCH 1/2] migration: track timing and received pages in MigrationIncomingState

[Trieu Huynh]

From: Trieu Huynh <vikingtc4@gmail.com>

Add start_time, total_time, received_normal_pages,
received_zero_pages, and received_xbzrle_pages to
MigrationIncomingState.

start_time is recorded when the incoming state transitions to ACTIVE;
total_time is recorded when it transitions to COMPLETED. The page
counters are incremented in ram_load_precopy() as each page type is
received from the source.

These fields will be used by a subsequent patch to populate
fill_destination_migration_info() for the migration COMPLETED case.

Signed-off-by: Trieu Huynh <vikingtc4@gmail.com>
---
 migration/migration.c |  2 ++
 migration/migration.h | 17 +++++++++++++++++
 migration/ram.c       |  3 +++
 3 files changed, 22 insertions(+)

diff --git a/migration/migration.c b/migration/migration.c
index 5c9aaa6e58..17c9a8b344 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -726,6 +726,7 @@ static void process_incoming_migration_bh(void *opaque)
      */
     migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
                       MIGRATION_STATUS_COMPLETED);
+    mis->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - mis->start_time;
     migration_incoming_state_destroy();
 }
 
@@ -758,6 +759,7 @@ process_incoming_migration_co(void *opaque)
     migrate_set_state(&mis->state, MIGRATION_STATUS_SETUP,
                       MIGRATION_STATUS_ACTIVE);
 
+    mis->start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     mis->loadvm_co = qemu_coroutine_self();
     ret = qemu_loadvm_state(mis->from_src_file, &local_err);
     mis->loadvm_co = NULL;
diff --git a/migration/migration.h b/migration/migration.h
index b6888daced..cd51ae452c 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -176,6 +176,23 @@ struct MigrationIncomingState {
     /* PostCopyFD's for external userfaultfds & handlers of shared memory */
     GArray   *postcopy_remote_fds;
 
+    /*
+     * Timestamps for reporting migration duration via query-migrate on the
+     * destination. start_time is recorded when the state moves to ACTIVE;
+     * total_time is recorded when it moves to COMPLETED.
+     */
+    int64_t start_time;
+    int64_t total_time;
+
+    /*
+     * Page counters for reporting RAM statistics via query-migrate on the
+     * destination.  Incremented in ram_load_precopy() as each page type
+     * is received.
+     */
+    uint64_t received_normal_pages;
+    uint64_t received_zero_pages;
+    uint64_t received_xbzrle_pages;
+
     MigrationStatus state;
 
     /*
diff --git a/migration/ram.c b/migration/ram.c
index 979751f61b..6646e5daad 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -4417,10 +4417,12 @@ static int ram_load_precopy(QEMUFile *f)
                 break;
             }
             ram_handle_zero(host, TARGET_PAGE_SIZE);
+            mis->received_zero_pages++;
             break;
 
         case RAM_SAVE_FLAG_PAGE:
             qemu_get_buffer(f, host, TARGET_PAGE_SIZE);
+            mis->received_normal_pages++;
             break;
 
         case RAM_SAVE_FLAG_XBZRLE:
@@ -4430,6 +4432,7 @@ static int ram_load_precopy(QEMUFile *f)
                 ret = -EINVAL;
                 break;
             }
+            mis->received_xbzrle_pages++;
             break;
         case RAM_SAVE_FLAG_MULTIFD_FLUSH:
             multifd_recv_sync_main();
-- 
2.43.0

[PATCH 2/2] migration: expose RAM stats and timing on destination via query-migrate

[Trieu Huynh]

From: Trieu Huynh <vikingtc4@gmail.com>

MigrationStats had all-required QAPI fields, so partially populating
info->ram on the destination would force display of source-only metrics
(dirty-sync-count, precopy-bytes, etc.) as misleading zeros.

Make those source-only fields optional with '*' in the QAPI schema.
Update populate_ram_info() to set them via has_* setters so source
output is unchanged.

As per schema's changes, fill_destination_migration_info() can now
populate info->ram for the COMPLETED case with the fields that are
meaningful on the receive side:

  * total, remaining, page-size         (RAM layout)
  * transferred, normal, normal-bytes   (bytes/pages received)
  * duplicate                           (zero pages received)
  * mbps                                (throughput: transferred*8
                                         /time_ms/1000)
  * pages-per-second                    (total pages / duration in
                                         seconds)

Others optional fields (dirty-sync-count, precopy-bytes,
downtime-bytes, postcopy-bytes, multifd-bytes, postcopy-requests,
dirty-sync-missed-zero-copy, dirty-pages-rate) are left unset and
are absent from the destination output.

On dst, {"execute":"query-migrate"}
* As-is:
{
    "return": {
        "status": "completed"
}
* To-be:
{
    "return": {
        "status": "completed",
        "total-time": 94,
        "ram": {
            "total": 554508288,
            "pages-per-second": 1440234,
            "page-size": 4096,
            "remaining": 0,
            "mbps": 97.955404255319152,
            "transferred": 1150976,
            "duplicate": 135101,
            "normal-bytes": 1150976,
            "normal": 281
        }
    }
}

Signed-off-by: Trieu Huynh <vikingtc4@gmail.com>
---
 migration/migration.c | 35 +++++++++++++++++++++++++++++++++++
 qapi/migration.json   | 14 +++++++-------
 2 files changed, 42 insertions(+), 7 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index 17c9a8b344..925d29890d 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1063,17 +1063,24 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
     info->ram->normal = qatomic_read(&mig_stats.normal_pages);
     info->ram->normal_bytes = info->ram->normal * page_size;
     info->ram->mbps = s->mbps;
+    info->ram->has_dirty_sync_count = true;
     info->ram->dirty_sync_count =
         qatomic_read(&mig_stats.dirty_sync_count);
+    info->ram->has_dirty_sync_missed_zero_copy = true;
     info->ram->dirty_sync_missed_zero_copy =
         qatomic_read(&mig_stats.dirty_sync_missed_zero_copy);
+    info->ram->has_postcopy_requests = true;
     info->ram->postcopy_requests =
         qatomic_read(&mig_stats.postcopy_requests);
     info->ram->page_size = page_size;
+    info->ram->has_multifd_bytes = true;
     info->ram->multifd_bytes = qatomic_read(&mig_stats.multifd_bytes);
     info->ram->pages_per_second = s->pages_per_second;
+    info->ram->has_precopy_bytes = true;
     info->ram->precopy_bytes = qatomic_read(&mig_stats.precopy_bytes);
+    info->ram->has_downtime_bytes = true;
     info->ram->downtime_bytes = qatomic_read(&mig_stats.downtime_bytes);
+    info->ram->has_postcopy_bytes = true;
     info->ram->postcopy_bytes = qatomic_read(&mig_stats.postcopy_bytes);
 
     if (migrate_xbzrle()) {
@@ -1094,6 +1101,7 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
 
     if (s->state != MIGRATION_STATUS_COMPLETED) {
         info->ram->remaining = ram_bytes_remaining();
+        info->ram->has_dirty_pages_rate = true;
         info->ram->dirty_pages_rate =
            qatomic_read(&mig_stats.dirty_pages_rate);
     }
@@ -1209,6 +1217,33 @@ static void fill_destination_migration_info(MigrationInfo *info)
     case MIGRATION_STATUS_COMPLETED:
         info->has_status = true;
         fill_destination_postcopy_migration_info(info);
+        if (mis->total_time > 0) {
+            info->has_total_time = true;
+            info->total_time = mis->total_time;
+        }
+        {
+            size_t page_size = qemu_target_page_size();
+            uint64_t normal  = mis->received_normal_pages;
+            uint64_t zero    = mis->received_zero_pages;
+            uint64_t xbzrle  = mis->received_xbzrle_pages;
+
+            info->ram = g_malloc0(sizeof(*info->ram));
+            info->ram->total        = ram_bytes_total();
+            info->ram->remaining    = 0;
+            info->ram->normal       = normal + xbzrle;
+            info->ram->normal_bytes = (normal + xbzrle) * page_size;
+            info->ram->duplicate    = zero;
+            info->ram->transferred  = (normal + xbzrle) * page_size;
+            info->ram->page_size    = page_size;
+
+            if (info->has_total_time) {
+                info->ram->mbps = info->ram->transferred * 8.0
+                                  / info->total_time / 1000.0;
+                info->ram->pages_per_second = (normal + xbzrle + zero)
+                                              * 1000.0 / info->total_time;
+            }
+            /* source-only optional omitted from output */
+        }
         break;
     default:
         return;
diff --git a/qapi/migration.json b/qapi/migration.json
index 7134d4ce47..a695d04a22 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -68,13 +68,13 @@
   'data': {'transferred': 'int', 'remaining': 'int', 'total': 'int' ,
            'duplicate': 'int',
            'normal': 'int',
-           'normal-bytes': 'int', 'dirty-pages-rate': 'int',
-           'mbps': 'number', 'dirty-sync-count': 'int',
-           'postcopy-requests': 'int', 'page-size': 'int',
-           'multifd-bytes': 'uint64', 'pages-per-second': 'uint64',
-           'precopy-bytes': 'uint64', 'downtime-bytes': 'uint64',
-           'postcopy-bytes': 'uint64',
-           'dirty-sync-missed-zero-copy': 'uint64' } }
+           'normal-bytes': 'int', '*dirty-pages-rate': 'int',
+           'mbps': 'number', '*dirty-sync-count': 'int',
+           '*postcopy-requests': 'int', 'page-size': 'int',
+           '*multifd-bytes': 'uint64', 'pages-per-second': 'uint64',
+           '*precopy-bytes': 'uint64', '*downtime-bytes': 'uint64',
+           '*postcopy-bytes': 'uint64',
+           '*dirty-sync-missed-zero-copy': 'uint64' } }
 
 ##
 # @XBZRLECacheStats:
-- 
2.43.0

Re: [PATCH 0/2] migration: include timing and RAM stats on destination when query-migrate

[Fabiano Rosas]

Trieu Huynh <vikingtc4@gmail.com> writes:

> From: Trieu Huynh <vikingtc4@gmail.com>
>
> When query-migrate is called on the *destination* QEMU after a precopy
> migration completes it returns only {"status": "completed"} — no timing,
> no RAM statistics.  The source correctly returns total-time, downtime,
> setup-time, and full ram stats.
>
> This series fixes the gap in two commits:
>
> * commit 1: Track start/end timestamps and per-page-type counters in
>   MigrationIncomingState, recorded in the receive path.
>
> * commit 2: Make source-only MigrationStats fields optional in the QAPI
>   schema so the destination can populate info->ram with only the fields
>   that are meaningful on the receive side.  Fields computed from tracked
>   counters (transferred, normal, duplicate, mbps, pages-per-second) are
>   shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are
>   absent.
>
> The QAPI change (commit 2) is ABI-compatible: optional fields that are
> not set are simply absent from the output.  Source-side callers are
> unaffected because populate_ram_info() sets all optional fields via
> has_* setters, so the source output is identical to before.
>
> On dst, {"execute":"query-migrate"}
> * As-is:
> {
>     "return": {
>         "status": "completed"
> }
> * To-be:
> {
>     "return": {
>         "status": "completed",
>         "total-time": 94,
>         "ram": {
>             "total": 554508288,
>             "pages-per-second": 1440234,
>             "page-size": 4096,
>             "remaining": 0,
>             "mbps": 97.955404255319152,
>             "transferred": 1150976,
>             "duplicate": 135101,
>             "normal-bytes": 1150976,
>             "normal": 281
>         }
>     }
> }
>
> Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
>
> Trieu Huynh (2):
>   migration: track timing and received pages in MigrationIncomingState
>   migration: expose RAM stats and timing on destination via
>     query-migrate
>
>  migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
>  migration/migration.h | 17 +++++++++++++++++
>  migration/ram.c       |  3 +++
>  qapi/migration.json   | 14 +++++++-------
>  4 files changed, 64 insertions(+), 7 deletions(-)

Hi, remember to copy the interested people in your series. Using
get_maintainers is correct, but when there's already a discussion about
the topic it's good to add some CCs manually.

+CC Daniel and Claudio

Re: [PATCH 1/2] migration: track timing and received pages in MigrationIncomingState

[Claudio Fontana]

Hello,

On 4/5/26 17:26, Trieu Huynh wrote:
> From: Trieu Huynh <vikingtc4@gmail.com>
> 
> Add start_time, total_time, received_normal_pages,
> received_zero_pages, and received_xbzrle_pages to
> MigrationIncomingState.
> 
> start_time is recorded when the incoming state transitions to ACTIVE;
> total_time is recorded when it transitions to COMPLETED. The page
> counters are incremented in ram_load_precopy() as each page type is
> received from the source.
> 
> These fields will be used by a subsequent patch to populate
> fill_destination_migration_info() for the migration COMPLETED case.
> 
> Signed-off-by: Trieu Huynh <vikingtc4@gmail.com>
> ---
>  migration/migration.c |  2 ++
>  migration/migration.h | 17 +++++++++++++++++
>  migration/ram.c       |  3 +++
>  3 files changed, 22 insertions(+)
> 
> diff --git a/migration/migration.c b/migration/migration.c
> index 5c9aaa6e58..17c9a8b344 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -726,6 +726,7 @@ static void process_incoming_migration_bh(void *opaque)
>       */
>      migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
>                        MIGRATION_STATUS_COMPLETED);
> +    mis->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - mis->start_time;
>      migration_incoming_state_destroy();
>  }
>  
> @@ -758,6 +759,7 @@ process_incoming_migration_co(void *opaque)
>      migrate_set_state(&mis->state, MIGRATION_STATUS_SETUP,
>                        MIGRATION_STATUS_ACTIVE);
>  
> +    mis->start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
>      mis->loadvm_co = qemu_coroutine_self();
>      ret = qemu_loadvm_state(mis->from_src_file, &local_err);
>      mis->loadvm_co = NULL;
> diff --git a/migration/migration.h b/migration/migration.h
> index b6888daced..cd51ae452c 100644
> --- a/migration/migration.h
> +++ b/migration/migration.h
> @@ -176,6 +176,23 @@ struct MigrationIncomingState {
>      /* PostCopyFD's for external userfaultfds & handlers of shared memory */
>      GArray   *postcopy_remote_fds;
>  
> +    /*
> +     * Timestamps for reporting migration duration via query-migrate on the
> +     * destination. start_time is recorded when the state moves to ACTIVE;
> +     * total_time is recorded when it moves to COMPLETED.
> +     */
> +    int64_t start_time;
> +    int64_t total_time;

nit: since this is a timestamp and these are directly linked to state changes,
why not call these:

int64_t migration_active_ts;
int64_t migration_completed_ts;

or something similar?

Thank you,

Claudio

> +
> +    /*
> +     * Page counters for reporting RAM statistics via query-migrate on the
> +     * destination.  Incremented in ram_load_precopy() as each page type
> +     * is received.
> +     */
> +    uint64_t received_normal_pages;
> +    uint64_t received_zero_pages;
> +    uint64_t received_xbzrle_pages;
> +
>      MigrationStatus state;
>  
>      /*
> diff --git a/migration/ram.c b/migration/ram.c
> index 979751f61b..6646e5daad 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -4417,10 +4417,12 @@ static int ram_load_precopy(QEMUFile *f)
>                  break;
>              }
>              ram_handle_zero(host, TARGET_PAGE_SIZE);
> +            mis->received_zero_pages++;
>              break;
>  
>          case RAM_SAVE_FLAG_PAGE:
>              qemu_get_buffer(f, host, TARGET_PAGE_SIZE);
> +            mis->received_normal_pages++;
>              break;
>  
>          case RAM_SAVE_FLAG_XBZRLE:
> @@ -4430,6 +4432,7 @@ static int ram_load_precopy(QEMUFile *f)
>                  ret = -EINVAL;
>                  break;
>              }
> +            mis->received_xbzrle_pages++;
>              break;
>          case RAM_SAVE_FLAG_MULTIFD_FLUSH:
>              multifd_recv_sync_main();

Re: [PATCH 1/2] migration: track timing and received pages in MigrationIncomingState

[Claudio Fontana]

On 4/7/26 14:02, Claudio Fontana wrote:
> Hello,
> 
> On 4/5/26 17:26, Trieu Huynh wrote:
>> From: Trieu Huynh <vikingtc4@gmail.com>
>>
>> Add start_time, total_time, received_normal_pages,
>> received_zero_pages, and received_xbzrle_pages to
>> MigrationIncomingState.
>>
>> start_time is recorded when the incoming state transitions to ACTIVE;
>> total_time is recorded when it transitions to COMPLETED. The page
>> counters are incremented in ram_load_precopy() as each page type is
>> received from the source.
>>
>> These fields will be used by a subsequent patch to populate
>> fill_destination_migration_info() for the migration COMPLETED case.
>>
>> Signed-off-by: Trieu Huynh <vikingtc4@gmail.com>
>> ---
>>  migration/migration.c |  2 ++
>>  migration/migration.h | 17 +++++++++++++++++
>>  migration/ram.c       |  3 +++
>>  3 files changed, 22 insertions(+)
>>
>> diff --git a/migration/migration.c b/migration/migration.c
>> index 5c9aaa6e58..17c9a8b344 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -726,6 +726,7 @@ static void process_incoming_migration_bh(void *opaque)
>>       */
>>      migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
>>                        MIGRATION_STATUS_COMPLETED);
>> +    mis->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - mis->start_time;
>>      migration_incoming_state_destroy();
>>  }
>>  
>> @@ -758,6 +759,7 @@ process_incoming_migration_co(void *opaque)
>>      migrate_set_state(&mis->state, MIGRATION_STATUS_SETUP,
>>                        MIGRATION_STATUS_ACTIVE);
>>  
>> +    mis->start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
>>      mis->loadvm_co = qemu_coroutine_self();
>>      ret = qemu_loadvm_state(mis->from_src_file, &local_err);
>>      mis->loadvm_co = NULL;
>> diff --git a/migration/migration.h b/migration/migration.h
>> index b6888daced..cd51ae452c 100644
>> --- a/migration/migration.h
>> +++ b/migration/migration.h
>> @@ -176,6 +176,23 @@ struct MigrationIncomingState {
>>      /* PostCopyFD's for external userfaultfds & handlers of shared memory */
>>      GArray   *postcopy_remote_fds;
>>  
>> +    /*
>> +     * Timestamps for reporting migration duration via query-migrate on the
>> +     * destination. start_time is recorded when the state moves to ACTIVE;
>> +     * total_time is recorded when it moves to COMPLETED.
>> +     */
>> +    int64_t start_time;
>> +    int64_t total_time;
> 
> nit: since this is a timestamp and these are directly linked to state changes,
> why not call these:
> 
> int64_t migration_active_ts;
> int64_t migration_completed_ts;

I got the second one wrong, it is not a timestamp, that is indeed a time.
So clarity in the name seems to be more than a nit. What about:

int64_t migration_active_ts;      /* ms */
int64_t migration_completed_time; /* ms */

Thanks,

Claudio

> 
>> +
>> +    /*
>> +     * Page counters for reporting RAM statistics via query-migrate on the
>> +     * destination.  Incremented in ram_load_precopy() as each page type
>> +     * is received.
>> +     */
>> +    uint64_t received_normal_pages;
>> +    uint64_t received_zero_pages;
>> +    uint64_t received_xbzrle_pages;
>> +
>>      MigrationStatus state;
>>  
>>      /*
>> diff --git a/migration/ram.c b/migration/ram.c
>> index 979751f61b..6646e5daad 100644
>> --- a/migration/ram.c
>> +++ b/migration/ram.c
>> @@ -4417,10 +4417,12 @@ static int ram_load_precopy(QEMUFile *f)
>>                  break;
>>              }
>>              ram_handle_zero(host, TARGET_PAGE_SIZE);
>> +            mis->received_zero_pages++;
>>              break;
>>  
>>          case RAM_SAVE_FLAG_PAGE:
>>              qemu_get_buffer(f, host, TARGET_PAGE_SIZE);
>> +            mis->received_normal_pages++;
>>              break;
>>  
>>          case RAM_SAVE_FLAG_XBZRLE:
>> @@ -4430,6 +4432,7 @@ static int ram_load_precopy(QEMUFile *f)
>>                  ret = -EINVAL;
>>                  break;
>>              }
>> +            mis->received_xbzrle_pages++;
>>              break;
>>          case RAM_SAVE_FLAG_MULTIFD_FLUSH:
>>              multifd_recv_sync_main();
> 

Re: [PATCH 1/2] migration: track timing and received pages in MigrationIncomingState

[Trieu Huynh]

On Tue, Apr 07, 2026 at 02:11:54PM +0200, Claudio Fontana wrote:
> On 4/7/26 14:02, Claudio Fontana wrote:
> > Hello,
> > 
> > On 4/5/26 17:26, Trieu Huynh wrote:
> >> From: Trieu Huynh <vikingtc4@gmail.com>
> >>
> >> Add start_time, total_time, received_normal_pages,
> >> received_zero_pages, and received_xbzrle_pages to
> >> MigrationIncomingState.
> >>
> >> start_time is recorded when the incoming state transitions to ACTIVE;
> >> total_time is recorded when it transitions to COMPLETED. The page
> >> counters are incremented in ram_load_precopy() as each page type is
> >> received from the source.
> >>
> >> These fields will be used by a subsequent patch to populate
> >> fill_destination_migration_info() for the migration COMPLETED case.
> >>
> >> Signed-off-by: Trieu Huynh <vikingtc4@gmail.com>
> >> ---
> >>  migration/migration.c |  2 ++
> >>  migration/migration.h | 17 +++++++++++++++++
> >>  migration/ram.c       |  3 +++
> >>  3 files changed, 22 insertions(+)
> >>
> >> diff --git a/migration/migration.c b/migration/migration.c
> >> index 5c9aaa6e58..17c9a8b344 100644
> >> --- a/migration/migration.c
> >> +++ b/migration/migration.c
> >> @@ -726,6 +726,7 @@ static void process_incoming_migration_bh(void *opaque)
> >>       */
> >>      migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
> >>                        MIGRATION_STATUS_COMPLETED);
> >> +    mis->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME) - mis->start_time;
> >>      migration_incoming_state_destroy();
> >>  }
> >>  
> >> @@ -758,6 +759,7 @@ process_incoming_migration_co(void *opaque)
> >>      migrate_set_state(&mis->state, MIGRATION_STATUS_SETUP,
> >>                        MIGRATION_STATUS_ACTIVE);
> >>  
> >> +    mis->start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
> >>      mis->loadvm_co = qemu_coroutine_self();
> >>      ret = qemu_loadvm_state(mis->from_src_file, &local_err);
> >>      mis->loadvm_co = NULL;
> >> diff --git a/migration/migration.h b/migration/migration.h
> >> index b6888daced..cd51ae452c 100644
> >> --- a/migration/migration.h
> >> +++ b/migration/migration.h
> >> @@ -176,6 +176,23 @@ struct MigrationIncomingState {
> >>      /* PostCopyFD's for external userfaultfds & handlers of shared memory */
> >>      GArray   *postcopy_remote_fds;
> >>  
> >> +    /*
> >> +     * Timestamps for reporting migration duration via query-migrate on the
> >> +     * destination. start_time is recorded when the state moves to ACTIVE;
> >> +     * total_time is recorded when it moves to COMPLETED.
> >> +     */
> >> +    int64_t start_time;
> >> +    int64_t total_time;
> > 
> > nit: since this is a timestamp and these are directly linked to state changes,
> > why not call these:
> > 
> > int64_t migration_active_ts;
> > int64_t migration_completed_ts;
> 
> I got the second one wrong, it is not a timestamp, that is indeed a time.
> So clarity in the name seems to be more than a nit. What about:
> 
> int64_t migration_active_ts;      /* ms */
> int64_t migration_completed_time; /* ms */
> 
make sense to me. will update in v2. thank you.
> Thanks,
> 
> Claudio
> 
> > 
> >> +
> >> +    /*
> >> +     * Page counters for reporting RAM statistics via query-migrate on the
> >> +     * destination.  Incremented in ram_load_precopy() as each page type
> >> +     * is received.
> >> +     */
> >> +    uint64_t received_normal_pages;
> >> +    uint64_t received_zero_pages;
> >> +    uint64_t received_xbzrle_pages;
> >> +
> >>      MigrationStatus state;
> >>  
> >>      /*
> >> diff --git a/migration/ram.c b/migration/ram.c
> >> index 979751f61b..6646e5daad 100644
> >> --- a/migration/ram.c
> >> +++ b/migration/ram.c
> >> @@ -4417,10 +4417,12 @@ static int ram_load_precopy(QEMUFile *f)
> >>                  break;
> >>              }
> >>              ram_handle_zero(host, TARGET_PAGE_SIZE);
> >> +            mis->received_zero_pages++;
> >>              break;
> >>  
> >>          case RAM_SAVE_FLAG_PAGE:
> >>              qemu_get_buffer(f, host, TARGET_PAGE_SIZE);
> >> +            mis->received_normal_pages++;
> >>              break;
> >>  
> >>          case RAM_SAVE_FLAG_XBZRLE:
> >> @@ -4430,6 +4432,7 @@ static int ram_load_precopy(QEMUFile *f)
> >>                  ret = -EINVAL;
> >>                  break;
> >>              }
> >> +            mis->received_xbzrle_pages++;
> >>              break;
> >>          case RAM_SAVE_FLAG_MULTIFD_FLUSH:
> >>              multifd_recv_sync_main();
> > 
> 
BRs,
Trieu Huynh

Re: [PATCH 2/2] migration: expose RAM stats and timing on destination via query-migrate

[Claudio Fontana]

Hello Trieu,

makes sense to me in general, but maintainers can give you a more proper review on this,
I can only point out a few sources of confusion from my side, which likely stem from lack of knowledge from my side:

On 4/5/26 17:26, Trieu Huynh wrote:
> From: Trieu Huynh <vikingtc4@gmail.com>
> 
> MigrationStats had all-required QAPI fields, so partially populating
> info->ram on the destination would force display of source-only metrics
> (dirty-sync-count, precopy-bytes, etc.) as misleading zeros.
> 
> Make those source-only fields optional with '*' in the QAPI schema.
> Update populate_ram_info() to set them via has_* setters so source
> output is unchanged.
> 
> As per schema's changes, fill_destination_migration_info() can now
> populate info->ram for the COMPLETED case with the fields that are
> meaningful on the receive side:
> 
>   * total, remaining, page-size         (RAM layout)

Does 'remaining' make sense if the migration is completed? Don't we just assume 0?
But maybe it makes sense to keep it for simplicity/consistency.

>   * transferred, normal, normal-bytes   (bytes/pages received)
>   * duplicate                           (zero pages received)

it is interesting that zero pages are called "duplicate".

>   * mbps                                (throughput: transferred*8
>                                          /time_ms/1000)
>   * pages-per-second                    (total pages / duration in
>                                          seconds)
> 
> Others optional fields (dirty-sync-count, precopy-bytes,
> downtime-bytes, postcopy-bytes, multifd-bytes, postcopy-requests,
> dirty-sync-missed-zero-copy, dirty-pages-rate) are left unset and
> are absent from the destination output.
> 
> On dst, {"execute":"query-migrate"}
> * As-is:
> {
>     "return": {
>         "status": "completed"
> }
> * To-be:
> {
>     "return": {
>         "status": "completed",
>         "total-time": 94,
>         "ram": {
>             "total": 554508288,
>             "pages-per-second": 1440234,
>             "page-size": 4096,
>             "remaining": 0,
>             "mbps": 97.955404255319152,
>             "transferred": 1150976,
>             "duplicate": 135101,
>             "normal-bytes": 1150976,
>             "normal": 281
>         }
>     }
> }
> 
> Signed-off-by: Trieu Huynh <vikingtc4@gmail.com>
> ---
>  migration/migration.c | 35 +++++++++++++++++++++++++++++++++++
>  qapi/migration.json   | 14 +++++++-------
>  2 files changed, 42 insertions(+), 7 deletions(-)
> 
> diff --git a/migration/migration.c b/migration/migration.c
> index 17c9a8b344..925d29890d 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -1063,17 +1063,24 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
>      info->ram->normal = qatomic_read(&mig_stats.normal_pages);
>      info->ram->normal_bytes = info->ram->normal * page_size;
>      info->ram->mbps = s->mbps;
> +    info->ram->has_dirty_sync_count = true;
>      info->ram->dirty_sync_count =
>          qatomic_read(&mig_stats.dirty_sync_count);
> +    info->ram->has_dirty_sync_missed_zero_copy = true;
>      info->ram->dirty_sync_missed_zero_copy =
>          qatomic_read(&mig_stats.dirty_sync_missed_zero_copy);
> +    info->ram->has_postcopy_requests = true;
>      info->ram->postcopy_requests =
>          qatomic_read(&mig_stats.postcopy_requests);
>      info->ram->page_size = page_size;
> +    info->ram->has_multifd_bytes = true;
>      info->ram->multifd_bytes = qatomic_read(&mig_stats.multifd_bytes);
>      info->ram->pages_per_second = s->pages_per_second;
> +    info->ram->has_precopy_bytes = true;
>      info->ram->precopy_bytes = qatomic_read(&mig_stats.precopy_bytes);
> +    info->ram->has_downtime_bytes = true;
>      info->ram->downtime_bytes = qatomic_read(&mig_stats.downtime_bytes);
> +    info->ram->has_postcopy_bytes = true;
>      info->ram->postcopy_bytes = qatomic_read(&mig_stats.postcopy_bytes);
>  
>      if (migrate_xbzrle()) {
> @@ -1094,6 +1101,7 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
>  
>      if (s->state != MIGRATION_STATUS_COMPLETED) {
>          info->ram->remaining = ram_bytes_remaining();
> +        info->ram->has_dirty_pages_rate = true;
>          info->ram->dirty_pages_rate =
>             qatomic_read(&mig_stats.dirty_pages_rate);
>      }
> @@ -1209,6 +1217,33 @@ static void fill_destination_migration_info(MigrationInfo *info)
>      case MIGRATION_STATUS_COMPLETED:
>          info->has_status = true;
>          fill_destination_postcopy_migration_info(info);
> +        if (mis->total_time > 0) {
> +            info->has_total_time = true;
> +            info->total_time = mis->total_time;
> +        }
> +        {
> +            size_t page_size = qemu_target_page_size();
> +            uint64_t normal  = mis->received_normal_pages;
> +            uint64_t zero    = mis->received_zero_pages;
> +            uint64_t xbzrle  = mis->received_xbzrle_pages;
> +
> +            info->ram = g_malloc0(sizeof(*info->ram));
> +            info->ram->total        = ram_bytes_total();
> +            info->ram->remaining    = 0;
> +            info->ram->normal       = normal + xbzrle;
> +            info->ram->normal_bytes = (normal + xbzrle) * page_size;
> +            info->ram->duplicate    = zero;
> +            info->ram->transferred  = (normal + xbzrle) * page_size;
> +            info->ram->page_size    = page_size;
> +
> +            if (info->has_total_time) {
> +                info->ram->mbps = info->ram->transferred * 8.0
> +                                  / info->total_time / 1000.0;
> +                info->ram->pages_per_second = (normal + xbzrle + zero)
> +                                              * 1000.0 / info->total_time;
> +            }
> +            /* source-only optional omitted from output */
> +        }
>          break;
>      default:
>          return;
> diff --git a/qapi/migration.json b/qapi/migration.json
> index 7134d4ce47..a695d04a22 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -68,13 +68,13 @@
>    'data': {'transferred': 'int', 'remaining': 'int', 'total': 'int' ,
>             'duplicate': 'int',
>             'normal': 'int',
> -           'normal-bytes': 'int', 'dirty-pages-rate': 'int',
> -           'mbps': 'number', 'dirty-sync-count': 'int',
> -           'postcopy-requests': 'int', 'page-size': 'int',
> -           'multifd-bytes': 'uint64', 'pages-per-second': 'uint64',
> -           'precopy-bytes': 'uint64', 'downtime-bytes': 'uint64',
> -           'postcopy-bytes': 'uint64',
> -           'dirty-sync-missed-zero-copy': 'uint64' } }
> +           'normal-bytes': 'int', '*dirty-pages-rate': 'int',
> +           'mbps': 'number', '*dirty-sync-count': 'int',
> +           '*postcopy-requests': 'int', 'page-size': 'int',
> +           '*multifd-bytes': 'uint64', 'pages-per-second': 'uint64',
> +           '*precopy-bytes': 'uint64', '*downtime-bytes': 'uint64',
> +           '*postcopy-bytes': 'uint64',
> +           '*dirty-sync-missed-zero-copy': 'uint64' } }
>  
>  ##
>  # @XBZRLECacheStats:

This looks good and makes sense to me,
but looking forward to more authoritative voices.

Does this work build ok on all archs? Any tests that could be extended to cover these new stats,
maybe tests/qtest/migration/ ?

Thank you!

Claudio

Re: [PATCH 2/2] migration: expose RAM stats and timing on destination via query-migrate

[Trieu Huynh]

On Wed, Apr 08, 2026 at 06:14:41PM +0200, Claudio Fontana wrote:
> Hello Trieu,
> 
> makes sense to me in general, but maintainers can give you a more proper review on this,
> I can only point out a few sources of confusion from my side, which likely stem from lack of knowledge from my side:
> 
> On 4/5/26 17:26, Trieu Huynh wrote:
> > From: Trieu Huynh <vikingtc4@gmail.com>
> > 
> > MigrationStats had all-required QAPI fields, so partially populating
> > info->ram on the destination would force display of source-only metrics
> > (dirty-sync-count, precopy-bytes, etc.) as misleading zeros.
> > 
> > Make those source-only fields optional with '*' in the QAPI schema.
> > Update populate_ram_info() to set them via has_* setters so source
> > output is unchanged.
> > 
> > As per schema's changes, fill_destination_migration_info() can now
> > populate info->ram for the COMPLETED case with the fields that are
> > meaningful on the receive side:
> > 
> >   * total, remaining, page-size         (RAM layout)
> 
> Does 'remaining' make sense if the migration is completed? Don't we just assume 0?
> But maybe it makes sense to keep it for simplicity/consistency.
> 
Indeed, it set to 0 when migration status is completed, just to be
consistent.
> >   * transferred, normal, normal-bytes   (bytes/pages received)
> >   * duplicate                           (zero pages received)
> 
> it is interesting that zero pages are called "duplicate".
> 
Yes, it is. IIUC, in migration stats, duplicate = "pages we didn't
need to send because dest's memory is zero-initialized on startup,
so any all-zero src page is already "duplicate" of what the dst has.
> >   * mbps                                (throughput: transferred*8
> >                                          /time_ms/1000)
> >   * pages-per-second                    (total pages / duration in
> >                                          seconds)
> > 
> > Others optional fields (dirty-sync-count, precopy-bytes,
> > downtime-bytes, postcopy-bytes, multifd-bytes, postcopy-requests,
> > dirty-sync-missed-zero-copy, dirty-pages-rate) are left unset and
> > are absent from the destination output.
> > 
> > On dst, {"execute":"query-migrate"}
> > * As-is:
> > {
> >     "return": {
> >         "status": "completed"
> > }
> > * To-be:
> > {
> >     "return": {
> >         "status": "completed",
> >         "total-time": 94,
> >         "ram": {
> >             "total": 554508288,
> >             "pages-per-second": 1440234,
> >             "page-size": 4096,
> >             "remaining": 0,
> >             "mbps": 97.955404255319152,
> >             "transferred": 1150976,
> >             "duplicate": 135101,
> >             "normal-bytes": 1150976,
> >             "normal": 281
> >         }
> >     }
> > }
> > 
> > Signed-off-by: Trieu Huynh <vikingtc4@gmail.com>
> > ---
> >  migration/migration.c | 35 +++++++++++++++++++++++++++++++++++
> >  qapi/migration.json   | 14 +++++++-------
> >  2 files changed, 42 insertions(+), 7 deletions(-)
> > 
> > diff --git a/migration/migration.c b/migration/migration.c
> > index 17c9a8b344..925d29890d 100644
> > --- a/migration/migration.c
> > +++ b/migration/migration.c
> > @@ -1063,17 +1063,24 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
> >      info->ram->normal = qatomic_read(&mig_stats.normal_pages);
> >      info->ram->normal_bytes = info->ram->normal * page_size;
> >      info->ram->mbps = s->mbps;
> > +    info->ram->has_dirty_sync_count = true;
> >      info->ram->dirty_sync_count =
> >          qatomic_read(&mig_stats.dirty_sync_count);
> > +    info->ram->has_dirty_sync_missed_zero_copy = true;
> >      info->ram->dirty_sync_missed_zero_copy =
> >          qatomic_read(&mig_stats.dirty_sync_missed_zero_copy);
> > +    info->ram->has_postcopy_requests = true;
> >      info->ram->postcopy_requests =
> >          qatomic_read(&mig_stats.postcopy_requests);
> >      info->ram->page_size = page_size;
> > +    info->ram->has_multifd_bytes = true;
> >      info->ram->multifd_bytes = qatomic_read(&mig_stats.multifd_bytes);
> >      info->ram->pages_per_second = s->pages_per_second;
> > +    info->ram->has_precopy_bytes = true;
> >      info->ram->precopy_bytes = qatomic_read(&mig_stats.precopy_bytes);
> > +    info->ram->has_downtime_bytes = true;
> >      info->ram->downtime_bytes = qatomic_read(&mig_stats.downtime_bytes);
> > +    info->ram->has_postcopy_bytes = true;
> >      info->ram->postcopy_bytes = qatomic_read(&mig_stats.postcopy_bytes);
> >  
> >      if (migrate_xbzrle()) {
> > @@ -1094,6 +1101,7 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
> >  
> >      if (s->state != MIGRATION_STATUS_COMPLETED) {
> >          info->ram->remaining = ram_bytes_remaining();
> > +        info->ram->has_dirty_pages_rate = true;
> >          info->ram->dirty_pages_rate =
> >             qatomic_read(&mig_stats.dirty_pages_rate);
> >      }
> > @@ -1209,6 +1217,33 @@ static void fill_destination_migration_info(MigrationInfo *info)
> >      case MIGRATION_STATUS_COMPLETED:
> >          info->has_status = true;
> >          fill_destination_postcopy_migration_info(info);
> > +        if (mis->total_time > 0) {
> > +            info->has_total_time = true;
> > +            info->total_time = mis->total_time;
> > +        }
> > +        {
> > +            size_t page_size = qemu_target_page_size();
> > +            uint64_t normal  = mis->received_normal_pages;
> > +            uint64_t zero    = mis->received_zero_pages;
> > +            uint64_t xbzrle  = mis->received_xbzrle_pages;
> > +
> > +            info->ram = g_malloc0(sizeof(*info->ram));
> > +            info->ram->total        = ram_bytes_total();
> > +            info->ram->remaining    = 0;
> > +            info->ram->normal       = normal + xbzrle;
> > +            info->ram->normal_bytes = (normal + xbzrle) * page_size;
> > +            info->ram->duplicate    = zero;
> > +            info->ram->transferred  = (normal + xbzrle) * page_size;
> > +            info->ram->page_size    = page_size;
> > +
> > +            if (info->has_total_time) {
> > +                info->ram->mbps = info->ram->transferred * 8.0
> > +                                  / info->total_time / 1000.0;
> > +                info->ram->pages_per_second = (normal + xbzrle + zero)
> > +                                              * 1000.0 / info->total_time;
> > +            }
> > +            /* source-only optional omitted from output */
> > +        }
> >          break;
> >      default:
> >          return;
> > diff --git a/qapi/migration.json b/qapi/migration.json
> > index 7134d4ce47..a695d04a22 100644
> > --- a/qapi/migration.json
> > +++ b/qapi/migration.json
> > @@ -68,13 +68,13 @@
> >    'data': {'transferred': 'int', 'remaining': 'int', 'total': 'int' ,
> >             'duplicate': 'int',
> >             'normal': 'int',
> > -           'normal-bytes': 'int', 'dirty-pages-rate': 'int',
> > -           'mbps': 'number', 'dirty-sync-count': 'int',
> > -           'postcopy-requests': 'int', 'page-size': 'int',
> > -           'multifd-bytes': 'uint64', 'pages-per-second': 'uint64',
> > -           'precopy-bytes': 'uint64', 'downtime-bytes': 'uint64',
> > -           'postcopy-bytes': 'uint64',
> > -           'dirty-sync-missed-zero-copy': 'uint64' } }
> > +           'normal-bytes': 'int', '*dirty-pages-rate': 'int',
> > +           'mbps': 'number', '*dirty-sync-count': 'int',
> > +           '*postcopy-requests': 'int', 'page-size': 'int',
> > +           '*multifd-bytes': 'uint64', 'pages-per-second': 'uint64',
> > +           '*precopy-bytes': 'uint64', '*downtime-bytes': 'uint64',
> > +           '*postcopy-bytes': 'uint64',
> > +           '*dirty-sync-missed-zero-copy': 'uint64' } }
> >  
> >  ##
> >  # @XBZRLECacheStats:
> 
> This looks good and makes sense to me,
> but looking forward to more authoritative voices.
> 
Thanks, let's waiting.
> Does this work build ok on all archs? Any tests that could be extended to cover these new stats,
Actually, I have tested x86_64 but not other archs. The patch are
arch-independent IIUC, so I expected it will work without impact any
existing callers on other archs. Anw, I will check it. 
> maybe tests/qtest/migration/ ?
> 
Thanks for the point. I think can reuse existing test_precopy_tcp_plain.
Will try to add it in v2. 
> Thank you!
> 
> Claudio
> 
BRs,
Trieu Huynh

Re: [PATCH 0/2] migration: include timing and RAM stats on destination when query-migrate

[Peter Xu]

On Mon, Apr 06, 2026 at 11:02:56AM -0300, Fabiano Rosas wrote:
> Trieu Huynh <vikingtc4@gmail.com> writes:
> 
> > From: Trieu Huynh <vikingtc4@gmail.com>
> >
> > When query-migrate is called on the *destination* QEMU after a precopy
> > migration completes it returns only {"status": "completed"} — no timing,
> > no RAM statistics.  The source correctly returns total-time, downtime,
> > setup-time, and full ram stats.
> >
> > This series fixes the gap in two commits:
> >
> > * commit 1: Track start/end timestamps and per-page-type counters in
> >   MigrationIncomingState, recorded in the receive path.
> >
> > * commit 2: Make source-only MigrationStats fields optional in the QAPI
> >   schema so the destination can populate info->ram with only the fields
> >   that are meaningful on the receive side.  Fields computed from tracked
> >   counters (transferred, normal, duplicate, mbps, pages-per-second) are
> >   shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are
> >   absent.
> >
> > The QAPI change (commit 2) is ABI-compatible: optional fields that are
> > not set are simply absent from the output.  Source-side callers are
> > unaffected because populate_ram_info() sets all optional fields via
> > has_* setters, so the source output is identical to before.
> >
> > On dst, {"execute":"query-migrate"}
> > * As-is:
> > {
> >     "return": {
> >         "status": "completed"
> > }
> > * To-be:
> > {
> >     "return": {
> >         "status": "completed",
> >         "total-time": 94,
> >         "ram": {
> >             "total": 554508288,
> >             "pages-per-second": 1440234,
> >             "page-size": 4096,
> >             "remaining": 0,
> >             "mbps": 97.955404255319152,
> >             "transferred": 1150976,
> >             "duplicate": 135101,
> >             "normal-bytes": 1150976,
> >             "normal": 281
> >         }
> >     }
> > }
> >
> > Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
> >
> > Trieu Huynh (2):
> >   migration: track timing and received pages in MigrationIncomingState
> >   migration: expose RAM stats and timing on destination via
> >     query-migrate
> >
> >  migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
> >  migration/migration.h | 17 +++++++++++++++++
> >  migration/ram.c       |  3 +++
> >  qapi/migration.json   | 14 +++++++-------
> >  4 files changed, 64 insertions(+), 7 deletions(-)
> 
> Hi, remember to copy the interested people in your series. Using
> get_maintainers is correct, but when there's already a discussion about
> the topic it's good to add some CCs manually.
> 
> +CC Daniel and Claudio

The request is a valid one.  Said that, redo accounting on both sides seem
to be a duplicated work and overkill to me, if src has everything.

Is there a way we can make Libvirt always query the results before src dies
and report it somewhere?  I recall these info were captured somewhere, do
we at least dump them into logs?

Thanks,

-- 
Peter Xu

Re: [PATCH 0/2] migration: include timing and RAM stats on destination when query-migrate

[Claudio Fontana]

On 4/8/26 22:04, Peter Xu wrote:
> On Mon, Apr 06, 2026 at 11:02:56AM -0300, Fabiano Rosas wrote:
>> Trieu Huynh <vikingtc4@gmail.com> writes:
>>
>>> From: Trieu Huynh <vikingtc4@gmail.com>
>>>
>>> When query-migrate is called on the *destination* QEMU after a precopy
>>> migration completes it returns only {"status": "completed"} — no timing,
>>> no RAM statistics.  The source correctly returns total-time, downtime,
>>> setup-time, and full ram stats.
>>>
>>> This series fixes the gap in two commits:
>>>
>>> * commit 1: Track start/end timestamps and per-page-type counters in
>>>   MigrationIncomingState, recorded in the receive path.
>>>
>>> * commit 2: Make source-only MigrationStats fields optional in the QAPI
>>>   schema so the destination can populate info->ram with only the fields
>>>   that are meaningful on the receive side.  Fields computed from tracked
>>>   counters (transferred, normal, duplicate, mbps, pages-per-second) are
>>>   shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are
>>>   absent.
>>>
>>> The QAPI change (commit 2) is ABI-compatible: optional fields that are
>>> not set are simply absent from the output.  Source-side callers are
>>> unaffected because populate_ram_info() sets all optional fields via
>>> has_* setters, so the source output is identical to before.
>>>
>>> On dst, {"execute":"query-migrate"}
>>> * As-is:
>>> {
>>>     "return": {
>>>         "status": "completed"
>>> }
>>> * To-be:
>>> {
>>>     "return": {
>>>         "status": "completed",
>>>         "total-time": 94,
>>>         "ram": {
>>>             "total": 554508288,
>>>             "pages-per-second": 1440234,
>>>             "page-size": 4096,
>>>             "remaining": 0,
>>>             "mbps": 97.955404255319152,
>>>             "transferred": 1150976,
>>>             "duplicate": 135101,
>>>             "normal-bytes": 1150976,
>>>             "normal": 281
>>>         }
>>>     }
>>> }
>>>
>>> Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
>>>
>>> Trieu Huynh (2):
>>>   migration: track timing and received pages in MigrationIncomingState
>>>   migration: expose RAM stats and timing on destination via
>>>     query-migrate
>>>
>>>  migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
>>>  migration/migration.h | 17 +++++++++++++++++
>>>  migration/ram.c       |  3 +++
>>>  qapi/migration.json   | 14 +++++++-------
>>>  4 files changed, 64 insertions(+), 7 deletions(-)
>>
>> Hi, remember to copy the interested people in your series. Using
>> get_maintainers is correct, but when there's already a discussion about
>> the topic it's good to add some CCs manually.
>>
>> +CC Daniel and Claudio
> 
> The request is a valid one.  Said that, redo accounting on both sides seem
> to be a duplicated work and overkill to me, if src has everything.
> 
> Is there a way we can make Libvirt always query the results before src dies
> and report it somewhere?  I recall these info were captured somewhere, do
> we at least dump them into logs?
> 
> Thanks,
> 

I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.

That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.

So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed,
so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
It might go beyond querying for stats using QEMU and libvirt.

Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:

VIR_MIGRATE_NO_SOURCE_SHUTOFF

So one might say:

flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF
new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)

and then we could do everything that needs to be done from the platform layer,
then issue a virDomainResume to start the domain on destination,
and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.

But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.

Thanks,

Claudio

Re: [PATCH 0/2] migration: include timing and RAM stats on destination when query-migrate

[Fabian Grünbichler]

On April 9, 2026 12:08 pm, Claudio Fontana wrote:
> On 4/8/26 22:04, Peter Xu wrote:
>> On Mon, Apr 06, 2026 at 11:02:56AM -0300, Fabiano Rosas wrote:
>>> Trieu Huynh <vikingtc4@gmail.com> writes:
>>>
>>>> From: Trieu Huynh <vikingtc4@gmail.com>
>>>>
>>>> When query-migrate is called on the *destination* QEMU after a precopy
>>>> migration completes it returns only {"status": "completed"} — no timing,
>>>> no RAM statistics.  The source correctly returns total-time, downtime,
>>>> setup-time, and full ram stats.

>>>> [..]
>>>>
>>>
>>> Hi, remember to copy the interested people in your series. Using
>>> get_maintainers is correct, but when there's already a discussion about
>>> the topic it's good to add some CCs manually.
>>>
>>> +CC Daniel and Claudio
>> 
>> The request is a valid one.  Said that, redo accounting on both sides seem
>> to be a duplicated work and overkill to me, if src has everything.
>> 
>> Is there a way we can make Libvirt always query the results before src dies
>> and report it somewhere?  I recall these info were captured somewhere, do
>> we at least dump them into logs?
>> 
>> Thanks,
>> 
> 
> I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
> 
> That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.

that is correct, we call query-migrate in a loop while migrating both to
log progress and to manage migration parameters, and manually switch
execution and ownership over to the target node once the migration has
converged:

https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/QemuMigrate.pm;h=f7ec322770a47dcc8b89109ecfa6f15ea140f30f;hb=HEAD#l1423

> So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed,
> so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
> It might go beyond querying for stats using QEMU and libvirt.
> 
> Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:
> 
> VIR_MIGRATE_NO_SOURCE_SHUTOFF
> 
> So one might say:
> 
> flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF
> new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)
> 
> and then we could do everything that needs to be done from the platform layer,
> then issue a virDomainResume to start the domain on destination,
> and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.
> 
> But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.
> 
> Thanks,
> 
> Claudio
> 
> 
> 

Re: [PATCH 0/2] migration: include timing and RAM stats on destination when query-migrate

[Claudio Fontana]

On 4/9/26 15:08, Fabian Grünbichler wrote:
> On April 9, 2026 12:08 pm, Claudio Fontana wrote:
>> On 4/8/26 22:04, Peter Xu wrote:
>>> On Mon, Apr 06, 2026 at 11:02:56AM -0300, Fabiano Rosas wrote:
>>>> Trieu Huynh <vikingtc4@gmail.com> writes:
>>>>
>>>>> From: Trieu Huynh <vikingtc4@gmail.com>
>>>>>
>>>>> When query-migrate is called on the *destination* QEMU after a precopy
>>>>> migration completes it returns only {"status": "completed"} — no timing,
>>>>> no RAM statistics.  The source correctly returns total-time, downtime,
>>>>> setup-time, and full ram stats.
> 
>>>>> [..]
>>>>>
>>>>
>>>> Hi, remember to copy the interested people in your series. Using
>>>> get_maintainers is correct, but when there's already a discussion about
>>>> the topic it's good to add some CCs manually.
>>>>
>>>> +CC Daniel and Claudio
>>>
>>> The request is a valid one.  Said that, redo accounting on both sides seem
>>> to be a duplicated work and overkill to me, if src has everything.
>>>
>>> Is there a way we can make Libvirt always query the results before src dies
>>> and report it somewhere?  I recall these info were captured somewhere, do
>>> we at least dump them into logs?
>>>
>>> Thanks,
>>>
>>
>> I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
>>
>> That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.
> 
> that is correct, we call query-migrate in a loop while migrating both to
> log progress and to manage migration parameters, and manually switch
> execution and ownership over to the target node once the migration has
> converged:
> 
> https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/QemuMigrate.pm;h=f7ec322770a47dcc8b89109ecfa6f15ea140f30f;hb=HEAD#l1423

Hello Fabian, thanks for the confirmation,
would it simplify your code if you had the stats already carried over by QEMU to the new QEMU instance?

I suspect you don't want to touch that code without a very compelling reason, but at least in principle it should make things easier?

> 
>> So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed,
>> so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
>> It might go beyond querying for stats using QEMU and libvirt.
>>
>> Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:
>>
>> VIR_MIGRATE_NO_SOURCE_SHUTOFF
>>
>> So one might say:
>>
>> flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF
>> new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)
>>
>> and then we could do everything that needs to be done from the platform layer,
>> then issue a virDomainResume to start the domain on destination,
>> and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.
>>
>> But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.
>>
>> Thanks,
>>
>> Claudio
>>
>>
>>
> 

Re: [PATCH 0/2] migration: include timing and RAM stats on destination when query-migrate

[Fabian Grünbichler]

On April 9, 2026 3:17 pm, Claudio Fontana wrote:
> On 4/9/26 15:08, Fabian Grünbichler wrote:
>> On April 9, 2026 12:08 pm, Claudio Fontana wrote:
>>> On 4/8/26 22:04, Peter Xu wrote:
>>>> On Mon, Apr 06, 2026 at 11:02:56AM -0300, Fabiano Rosas wrote:
>>>>> Trieu Huynh <vikingtc4@gmail.com> writes:
>>>>>
>>>>>> From: Trieu Huynh <vikingtc4@gmail.com>
>>>>>>
>>>>>> When query-migrate is called on the *destination* QEMU after a precopy
>>>>>> migration completes it returns only {"status": "completed"} — no timing,
>>>>>> no RAM statistics.  The source correctly returns total-time, downtime,
>>>>>> setup-time, and full ram stats.
>> 
>>>>>> [..]
>>>>>>
>>>>>
>>>>> Hi, remember to copy the interested people in your series. Using
>>>>> get_maintainers is correct, but when there's already a discussion about
>>>>> the topic it's good to add some CCs manually.
>>>>>
>>>>> +CC Daniel and Claudio
>>>>
>>>> The request is a valid one.  Said that, redo accounting on both sides seem
>>>> to be a duplicated work and overkill to me, if src has everything.
>>>>
>>>> Is there a way we can make Libvirt always query the results before src dies
>>>> and report it somewhere?  I recall these info were captured somewhere, do
>>>> we at least dump them into logs?
>>>>
>>>> Thanks,
>>>>
>>>
>>> I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
>>>
>>> That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.
>> 
>> that is correct, we call query-migrate in a loop while migrating both to
>> log progress and to manage migration parameters, and manually switch
>> execution and ownership over to the target node once the migration has
>> converged:
>> 
>> https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/QemuMigrate.pm;h=f7ec322770a47dcc8b89109ecfa6f15ea140f30f;hb=HEAD#l1423
> 
> Hello Fabian, thanks for the confirmation,
> would it simplify your code if you had the stats already carried over by QEMU to the new QEMU instance?

not really - the source node is driving the migration anyway, so it can
also do the polling + logging. there is no management task running on
the target node/side, after the initial spawning of the VM process.

> I suspect you don't want to touch that code without a very compelling reason, but at least in principle it should make things easier?

Re: [PATCH 0/2] migration: include timing and RAM stats on destination when query-migrate

[Peter Xu]

On Thu, Apr 09, 2026 at 12:08:34PM +0200, Claudio Fontana wrote:
> On 4/8/26 22:04, Peter Xu wrote:
> > On Mon, Apr 06, 2026 at 11:02:56AM -0300, Fabiano Rosas wrote:
> >> Trieu Huynh <vikingtc4@gmail.com> writes:
> >>
> >>> From: Trieu Huynh <vikingtc4@gmail.com>
> >>>
> >>> When query-migrate is called on the *destination* QEMU after a precopy
> >>> migration completes it returns only {"status": "completed"} — no timing,
> >>> no RAM statistics.  The source correctly returns total-time, downtime,
> >>> setup-time, and full ram stats.
> >>>
> >>> This series fixes the gap in two commits:
> >>>
> >>> * commit 1: Track start/end timestamps and per-page-type counters in
> >>>   MigrationIncomingState, recorded in the receive path.
> >>>
> >>> * commit 2: Make source-only MigrationStats fields optional in the QAPI
> >>>   schema so the destination can populate info->ram with only the fields
> >>>   that are meaningful on the receive side.  Fields computed from tracked
> >>>   counters (transferred, normal, duplicate, mbps, pages-per-second) are
> >>>   shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are
> >>>   absent.
> >>>
> >>> The QAPI change (commit 2) is ABI-compatible: optional fields that are
> >>> not set are simply absent from the output.  Source-side callers are
> >>> unaffected because populate_ram_info() sets all optional fields via
> >>> has_* setters, so the source output is identical to before.
> >>>
> >>> On dst, {"execute":"query-migrate"}
> >>> * As-is:
> >>> {
> >>>     "return": {
> >>>         "status": "completed"
> >>> }
> >>> * To-be:
> >>> {
> >>>     "return": {
> >>>         "status": "completed",
> >>>         "total-time": 94,
> >>>         "ram": {
> >>>             "total": 554508288,
> >>>             "pages-per-second": 1440234,
> >>>             "page-size": 4096,
> >>>             "remaining": 0,
> >>>             "mbps": 97.955404255319152,
> >>>             "transferred": 1150976,
> >>>             "duplicate": 135101,
> >>>             "normal-bytes": 1150976,
> >>>             "normal": 281
> >>>         }
> >>>     }
> >>> }
> >>>
> >>> Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
> >>>
> >>> Trieu Huynh (2):
> >>>   migration: track timing and received pages in MigrationIncomingState
> >>>   migration: expose RAM stats and timing on destination via
> >>>     query-migrate
> >>>
> >>>  migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
> >>>  migration/migration.h | 17 +++++++++++++++++
> >>>  migration/ram.c       |  3 +++
> >>>  qapi/migration.json   | 14 +++++++-------
> >>>  4 files changed, 64 insertions(+), 7 deletions(-)
> >>
> >> Hi, remember to copy the interested people in your series. Using
> >> get_maintainers is correct, but when there's already a discussion about
> >> the topic it's good to add some CCs manually.
> >>
> >> +CC Daniel and Claudio
> > 
> > The request is a valid one.  Said that, redo accounting on both sides seem
> > to be a duplicated work and overkill to me, if src has everything.
> > 
> > Is there a way we can make Libvirt always query the results before src dies
> > and report it somewhere?  I recall these info were captured somewhere, do
> > we at least dump them into logs?
> > 
> > Thanks,
> > 
> 
> I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
> 
> That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.

The problem is here from QEMU's perspective we have all the information,
and "not quit" is the default behavior for src QEMU.  It means by default
all info is available.. this is true for no matter what upper mgmt is in
use.

> 
> So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed,
> so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
> It might go beyond querying for stats using QEMU and libvirt.
> 
> Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:
> 
> VIR_MIGRATE_NO_SOURCE_SHUTOFF
> 
> So one might say:
> 
> flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF
> new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)
> 
> and then we could do everything that needs to be done from the platform layer,
> then issue a virDomainResume to start the domain on destination,
> and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.
> 
> But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.

I still think it'll be great we only do accounting once, and above sounds
like a libvirt question that I don't know the answer. Adding Jiri and
libvirt list in case we can collect some inputs.

If you're looking for this feature, IMHO it'll be great if you can have a
look at least from libvirt side to see if there's any blocker for
collecting this info from src, then we can evaluate the rest. E.g. there's
at least another option to forward statistic data from src to dest.  Then
we don't worry on mismatched accounting, or when fixing / add new
accounting we don't need to do it both sides, which will be awkward.

Thanks,

> 
> Thanks,
> 
> Claudio
> 

-- 
Peter Xu

Re: [PATCH 0/2] migration: include timing and RAM stats on destination when query-migrate

[Trieu Huynh]

On Mon, Apr 13, 2026 at 05:07:00PM -0400, Peter Xu wrote:
> On Thu, Apr 09, 2026 at 12:08:34PM +0200, Claudio Fontana wrote:
> > On 4/8/26 22:04, Peter Xu wrote:
> > > On Mon, Apr 06, 2026 at 11:02:56AM -0300, Fabiano Rosas wrote:
> > >> Trieu Huynh <vikingtc4@gmail.com> writes:
> > >>
> > >>> From: Trieu Huynh <vikingtc4@gmail.com>
> > >>>
> > >>> When query-migrate is called on the *destination* QEMU after a precopy
> > >>> migration completes it returns only {"status": "completed"} — no timing,
> > >>> no RAM statistics.  The source correctly returns total-time, downtime,
> > >>> setup-time, and full ram stats.
> > >>>
> > >>> This series fixes the gap in two commits:
> > >>>
> > >>> * commit 1: Track start/end timestamps and per-page-type counters in
> > >>>   MigrationIncomingState, recorded in the receive path.
> > >>>
> > >>> * commit 2: Make source-only MigrationStats fields optional in the QAPI
> > >>>   schema so the destination can populate info->ram with only the fields
> > >>>   that are meaningful on the receive side.  Fields computed from tracked
> > >>>   counters (transferred, normal, duplicate, mbps, pages-per-second) are
> > >>>   shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are
> > >>>   absent.
> > >>>
> > >>> The QAPI change (commit 2) is ABI-compatible: optional fields that are
> > >>> not set are simply absent from the output.  Source-side callers are
> > >>> unaffected because populate_ram_info() sets all optional fields via
> > >>> has_* setters, so the source output is identical to before.
> > >>>
> > >>> On dst, {"execute":"query-migrate"}
> > >>> * As-is:
> > >>> {
> > >>>     "return": {
> > >>>         "status": "completed"
> > >>> }
> > >>> * To-be:
> > >>> {
> > >>>     "return": {
> > >>>         "status": "completed",
> > >>>         "total-time": 94,
> > >>>         "ram": {
> > >>>             "total": 554508288,
> > >>>             "pages-per-second": 1440234,
> > >>>             "page-size": 4096,
> > >>>             "remaining": 0,
> > >>>             "mbps": 97.955404255319152,
> > >>>             "transferred": 1150976,
> > >>>             "duplicate": 135101,
> > >>>             "normal-bytes": 1150976,
> > >>>             "normal": 281
> > >>>         }
> > >>>     }
> > >>> }
> > >>>
> > >>> Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
> > >>>
> > >>> Trieu Huynh (2):
> > >>>   migration: track timing and received pages in MigrationIncomingState
> > >>>   migration: expose RAM stats and timing on destination via
> > >>>     query-migrate
> > >>>
> > >>>  migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
> > >>>  migration/migration.h | 17 +++++++++++++++++
> > >>>  migration/ram.c       |  3 +++
> > >>>  qapi/migration.json   | 14 +++++++-------
> > >>>  4 files changed, 64 insertions(+), 7 deletions(-)
> > >>
> > >> Hi, remember to copy the interested people in your series. Using
> > >> get_maintainers is correct, but when there's already a discussion about
> > >> the topic it's good to add some CCs manually.
> > >>
> > >> +CC Daniel and Claudio
> > > 
> > > The request is a valid one.  Said that, redo accounting on both sides seem
> > > to be a duplicated work and overkill to me, if src has everything.
> > > 
> > > Is there a way we can make Libvirt always query the results before src dies
> > > and report it somewhere?  I recall these info were captured somewhere, do
> > > we at least dump them into logs?
> > > 
> > > Thanks,
> > > 
> > 
> > I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
> > 
> > That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.
> 
> The problem is here from QEMU's perspective we have all the information,
> and "not quit" is the default behavior for src QEMU.  It means by default
> all info is available.. this is true for no matter what upper mgmt is in
> use.
> 
> > 
> > So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed,
> > so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
> > It might go beyond querying for stats using QEMU and libvirt.
> > 
> > Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:
> > 
> > VIR_MIGRATE_NO_SOURCE_SHUTOFF
> > 
> > So one might say:
> > 
> > flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF
> > new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)
> > 
> > and then we could do everything that needs to be done from the platform layer,
> > then issue a virDomainResume to start the domain on destination,
> > and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.
> > 
> > But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.
> 
> I still think it'll be great we only do accounting once, and above sounds
> like a libvirt question that I don't know the answer. Adding Jiri and
> libvirt list in case we can collect some inputs.
> 
> If you're looking for this feature, IMHO it'll be great if you can have a
> look at least from libvirt side to see if there's any blocker for
> collecting this info from src, then we can evaluate the rest. E.g. there's
> at least another option to forward statistic data from src to dest.  Then
> we don't worry on mismatched accounting, or when fixing / add new
> accounting we don't need to do it both sides, which will be awkward.
Hello Peter,
One specific use case where the source is completely unavailable is loading 
from a migration snapshot (non-live).
In this scenario, there is no active source QEMU to query_for_stats. The 
dst (IMHO) is the only entity that can report how long the loading took 
and how much data was actually processed from the snapshot file.
This use case is a little bit niche but still a valid one though, isn't it?
> 
> Thanks,
> 
> > 
> > Thanks,
> > 
> > Claudio
> > 
> 
> -- 
> Peter Xu
> 
BRs,
Trieu Huynh

Re: [PATCH 0/2] migration: include timing and RAM stats on destination when query-migrate

[Peter Xu]

On Thu, Apr 16, 2026 at 11:53:55PM +0700, Trieu Huynh wrote:
> Hello Peter,

Hi, Trieu,

> One specific use case where the source is completely unavailable is loading 
> from a migration snapshot (non-live).
> In this scenario, there is no active source QEMU to query_for_stats. The 
> dst (IMHO) is the only entity that can report how long the loading took 
> and how much data was actually processed from the snapshot file.
> This use case is a little bit niche but still a valid one though, isn't it?

Yes it's in general valid, but it also depends.  There's a reason that QEMU
supported load snapshot over the years and nobody was asking for it.  I
believe it's because people simply don't need it.

Load snapshot, unlike generic form of live migration, doesn't have much
uncertainty on its own.  It should either succeed after a while, or fail
with an error.  People should not expect to need to query anything.

It will likely, and should, hold true even if with async load snapshot in
the future.

It is only valid in that it might be useful for developers who works on
snapshot, especially tuning the performance.  But then it also means we
shouldn't expose it to an user API (QMP is in this case).  For dev only
reportings, we should rely on trace points unless we have good reason to
make it visible to users.

In general, we don't need to add unnecessary API maint burden to ourselves
when not needed.

Thanks,

-- 
Peter Xu

Re: [PATCH 0/2] migration: include timing and RAM stats on destination when query-migrate

[Trieu Huynh]

On Thu, Apr 16, 2026 at 02:50:29PM -0400, Peter Xu wrote:
> On Thu, Apr 16, 2026 at 11:53:55PM +0700, Trieu Huynh wrote:
> > Hello Peter,
> 
> Hi, Trieu,
> 
> > One specific use case where the source is completely unavailable is loading 
> > from a migration snapshot (non-live).
> > In this scenario, there is no active source QEMU to query_for_stats. The 
> > dst (IMHO) is the only entity that can report how long the loading took 
> > and how much data was actually processed from the snapshot file.
> > This use case is a little bit niche but still a valid one though, isn't it?
> 
> Yes it's in general valid, but it also depends.  There's a reason that QEMU
> supported load snapshot over the years and nobody was asking for it.  I
> believe it's because people simply don't need it.
> 
> Load snapshot, unlike generic form of live migration, doesn't have much
> uncertainty on its own.  It should either succeed after a while, or fail
> with an error.  People should not expect to need to query anything.
> 
> It will likely, and should, hold true even if with async load snapshot in
> the future.
> 
> It is only valid in that it might be useful for developers who works on
> snapshot, especially tuning the performance.  But then it also means we
> shouldn't expose it to an user API (QMP is in this case).  For dev only
> reportings, we should rely on trace points unless we have good reason to
> make it visible to users.
> 
> In general, we don't need to add unnecessary API maint burden to ourselves
> when not needed.
Thanks for your detailed feedback, I appreciate it.
Taking your comments from the other thread into account, I might drop this
series for now. As I don't have a deep understanding of libvirt or similar
management tools, the specific use case remains a bit ambiguous to me. I’ll
leave this series/patches here for now in case Claudio and other experts
want to chime in or if further concerns arise. Thanks again!
> 
> Thanks,
> 
> -- 
> Peter Xu
> 
Brs,
Trieu Huynh

Re: [PATCH 0/2] migration: include timing and RAM stats on destination when query-migrate

[Fabiano Rosas]

Peter Xu <peterx@redhat.com> writes:

> On Thu, Apr 09, 2026 at 12:08:34PM +0200, Claudio Fontana wrote:
>> On 4/8/26 22:04, Peter Xu wrote:
>> > On Mon, Apr 06, 2026 at 11:02:56AM -0300, Fabiano Rosas wrote:
>> >> Trieu Huynh <vikingtc4@gmail.com> writes:
>> >>
>> >>> From: Trieu Huynh <vikingtc4@gmail.com>
>> >>>
>> >>> When query-migrate is called on the *destination* QEMU after a precopy
>> >>> migration completes it returns only {"status": "completed"} — no timing,
>> >>> no RAM statistics.  The source correctly returns total-time, downtime,
>> >>> setup-time, and full ram stats.
>> >>>
>> >>> This series fixes the gap in two commits:
>> >>>
>> >>> * commit 1: Track start/end timestamps and per-page-type counters in
>> >>>   MigrationIncomingState, recorded in the receive path.
>> >>>
>> >>> * commit 2: Make source-only MigrationStats fields optional in the QAPI
>> >>>   schema so the destination can populate info->ram with only the fields
>> >>>   that are meaningful on the receive side.  Fields computed from tracked
>> >>>   counters (transferred, normal, duplicate, mbps, pages-per-second) are
>> >>>   shown; source-only fields (dirty-sync-count, precopy-bytes, etc.) are
>> >>>   absent.
>> >>>
>> >>> The QAPI change (commit 2) is ABI-compatible: optional fields that are
>> >>> not set are simply absent from the output.  Source-side callers are
>> >>> unaffected because populate_ram_info() sets all optional fields via
>> >>> has_* setters, so the source output is identical to before.
>> >>>
>> >>> On dst, {"execute":"query-migrate"}
>> >>> * As-is:
>> >>> {
>> >>>     "return": {
>> >>>         "status": "completed"
>> >>> }
>> >>> * To-be:
>> >>> {
>> >>>     "return": {
>> >>>         "status": "completed",
>> >>>         "total-time": 94,
>> >>>         "ram": {
>> >>>             "total": 554508288,
>> >>>             "pages-per-second": 1440234,
>> >>>             "page-size": 4096,
>> >>>             "remaining": 0,
>> >>>             "mbps": 97.955404255319152,
>> >>>             "transferred": 1150976,
>> >>>             "duplicate": 135101,
>> >>>             "normal-bytes": 1150976,
>> >>>             "normal": 281
>> >>>         }
>> >>>     }
>> >>> }
>> >>>
>> >>> Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3151
>> >>>
>> >>> Trieu Huynh (2):
>> >>>   migration: track timing and received pages in MigrationIncomingState
>> >>>   migration: expose RAM stats and timing on destination via
>> >>>     query-migrate
>> >>>
>> >>>  migration/migration.c | 37 +++++++++++++++++++++++++++++++++++++
>> >>>  migration/migration.h | 17 +++++++++++++++++
>> >>>  migration/ram.c       |  3 +++
>> >>>  qapi/migration.json   | 14 +++++++-------
>> >>>  4 files changed, 64 insertions(+), 7 deletions(-)
>> >>
>> >> Hi, remember to copy the interested people in your series. Using
>> >> get_maintainers is correct, but when there's already a discussion about
>> >> the topic it's good to add some CCs manually.
>> >>
>> >> +CC Daniel and Claudio
>> > 
>> > The request is a valid one.  Said that, redo accounting on both sides seem
>> > to be a duplicated work and overkill to me, if src has everything.
>> > 
>> > Is there a way we can make Libvirt always query the results before src dies
>> > and report it somewhere?  I recall these info were captured somewhere, do
>> > we at least dump them into logs?
>> > 
>> > Thanks,
>> > 
>> 
>> I am interested in the outcome, but I think the approach shown here is fine, the amount of duplication seems minimal to me.
>> 
>> That said if we _have_ to keep libvirt in the picture, I think I see a possible alternative, but then the feature is not available to anyone that does not use libvirt. At the same time, if libvirt is not used, one is free to keep source QEMU alive and query the stats, like I think Proxmox does.
>
> The problem is here from QEMU's perspective we have all the information,
> and "not quit" is the default behavior for src QEMU.  It means by default
> all info is available.. this is true for no matter what upper mgmt is in
> use.
>
>> 
>> So with libvirt, I think we need a way to tell libvirt to keep the source QEMU alive before the actual switchover and the source QEMU is destroyed,
>> so that we can do everything that needs to be done from the platform layer before the source QEMU is lost.
>> It might go beyond querying for stats using QEMU and libvirt.
>> 
>> Something similar to (and potentially combined with) virDomainMigrateFlags VIR_MIGRATE_PAUSE, for example:
>> 
>> VIR_MIGRATE_NO_SOURCE_SHUTOFF
>> 
>> So one might say:
>> 
>> flags = VIR_MIGRATE_LIVE | VIR_MIGRATE_PERSIST_DEST | VIR_MIGRATE_UNDEFINE_SOURCE | VIR_MIGRATE_PAUSE | VIR_MIGRATE_NO_SOURCE_SHUTOFF
>> new_domain = virDomainMigrate3(old_domain, conn, params, nparams, flags)
>> 
>> and then we could do everything that needs to be done from the platform layer,
>> then issue a virDomainResume to start the domain on destination,
>> and maybe a virDomainDestroyFlags on the original domain to terminate it on the source.
>> 
>> But I would still favor the simplicity and the overall applicability of this series from Trieu Huynh.
>
> I still think it'll be great we only do accounting once, and above sounds
> like a libvirt question that I don't know the answer. Adding Jiri and
> libvirt list in case we can collect some inputs.
>
> If you're looking for this feature, IMHO it'll be great if you can have a
> look at least from libvirt side to see if there's any blocker for
> collecting this info from src, then we can evaluate the rest. E.g. there's
> at least another option to forward statistic data from src to dest.  Then
> we don't worry on mismatched accounting, or when fixing / add new
> accounting we don't need to do it both sides, which will be awkward.

I came to the thread to suggest this, but I see you already mentioned
it. I'm not sure what you mean by "forward", but I was actually thinking
"migrate". Do we have precedent for migrating a QAPI object? Do we want
to have it? =) Put that MigrationInfo at the end of the stream and we'd
have a nice symmetric interface for query-migrate in both src and dst.

(we have plans to deal with QEMU_VM_VMDESCRIPTION at some point anyway,
we could maybe rework that end section into something useful)

>
> Thanks,
>
>> 
>> Thanks,
>> 
>> Claudio
>> 

Re: [PATCH 0/2] migration: include timing and RAM stats on destination when query-migrate

[Peter Xu]

On Wed, Apr 22, 2026 at 07:46:45PM -0300, Fabiano Rosas wrote:
> I came to the thread to suggest this, but I see you already mentioned
> it. I'm not sure what you mean by "forward", but I was actually thinking
> "migrate". Do we have precedent for migrating a QAPI object? Do we want
> to have it? =) Put that MigrationInfo at the end of the stream and we'd
> have a nice symmetric interface for query-migrate in both src and dst.

Do we need to "migrate" it, say, using VMSD data structures?

We could, but I'm not yet sure if it's a good idea, especially if we should
make that as part of ABI.  It may add some unnecessary burden to us.

> 
> (we have plans to deal with QEMU_VM_VMDESCRIPTION at some point anyway,
> we could maybe rework that end section into something useful)

That might still be a bit early phase to contain all the stats info. For
example, the real downtime needs to be calculated until src receiving the
ACK, that should be after dest receive QEMU_VM_VMDESCRIPTION, IIRC.  So if
we want to at least forward the downtime info, that might be too early.

In all cases, I still think we should evaluate libvirt solution before any
other explorations.. because so far we still don't know why it can't be
done, and it's the simplest afaiu.

-- 
Peter Xu