[PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Peter Xu]

CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907

This series introduces a generic way to specify migration parameters that
can be used even during the early boot phase of QEMU.

One example use case that already existed is CPR-transfer / CPR-exec.
currently QEMU has a temporary global variable (incoming_mode) to achieve
this, but it's hard to understand and this hack bleeded into quite a few
places that we could have avoided.  The lines in patch 2 touched may
provide some idea.

With a generic approach of setting migration parameters with cmdlines, we
can remove this hack meanwhile QEMU should be able to keep the CPR behavior
as before.  To CPR maintainers and reviewers: please have a closer look,
even better if it can be smoke tested, to see if this works for Oracle's
environment, TIA.

The 1st patch implemented that new semantics.  It is straightforward: now
we can setup any migration parameter using an extra line of:

  -incoming config:key1=value1,key2=value2,...

So far only one such instance is allowed for simplicity, but it should be
enough.  We still allow multiple -incoming for specifying channels, used
together with one "-incoming config:*".

Then parameters set this way will be visible almost anytime for QEMU, for
example, during initialization of device backends (which is before
migration object created).

I posted this series majorly because I want to see if this will make a
possible new user for the new "local" migration parameter proposed in
Vladimir's series:

  https://lore.kernel.org/r/20260522120534.77653-1-vsementsov@yandex-team.ru

Especially, there're some context on this idea too in this email:

  https://lore.kernel.org/all/ahdI7Vl5KraK566D@x1.local/

With this series, we should be able to drop "incoming-fds" TAP property
from the other series, instead relying on the existing "local" parameters
both in migration core and in TAP's property should suffice.

One thing to mention is I didn't further make only-migratable into a
migration parameter.  Logically it will also work now with only-migratable,
but it then also means I'll need to convert it to a parameter, which will
be mutable even after VM started.  It will change how only-migratable used
to work, hence I skipped.

After this, we also almost have no reason to use -global for migration
parameters.  Capabilities are still not supported in -incoming cmdline,
though.

Thanks,

Peter Xu (2):
  migration/vl: Allow set parameters with -incoming config:*
  migration/cpr: Opt-in "mode" parameter for early boot access

 include/migration/cpr.h  |  3 --
 include/migration/misc.h |  5 +++
 migration/migration.h    |  3 ++
 migration/cpr.c          | 18 ++++----
 migration/migration.c    | 91 +++++++++++++++++++++++++++++++++++++++-
 migration/options.c      | 10 ++---
 system/vl.c              |  7 ++++
 qemu-options.hx          | 18 +++++++-
 8 files changed, 133 insertions(+), 22 deletions(-)

-- 
2.53.0

[PATCH RFC 1/2] migration/vl: Allow set parameters with -incoming config:*

[Peter Xu]

QEMU doesn't yet have a good way to specify migration parameters so that
they can be available even during early stage of QEMU boots.  It is because
the migration object (who owns the migration parameters) will only be
created after PHASE_LATE_BACKENDS_CREATED.  It means anything before it
reading migration parameters is illegal.

However, QEMU does have special use cases for such, namely only-migratable
flag, and cpr-transfer. Recently, we have one more possible user to read a
to-be-introduced new migration parameters during backend initialization
phase.  We can introduce yet another global variable (or per-device
parameter) to bypass this limitation, but we can also seek for a generic
solution that we can setup migration parameters very early, even during
backend initializations.

See this discussion for more details on the context of the problem:

  https://lore.kernel.org/r/ahdI7Vl5KraK566D@x1.local

This patch wants to take the latter approach.

As a start, introduce a new way to specify migration parameters in QEMU
boot commandline, as proposed in the above discussion:

  -incoming config:key1=value1,key2=value2,...

When specified, QEMU will parse a string formatted MigrationParameters and
keep it.  When migration object is created, it will apply all the settings
as initial value.

Since the application of boot parameters will be after object_new()
completes, it means it happens after all machine compat properties or
-global settings (which should be done during instance_post_init()).

So far, it's still only a way to specify parameters.  All parameters are
not visible before migration object created, like before.  Any parameter
that needs to be visible during boot will need to opt-in this feature.
Follow up patches will switch the current users to use this model.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/migration/misc.h |  5 ++++
 migration/migration.c    | 54 ++++++++++++++++++++++++++++++++++++++++
 system/vl.c              |  7 ++++++
 qemu-options.hx          | 18 ++++++++++++--
 4 files changed, 82 insertions(+), 2 deletions(-)

diff --git a/include/migration/misc.h b/include/migration/misc.h
index 3159a5e53c..aff79b1380 100644
--- a/include/migration/misc.h
+++ b/include/migration/misc.h
@@ -133,6 +133,11 @@ bool migrate_is_uri(const char *uri);
 /* Parse @uri and return @channel, returning true on success */
 bool migrate_uri_parse(const char *uri, MigrationChannel **channel,
                        Error **errp);
+/*
+ * Parse @config_str in form of "config:key1=value1,..." to initialize
+ * migration parameters.
+ */
+bool migration_parameters_boot_parse(const char *config_str, Error **errp);
 
 /* migration/multifd-device-state.c */
 typedef struct SaveCompletePrecopyThreadData {
diff --git a/migration/migration.c b/migration/migration.c
index 074d3f2c69..d918be7a44 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -41,6 +41,7 @@
 #include "qapi/qapi-commands-migration.h"
 #include "qapi/qapi-events-migration.h"
 #include "qapi/qmp/qerror.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qobject/qnull.h"
 #include "qemu/rcu.h"
 #include "postcopy-ram.h"
@@ -94,6 +95,9 @@ enum mig_rp_message_type {
 static MigrationState *current_migration;
 static MigrationIncomingState *current_incoming;
 
+/* Only used during boot, destroyed after migration object initialized */
+static MigrationParameters *mig_boot_params;
+
 static GSList *migration_blockers[MIG_MODE__MAX];
 
 static bool migration_object_check(MigrationState *ms, Error **errp);
@@ -102,6 +106,45 @@ static bool stop_return_path_thread_on_source(MigrationState *s);
 static void migration_release_dst_files(MigrationState *ms);
 static void migration_completion_end(MigrationState *s);
 
+bool migration_parameters_boot_parse(const char *config_str, Error **errp)
+{
+    Visitor *v;
+
+    if (mig_boot_params) {
+        error_setg(errp, "Only one -incoming config:* is allowed.");
+        return false;
+    }
+
+    v = qobject_input_visitor_new_str(config_str, NULL, errp);
+    if (!v) {
+        goto fail;
+    }
+
+    if (!visit_type_MigrationParameters(v, NULL, &mig_boot_params, errp)) {
+        goto fail;
+    }
+
+    visit_free(v);
+    return true;
+
+fail:
+    visit_free(v);
+    return false;
+}
+
+static void migration_parameters_boot_apply(void)
+{
+    if (mig_boot_params) {
+        /*
+         * This can fail, because qobject visitor doesn't do sanity check
+         * on values while parsing.  It's not too late; we're still in boot
+         * phase.
+         */
+        qmp_migrate_set_parameters(mig_boot_params, &error_abort);
+        g_clear_pointer(&mig_boot_params, qapi_free_MigrationParameters);
+    }
+}
+
 static void migration_downtime_start(MigrationState *s)
 {
     trace_vmstate_downtime_checkpoint("src-downtime-start");
@@ -322,6 +365,17 @@ void migration_object_init(void)
 
     current_incoming->exit_on_error = INMIGRATE_DEFAULT_EXIT_ON_ERROR;
 
+    /*
+     * Apply boot migration parameters in case the user specified some via
+     * command line "-incoming config:*". NOTE: this will overwrite machine
+     * type compat properties and -global settings!
+     */
+    migration_parameters_boot_apply();
+
+    /*
+     * The boot parameters should have been verified already, but leave it
+     * after applying boot parameters to do one check for everything.
+     */
     migration_object_check(current_migration, &error_fatal);
 
     ram_mig_init();
diff --git a/system/vl.c b/system/vl.c
index da36b2c6e1..49f5fa0c7b 100644
--- a/system/vl.c
+++ b/system/vl.c
@@ -1838,6 +1838,13 @@ static void incoming_option_parse(const char *str)
 
     if (!strcmp(str, "defer")) {
         channel = NULL;
+    } else if (!strncmp(str, "config:", 7)) {
+        /*
+         * This is not a channel setup, but configuration to incoming
+         * migration parameters to make them available during early boot.
+         */
+        migration_parameters_boot_parse(str + 7, &error_fatal);
+        return;
     } else if (migrate_is_uri(str)) {
         migrate_uri_parse(str, &channel, &error_fatal);
     } else {
diff --git a/qemu-options.hx b/qemu-options.hx
index 96ae41f787..1fc92a409a 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -5366,11 +5366,15 @@ DEF("incoming", HAS_ARG, QEMU_OPTION_incoming, \
     "-incoming <channel>\n" \
     "                accept incoming migration on the migration channel\n" \
     "-incoming defer\n" \
-    "                wait for the URI to be specified via migrate_incoming\n",
+    "                wait for the URI to be specified via migrate_incoming\n"
+    "-incoming config:key1=value1,key2=value2,...\n" \
+    "                specify migration parameters valid even during boot\n",
     QEMU_ARCH_ALL)
 SRST
+
 The -incoming option specifies the migration channel for an incoming
-migration.  It may be used multiple times to specify multiple
+migration, or can also be used to setup migration parameters for the
+incoming migration.  It may be used multiple times to specify multiple
 migration channel types.  The channel type is specified in <channel>,
 or is 'main' for all other forms of -incoming.  If multiple -incoming
 options are specified for a channel type, the last one takes precedence.
@@ -5411,6 +5415,16 @@ options are specified for a channel type, the last one takes precedence.
     Wait for the URI to be specified via migrate\_incoming. The monitor
     can be used to change settings (such as migration parameters) prior
     to issuing the migrate\_incoming to allow the migration to begin.
+
+``-incoming config:key1=value1[,key2=value2,...]``
+
+    Specify migration parameters in QEMU commandlines, so that these
+    parameters will be available even during very early boot of QEMU.
+    They will be applied properly after QEMU boots and when the migration
+    core is initialized.  From that POV, it has similar effect as setting
+    these parameters using QMP command ``migrate-set-parameters`` or HMP
+    command ``migrate_set_parameter``.
+
 ERST
 
 DEF("only-migratable", 0, QEMU_OPTION_only_migratable, \
-- 
2.53.0

[PATCH RFC 2/2] migration/cpr: Opt-in "mode" parameter for early boot access

[Peter Xu]

Make "mode" to be the first parameter to opt-in for early boot access.  CPR
will start to consume this early boot parameter.  With this, we can remove
the special "incoming_mode" variable together with cpr_get_incoming_mode().

One thing to mention is, to make cpr_is_incoming() work like before, we
need to do extra check on INMIGRATE runstate to make sure it only returns
true while during the incoming migration.  It used to be achieved by a
pretty hackish "cpr_set_incoming_mode(MIG_MODE_NONE)" when incoming
migration destroys.  Now we can remove it.

Another good side effect is, we can finally drop MIG_MODE_NONE: it was
never a valid value, only used by the temp global variable.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/migration/cpr.h |  3 ---
 migration/migration.h   |  3 +++
 migration/cpr.c         | 18 +++++++++---------
 migration/migration.c   | 37 ++++++++++++++++++++++++++++++++++++-
 migration/options.c     | 10 +++-------
 5 files changed, 51 insertions(+), 20 deletions(-)

diff --git a/include/migration/cpr.h b/include/migration/cpr.h
index 56fb67e6b4..27061ad629 100644
--- a/include/migration/cpr.h
+++ b/include/migration/cpr.h
@@ -12,8 +12,6 @@
 #include "io/channel.h"
 #include "qemu/queue.h"
 
-#define MIG_MODE_NONE           -1
-
 #define QEMU_CPR_FILE_MAGIC     0x51435052
 #define QEMU_CPR_FILE_VERSION   0x00000001
 #define CPR_STATE "CprState"
@@ -38,7 +36,6 @@ int cpr_open_fd(const char *path, int flags, const char *name, int id,
 typedef bool (*cpr_walk_fd_cb)(int fd);
 bool cpr_walk_fd(cpr_walk_fd_cb cb);
 
-MigMode cpr_get_incoming_mode(void);
 void cpr_set_incoming_mode(MigMode mode);
 bool cpr_is_incoming(void);
 
diff --git a/migration/migration.h b/migration/migration.h
index 841f49b215..9fa97f6d9a 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -609,4 +609,7 @@ void migration_bitmap_sync_precopy(bool last_stage);
 void dirty_bitmap_mig_init(void);
 bool should_send_vmdesc(void);
 
+void migration_parameters_boot_set_mode(MigMode mode);
+MigrationParameters *migration_get_parameters(void);
+
 #endif
diff --git a/migration/cpr.c b/migration/cpr.c
index bca43e9bf3..1f49afe109 100644
--- a/migration/cpr.c
+++ b/migration/cpr.c
@@ -16,6 +16,7 @@
 #include "migration/qemu-file.h"
 #include "migration/savevm.h"
 #include "migration/vmstate.h"
+#include "migration/migration.h"
 #include "monitor/monitor.h"
 #include "system/runstate.h"
 #include "trace.h"
@@ -239,21 +240,20 @@ QIOChannel *cpr_state_ioc(void)
     return qemu_file_get_ioc(cpr_state_file);
 }
 
-static MigMode incoming_mode = MIG_MODE_NONE;
-
-MigMode cpr_get_incoming_mode(void)
-{
-    return incoming_mode;
-}
-
 void cpr_set_incoming_mode(MigMode mode)
 {
-    incoming_mode = mode;
+    migration_parameters_boot_set_mode(mode);
 }
 
 bool cpr_is_incoming(void)
 {
-    return incoming_mode != MIG_MODE_NONE;
+    MigMode mode = migrate_mode();
+
+    if (!runstate_check(RUN_STATE_INMIGRATE)) {
+        return false;
+    }
+
+    return mode == MIG_MODE_CPR_EXEC || mode == MIG_MODE_CPR_TRANSFER;
 }
 
 bool cpr_state_save(MigrationChannel *channel, Error **errp)
diff --git a/migration/migration.c b/migration/migration.c
index d918be7a44..d7591571ed 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -145,6 +145,42 @@ static void migration_parameters_boot_apply(void)
     }
 }
 
+static void migration_parameters_boot_init(void)
+{
+    if (!mig_boot_params) {
+        mig_boot_params = g_new0(MigrationParameters, 1);
+    }
+}
+
+/*
+ * This should only be used during boot by CPR.  NOTE: This is only needed
+ * to be compatible with old CPR use case, if we decide to have users
+ * switch to -incoming config:mode=cpr-* then this can be removed.
+ */
+void migration_parameters_boot_set_mode(MigMode mode)
+{
+    assert(!current_migration);
+    migration_parameters_boot_init();
+    mig_boot_params->has_mode = true;
+    mig_boot_params->mode = mode;
+}
+
+/*
+ * Get the effective migration parameter object.
+ *
+ * Three possibilities:
+ * - Migration object has been initialized, always use it, or,
+ * - Migration boot parameters are initialized, then use it, or,
+ * - return NULL
+ *
+ * Callers should always check non-NULL pointer first before use.
+ */
+MigrationParameters *migration_get_parameters(void)
+{
+    return current_migration ?
+        &current_migration->parameters : mig_boot_params;
+}
+
 static void migration_downtime_start(MigrationState *s)
 {
     trace_vmstate_downtime_checkpoint("src-downtime-start");
@@ -546,7 +582,6 @@ void migration_incoming_state_destroy(void)
         mis->postcopy_qemufile_dst = NULL;
     }
 
-    cpr_set_incoming_mode(MIG_MODE_NONE);
     yank_unregister_instance(MIGRATION_YANK_INSTANCE);
 }
 
diff --git a/migration/options.c b/migration/options.c
index 5cbfd29099..c1209cbec3 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -886,16 +886,12 @@ uint64_t migrate_max_postcopy_bandwidth(void)
     return s->parameters.max_postcopy_bandwidth;
 }
 
+/* Opt-in "mode" parameter to be available even during boot */
 MigMode migrate_mode(void)
 {
-    MigMode mode = cpr_get_incoming_mode();
+    MigrationParameters *params = migration_get_parameters();
 
-    if (mode == MIG_MODE_NONE) {
-        mode = migrate_get_current()->parameters.mode;
-    }
-
-    assert(mode >= 0 && mode < MIG_MODE__MAX);
-    return mode;
+    return params ? params->mode : MIG_MODE_NORMAL;
 }
 
 int migrate_multifd_channels(void)
-- 
2.53.0

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Fabiano Rosas]

Peter Xu <peterx@redhat.com> writes:

> CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907
>
> This series introduces a generic way to specify migration parameters that
> can be used even during the early boot phase of QEMU.
>
> One example use case that already existed is CPR-transfer / CPR-exec.
> currently QEMU has a temporary global variable (incoming_mode) to achieve
> this, but it's hard to understand and this hack bleeded into quite a few
> places that we could have avoided.  The lines in patch 2 touched may
> provide some idea.
>
> With a generic approach of setting migration parameters with cmdlines, we
> can remove this hack meanwhile QEMU should be able to keep the CPR behavior
> as before.  To CPR maintainers and reviewers: please have a closer look,
> even better if it can be smoke tested, to see if this works for Oracle's
> environment, TIA.
>
> The 1st patch implemented that new semantics.  It is straightforward: now
> we can setup any migration parameter using an extra line of:
>
>   -incoming config:key1=value1,key2=value2,...

Just merge my code mate.
https://lore.kernel.org/r/20251215220041.12657-25-farosas@suse.de

A couple of differences from my patch:

- old-style keyval vs. json;

Isn't json preferred nowadays?

- using "config:"

This makes it non-uniform with uri and channels which don't have a
keyword in front of them. I guess I could live with it, but it seems
odd. I see that it makes parsing way easier.

>
> So far only one such instance is allowed for simplicity, but it should be
> enough.  We still allow multiple -incoming for specifying channels, used
> together with one "-incoming config:*".
>
> Then parameters set this way will be visible almost anytime for QEMU, for
> example, during initialization of device backends (which is before
> migration object created).
>
> I posted this series majorly because I want to see if this will make a
> possible new user for the new "local" migration parameter proposed in
> Vladimir's series:
>
>   https://lore.kernel.org/r/20260522120534.77653-1-vsementsov@yandex-team.ru
>
> Especially, there're some context on this idea too in this email:
>
>   https://lore.kernel.org/all/ahdI7Vl5KraK566D@x1.local/
>

I like the idea overall.

> With this series, we should be able to drop "incoming-fds" TAP property
> from the other series, instead relying on the existing "local" parameters
> both in migration core and in TAP's property should suffice.
>
> One thing to mention is I didn't further make only-migratable into a
> migration parameter.  Logically it will also work now with only-migratable,
> but it then also means I'll need to convert it to a parameter, which will
> be mutable even after VM started.  It will change how only-migratable used
> to work, hence I skipped.
>
> After this, we also almost have no reason to use -global for migration
> parameters.  Capabilities are still not supported in -incoming cmdline,
> though.
>

Will we still merge capabilities and parameters? Then it would be a free
upgrade.

> Thanks,
>
> Peter Xu (2):
>   migration/vl: Allow set parameters with -incoming config:*
>   migration/cpr: Opt-in "mode" parameter for early boot access
>
>  include/migration/cpr.h  |  3 --
>  include/migration/misc.h |  5 +++
>  migration/migration.h    |  3 ++
>  migration/cpr.c          | 18 ++++----
>  migration/migration.c    | 91 +++++++++++++++++++++++++++++++++++++++-
>  migration/options.c      | 10 ++---
>  system/vl.c              |  7 ++++
>  qemu-options.hx          | 18 +++++++-
>  8 files changed, 133 insertions(+), 22 deletions(-)

Re: [PATCH RFC 1/2] migration/vl: Allow set parameters with -incoming config:*

[Fabiano Rosas]

Peter Xu <peterx@redhat.com> writes:

> QEMU doesn't yet have a good way to specify migration parameters so that
> they can be available even during early stage of QEMU boots.  It is because
> the migration object (who owns the migration parameters) will only be
> created after PHASE_LATE_BACKENDS_CREATED.  It means anything before it
> reading migration parameters is illegal.
>
> However, QEMU does have special use cases for such, namely only-migratable
> flag, and cpr-transfer. Recently, we have one more possible user to read a
> to-be-introduced new migration parameters during backend initialization
> phase.  We can introduce yet another global variable (or per-device
> parameter) to bypass this limitation, but we can also seek for a generic
> solution that we can setup migration parameters very early, even during
> backend initializations.
>
> See this discussion for more details on the context of the problem:
>
>   https://lore.kernel.org/r/ahdI7Vl5KraK566D@x1.local
>
> This patch wants to take the latter approach.
>
> As a start, introduce a new way to specify migration parameters in QEMU
> boot commandline, as proposed in the above discussion:
>
>   -incoming config:key1=value1,key2=value2,...
>
> When specified, QEMU will parse a string formatted MigrationParameters and
> keep it.  When migration object is created, it will apply all the settings
> as initial value.
>
> Since the application of boot parameters will be after object_new()
> completes, it means it happens after all machine compat properties or
> -global settings (which should be done during instance_post_init()).
>
> So far, it's still only a way to specify parameters.  All parameters are
> not visible before migration object created, like before.  Any parameter
> that needs to be visible during boot will need to opt-in this feature.
> Follow up patches will switch the current users to use this model.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  include/migration/misc.h |  5 ++++
>  migration/migration.c    | 54 ++++++++++++++++++++++++++++++++++++++++
>  system/vl.c              |  7 ++++++
>  qemu-options.hx          | 18 ++++++++++++--
>  4 files changed, 82 insertions(+), 2 deletions(-)
>
> diff --git a/include/migration/misc.h b/include/migration/misc.h
> index 3159a5e53c..aff79b1380 100644
> --- a/include/migration/misc.h
> +++ b/include/migration/misc.h
> @@ -133,6 +133,11 @@ bool migrate_is_uri(const char *uri);
>  /* Parse @uri and return @channel, returning true on success */
>  bool migrate_uri_parse(const char *uri, MigrationChannel **channel,
>                         Error **errp);
> +/*
> + * Parse @config_str in form of "config:key1=value1,..." to initialize
> + * migration parameters.
> + */
> +bool migration_parameters_boot_parse(const char *config_str, Error **errp);
>  

'boot' is not the right word. Maybe 'early'?

>  /* migration/multifd-device-state.c */
>  typedef struct SaveCompletePrecopyThreadData {
> diff --git a/migration/migration.c b/migration/migration.c
> index 074d3f2c69..d918be7a44 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -41,6 +41,7 @@
>  #include "qapi/qapi-commands-migration.h"
>  #include "qapi/qapi-events-migration.h"
>  #include "qapi/qmp/qerror.h"
> +#include "qapi/qobject-input-visitor.h"
>  #include "qobject/qnull.h"
>  #include "qemu/rcu.h"
>  #include "postcopy-ram.h"
> @@ -94,6 +95,9 @@ enum mig_rp_message_type {
>  static MigrationState *current_migration;
>  static MigrationIncomingState *current_incoming;
>  
> +/* Only used during boot, destroyed after migration object initialized */
> +static MigrationParameters *mig_boot_params;
> +
>  static GSList *migration_blockers[MIG_MODE__MAX];
>  
>  static bool migration_object_check(MigrationState *ms, Error **errp);
> @@ -102,6 +106,45 @@ static bool stop_return_path_thread_on_source(MigrationState *s);
>  static void migration_release_dst_files(MigrationState *ms);
>  static void migration_completion_end(MigrationState *s);
>  
> +bool migration_parameters_boot_parse(const char *config_str, Error **errp)
> +{
> +    Visitor *v;
> +
> +    if (mig_boot_params) {
> +        error_setg(errp, "Only one -incoming config:* is allowed.");
> +        return false;
> +    }
> +
> +    v = qobject_input_visitor_new_str(config_str, NULL, errp);
> +    if (!v) {
> +        goto fail;
> +    }
> +
> +    if (!visit_type_MigrationParameters(v, NULL, &mig_boot_params, errp)) {
> +        goto fail;
> +    }
> +
> +    visit_free(v);
> +    return true;
> +
> +fail:
> +    visit_free(v);
> +    return false;
> +}
> +
> +static void migration_parameters_boot_apply(void)
> +{
> +    if (mig_boot_params) {
> +        /*
> +         * This can fail, because qobject visitor doesn't do sanity check
> +         * on values while parsing.  It's not too late; we're still in boot
> +         * phase.
> +         */
> +        qmp_migrate_set_parameters(mig_boot_params, &error_abort);
> +        g_clear_pointer(&mig_boot_params, qapi_free_MigrationParameters);
> +    }
> +}
> +
>  static void migration_downtime_start(MigrationState *s)
>  {
>      trace_vmstate_downtime_checkpoint("src-downtime-start");
> @@ -322,6 +365,17 @@ void migration_object_init(void)
>  
>      current_incoming->exit_on_error = INMIGRATE_DEFAULT_EXIT_ON_ERROR;
>  
> +    /*
> +     * Apply boot migration parameters in case the user specified some via
> +     * command line "-incoming config:*". NOTE: this will overwrite machine
> +     * type compat properties and -global settings!
> +     */

Hmm, but then it's exactly the same as -global migration.* ? Why do we
need to extend -incoming?

Also, I don't see how qmp_migrate_set_parameters can overwrite compat
properties. They're not part of MigrationParameters.

> +    migration_parameters_boot_apply();
> +
> +    /*
> +     * The boot parameters should have been verified already, but leave it
> +     * after applying boot parameters to do one check for everything.
> +     */

Will it get expensive? We do that for every QEMU start.

>      migration_object_check(current_migration, &error_fatal);
>  
>      ram_mig_init();
> diff --git a/system/vl.c b/system/vl.c
> index da36b2c6e1..49f5fa0c7b 100644
> --- a/system/vl.c
> +++ b/system/vl.c
> @@ -1838,6 +1838,13 @@ static void incoming_option_parse(const char *str)
>  
>      if (!strcmp(str, "defer")) {
>          channel = NULL;
> +    } else if (!strncmp(str, "config:", 7)) {
> +        /*
> +         * This is not a channel setup, but configuration to incoming
> +         * migration parameters to make them available during early boot.
> +         */
> +        migration_parameters_boot_parse(str + 7, &error_fatal);
> +        return;
>      } else if (migrate_is_uri(str)) {
>          migrate_uri_parse(str, &channel, &error_fatal);
>      } else {
> diff --git a/qemu-options.hx b/qemu-options.hx
> index 96ae41f787..1fc92a409a 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -5366,11 +5366,15 @@ DEF("incoming", HAS_ARG, QEMU_OPTION_incoming, \
>      "-incoming <channel>\n" \
>      "                accept incoming migration on the migration channel\n" \
>      "-incoming defer\n" \
> -    "                wait for the URI to be specified via migrate_incoming\n",
> +    "                wait for the URI to be specified via migrate_incoming\n"
> +    "-incoming config:key1=value1,key2=value2,...\n" \
> +    "                specify migration parameters valid even during boot\n",
>      QEMU_ARCH_ALL)
>  SRST
> +
>  The -incoming option specifies the migration channel for an incoming
> -migration.  It may be used multiple times to specify multiple
> +migration, or can also be used to setup migration parameters for the
> +incoming migration.  It may be used multiple times to specify multiple
>  migration channel types.  The channel type is specified in <channel>,
>  or is 'main' for all other forms of -incoming.  If multiple -incoming
>  options are specified for a channel type, the last one takes precedence.
> @@ -5411,6 +5415,16 @@ options are specified for a channel type, the last one takes precedence.
>      Wait for the URI to be specified via migrate\_incoming. The monitor
>      can be used to change settings (such as migration parameters) prior
>      to issuing the migrate\_incoming to allow the migration to begin.
> +
> +``-incoming config:key1=value1[,key2=value2,...]``
> +
> +    Specify migration parameters in QEMU commandlines, so that these

command line

> +    parameters will be available even during very early boot of QEMU.
> +    They will be applied properly after QEMU boots and when the migration

The "applied properly" part is implementation detail.

> +    core is initialized.  From that POV, it has similar effect as setting
> +    these parameters using QMP command ``migrate-set-parameters`` or HMP
> +    command ``migrate_set_parameter``.
> +
>  ERST
>  
>  DEF("only-migratable", 0, QEMU_OPTION_only_migratable, \

Re: [PATCH RFC 2/2] migration/cpr: Opt-in "mode" parameter for early boot access

[Fabiano Rosas]

Peter Xu <peterx@redhat.com> writes:

> Make "mode" to be the first parameter to opt-in for early boot access.  CPR
> will start to consume this early boot parameter.  With this, we can remove
> the special "incoming_mode" variable together with cpr_get_incoming_mode().
>

Hmm, it's now possible to invoke QEMU with:

-incoming config:mode=cpr-exec,cpr-exec-command=-incoming config:

Seems like it could be an issue.

> One thing to mention is, to make cpr_is_incoming() work like before, we
> need to do extra check on INMIGRATE runstate to make sure it only returns
> true while during the incoming migration.

...

> It used to be achieved by a
> pretty hackish "cpr_set_incoming_mode(MIG_MODE_NONE)" when incoming
> migration destroys.  Now we can remove it.

That's not it, I think. cpr_is_incoming() only returns true on incoming
migration because on outgoing there's no parsing of -incoming and
therefore cpr_state_load() doesn't call cpr_set_incoming_mode().

>
> Another good side effect is, we can finally drop MIG_MODE_NONE: it was
> never a valid value, only used by the temp global variable.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  include/migration/cpr.h |  3 ---
>  migration/migration.h   |  3 +++
>  migration/cpr.c         | 18 +++++++++---------
>  migration/migration.c   | 37 ++++++++++++++++++++++++++++++++++++-
>  migration/options.c     | 10 +++-------
>  5 files changed, 51 insertions(+), 20 deletions(-)
>
> diff --git a/include/migration/cpr.h b/include/migration/cpr.h
> index 56fb67e6b4..27061ad629 100644
> --- a/include/migration/cpr.h
> +++ b/include/migration/cpr.h
> @@ -12,8 +12,6 @@
>  #include "io/channel.h"
>  #include "qemu/queue.h"
>  
> -#define MIG_MODE_NONE           -1
> -
>  #define QEMU_CPR_FILE_MAGIC     0x51435052
>  #define QEMU_CPR_FILE_VERSION   0x00000001
>  #define CPR_STATE "CprState"
> @@ -38,7 +36,6 @@ int cpr_open_fd(const char *path, int flags, const char *name, int id,
>  typedef bool (*cpr_walk_fd_cb)(int fd);
>  bool cpr_walk_fd(cpr_walk_fd_cb cb);
>  
> -MigMode cpr_get_incoming_mode(void);
>  void cpr_set_incoming_mode(MigMode mode);
>  bool cpr_is_incoming(void);
>  
> diff --git a/migration/migration.h b/migration/migration.h
> index 841f49b215..9fa97f6d9a 100644
> --- a/migration/migration.h
> +++ b/migration/migration.h
> @@ -609,4 +609,7 @@ void migration_bitmap_sync_precopy(bool last_stage);
>  void dirty_bitmap_mig_init(void);
>  bool should_send_vmdesc(void);
>  
> +void migration_parameters_boot_set_mode(MigMode mode);
> +MigrationParameters *migration_get_parameters(void);
> +
>  #endif
> diff --git a/migration/cpr.c b/migration/cpr.c
> index bca43e9bf3..1f49afe109 100644
> --- a/migration/cpr.c
> +++ b/migration/cpr.c
> @@ -16,6 +16,7 @@
>  #include "migration/qemu-file.h"
>  #include "migration/savevm.h"
>  #include "migration/vmstate.h"
> +#include "migration/migration.h"
>  #include "monitor/monitor.h"
>  #include "system/runstate.h"
>  #include "trace.h"
> @@ -239,21 +240,20 @@ QIOChannel *cpr_state_ioc(void)
>      return qemu_file_get_ioc(cpr_state_file);
>  }
>  
> -static MigMode incoming_mode = MIG_MODE_NONE;
> -
> -MigMode cpr_get_incoming_mode(void)
> -{
> -    return incoming_mode;
> -}
> -
>  void cpr_set_incoming_mode(MigMode mode)
>  {
> -    incoming_mode = mode;
> +    migration_parameters_boot_set_mode(mode);
>  }
>  
>  bool cpr_is_incoming(void)
>  {
> -    return incoming_mode != MIG_MODE_NONE;
> +    MigMode mode = migrate_mode();
> +
> +    if (!runstate_check(RUN_STATE_INMIGRATE)) {
> +        return false;
> +    }
> +
> +    return mode == MIG_MODE_CPR_EXEC || mode == MIG_MODE_CPR_TRANSFER;
>  }
>  
>  bool cpr_state_save(MigrationChannel *channel, Error **errp)
> diff --git a/migration/migration.c b/migration/migration.c
> index d918be7a44..d7591571ed 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -145,6 +145,42 @@ static void migration_parameters_boot_apply(void)
>      }
>  }
>  
> +static void migration_parameters_boot_init(void)
> +{
> +    if (!mig_boot_params) {
> +        mig_boot_params = g_new0(MigrationParameters, 1);
> +    }
> +}
> +
> +/*
> + * This should only be used during boot by CPR.  NOTE: This is only needed
> + * to be compatible with old CPR use case, if we decide to have users
> + * switch to -incoming config:mode=cpr-* then this can be removed.

This new scheme does create some usability weirdness of allowing a
parameter to be set in -incoming and then to be changed later with
migrate-set-parameters. Hopefully it's ok.

> + */
> +void migration_parameters_boot_set_mode(MigMode mode)
> +{
> +    assert(!current_migration);
> +    migration_parameters_boot_init();
> +    mig_boot_params->has_mode = true;
> +    mig_boot_params->mode = mode;

I bet there's some visitor trick that could make this function generic
for all parameters.

> +}
> +
> +/*
> + * Get the effective migration parameter object.
> + *
> + * Three possibilities:
> + * - Migration object has been initialized, always use it, or,
> + * - Migration boot parameters are initialized, then use it, or,
> + * - return NULL

Hmm, can't we create the early parameters unconditionally so there's no
NULL case here? A bit annoying having to check for NULL for (eventually)
every parameter, even on the source side.

> + *
> + * Callers should always check non-NULL pointer first before use.
> + */
> +MigrationParameters *migration_get_parameters(void)
> +{
> +    return current_migration ?
> +        &current_migration->parameters : mig_boot_params;
> +}

Feels like time to drop the global parameters already. It's a weird
cognitive load having to grasp that there's two MigrationParameters,
both used in outgoing and incoming, but one can only be set in
incoming. The one stored in current_migration might be used
interchangeably with the one that's flying around. One lives inside and
shares lifetime with MigrationState, the other is dynamically allocated
and has a shorter lifetime.

> +
>  static void migration_downtime_start(MigrationState *s)
>  {
>      trace_vmstate_downtime_checkpoint("src-downtime-start");
> @@ -546,7 +582,6 @@ void migration_incoming_state_destroy(void)
>          mis->postcopy_qemufile_dst = NULL;
>      }
>  
> -    cpr_set_incoming_mode(MIG_MODE_NONE);
>      yank_unregister_instance(MIGRATION_YANK_INSTANCE);
>  }
>  
> diff --git a/migration/options.c b/migration/options.c
> index 5cbfd29099..c1209cbec3 100644
> --- a/migration/options.c
> +++ b/migration/options.c
> @@ -886,16 +886,12 @@ uint64_t migrate_max_postcopy_bandwidth(void)
>      return s->parameters.max_postcopy_bandwidth;
>  }
>  
> +/* Opt-in "mode" parameter to be available even during boot */
>  MigMode migrate_mode(void)
>  {
> -    MigMode mode = cpr_get_incoming_mode();
> +    MigrationParameters *params = migration_get_parameters();
>  
> -    if (mode == MIG_MODE_NONE) {
> -        mode = migrate_get_current()->parameters.mode;
> -    }
> -
> -    assert(mode >= 0 && mode < MIG_MODE__MAX);
> -    return mode;
> +    return params ? params->mode : MIG_MODE_NORMAL;
>  }
>  
>  int migrate_multifd_channels(void)

Re: [PATCH RFC 1/2] migration/vl: Allow set parameters with -incoming config:*

[Vladimir Sementsov-Ogievskiy]

On 29.05.26 00:29, Peter Xu wrote:
> QEMU doesn't yet have a good way to specify migration parameters so that
> they can be available even during early stage of QEMU boots.  It is because
> the migration object (who owns the migration parameters) will only be
> created after PHASE_LATE_BACKENDS_CREATED.  It means anything before it
> reading migration parameters is illegal.
> 
> However, QEMU does have special use cases for such, namely only-migratable
> flag, and cpr-transfer. Recently, we have one more possible user to read a
> to-be-introduced new migration parameters during backend initialization
> phase.  We can introduce yet another global variable (or per-device
> parameter) to bypass this limitation, but we can also seek for a generic
> solution that we can setup migration parameters very early, even during
> backend initializations.
> 
> See this discussion for more details on the context of the problem:
> 
>    https://lore.kernel.org/r/ahdI7Vl5KraK566D@x1.local
> 
> This patch wants to take the latter approach.
> 
> As a start, introduce a new way to specify migration parameters in QEMU
> boot commandline, as proposed in the above discussion:
> 
>    -incoming config:key1=value1,key2=value2,...

Do I understand correctly, that with qobject_input_visitor_new_str(),
it automatically suuport also json-like syntax

      -incoming config:'{"key1": "value1", "key2": {some more complex structure}}'

?

> 
> When specified, QEMU will parse a string formatted MigrationParameters and
> keep it.  When migration object is created, it will apply all the settings
> as initial value.
> 
> Since the application of boot parameters will be after object_new()
> completes, it means it happens after all machine compat properties or
> -global settings (which should be done during instance_post_init()).

[..]

>       to issuing the migrate\_incoming to allow the migration to begin.
> +
> +``-incoming config:key1=value1[,key2=value2,...]``
> +
> +    Specify migration parameters in QEMU commandlines, so that these
> +    parameters will be available even during very early boot of QEMU.
> +    They will be applied properly after QEMU boots and when the migration
> +    core is initialized.  From that POV, it has similar effect as setting
> +    these parameters using QMP command ``migrate-set-parameters`` or HMP
> +    command ``migrate_set_parameter``.
> +
>   ERST
>   
>   DEF("only-migratable", 0, QEMU_OPTION_only_migratable, \

So, to have a "defer" migration with preset parameter, I need two options

     -incoming defer -incoming config:key=val

right?

-- 
Best regards,
Vladimir

Re: [PATCH RFC 1/2] migration/vl: Allow set parameters with -incoming config:*

[Vladimir Sementsov-Ogievskiy]

On 29.05.26 00:29, Peter Xu wrote:
> QEMU doesn't yet have a good way to specify migration parameters so that
> they can be available even during early stage of QEMU boots.  It is because
> the migration object (who owns the migration parameters) will only be
> created after PHASE_LATE_BACKENDS_CREATED.  It means anything before it
> reading migration parameters is illegal.

But with this series, we actually do create a part of this migration object,
which is created earlier.

Is it possible instead to initialize the whole migration object earlier?
Or maybe, somehow "partly" initialize it (if full initialization is not
possible for some reasons), but still keep one object, without external
part, stored in separate global variable?

-- 
Best regards,
Vladimir

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Peter Xu]

On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote:
> Peter Xu <peterx@redhat.com> writes:
> 
> > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907
> >
> > This series introduces a generic way to specify migration parameters that
> > can be used even during the early boot phase of QEMU.
> >
> > One example use case that already existed is CPR-transfer / CPR-exec.
> > currently QEMU has a temporary global variable (incoming_mode) to achieve
> > this, but it's hard to understand and this hack bleeded into quite a few
> > places that we could have avoided.  The lines in patch 2 touched may
> > provide some idea.
> >
> > With a generic approach of setting migration parameters with cmdlines, we
> > can remove this hack meanwhile QEMU should be able to keep the CPR behavior
> > as before.  To CPR maintainers and reviewers: please have a closer look,
> > even better if it can be smoke tested, to see if this works for Oracle's
> > environment, TIA.
> >
> > The 1st patch implemented that new semantics.  It is straightforward: now
> > we can setup any migration parameter using an extra line of:
> >
> >   -incoming config:key1=value1,key2=value2,...
> 
> Just merge my code mate.
> https://lore.kernel.org/r/20251215220041.12657-25-farosas@suse.de

Heh, I apologize. I totally forgot it even if I seem to have read it..

The goal is differnent, though, which is to make it available before
migration object initialization.  Say, I wished -global would work too but
it doesn't, and it just can't.  So it's not "something good to have"
anymore, but functionally required for either CPR and TAP in the future if
we adopt this, they'll all read them early.

> 
> A couple of differences from my patch:
> 
> - old-style keyval vs. json;
> 
> Isn't json preferred nowadays?

I didn't mention it in doc, but yeah this series also supports JSON due to
the same qapi helper used.  I don't suggest json though for this entry (and
that's also why I didn't mention it..) because 99% of migration parameters
are scalars and I don't see why we should use JSON here.. the only one not
scalar is block-bitmap-mapping but it's not intended to be used in
-incoming at all (even if it'll work..).

> 
> - using "config:"
> 
> This makes it non-uniform with uri and channels which don't have a
> keyword in front of them. I guess I could live with it, but it seems
> odd. I see that it makes parsing way easier.

Yeah, having some identifier would be nice.  I wished channels also have
identifiers if we don't need to keep compatibility.

> 
> >
> > So far only one such instance is allowed for simplicity, but it should be
> > enough.  We still allow multiple -incoming for specifying channels, used
> > together with one "-incoming config:*".
> >
> > Then parameters set this way will be visible almost anytime for QEMU, for
> > example, during initialization of device backends (which is before
> > migration object created).
> >
> > I posted this series majorly because I want to see if this will make a
> > possible new user for the new "local" migration parameter proposed in
> > Vladimir's series:
> >
> >   https://lore.kernel.org/r/20260522120534.77653-1-vsementsov@yandex-team.ru
> >
> > Especially, there're some context on this idea too in this email:
> >
> >   https://lore.kernel.org/all/ahdI7Vl5KraK566D@x1.local/
> >
> 
> I like the idea overall.
> 
> > With this series, we should be able to drop "incoming-fds" TAP property
> > from the other series, instead relying on the existing "local" parameters
> > both in migration core and in TAP's property should suffice.
> >
> > One thing to mention is I didn't further make only-migratable into a
> > migration parameter.  Logically it will also work now with only-migratable,
> > but it then also means I'll need to convert it to a parameter, which will
> > be mutable even after VM started.  It will change how only-migratable used
> > to work, hence I skipped.
> >
> > After this, we also almost have no reason to use -global for migration
> > parameters.  Capabilities are still not supported in -incoming cmdline,
> > though.
> >
> 
> Will we still merge capabilities and parameters? Then it would be a free
> upgrade.

Yes!  /me waiting for your patches.

-- 
Peter Xu

Re: [PATCH RFC 1/2] migration/vl: Allow set parameters with -incoming config:*

[Peter Xu]

On Fri, May 29, 2026 at 10:19:22AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 29.05.26 00:29, Peter Xu wrote:
> > QEMU doesn't yet have a good way to specify migration parameters so that
> > they can be available even during early stage of QEMU boots.  It is because
> > the migration object (who owns the migration parameters) will only be
> > created after PHASE_LATE_BACKENDS_CREATED.  It means anything before it
> > reading migration parameters is illegal.
> > 
> > However, QEMU does have special use cases for such, namely only-migratable
> > flag, and cpr-transfer. Recently, we have one more possible user to read a
> > to-be-introduced new migration parameters during backend initialization
> > phase.  We can introduce yet another global variable (or per-device
> > parameter) to bypass this limitation, but we can also seek for a generic
> > solution that we can setup migration parameters very early, even during
> > backend initializations.
> > 
> > See this discussion for more details on the context of the problem:
> > 
> >    https://lore.kernel.org/r/ahdI7Vl5KraK566D@x1.local
> > 
> > This patch wants to take the latter approach.
> > 
> > As a start, introduce a new way to specify migration parameters in QEMU
> > boot commandline, as proposed in the above discussion:
> > 
> >    -incoming config:key1=value1,key2=value2,...
> 
> Do I understand correctly, that with qobject_input_visitor_new_str(),
> it automatically suuport also json-like syntax
> 
>      -incoming config:'{"key1": "value1", "key2": {some more complex structure}}'
> 
> ?

Yes.

> 
> > 
> > When specified, QEMU will parse a string formatted MigrationParameters and
> > keep it.  When migration object is created, it will apply all the settings
> > as initial value.
> > 
> > Since the application of boot parameters will be after object_new()
> > completes, it means it happens after all machine compat properties or
> > -global settings (which should be done during instance_post_init()).
> 
> [..]
> 
> >       to issuing the migrate\_incoming to allow the migration to begin.
> > +
> > +``-incoming config:key1=value1[,key2=value2,...]``
> > +
> > +    Specify migration parameters in QEMU commandlines, so that these
> > +    parameters will be available even during very early boot of QEMU.
> > +    They will be applied properly after QEMU boots and when the migration
> > +    core is initialized.  From that POV, it has similar effect as setting
> > +    these parameters using QMP command ``migrate-set-parameters`` or HMP
> > +    command ``migrate_set_parameter``.
> > +
> >   ERST
> >   DEF("only-migratable", 0, QEMU_OPTION_only_migratable, \
> 
> So, to have a "defer" migration with preset parameter, I need two options
> 
>     -incoming defer -incoming config:key=val
> 
> right?

Yes.

-- 
Peter Xu

Re: [PATCH RFC 1/2] migration/vl: Allow set parameters with -incoming config:*

[Peter Xu]

On Fri, May 29, 2026 at 10:26:26AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 29.05.26 00:29, Peter Xu wrote:
> > QEMU doesn't yet have a good way to specify migration parameters so that
> > they can be available even during early stage of QEMU boots.  It is because
> > the migration object (who owns the migration parameters) will only be
> > created after PHASE_LATE_BACKENDS_CREATED.  It means anything before it
> > reading migration parameters is illegal.
> 
> But with this series, we actually do create a part of this migration object,
> which is created earlier.
> 
> Is it possible instead to initialize the whole migration object earlier?
> Or maybe, somehow "partly" initialize it (if full initialization is not
> possible for some reasons), but still keep one object, without external
> part, stored in separate global variable?

It needs to be there because migration object currently depends on machine
compat properties to apply some parameters on top, see:

$ git grep "\"migration\"" hw/core/machine.c
hw/core/machine.c:    { "migration", "multifd-clean-tls-termination", "false" },
hw/core/machine.c:    { "migration", "send-switchover-start", "off"},
hw/core/machine.c:    { "migration", "zero-page-detection", "legacy"},
hw/core/machine.c:    { "migration", "multifd-flush-after-each-section", "on"},
hw/core/machine.c:    { "migration", "x-preempt-pre-7-2", "true" },

While machine compat properties are initialized in different phases.  The
one we really depend on (per above) is qemu_create_machine(), where
object_set_machine_compat_props() updates object_compat_props[1].

Logically it's fully initialized after all three entries updated, which
means we'd better initialize migration objects after at least
configure_accelerators() which updates object_compat_props[2]..

-- 
Peter Xu

Re: [PATCH RFC 2/2] migration/cpr: Opt-in "mode" parameter for early boot access

[Peter Xu]

On Thu, May 28, 2026 at 08:24:23PM -0300, Fabiano Rosas wrote:
> Peter Xu <peterx@redhat.com> writes:
> 
> > Make "mode" to be the first parameter to opt-in for early boot access.  CPR
> > will start to consume this early boot parameter.  With this, we can remove
> > the special "incoming_mode" variable together with cpr_get_incoming_mode().
> >
> 
> Hmm, it's now possible to invoke QEMU with:
> 
> -incoming config:mode=cpr-exec,cpr-exec-command=-incoming config:
> 
> Seems like it could be an issue.

Do you mean when being abused, or some issue that can happen with existing
cpr-exec?

For normal cpr-exec operation, we should only pass in -incoming file:*.  No
config should be passed in.

But indeed if we want to decouple the implied "mode=" change for cpr-exec
and cpr-transfer, we may want to also have "-incoming config:mode=cpr-exec"
in the cpr-exec-command=.  But still, I don't see an issue so far.

If we think that's too much, we can still stick with the
cpr_set_incoming_mode() implicitly setting mode for cpr-*.

> 
> > One thing to mention is, to make cpr_is_incoming() work like before, we
> > need to do extra check on INMIGRATE runstate to make sure it only returns
> > true while during the incoming migration.
> 
> ...
> 
> > It used to be achieved by a
> > pretty hackish "cpr_set_incoming_mode(MIG_MODE_NONE)" when incoming
> > migration destroys.  Now we can remove it.
> 
> That's not it, I think. cpr_is_incoming() only returns true on incoming
> migration because on outgoing there's no parsing of -incoming and
> therefore cpr_state_load() doesn't call cpr_set_incoming_mode().

Yes, I believe it's done by both these steps: cpr_set_incoming_mode() start
to make it return true, then the other set of MIG_MODE_NONE (that I
mentioned above) to explicitly turn it off.  Now with INMIGRATE check it
should hopefully keep the exact same behavior.

> 
> >
> > Another good side effect is, we can finally drop MIG_MODE_NONE: it was
> > never a valid value, only used by the temp global variable.
> >
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  include/migration/cpr.h |  3 ---
> >  migration/migration.h   |  3 +++
> >  migration/cpr.c         | 18 +++++++++---------
> >  migration/migration.c   | 37 ++++++++++++++++++++++++++++++++++++-
> >  migration/options.c     | 10 +++-------
> >  5 files changed, 51 insertions(+), 20 deletions(-)
> >
> > diff --git a/include/migration/cpr.h b/include/migration/cpr.h
> > index 56fb67e6b4..27061ad629 100644
> > --- a/include/migration/cpr.h
> > +++ b/include/migration/cpr.h
> > @@ -12,8 +12,6 @@
> >  #include "io/channel.h"
> >  #include "qemu/queue.h"
> >  
> > -#define MIG_MODE_NONE           -1
> > -
> >  #define QEMU_CPR_FILE_MAGIC     0x51435052
> >  #define QEMU_CPR_FILE_VERSION   0x00000001
> >  #define CPR_STATE "CprState"
> > @@ -38,7 +36,6 @@ int cpr_open_fd(const char *path, int flags, const char *name, int id,
> >  typedef bool (*cpr_walk_fd_cb)(int fd);
> >  bool cpr_walk_fd(cpr_walk_fd_cb cb);
> >  
> > -MigMode cpr_get_incoming_mode(void);
> >  void cpr_set_incoming_mode(MigMode mode);
> >  bool cpr_is_incoming(void);
> >  
> > diff --git a/migration/migration.h b/migration/migration.h
> > index 841f49b215..9fa97f6d9a 100644
> > --- a/migration/migration.h
> > +++ b/migration/migration.h
> > @@ -609,4 +609,7 @@ void migration_bitmap_sync_precopy(bool last_stage);
> >  void dirty_bitmap_mig_init(void);
> >  bool should_send_vmdesc(void);
> >  
> > +void migration_parameters_boot_set_mode(MigMode mode);
> > +MigrationParameters *migration_get_parameters(void);
> > +
> >  #endif
> > diff --git a/migration/cpr.c b/migration/cpr.c
> > index bca43e9bf3..1f49afe109 100644
> > --- a/migration/cpr.c
> > +++ b/migration/cpr.c
> > @@ -16,6 +16,7 @@
> >  #include "migration/qemu-file.h"
> >  #include "migration/savevm.h"
> >  #include "migration/vmstate.h"
> > +#include "migration/migration.h"
> >  #include "monitor/monitor.h"
> >  #include "system/runstate.h"
> >  #include "trace.h"
> > @@ -239,21 +240,20 @@ QIOChannel *cpr_state_ioc(void)
> >      return qemu_file_get_ioc(cpr_state_file);
> >  }
> >  
> > -static MigMode incoming_mode = MIG_MODE_NONE;
> > -
> > -MigMode cpr_get_incoming_mode(void)
> > -{
> > -    return incoming_mode;
> > -}
> > -
> >  void cpr_set_incoming_mode(MigMode mode)
> >  {
> > -    incoming_mode = mode;
> > +    migration_parameters_boot_set_mode(mode);
> >  }
> >  
> >  bool cpr_is_incoming(void)
> >  {
> > -    return incoming_mode != MIG_MODE_NONE;
> > +    MigMode mode = migrate_mode();
> > +
> > +    if (!runstate_check(RUN_STATE_INMIGRATE)) {
> > +        return false;
> > +    }
> > +
> > +    return mode == MIG_MODE_CPR_EXEC || mode == MIG_MODE_CPR_TRANSFER;
> >  }
> >  
> >  bool cpr_state_save(MigrationChannel *channel, Error **errp)
> > diff --git a/migration/migration.c b/migration/migration.c
> > index d918be7a44..d7591571ed 100644
> > --- a/migration/migration.c
> > +++ b/migration/migration.c
> > @@ -145,6 +145,42 @@ static void migration_parameters_boot_apply(void)
> >      }
> >  }
> >  
> > +static void migration_parameters_boot_init(void)
> > +{
> > +    if (!mig_boot_params) {
> > +        mig_boot_params = g_new0(MigrationParameters, 1);
> > +    }
> > +}
> > +
> > +/*
> > + * This should only be used during boot by CPR.  NOTE: This is only needed
> > + * to be compatible with old CPR use case, if we decide to have users
> > + * switch to -incoming config:mode=cpr-* then this can be removed.
> 
> This new scheme does create some usability weirdness of allowing a
> parameter to be set in -incoming and then to be changed later with
> migrate-set-parameters. Hopefully it's ok.

I assume it's OK?  It's the same as -global migration.* in this case,
updates to global parameters.

> 
> > + */
> > +void migration_parameters_boot_set_mode(MigMode mode)
> > +{
> > +    assert(!current_migration);
> > +    migration_parameters_boot_init();
> > +    mig_boot_params->has_mode = true;
> > +    mig_boot_params->mode = mode;
> 
> I bet there's some visitor trick that could make this function generic
> for all parameters.

Yeah, that's why I relied on the qmp_migrate_set_parameters() API to apply
these configs in patch 1, waiting for you to introduce that trick in your
other series to not hard-code has_* checks. :)

But IMHO it's not needed here: we shouldn't even need this helper if we
don't implicitly set mode..  The plan is we should never need another
parameter for such helper.. hence I made it directly accessing it.

> 
> > +}
> > +
> > +/*
> > + * Get the effective migration parameter object.
> > + *
> > + * Three possibilities:
> > + * - Migration object has been initialized, always use it, or,
> > + * - Migration boot parameters are initialized, then use it, or,
> > + * - return NULL
> 
> Hmm, can't we create the early parameters unconditionally so there's no
> NULL case here? A bit annoying having to check for NULL for (eventually)
> every parameter, even on the source side.

We could do that, but remember the default values for parameters are only
applied so far until migration objects init (migration_properties, when
qdev walks it over; unfortunately still a TYPE_DEVICE).

If you want me to finish that part first I can try too; If we plan to merge
the other incoming-fds to TAP then there's also no rush for this work, we
can add some pre-requisite changes.

Or, this series can also be the minimum to drop TAP's incoming-fds, then we
cleanup this NULL case later.

> 
> > + *
> > + * Callers should always check non-NULL pointer first before use.
> > + */
> > +MigrationParameters *migration_get_parameters(void)
> > +{
> > +    return current_migration ?
> > +        &current_migration->parameters : mig_boot_params;
> > +}
> 
> Feels like time to drop the global parameters already. It's a weird
> cognitive load having to grasp that there's two MigrationParameters,
> both used in outgoing and incoming, but one can only be set in
> incoming. The one stored in current_migration might be used
> interchangeably with the one that's flying around. One lives inside and
> shares lifetime with MigrationState, the other is dynamically allocated
> and has a shorter lifetime.

The idea is making mig_boot_params so temprary so nobody notices.

Note that we have at least these layers of values to apply on top of the
parameters:

a) default value (via migration_properties, discussed above)
b) machine type compat value (via instance_post_init() of mig obj)
c) "-incoming config:*" that this series introduced

IIUC, the sane way to prioritize these values is having 3) higher priority
than 1) and 2).  In this case, we will need a buffering mechanism anyway
when:

- take early migration parameters, we make sure parameters available almost
  since the start

- ... qemu initializes machine compat properties ...

- migration object init, apply machine compat properties

When applying machine compat properties, the migration parameter object
cannot be the same that persisted the boot values, so we likely always need
two of them so that after apply compat properties we use boot parameters to
overwrite them.

> 
> > +
> >  static void migration_downtime_start(MigrationState *s)
> >  {
> >      trace_vmstate_downtime_checkpoint("src-downtime-start");
> > @@ -546,7 +582,6 @@ void migration_incoming_state_destroy(void)
> >          mis->postcopy_qemufile_dst = NULL;
> >      }
> >  
> > -    cpr_set_incoming_mode(MIG_MODE_NONE);
> >      yank_unregister_instance(MIGRATION_YANK_INSTANCE);
> >  }
> >  
> > diff --git a/migration/options.c b/migration/options.c
> > index 5cbfd29099..c1209cbec3 100644
> > --- a/migration/options.c
> > +++ b/migration/options.c
> > @@ -886,16 +886,12 @@ uint64_t migrate_max_postcopy_bandwidth(void)
> >      return s->parameters.max_postcopy_bandwidth;
> >  }
> >  
> > +/* Opt-in "mode" parameter to be available even during boot */
> >  MigMode migrate_mode(void)
> >  {
> > -    MigMode mode = cpr_get_incoming_mode();
> > +    MigrationParameters *params = migration_get_parameters();
> >  
> > -    if (mode == MIG_MODE_NONE) {
> > -        mode = migrate_get_current()->parameters.mode;
> > -    }
> > -
> > -    assert(mode >= 0 && mode < MIG_MODE__MAX);
> > -    return mode;
> > +    return params ? params->mode : MIG_MODE_NORMAL;
> >  }
> >  
> >  int migrate_multifd_channels(void)
> 

-- 
Peter Xu

Re: [PATCH RFC 1/2] migration/vl: Allow set parameters with -incoming config:*

[Peter Xu]

On Thu, May 28, 2026 at 07:16:57PM -0300, Fabiano Rosas wrote:
> Peter Xu <peterx@redhat.com> writes:
> 
> > QEMU doesn't yet have a good way to specify migration parameters so that
> > they can be available even during early stage of QEMU boots.  It is because
> > the migration object (who owns the migration parameters) will only be
> > created after PHASE_LATE_BACKENDS_CREATED.  It means anything before it
> > reading migration parameters is illegal.
> >
> > However, QEMU does have special use cases for such, namely only-migratable
> > flag, and cpr-transfer. Recently, we have one more possible user to read a
> > to-be-introduced new migration parameters during backend initialization
> > phase.  We can introduce yet another global variable (or per-device
> > parameter) to bypass this limitation, but we can also seek for a generic
> > solution that we can setup migration parameters very early, even during
> > backend initializations.
> >
> > See this discussion for more details on the context of the problem:
> >
> >   https://lore.kernel.org/r/ahdI7Vl5KraK566D@x1.local
> >
> > This patch wants to take the latter approach.
> >
> > As a start, introduce a new way to specify migration parameters in QEMU
> > boot commandline, as proposed in the above discussion:
> >
> >   -incoming config:key1=value1,key2=value2,...
> >
> > When specified, QEMU will parse a string formatted MigrationParameters and
> > keep it.  When migration object is created, it will apply all the settings
> > as initial value.
> >
> > Since the application of boot parameters will be after object_new()
> > completes, it means it happens after all machine compat properties or
> > -global settings (which should be done during instance_post_init()).
> >
> > So far, it's still only a way to specify parameters.  All parameters are
> > not visible before migration object created, like before.  Any parameter
> > that needs to be visible during boot will need to opt-in this feature.
> > Follow up patches will switch the current users to use this model.
> >
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  include/migration/misc.h |  5 ++++
> >  migration/migration.c    | 54 ++++++++++++++++++++++++++++++++++++++++
> >  system/vl.c              |  7 ++++++
> >  qemu-options.hx          | 18 ++++++++++++--
> >  4 files changed, 82 insertions(+), 2 deletions(-)
> >
> > diff --git a/include/migration/misc.h b/include/migration/misc.h
> > index 3159a5e53c..aff79b1380 100644
> > --- a/include/migration/misc.h
> > +++ b/include/migration/misc.h
> > @@ -133,6 +133,11 @@ bool migrate_is_uri(const char *uri);
> >  /* Parse @uri and return @channel, returning true on success */
> >  bool migrate_uri_parse(const char *uri, MigrationChannel **channel,
> >                         Error **errp);
> > +/*
> > + * Parse @config_str in form of "config:key1=value1,..." to initialize
> > + * migration parameters.
> > + */
> > +bool migration_parameters_boot_parse(const char *config_str, Error **errp);
> >  
> 
> 'boot' is not the right word. Maybe 'early'?

I am ok to switch to "early".  Another option is.. "cmdline"?

> 
> >  /* migration/multifd-device-state.c */
> >  typedef struct SaveCompletePrecopyThreadData {
> > diff --git a/migration/migration.c b/migration/migration.c
> > index 074d3f2c69..d918be7a44 100644
> > --- a/migration/migration.c
> > +++ b/migration/migration.c
> > @@ -41,6 +41,7 @@
> >  #include "qapi/qapi-commands-migration.h"
> >  #include "qapi/qapi-events-migration.h"
> >  #include "qapi/qmp/qerror.h"
> > +#include "qapi/qobject-input-visitor.h"
> >  #include "qobject/qnull.h"
> >  #include "qemu/rcu.h"
> >  #include "postcopy-ram.h"
> > @@ -94,6 +95,9 @@ enum mig_rp_message_type {
> >  static MigrationState *current_migration;
> >  static MigrationIncomingState *current_incoming;
> >  
> > +/* Only used during boot, destroyed after migration object initialized */
> > +static MigrationParameters *mig_boot_params;
> > +
> >  static GSList *migration_blockers[MIG_MODE__MAX];
> >  
> >  static bool migration_object_check(MigrationState *ms, Error **errp);
> > @@ -102,6 +106,45 @@ static bool stop_return_path_thread_on_source(MigrationState *s);
> >  static void migration_release_dst_files(MigrationState *ms);
> >  static void migration_completion_end(MigrationState *s);
> >  
> > +bool migration_parameters_boot_parse(const char *config_str, Error **errp)
> > +{
> > +    Visitor *v;
> > +
> > +    if (mig_boot_params) {
> > +        error_setg(errp, "Only one -incoming config:* is allowed.");
> > +        return false;
> > +    }
> > +
> > +    v = qobject_input_visitor_new_str(config_str, NULL, errp);
> > +    if (!v) {
> > +        goto fail;
> > +    }
> > +
> > +    if (!visit_type_MigrationParameters(v, NULL, &mig_boot_params, errp)) {
> > +        goto fail;
> > +    }
> > +
> > +    visit_free(v);
> > +    return true;
> > +
> > +fail:
> > +    visit_free(v);
> > +    return false;
> > +}
> > +
> > +static void migration_parameters_boot_apply(void)
> > +{
> > +    if (mig_boot_params) {
> > +        /*
> > +         * This can fail, because qobject visitor doesn't do sanity check
> > +         * on values while parsing.  It's not too late; we're still in boot
> > +         * phase.
> > +         */
> > +        qmp_migrate_set_parameters(mig_boot_params, &error_abort);
> > +        g_clear_pointer(&mig_boot_params, qapi_free_MigrationParameters);
> > +    }
> > +}
> > +
> >  static void migration_downtime_start(MigrationState *s)
> >  {
> >      trace_vmstate_downtime_checkpoint("src-downtime-start");
> > @@ -322,6 +365,17 @@ void migration_object_init(void)
> >  
> >      current_incoming->exit_on_error = INMIGRATE_DEFAULT_EXIT_ON_ERROR;
> >  
> > +    /*
> > +     * Apply boot migration parameters in case the user specified some via
> > +     * command line "-incoming config:*". NOTE: this will overwrite machine
> > +     * type compat properties and -global settings!
> > +     */
> 
> Hmm, but then it's exactly the same as -global migration.* ? Why do we
> need to extend -incoming?
> 
> Also, I don't see how qmp_migrate_set_parameters can overwrite compat
> properties. They're not part of MigrationParameters.

Some of this should have been mentioned in my other replies, let me know if
there's still concern over there after reading those.

In this case, after we set current_migration pointer compat properties are
applied, due to instance_post_init() happening within object_new().

> 
> > +    migration_parameters_boot_apply();
> > +
> > +    /*
> > +     * The boot parameters should have been verified already, but leave it
> > +     * after applying boot parameters to do one check for everything.
> > +     */
> 
> Will it get expensive? We do that for every QEMU start.

Checking caps and parameters?  Why expensive?

Here I should have mentioned we must do one shot check here, because
sometimes there're illegal mixtures of "-incoming config*" and default
values only after they merged together.  I'll update the comment for it,
IOW we must check here anyway.

> 
> >      migration_object_check(current_migration, &error_fatal);
> >  
> >      ram_mig_init();
> > diff --git a/system/vl.c b/system/vl.c
> > index da36b2c6e1..49f5fa0c7b 100644
> > --- a/system/vl.c
> > +++ b/system/vl.c
> > @@ -1838,6 +1838,13 @@ static void incoming_option_parse(const char *str)
> >  
> >      if (!strcmp(str, "defer")) {
> >          channel = NULL;
> > +    } else if (!strncmp(str, "config:", 7)) {
> > +        /*
> > +         * This is not a channel setup, but configuration to incoming
> > +         * migration parameters to make them available during early boot.
> > +         */
> > +        migration_parameters_boot_parse(str + 7, &error_fatal);
> > +        return;
> >      } else if (migrate_is_uri(str)) {
> >          migrate_uri_parse(str, &channel, &error_fatal);
> >      } else {
> > diff --git a/qemu-options.hx b/qemu-options.hx
> > index 96ae41f787..1fc92a409a 100644
> > --- a/qemu-options.hx
> > +++ b/qemu-options.hx
> > @@ -5366,11 +5366,15 @@ DEF("incoming", HAS_ARG, QEMU_OPTION_incoming, \
> >      "-incoming <channel>\n" \
> >      "                accept incoming migration on the migration channel\n" \
> >      "-incoming defer\n" \
> > -    "                wait for the URI to be specified via migrate_incoming\n",
> > +    "                wait for the URI to be specified via migrate_incoming\n"
> > +    "-incoming config:key1=value1,key2=value2,...\n" \
> > +    "                specify migration parameters valid even during boot\n",
> >      QEMU_ARCH_ALL)
> >  SRST
> > +
> >  The -incoming option specifies the migration channel for an incoming
> > -migration.  It may be used multiple times to specify multiple
> > +migration, or can also be used to setup migration parameters for the
> > +incoming migration.  It may be used multiple times to specify multiple
> >  migration channel types.  The channel type is specified in <channel>,
> >  or is 'main' for all other forms of -incoming.  If multiple -incoming
> >  options are specified for a channel type, the last one takes precedence.
> > @@ -5411,6 +5415,16 @@ options are specified for a channel type, the last one takes precedence.
> >      Wait for the URI to be specified via migrate\_incoming. The monitor
> >      can be used to change settings (such as migration parameters) prior
> >      to issuing the migrate\_incoming to allow the migration to begin.
> > +
> > +``-incoming config:key1=value1[,key2=value2,...]``
> > +
> > +    Specify migration parameters in QEMU commandlines, so that these
> 
> command line

Fixed.

> 
> > +    parameters will be available even during very early boot of QEMU.
> > +    They will be applied properly after QEMU boots and when the migration
> 
> The "applied properly" part is implementation detail.

True, I simplified it:

``-incoming config:key1=value1[,key2=value2,...]``

    Specify migration parameters in QEMU command line, so that these
    parameters will be available even during very early boot of QEMU.  It
    has similar effect as setting these parameters using QMP command
    ``migrate-set-parameters`` or HMP command ``migrate_set_parameter``.

> 
> > +    core is initialized.  From that POV, it has similar effect as setting
> > +    these parameters using QMP command ``migrate-set-parameters`` or HMP
> > +    command ``migrate_set_parameter``.
> > +
> >  ERST
> >  
> >  DEF("only-migratable", 0, QEMU_OPTION_only_migratable, \
> 

-- 
Peter Xu

Re: [PATCH RFC 1/2] migration/vl: Allow set parameters with -incoming config:*

[Fabiano Rosas]

Peter Xu <peterx@redhat.com> writes:

> On Thu, May 28, 2026 at 07:16:57PM -0300, Fabiano Rosas wrote:
>> Peter Xu <peterx@redhat.com> writes:
>> 
>> > QEMU doesn't yet have a good way to specify migration parameters so that
>> > they can be available even during early stage of QEMU boots.  It is because
>> > the migration object (who owns the migration parameters) will only be
>> > created after PHASE_LATE_BACKENDS_CREATED.  It means anything before it
>> > reading migration parameters is illegal.
>> >
>> > However, QEMU does have special use cases for such, namely only-migratable
>> > flag, and cpr-transfer. Recently, we have one more possible user to read a
>> > to-be-introduced new migration parameters during backend initialization
>> > phase.  We can introduce yet another global variable (or per-device
>> > parameter) to bypass this limitation, but we can also seek for a generic
>> > solution that we can setup migration parameters very early, even during
>> > backend initializations.
>> >
>> > See this discussion for more details on the context of the problem:
>> >
>> >   https://lore.kernel.org/r/ahdI7Vl5KraK566D@x1.local
>> >
>> > This patch wants to take the latter approach.
>> >
>> > As a start, introduce a new way to specify migration parameters in QEMU
>> > boot commandline, as proposed in the above discussion:
>> >
>> >   -incoming config:key1=value1,key2=value2,...
>> >
>> > When specified, QEMU will parse a string formatted MigrationParameters and
>> > keep it.  When migration object is created, it will apply all the settings
>> > as initial value.
>> >
>> > Since the application of boot parameters will be after object_new()
>> > completes, it means it happens after all machine compat properties or
>> > -global settings (which should be done during instance_post_init()).
>> >
>> > So far, it's still only a way to specify parameters.  All parameters are
>> > not visible before migration object created, like before.  Any parameter
>> > that needs to be visible during boot will need to opt-in this feature.
>> > Follow up patches will switch the current users to use this model.
>> >
>> > Signed-off-by: Peter Xu <peterx@redhat.com>
>> > ---
>> >  include/migration/misc.h |  5 ++++
>> >  migration/migration.c    | 54 ++++++++++++++++++++++++++++++++++++++++
>> >  system/vl.c              |  7 ++++++
>> >  qemu-options.hx          | 18 ++++++++++++--
>> >  4 files changed, 82 insertions(+), 2 deletions(-)
>> >
>> > diff --git a/include/migration/misc.h b/include/migration/misc.h
>> > index 3159a5e53c..aff79b1380 100644
>> > --- a/include/migration/misc.h
>> > +++ b/include/migration/misc.h
>> > @@ -133,6 +133,11 @@ bool migrate_is_uri(const char *uri);
>> >  /* Parse @uri and return @channel, returning true on success */
>> >  bool migrate_uri_parse(const char *uri, MigrationChannel **channel,
>> >                         Error **errp);
>> > +/*
>> > + * Parse @config_str in form of "config:key1=value1,..." to initialize
>> > + * migration parameters.
>> > + */
>> > +bool migration_parameters_boot_parse(const char *config_str, Error **errp);
>> >  
>> 
>> 'boot' is not the right word. Maybe 'early'?
>
> I am ok to switch to "early".  Another option is.. "cmdline"?
>

Either is fine.

>> 
>> >  /* migration/multifd-device-state.c */
>> >  typedef struct SaveCompletePrecopyThreadData {
>> > diff --git a/migration/migration.c b/migration/migration.c
>> > index 074d3f2c69..d918be7a44 100644
>> > --- a/migration/migration.c
>> > +++ b/migration/migration.c
>> > @@ -41,6 +41,7 @@
>> >  #include "qapi/qapi-commands-migration.h"
>> >  #include "qapi/qapi-events-migration.h"
>> >  #include "qapi/qmp/qerror.h"
>> > +#include "qapi/qobject-input-visitor.h"
>> >  #include "qobject/qnull.h"
>> >  #include "qemu/rcu.h"
>> >  #include "postcopy-ram.h"
>> > @@ -94,6 +95,9 @@ enum mig_rp_message_type {
>> >  static MigrationState *current_migration;
>> >  static MigrationIncomingState *current_incoming;
>> >  
>> > +/* Only used during boot, destroyed after migration object initialized */
>> > +static MigrationParameters *mig_boot_params;
>> > +
>> >  static GSList *migration_blockers[MIG_MODE__MAX];
>> >  
>> >  static bool migration_object_check(MigrationState *ms, Error **errp);
>> > @@ -102,6 +106,45 @@ static bool stop_return_path_thread_on_source(MigrationState *s);
>> >  static void migration_release_dst_files(MigrationState *ms);
>> >  static void migration_completion_end(MigrationState *s);
>> >  
>> > +bool migration_parameters_boot_parse(const char *config_str, Error **errp)
>> > +{
>> > +    Visitor *v;
>> > +
>> > +    if (mig_boot_params) {
>> > +        error_setg(errp, "Only one -incoming config:* is allowed.");
>> > +        return false;
>> > +    }
>> > +
>> > +    v = qobject_input_visitor_new_str(config_str, NULL, errp);
>> > +    if (!v) {
>> > +        goto fail;
>> > +    }
>> > +
>> > +    if (!visit_type_MigrationParameters(v, NULL, &mig_boot_params, errp)) {
>> > +        goto fail;
>> > +    }
>> > +
>> > +    visit_free(v);
>> > +    return true;
>> > +
>> > +fail:
>> > +    visit_free(v);
>> > +    return false;
>> > +}
>> > +
>> > +static void migration_parameters_boot_apply(void)
>> > +{
>> > +    if (mig_boot_params) {
>> > +        /*
>> > +         * This can fail, because qobject visitor doesn't do sanity check
>> > +         * on values while parsing.  It's not too late; we're still in boot
>> > +         * phase.
>> > +         */
>> > +        qmp_migrate_set_parameters(mig_boot_params, &error_abort);
>> > +        g_clear_pointer(&mig_boot_params, qapi_free_MigrationParameters);
>> > +    }
>> > +}
>> > +
>> >  static void migration_downtime_start(MigrationState *s)
>> >  {
>> >      trace_vmstate_downtime_checkpoint("src-downtime-start");
>> > @@ -322,6 +365,17 @@ void migration_object_init(void)
>> >  
>> >      current_incoming->exit_on_error = INMIGRATE_DEFAULT_EXIT_ON_ERROR;
>> >  
>> > +    /*
>> > +     * Apply boot migration parameters in case the user specified some via
>> > +     * command line "-incoming config:*". NOTE: this will overwrite machine
>> > +     * type compat properties and -global settings!
>> > +     */
>> 
>> Hmm, but then it's exactly the same as -global migration.* ? Why do we
>> need to extend -incoming?
>> 
>> Also, I don't see how qmp_migrate_set_parameters can overwrite compat
>> properties. They're not part of MigrationParameters.
>
> Some of this should have been mentioned in my other replies, let me know if
> there's still concern over there after reading those.
>

The question about -global is non-sensical, forget about it.

> In this case, after we set current_migration pointer compat properties are
> applied, due to instance_post_init() happening within object_new().
>

Yes, but migration_properties has two distinct sets of Properties, one
contains only compat properties that match machine.c properties and the
other has properties that go into MigrationParameters. I was under the
impression that we never tied compat to parameters.

The only machine compat property that is also a migration parameter I
see is zero-page-detection. Maybe we shouldn't have done that. Now we
can't decouple MigrationParameters from -global. Unless, of course we
declare that from now on compat needs to be done with a new property
that's not part of MigrationParameters; and duplicate
zero-page-detection.

>> 
>> > +    migration_parameters_boot_apply();
>> > +
>> > +    /*
>> > +     * The boot parameters should have been verified already, but leave it
>> > +     * after applying boot parameters to do one check for everything.
>> > +     */
>> 
>> Will it get expensive? We do that for every QEMU start.
>
> Checking caps and parameters?  Why expensive?
>

Just because we're putting more migration code in QEMU's startup
path. Most QEMU invocations don't care about migration. Eventually,
checking every parameter twice might have an impact.

> Here I should have mentioned we must do one shot check here, because
> sometimes there're illegal mixtures of "-incoming config*" and default
> values only after they merged together.  I'll update the comment for it,
> IOW we must check here anyway.
>

Right.

>> 
>> >      migration_object_check(current_migration, &error_fatal);
>> >  
>> >      ram_mig_init();
>> > diff --git a/system/vl.c b/system/vl.c
>> > index da36b2c6e1..49f5fa0c7b 100644
>> > --- a/system/vl.c
>> > +++ b/system/vl.c
>> > @@ -1838,6 +1838,13 @@ static void incoming_option_parse(const char *str)
>> >  
>> >      if (!strcmp(str, "defer")) {
>> >          channel = NULL;
>> > +    } else if (!strncmp(str, "config:", 7)) {
>> > +        /*
>> > +         * This is not a channel setup, but configuration to incoming
>> > +         * migration parameters to make them available during early boot.
>> > +         */
>> > +        migration_parameters_boot_parse(str + 7, &error_fatal);
>> > +        return;
>> >      } else if (migrate_is_uri(str)) {
>> >          migrate_uri_parse(str, &channel, &error_fatal);
>> >      } else {
>> > diff --git a/qemu-options.hx b/qemu-options.hx
>> > index 96ae41f787..1fc92a409a 100644
>> > --- a/qemu-options.hx
>> > +++ b/qemu-options.hx
>> > @@ -5366,11 +5366,15 @@ DEF("incoming", HAS_ARG, QEMU_OPTION_incoming, \
>> >      "-incoming <channel>\n" \
>> >      "                accept incoming migration on the migration channel\n" \
>> >      "-incoming defer\n" \
>> > -    "                wait for the URI to be specified via migrate_incoming\n",
>> > +    "                wait for the URI to be specified via migrate_incoming\n"
>> > +    "-incoming config:key1=value1,key2=value2,...\n" \
>> > +    "                specify migration parameters valid even during boot\n",
>> >      QEMU_ARCH_ALL)
>> >  SRST
>> > +
>> >  The -incoming option specifies the migration channel for an incoming
>> > -migration.  It may be used multiple times to specify multiple
>> > +migration, or can also be used to setup migration parameters for the
>> > +incoming migration.  It may be used multiple times to specify multiple
>> >  migration channel types.  The channel type is specified in <channel>,
>> >  or is 'main' for all other forms of -incoming.  If multiple -incoming
>> >  options are specified for a channel type, the last one takes precedence.
>> > @@ -5411,6 +5415,16 @@ options are specified for a channel type, the last one takes precedence.
>> >      Wait for the URI to be specified via migrate\_incoming. The monitor
>> >      can be used to change settings (such as migration parameters) prior
>> >      to issuing the migrate\_incoming to allow the migration to begin.
>> > +
>> > +``-incoming config:key1=value1[,key2=value2,...]``
>> > +
>> > +    Specify migration parameters in QEMU commandlines, so that these
>> 
>> command line
>
> Fixed.
>
>> 
>> > +    parameters will be available even during very early boot of QEMU.
>> > +    They will be applied properly after QEMU boots and when the migration
>> 
>> The "applied properly" part is implementation detail.
>
> True, I simplified it:
>
> ``-incoming config:key1=value1[,key2=value2,...]``
>
>     Specify migration parameters in QEMU command line, so that these
>     parameters will be available even during very early boot of QEMU.  It
>     has similar effect as setting these parameters using QMP command
>     ``migrate-set-parameters`` or HMP command ``migrate_set_parameter``.
>

ack.

>> 
>> > +    core is initialized.  From that POV, it has similar effect as setting
>> > +    these parameters using QMP command ``migrate-set-parameters`` or HMP
>> > +    command ``migrate_set_parameter``.
>> > +
>> >  ERST
>> >  
>> >  DEF("only-migratable", 0, QEMU_OPTION_only_migratable, \
>> 

Re: [PATCH RFC 2/2] migration/cpr: Opt-in "mode" parameter for early boot access

[Fabiano Rosas]

Peter Xu <peterx@redhat.com> writes:

> On Thu, May 28, 2026 at 08:24:23PM -0300, Fabiano Rosas wrote:
>> Peter Xu <peterx@redhat.com> writes:
>> 
>> > Make "mode" to be the first parameter to opt-in for early boot access.  CPR
>> > will start to consume this early boot parameter.  With this, we can remove
>> > the special "incoming_mode" variable together with cpr_get_incoming_mode().
>> >
>> 
>> Hmm, it's now possible to invoke QEMU with:
>> 
>> -incoming config:mode=cpr-exec,cpr-exec-command=-incoming config:
>> 
>> Seems like it could be an issue.
>
> Do you mean when being abused, or some issue that can happen with existing
> cpr-exec?
>

Abused. Or just overall user confusion over the syntax.

You imply in this patch that we want the users to eventually set the
mode in the -incoming option instead of relying in the automatic
detection at cpr_state_load(). Ok.

Then it's natural that the user will also want to set cpr-exec-command
via the -incoming option. But the definition of cpr-exec-command says
that it must include the -incoming option for the new machine. So now
the -incoming option has to deal with a potentially huge string. It's
also undefined what happens if the second commad line also contains
cpr-exec-command in it.

> For normal cpr-exec operation, we should only pass in -incoming file:*.  No
> config should be passed in.
>

Hmm, but there's no "should" here, is there? This patch allows the mode
to be set.

> But indeed if we want to decouple the implied "mode=" change for cpr-exec
> and cpr-transfer, we may want to also have "-incoming config:mode=cpr-exec"
> in the cpr-exec-command=.  But still, I don't see an issue so far.
>

Ok, keep it in mind though.

> If we think that's too much, we can still stick with the
> cpr_set_incoming_mode() implicitly setting mode for cpr-*.
>

Meh.. It's fine either way I think.

>> 
>> > One thing to mention is, to make cpr_is_incoming() work like before, we
>> > need to do extra check on INMIGRATE runstate to make sure it only returns
>> > true while during the incoming migration.
>> 
>> ...
>> 
>> > It used to be achieved by a
>> > pretty hackish "cpr_set_incoming_mode(MIG_MODE_NONE)" when incoming
>> > migration destroys.  Now we can remove it.
>> 
>> That's not it, I think. cpr_is_incoming() only returns true on incoming
>> migration because on outgoing there's no parsing of -incoming and
>> therefore cpr_state_load() doesn't call cpr_set_incoming_mode().
>
> Yes, I believe it's done by both these steps: cpr_set_incoming_mode() start
> to make it return true, then the other set of MIG_MODE_NONE (that I
> mentioned above) to explicitly turn it off.  Now with INMIGRATE check it
> should hopefully keep the exact same behavior.
>

Well, the new QEMU could be used to live update again, then the mode
should indeed be reset. Might as well have been made part of
migrate_prepare. I don't see the need to clear it like that aside from a
general cleanup action. Whatever, it's not relevant to this discussion.

>> 
>> >
>> > Another good side effect is, we can finally drop MIG_MODE_NONE: it was
>> > never a valid value, only used by the temp global variable.
>> >
>> > Signed-off-by: Peter Xu <peterx@redhat.com>
>> > ---
>> >  include/migration/cpr.h |  3 ---
>> >  migration/migration.h   |  3 +++
>> >  migration/cpr.c         | 18 +++++++++---------
>> >  migration/migration.c   | 37 ++++++++++++++++++++++++++++++++++++-
>> >  migration/options.c     | 10 +++-------
>> >  5 files changed, 51 insertions(+), 20 deletions(-)
>> >
>> > diff --git a/include/migration/cpr.h b/include/migration/cpr.h
>> > index 56fb67e6b4..27061ad629 100644
>> > --- a/include/migration/cpr.h
>> > +++ b/include/migration/cpr.h
>> > @@ -12,8 +12,6 @@
>> >  #include "io/channel.h"
>> >  #include "qemu/queue.h"
>> >  
>> > -#define MIG_MODE_NONE           -1
>> > -
>> >  #define QEMU_CPR_FILE_MAGIC     0x51435052
>> >  #define QEMU_CPR_FILE_VERSION   0x00000001
>> >  #define CPR_STATE "CprState"
>> > @@ -38,7 +36,6 @@ int cpr_open_fd(const char *path, int flags, const char *name, int id,
>> >  typedef bool (*cpr_walk_fd_cb)(int fd);
>> >  bool cpr_walk_fd(cpr_walk_fd_cb cb);
>> >  
>> > -MigMode cpr_get_incoming_mode(void);
>> >  void cpr_set_incoming_mode(MigMode mode);
>> >  bool cpr_is_incoming(void);
>> >  
>> > diff --git a/migration/migration.h b/migration/migration.h
>> > index 841f49b215..9fa97f6d9a 100644
>> > --- a/migration/migration.h
>> > +++ b/migration/migration.h
>> > @@ -609,4 +609,7 @@ void migration_bitmap_sync_precopy(bool last_stage);
>> >  void dirty_bitmap_mig_init(void);
>> >  bool should_send_vmdesc(void);
>> >  
>> > +void migration_parameters_boot_set_mode(MigMode mode);
>> > +MigrationParameters *migration_get_parameters(void);
>> > +
>> >  #endif
>> > diff --git a/migration/cpr.c b/migration/cpr.c
>> > index bca43e9bf3..1f49afe109 100644
>> > --- a/migration/cpr.c
>> > +++ b/migration/cpr.c
>> > @@ -16,6 +16,7 @@
>> >  #include "migration/qemu-file.h"
>> >  #include "migration/savevm.h"
>> >  #include "migration/vmstate.h"
>> > +#include "migration/migration.h"
>> >  #include "monitor/monitor.h"
>> >  #include "system/runstate.h"
>> >  #include "trace.h"
>> > @@ -239,21 +240,20 @@ QIOChannel *cpr_state_ioc(void)
>> >      return qemu_file_get_ioc(cpr_state_file);
>> >  }
>> >  
>> > -static MigMode incoming_mode = MIG_MODE_NONE;
>> > -
>> > -MigMode cpr_get_incoming_mode(void)
>> > -{
>> > -    return incoming_mode;
>> > -}
>> > -
>> >  void cpr_set_incoming_mode(MigMode mode)
>> >  {
>> > -    incoming_mode = mode;
>> > +    migration_parameters_boot_set_mode(mode);
>> >  }
>> >  
>> >  bool cpr_is_incoming(void)
>> >  {
>> > -    return incoming_mode != MIG_MODE_NONE;
>> > +    MigMode mode = migrate_mode();
>> > +
>> > +    if (!runstate_check(RUN_STATE_INMIGRATE)) {
>> > +        return false;
>> > +    }
>> > +
>> > +    return mode == MIG_MODE_CPR_EXEC || mode == MIG_MODE_CPR_TRANSFER;
>> >  }
>> >  
>> >  bool cpr_state_save(MigrationChannel *channel, Error **errp)
>> > diff --git a/migration/migration.c b/migration/migration.c
>> > index d918be7a44..d7591571ed 100644
>> > --- a/migration/migration.c
>> > +++ b/migration/migration.c
>> > @@ -145,6 +145,42 @@ static void migration_parameters_boot_apply(void)
>> >      }
>> >  }
>> >  
>> > +static void migration_parameters_boot_init(void)
>> > +{
>> > +    if (!mig_boot_params) {
>> > +        mig_boot_params = g_new0(MigrationParameters, 1);
>> > +    }
>> > +}
>> > +
>> > +/*
>> > + * This should only be used during boot by CPR.  NOTE: This is only needed
>> > + * to be compatible with old CPR use case, if we decide to have users
>> > + * switch to -incoming config:mode=cpr-* then this can be removed.
>> 
>> This new scheme does create some usability weirdness of allowing a
>> parameter to be set in -incoming and then to be changed later with
>> migrate-set-parameters. Hopefully it's ok.
>
> I assume it's OK?  It's the same as -global migration.* in this case,
> updates to global parameters.
>
>> 
>> > + */
>> > +void migration_parameters_boot_set_mode(MigMode mode)
>> > +{
>> > +    assert(!current_migration);
>> > +    migration_parameters_boot_init();
>> > +    mig_boot_params->has_mode = true;
>> > +    mig_boot_params->mode = mode;
>> 
>> I bet there's some visitor trick that could make this function generic
>> for all parameters.
>
> Yeah, that's why I relied on the qmp_migrate_set_parameters() API to apply
> these configs in patch 1, waiting for you to introduce that trick in your
> other series to not hard-code has_* checks. :)
>
> But IMHO it's not needed here: we shouldn't even need this helper if we
> don't implicitly set mode..  The plan is we should never need another
> parameter for such helper.. hence I made it directly accessing it.
>

ok

>> 
>> > +}
>> > +
>> > +/*
>> > + * Get the effective migration parameter object.
>> > + *
>> > + * Three possibilities:
>> > + * - Migration object has been initialized, always use it, or,
>> > + * - Migration boot parameters are initialized, then use it, or,
>> > + * - return NULL
>> 
>> Hmm, can't we create the early parameters unconditionally so there's no
>> NULL case here? A bit annoying having to check for NULL for (eventually)
>> every parameter, even on the source side.
>
> We could do that, but remember the default values for parameters are only
> applied so far until migration objects init (migration_properties, when
> qdev walks it over; unfortunately still a TYPE_DEVICE).
>

Sorry, you kind of lost me. What do the defaults have to do with
allocating mig_boot_params? Do you mean mig_boot_params cannot really be
used interchangeably with s->parameters because it doesn't have any
defaults set?

> If you want me to finish that part first I can try too; If we plan to merge
> the other incoming-fds to TAP then there's also no rush for this work, we
> can add some pre-requisite changes.
>
> Or, this series can also be the minimum to drop TAP's incoming-fds, then we
> cleanup this NULL case later.
>

Sure, let's help with Vladimir's work. But "migration_get_parameters"
returning something other than a valid MigrationParameters is prone to
cause issues down the line. It's not obvious that the early parameters
are only valid on the incoming side and also not obvious that the
function might be called too early even for early parameters.

>> 
>> > + *
>> > + * Callers should always check non-NULL pointer first before use.
>> > + */
>> > +MigrationParameters *migration_get_parameters(void)
>> > +{
>> > +    return current_migration ?
>> > +        &current_migration->parameters : mig_boot_params;
>> > +}
>> 
>> Feels like time to drop the global parameters already. It's a weird
>> cognitive load having to grasp that there's two MigrationParameters,
>> both used in outgoing and incoming, but one can only be set in
>> incoming. The one stored in current_migration might be used
>> interchangeably with the one that's flying around. One lives inside and
>> shares lifetime with MigrationState, the other is dynamically allocated
>> and has a shorter lifetime.
>
> The idea is making mig_boot_params so temprary so nobody notices.
>
> Note that we have at least these layers of values to apply on top of the
> parameters:
>
> a) default value (via migration_properties, discussed above)
> b) machine type compat value (via instance_post_init() of mig obj)
> c) "-incoming config:*" that this series introduced
>
> IIUC, the sane way to prioritize these values is having 3) higher priority
> than 1) and 2).  In this case, we will need a buffering mechanism anyway
> when:
>
> - take early migration parameters, we make sure parameters available almost
>   since the start
>
> - ... qemu initializes machine compat properties ...
>
> - migration object init, apply machine compat properties
>
> When applying machine compat properties, the migration parameter object
> cannot be the same that persisted the boot values, so we likely always need
> two of them so that after apply compat properties we use boot parameters to
> overwrite them.
>

Or we move MigrationParameters out of MigrationState and use it however
we want. At migration_instance_init() add a:

params->zero_page_detection = ms->zero_page_detection;

and that's it.

Of course, it disables the -global usage for setting random
parameters. But the compat properties remain in migration_properties
working like today.

I understand it's a whole can of worms by itself, but don't you think
it's worth it? It would allow to use a single MigrationParameters object
all over.

>> 
>> > +
>> >  static void migration_downtime_start(MigrationState *s)
>> >  {
>> >      trace_vmstate_downtime_checkpoint("src-downtime-start");
>> > @@ -546,7 +582,6 @@ void migration_incoming_state_destroy(void)
>> >          mis->postcopy_qemufile_dst = NULL;
>> >      }
>> >  
>> > -    cpr_set_incoming_mode(MIG_MODE_NONE);
>> >      yank_unregister_instance(MIGRATION_YANK_INSTANCE);
>> >  }
>> >  
>> > diff --git a/migration/options.c b/migration/options.c
>> > index 5cbfd29099..c1209cbec3 100644
>> > --- a/migration/options.c
>> > +++ b/migration/options.c
>> > @@ -886,16 +886,12 @@ uint64_t migrate_max_postcopy_bandwidth(void)
>> >      return s->parameters.max_postcopy_bandwidth;
>> >  }
>> >  
>> > +/* Opt-in "mode" parameter to be available even during boot */
>> >  MigMode migrate_mode(void)
>> >  {
>> > -    MigMode mode = cpr_get_incoming_mode();
>> > +    MigrationParameters *params = migration_get_parameters();
>> >  
>> > -    if (mode == MIG_MODE_NONE) {
>> > -        mode = migrate_get_current()->parameters.mode;
>> > -    }
>> > -
>> > -    assert(mode >= 0 && mode < MIG_MODE__MAX);
>> > -    return mode;
>> > +    return params ? params->mode : MIG_MODE_NORMAL;
>> >  }
>> >  
>> >  int migrate_multifd_channels(void)
>> 

Re: [PATCH RFC 1/2] migration/vl: Allow set parameters with -incoming config:*

[Peter Xu]

On Fri, May 29, 2026 at 01:44:11PM -0300, Fabiano Rosas wrote:
> Peter Xu <peterx@redhat.com> writes:
> 
> > On Thu, May 28, 2026 at 07:16:57PM -0300, Fabiano Rosas wrote:
> >> Peter Xu <peterx@redhat.com> writes:
> >> 
> >> > QEMU doesn't yet have a good way to specify migration parameters so that
> >> > they can be available even during early stage of QEMU boots.  It is because
> >> > the migration object (who owns the migration parameters) will only be
> >> > created after PHASE_LATE_BACKENDS_CREATED.  It means anything before it
> >> > reading migration parameters is illegal.
> >> >
> >> > However, QEMU does have special use cases for such, namely only-migratable
> >> > flag, and cpr-transfer. Recently, we have one more possible user to read a
> >> > to-be-introduced new migration parameters during backend initialization
> >> > phase.  We can introduce yet another global variable (or per-device
> >> > parameter) to bypass this limitation, but we can also seek for a generic
> >> > solution that we can setup migration parameters very early, even during
> >> > backend initializations.
> >> >
> >> > See this discussion for more details on the context of the problem:
> >> >
> >> >   https://lore.kernel.org/r/ahdI7Vl5KraK566D@x1.local
> >> >
> >> > This patch wants to take the latter approach.
> >> >
> >> > As a start, introduce a new way to specify migration parameters in QEMU
> >> > boot commandline, as proposed in the above discussion:
> >> >
> >> >   -incoming config:key1=value1,key2=value2,...
> >> >
> >> > When specified, QEMU will parse a string formatted MigrationParameters and
> >> > keep it.  When migration object is created, it will apply all the settings
> >> > as initial value.
> >> >
> >> > Since the application of boot parameters will be after object_new()
> >> > completes, it means it happens after all machine compat properties or
> >> > -global settings (which should be done during instance_post_init()).
> >> >
> >> > So far, it's still only a way to specify parameters.  All parameters are
> >> > not visible before migration object created, like before.  Any parameter
> >> > that needs to be visible during boot will need to opt-in this feature.
> >> > Follow up patches will switch the current users to use this model.
> >> >
> >> > Signed-off-by: Peter Xu <peterx@redhat.com>
> >> > ---
> >> >  include/migration/misc.h |  5 ++++
> >> >  migration/migration.c    | 54 ++++++++++++++++++++++++++++++++++++++++
> >> >  system/vl.c              |  7 ++++++
> >> >  qemu-options.hx          | 18 ++++++++++++--
> >> >  4 files changed, 82 insertions(+), 2 deletions(-)
> >> >
> >> > diff --git a/include/migration/misc.h b/include/migration/misc.h
> >> > index 3159a5e53c..aff79b1380 100644
> >> > --- a/include/migration/misc.h
> >> > +++ b/include/migration/misc.h
> >> > @@ -133,6 +133,11 @@ bool migrate_is_uri(const char *uri);
> >> >  /* Parse @uri and return @channel, returning true on success */
> >> >  bool migrate_uri_parse(const char *uri, MigrationChannel **channel,
> >> >                         Error **errp);
> >> > +/*
> >> > + * Parse @config_str in form of "config:key1=value1,..." to initialize
> >> > + * migration parameters.
> >> > + */
> >> > +bool migration_parameters_boot_parse(const char *config_str, Error **errp);
> >> >  
> >> 
> >> 'boot' is not the right word. Maybe 'early'?
> >
> > I am ok to switch to "early".  Another option is.. "cmdline"?
> >
> 
> Either is fine.
> 
> >> 
> >> >  /* migration/multifd-device-state.c */
> >> >  typedef struct SaveCompletePrecopyThreadData {
> >> > diff --git a/migration/migration.c b/migration/migration.c
> >> > index 074d3f2c69..d918be7a44 100644
> >> > --- a/migration/migration.c
> >> > +++ b/migration/migration.c
> >> > @@ -41,6 +41,7 @@
> >> >  #include "qapi/qapi-commands-migration.h"
> >> >  #include "qapi/qapi-events-migration.h"
> >> >  #include "qapi/qmp/qerror.h"
> >> > +#include "qapi/qobject-input-visitor.h"
> >> >  #include "qobject/qnull.h"
> >> >  #include "qemu/rcu.h"
> >> >  #include "postcopy-ram.h"
> >> > @@ -94,6 +95,9 @@ enum mig_rp_message_type {
> >> >  static MigrationState *current_migration;
> >> >  static MigrationIncomingState *current_incoming;
> >> >  
> >> > +/* Only used during boot, destroyed after migration object initialized */
> >> > +static MigrationParameters *mig_boot_params;
> >> > +
> >> >  static GSList *migration_blockers[MIG_MODE__MAX];
> >> >  
> >> >  static bool migration_object_check(MigrationState *ms, Error **errp);
> >> > @@ -102,6 +106,45 @@ static bool stop_return_path_thread_on_source(MigrationState *s);
> >> >  static void migration_release_dst_files(MigrationState *ms);
> >> >  static void migration_completion_end(MigrationState *s);
> >> >  
> >> > +bool migration_parameters_boot_parse(const char *config_str, Error **errp)
> >> > +{
> >> > +    Visitor *v;
> >> > +
> >> > +    if (mig_boot_params) {
> >> > +        error_setg(errp, "Only one -incoming config:* is allowed.");
> >> > +        return false;
> >> > +    }
> >> > +
> >> > +    v = qobject_input_visitor_new_str(config_str, NULL, errp);
> >> > +    if (!v) {
> >> > +        goto fail;
> >> > +    }
> >> > +
> >> > +    if (!visit_type_MigrationParameters(v, NULL, &mig_boot_params, errp)) {
> >> > +        goto fail;
> >> > +    }
> >> > +
> >> > +    visit_free(v);
> >> > +    return true;
> >> > +
> >> > +fail:
> >> > +    visit_free(v);
> >> > +    return false;
> >> > +}
> >> > +
> >> > +static void migration_parameters_boot_apply(void)
> >> > +{
> >> > +    if (mig_boot_params) {
> >> > +        /*
> >> > +         * This can fail, because qobject visitor doesn't do sanity check
> >> > +         * on values while parsing.  It's not too late; we're still in boot
> >> > +         * phase.
> >> > +         */
> >> > +        qmp_migrate_set_parameters(mig_boot_params, &error_abort);
> >> > +        g_clear_pointer(&mig_boot_params, qapi_free_MigrationParameters);
> >> > +    }
> >> > +}
> >> > +
> >> >  static void migration_downtime_start(MigrationState *s)
> >> >  {
> >> >      trace_vmstate_downtime_checkpoint("src-downtime-start");
> >> > @@ -322,6 +365,17 @@ void migration_object_init(void)
> >> >  
> >> >      current_incoming->exit_on_error = INMIGRATE_DEFAULT_EXIT_ON_ERROR;
> >> >  
> >> > +    /*
> >> > +     * Apply boot migration parameters in case the user specified some via
> >> > +     * command line "-incoming config:*". NOTE: this will overwrite machine
> >> > +     * type compat properties and -global settings!
> >> > +     */
> >> 
> >> Hmm, but then it's exactly the same as -global migration.* ? Why do we
> >> need to extend -incoming?
> >> 
> >> Also, I don't see how qmp_migrate_set_parameters can overwrite compat
> >> properties. They're not part of MigrationParameters.
> >
> > Some of this should have been mentioned in my other replies, let me know if
> > there's still concern over there after reading those.
> >
> 
> The question about -global is non-sensical, forget about it.
> 
> > In this case, after we set current_migration pointer compat properties are
> > applied, due to instance_post_init() happening within object_new().
> >
> 
> Yes, but migration_properties has two distinct sets of Properties, one
> contains only compat properties that match machine.c properties and the
> other has properties that go into MigrationParameters. I was under the
> impression that we never tied compat to parameters.
> 
> The only machine compat property that is also a migration parameter I
> see is zero-page-detection. Maybe we shouldn't have done that. Now we
> can't decouple MigrationParameters from -global. Unless, of course we
> declare that from now on compat needs to be done with a new property
> that's not part of MigrationParameters; and duplicate
> zero-page-detection.

I always treated migration parameters to be a subset of what we can apply
to compat properties in the past.

For example, -global almost works the same as compat properties and also at
the same time, currently both done in device_post_init().

In the future, if your series to convert caps to parameters work, then I
also want to add even more "parameters" (which used to be caps, like
postcopy preempt mode) into lists of machine compat properties, so e.g. we
can enable preempt mode by default.

I hope that makes it useful, but if you still see some benefits of
splitting the two things (compat properties / parameters), let me know.  I
may not have fully caught what you're visioning.

> 
> >> 
> >> > +    migration_parameters_boot_apply();
> >> > +
> >> > +    /*
> >> > +     * The boot parameters should have been verified already, but leave it
> >> > +     * after applying boot parameters to do one check for everything.
> >> > +     */
> >> 
> >> Will it get expensive? We do that for every QEMU start.
> >
> > Checking caps and parameters?  Why expensive?
> >
> 
> Just because we're putting more migration code in QEMU's startup
> path. Most QEMU invocations don't care about migration. Eventually,
> checking every parameter twice might have an impact.

I would expect most QEMU invocations should care about migration, but
irrelevant of that.. I never expected these checks to be expensive.

postcopy_ram_supported_by_host() might be the heaviest, but it's
conditional and I still don't see it an issue. We also don't run any loop.

What made you worry on this?

> 
> > Here I should have mentioned we must do one shot check here, because
> > sometimes there're illegal mixtures of "-incoming config*" and default
> > values only after they merged together.  I'll update the comment for it,
> > IOW we must check here anyway.
> >
> 
> Right.
> 
> >> 
> >> >      migration_object_check(current_migration, &error_fatal);
> >> >  
> >> >      ram_mig_init();
> >> > diff --git a/system/vl.c b/system/vl.c
> >> > index da36b2c6e1..49f5fa0c7b 100644
> >> > --- a/system/vl.c
> >> > +++ b/system/vl.c
> >> > @@ -1838,6 +1838,13 @@ static void incoming_option_parse(const char *str)
> >> >  
> >> >      if (!strcmp(str, "defer")) {
> >> >          channel = NULL;
> >> > +    } else if (!strncmp(str, "config:", 7)) {
> >> > +        /*
> >> > +         * This is not a channel setup, but configuration to incoming
> >> > +         * migration parameters to make them available during early boot.
> >> > +         */
> >> > +        migration_parameters_boot_parse(str + 7, &error_fatal);
> >> > +        return;
> >> >      } else if (migrate_is_uri(str)) {
> >> >          migrate_uri_parse(str, &channel, &error_fatal);
> >> >      } else {
> >> > diff --git a/qemu-options.hx b/qemu-options.hx
> >> > index 96ae41f787..1fc92a409a 100644
> >> > --- a/qemu-options.hx
> >> > +++ b/qemu-options.hx
> >> > @@ -5366,11 +5366,15 @@ DEF("incoming", HAS_ARG, QEMU_OPTION_incoming, \
> >> >      "-incoming <channel>\n" \
> >> >      "                accept incoming migration on the migration channel\n" \
> >> >      "-incoming defer\n" \
> >> > -    "                wait for the URI to be specified via migrate_incoming\n",
> >> > +    "                wait for the URI to be specified via migrate_incoming\n"
> >> > +    "-incoming config:key1=value1,key2=value2,...\n" \
> >> > +    "                specify migration parameters valid even during boot\n",
> >> >      QEMU_ARCH_ALL)
> >> >  SRST
> >> > +
> >> >  The -incoming option specifies the migration channel for an incoming
> >> > -migration.  It may be used multiple times to specify multiple
> >> > +migration, or can also be used to setup migration parameters for the
> >> > +incoming migration.  It may be used multiple times to specify multiple
> >> >  migration channel types.  The channel type is specified in <channel>,
> >> >  or is 'main' for all other forms of -incoming.  If multiple -incoming
> >> >  options are specified for a channel type, the last one takes precedence.
> >> > @@ -5411,6 +5415,16 @@ options are specified for a channel type, the last one takes precedence.
> >> >      Wait for the URI to be specified via migrate\_incoming. The monitor
> >> >      can be used to change settings (such as migration parameters) prior
> >> >      to issuing the migrate\_incoming to allow the migration to begin.
> >> > +
> >> > +``-incoming config:key1=value1[,key2=value2,...]``
> >> > +
> >> > +    Specify migration parameters in QEMU commandlines, so that these
> >> 
> >> command line
> >
> > Fixed.
> >
> >> 
> >> > +    parameters will be available even during very early boot of QEMU.
> >> > +    They will be applied properly after QEMU boots and when the migration
> >> 
> >> The "applied properly" part is implementation detail.
> >
> > True, I simplified it:
> >
> > ``-incoming config:key1=value1[,key2=value2,...]``
> >
> >     Specify migration parameters in QEMU command line, so that these
> >     parameters will be available even during very early boot of QEMU.  It
> >     has similar effect as setting these parameters using QMP command
> >     ``migrate-set-parameters`` or HMP command ``migrate_set_parameter``.
> >
> 
> ack.
> 
> >> 
> >> > +    core is initialized.  From that POV, it has similar effect as setting
> >> > +    these parameters using QMP command ``migrate-set-parameters`` or HMP
> >> > +    command ``migrate_set_parameter``.
> >> > +
> >> >  ERST
> >> >  
> >> >  DEF("only-migratable", 0, QEMU_OPTION_only_migratable, \
> >> 
> 

-- 
Peter Xu

Re: [PATCH RFC 1/2] migration/vl: Allow set parameters with -incoming config:*

[Fabiano Rosas]

Peter Xu <peterx@redhat.com> writes:

> On Fri, May 29, 2026 at 01:44:11PM -0300, Fabiano Rosas wrote:
>> Peter Xu <peterx@redhat.com> writes:
>> 
>> > On Thu, May 28, 2026 at 07:16:57PM -0300, Fabiano Rosas wrote:
>> >> Peter Xu <peterx@redhat.com> writes:
>> >> 
>> >> > QEMU doesn't yet have a good way to specify migration parameters so that
>> >> > they can be available even during early stage of QEMU boots.  It is because
>> >> > the migration object (who owns the migration parameters) will only be
>> >> > created after PHASE_LATE_BACKENDS_CREATED.  It means anything before it
>> >> > reading migration parameters is illegal.
>> >> >
>> >> > However, QEMU does have special use cases for such, namely only-migratable
>> >> > flag, and cpr-transfer. Recently, we have one more possible user to read a
>> >> > to-be-introduced new migration parameters during backend initialization
>> >> > phase.  We can introduce yet another global variable (or per-device
>> >> > parameter) to bypass this limitation, but we can also seek for a generic
>> >> > solution that we can setup migration parameters very early, even during
>> >> > backend initializations.
>> >> >
>> >> > See this discussion for more details on the context of the problem:
>> >> >
>> >> >   https://lore.kernel.org/r/ahdI7Vl5KraK566D@x1.local
>> >> >
>> >> > This patch wants to take the latter approach.
>> >> >
>> >> > As a start, introduce a new way to specify migration parameters in QEMU
>> >> > boot commandline, as proposed in the above discussion:
>> >> >
>> >> >   -incoming config:key1=value1,key2=value2,...
>> >> >
>> >> > When specified, QEMU will parse a string formatted MigrationParameters and
>> >> > keep it.  When migration object is created, it will apply all the settings
>> >> > as initial value.
>> >> >
>> >> > Since the application of boot parameters will be after object_new()
>> >> > completes, it means it happens after all machine compat properties or
>> >> > -global settings (which should be done during instance_post_init()).
>> >> >
>> >> > So far, it's still only a way to specify parameters.  All parameters are
>> >> > not visible before migration object created, like before.  Any parameter
>> >> > that needs to be visible during boot will need to opt-in this feature.
>> >> > Follow up patches will switch the current users to use this model.
>> >> >
>> >> > Signed-off-by: Peter Xu <peterx@redhat.com>
>> >> > ---
>> >> >  include/migration/misc.h |  5 ++++
>> >> >  migration/migration.c    | 54 ++++++++++++++++++++++++++++++++++++++++
>> >> >  system/vl.c              |  7 ++++++
>> >> >  qemu-options.hx          | 18 ++++++++++++--
>> >> >  4 files changed, 82 insertions(+), 2 deletions(-)
>> >> >
>> >> > diff --git a/include/migration/misc.h b/include/migration/misc.h
>> >> > index 3159a5e53c..aff79b1380 100644
>> >> > --- a/include/migration/misc.h
>> >> > +++ b/include/migration/misc.h
>> >> > @@ -133,6 +133,11 @@ bool migrate_is_uri(const char *uri);
>> >> >  /* Parse @uri and return @channel, returning true on success */
>> >> >  bool migrate_uri_parse(const char *uri, MigrationChannel **channel,
>> >> >                         Error **errp);
>> >> > +/*
>> >> > + * Parse @config_str in form of "config:key1=value1,..." to initialize
>> >> > + * migration parameters.
>> >> > + */
>> >> > +bool migration_parameters_boot_parse(const char *config_str, Error **errp);
>> >> >  
>> >> 
>> >> 'boot' is not the right word. Maybe 'early'?
>> >
>> > I am ok to switch to "early".  Another option is.. "cmdline"?
>> >
>> 
>> Either is fine.
>> 
>> >> 
>> >> >  /* migration/multifd-device-state.c */
>> >> >  typedef struct SaveCompletePrecopyThreadData {
>> >> > diff --git a/migration/migration.c b/migration/migration.c
>> >> > index 074d3f2c69..d918be7a44 100644
>> >> > --- a/migration/migration.c
>> >> > +++ b/migration/migration.c
>> >> > @@ -41,6 +41,7 @@
>> >> >  #include "qapi/qapi-commands-migration.h"
>> >> >  #include "qapi/qapi-events-migration.h"
>> >> >  #include "qapi/qmp/qerror.h"
>> >> > +#include "qapi/qobject-input-visitor.h"
>> >> >  #include "qobject/qnull.h"
>> >> >  #include "qemu/rcu.h"
>> >> >  #include "postcopy-ram.h"
>> >> > @@ -94,6 +95,9 @@ enum mig_rp_message_type {
>> >> >  static MigrationState *current_migration;
>> >> >  static MigrationIncomingState *current_incoming;
>> >> >  
>> >> > +/* Only used during boot, destroyed after migration object initialized */
>> >> > +static MigrationParameters *mig_boot_params;
>> >> > +
>> >> >  static GSList *migration_blockers[MIG_MODE__MAX];
>> >> >  
>> >> >  static bool migration_object_check(MigrationState *ms, Error **errp);
>> >> > @@ -102,6 +106,45 @@ static bool stop_return_path_thread_on_source(MigrationState *s);
>> >> >  static void migration_release_dst_files(MigrationState *ms);
>> >> >  static void migration_completion_end(MigrationState *s);
>> >> >  
>> >> > +bool migration_parameters_boot_parse(const char *config_str, Error **errp)
>> >> > +{
>> >> > +    Visitor *v;
>> >> > +
>> >> > +    if (mig_boot_params) {
>> >> > +        error_setg(errp, "Only one -incoming config:* is allowed.");
>> >> > +        return false;
>> >> > +    }
>> >> > +
>> >> > +    v = qobject_input_visitor_new_str(config_str, NULL, errp);
>> >> > +    if (!v) {
>> >> > +        goto fail;
>> >> > +    }
>> >> > +
>> >> > +    if (!visit_type_MigrationParameters(v, NULL, &mig_boot_params, errp)) {
>> >> > +        goto fail;
>> >> > +    }
>> >> > +
>> >> > +    visit_free(v);
>> >> > +    return true;
>> >> > +
>> >> > +fail:
>> >> > +    visit_free(v);
>> >> > +    return false;
>> >> > +}
>> >> > +
>> >> > +static void migration_parameters_boot_apply(void)
>> >> > +{
>> >> > +    if (mig_boot_params) {
>> >> > +        /*
>> >> > +         * This can fail, because qobject visitor doesn't do sanity check
>> >> > +         * on values while parsing.  It's not too late; we're still in boot
>> >> > +         * phase.
>> >> > +         */
>> >> > +        qmp_migrate_set_parameters(mig_boot_params, &error_abort);
>> >> > +        g_clear_pointer(&mig_boot_params, qapi_free_MigrationParameters);
>> >> > +    }
>> >> > +}
>> >> > +
>> >> >  static void migration_downtime_start(MigrationState *s)
>> >> >  {
>> >> >      trace_vmstate_downtime_checkpoint("src-downtime-start");
>> >> > @@ -322,6 +365,17 @@ void migration_object_init(void)
>> >> >  
>> >> >      current_incoming->exit_on_error = INMIGRATE_DEFAULT_EXIT_ON_ERROR;
>> >> >  
>> >> > +    /*
>> >> > +     * Apply boot migration parameters in case the user specified some via
>> >> > +     * command line "-incoming config:*". NOTE: this will overwrite machine
>> >> > +     * type compat properties and -global settings!
>> >> > +     */
>> >> 
>> >> Hmm, but then it's exactly the same as -global migration.* ? Why do we
>> >> need to extend -incoming?
>> >> 
>> >> Also, I don't see how qmp_migrate_set_parameters can overwrite compat
>> >> properties. They're not part of MigrationParameters.
>> >
>> > Some of this should have been mentioned in my other replies, let me know if
>> > there's still concern over there after reading those.
>> >
>> 
>> The question about -global is non-sensical, forget about it.
>> 
>> > In this case, after we set current_migration pointer compat properties are
>> > applied, due to instance_post_init() happening within object_new().
>> >
>> 
>> Yes, but migration_properties has two distinct sets of Properties, one
>> contains only compat properties that match machine.c properties and the
>> other has properties that go into MigrationParameters. I was under the
>> impression that we never tied compat to parameters.
>> 
>> The only machine compat property that is also a migration parameter I
>> see is zero-page-detection. Maybe we shouldn't have done that. Now we
>> can't decouple MigrationParameters from -global. Unless, of course we
>> declare that from now on compat needs to be done with a new property
>> that's not part of MigrationParameters; and duplicate
>> zero-page-detection.
>
> I always treated migration parameters to be a subset of what we can apply
> to compat properties in the past.
>

Good to know.

> For example, -global almost works the same as compat properties and also at
> the same time, currently both done in device_post_init().
>
> In the future, if your series to convert caps to parameters work, then I
> also want to add even more "parameters" (which used to be caps, like
> postcopy preempt mode) into lists of machine compat properties, so e.g. we
> can enable preempt mode by default.
>
> I hope that makes it useful, but if you still see some benefits of
> splitting the two things (compat properties / parameters), let me know.  I
> may not have fully caught what you're visioning.
>

The problem I see is a long-standing one: MigrationParameters needs to
be embedded into MigrationState because of -global (as both compat and
debug feature). As long as that holds, we'll need workarounds such as
having this extra early state.

But let's not make this a problem to be solved in this series, please
disconsider this discussion.

Just to wrap my thoughts, for reference:

What I want is a migration-owned MigrationParameters that we can do
anything with, decoupled from whatever is needed to expose compat
properties. There's two ways to achieve this: One is to declare
parameters are separate from compat and drop the debug usage (what I
suggested in my previous email). The other is to make MigrationState
stop being a QOM object and instead make MigrationParameters itself be a
QOM object. Then we could reference its members directly from qdev.

>> 
>> >> 
>> >> > +    migration_parameters_boot_apply();
>> >> > +
>> >> > +    /*
>> >> > +     * The boot parameters should have been verified already, but leave it
>> >> > +     * after applying boot parameters to do one check for everything.
>> >> > +     */
>> >> 
>> >> Will it get expensive? We do that for every QEMU start.
>> >
>> > Checking caps and parameters?  Why expensive?
>> >
>> 
>> Just because we're putting more migration code in QEMU's startup
>> path. Most QEMU invocations don't care about migration. Eventually,
>> checking every parameter twice might have an impact.
>
> I would expect most QEMU invocations should care about migration, but
> irrelevant of that.. I never expected these checks to be expensive.
>

Well, only a small subset of invocations will actually result in a
migration. There's many uses of QEMU that don't require, or even cannot
migrate.

> postcopy_ram_supported_by_host() might be the heaviest, but it's
> conditional and I still don't see it an issue. We also don't run any loop.
>
> What made you worry on this?
>

I don't know. The voices in my head.

>> 
>> > Here I should have mentioned we must do one shot check here, because
>> > sometimes there're illegal mixtures of "-incoming config*" and default
>> > values only after they merged together.  I'll update the comment for it,
>> > IOW we must check here anyway.
>> >
>> 
>> Right.
>> 
>> >> 
>> >> >      migration_object_check(current_migration, &error_fatal);
>> >> >  
>> >> >      ram_mig_init();
>> >> > diff --git a/system/vl.c b/system/vl.c
>> >> > index da36b2c6e1..49f5fa0c7b 100644
>> >> > --- a/system/vl.c
>> >> > +++ b/system/vl.c
>> >> > @@ -1838,6 +1838,13 @@ static void incoming_option_parse(const char *str)
>> >> >  
>> >> >      if (!strcmp(str, "defer")) {
>> >> >          channel = NULL;
>> >> > +    } else if (!strncmp(str, "config:", 7)) {
>> >> > +        /*
>> >> > +         * This is not a channel setup, but configuration to incoming
>> >> > +         * migration parameters to make them available during early boot.
>> >> > +         */
>> >> > +        migration_parameters_boot_parse(str + 7, &error_fatal);
>> >> > +        return;
>> >> >      } else if (migrate_is_uri(str)) {
>> >> >          migrate_uri_parse(str, &channel, &error_fatal);
>> >> >      } else {
>> >> > diff --git a/qemu-options.hx b/qemu-options.hx
>> >> > index 96ae41f787..1fc92a409a 100644
>> >> > --- a/qemu-options.hx
>> >> > +++ b/qemu-options.hx
>> >> > @@ -5366,11 +5366,15 @@ DEF("incoming", HAS_ARG, QEMU_OPTION_incoming, \
>> >> >      "-incoming <channel>\n" \
>> >> >      "                accept incoming migration on the migration channel\n" \
>> >> >      "-incoming defer\n" \
>> >> > -    "                wait for the URI to be specified via migrate_incoming\n",
>> >> > +    "                wait for the URI to be specified via migrate_incoming\n"
>> >> > +    "-incoming config:key1=value1,key2=value2,...\n" \
>> >> > +    "                specify migration parameters valid even during boot\n",
>> >> >      QEMU_ARCH_ALL)
>> >> >  SRST
>> >> > +
>> >> >  The -incoming option specifies the migration channel for an incoming
>> >> > -migration.  It may be used multiple times to specify multiple
>> >> > +migration, or can also be used to setup migration parameters for the
>> >> > +incoming migration.  It may be used multiple times to specify multiple
>> >> >  migration channel types.  The channel type is specified in <channel>,
>> >> >  or is 'main' for all other forms of -incoming.  If multiple -incoming
>> >> >  options are specified for a channel type, the last one takes precedence.
>> >> > @@ -5411,6 +5415,16 @@ options are specified for a channel type, the last one takes precedence.
>> >> >      Wait for the URI to be specified via migrate\_incoming. The monitor
>> >> >      can be used to change settings (such as migration parameters) prior
>> >> >      to issuing the migrate\_incoming to allow the migration to begin.
>> >> > +
>> >> > +``-incoming config:key1=value1[,key2=value2,...]``
>> >> > +
>> >> > +    Specify migration parameters in QEMU commandlines, so that these
>> >> 
>> >> command line
>> >
>> > Fixed.
>> >
>> >> 
>> >> > +    parameters will be available even during very early boot of QEMU.
>> >> > +    They will be applied properly after QEMU boots and when the migration
>> >> 
>> >> The "applied properly" part is implementation detail.
>> >
>> > True, I simplified it:
>> >
>> > ``-incoming config:key1=value1[,key2=value2,...]``
>> >
>> >     Specify migration parameters in QEMU command line, so that these
>> >     parameters will be available even during very early boot of QEMU.  It
>> >     has similar effect as setting these parameters using QMP command
>> >     ``migrate-set-parameters`` or HMP command ``migrate_set_parameter``.
>> >
>> 
>> ack.
>> 
>> >> 
>> >> > +    core is initialized.  From that POV, it has similar effect as setting
>> >> > +    these parameters using QMP command ``migrate-set-parameters`` or HMP
>> >> > +    command ``migrate_set_parameter``.
>> >> > +
>> >> >  ERST
>> >> >  
>> >> >  DEF("only-migratable", 0, QEMU_OPTION_only_migratable, \
>> >> 
>> 

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Daniel P. Berrangé]

On Fri, May 29, 2026 at 10:15:35AM -0400, Peter Xu wrote:
> On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote:
> > Peter Xu <peterx@redhat.com> writes:
> > 
> > > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907
> > >
> > > This series introduces a generic way to specify migration parameters that
> > > can be used even during the early boot phase of QEMU.
> > >
> > > One example use case that already existed is CPR-transfer / CPR-exec.
> > > currently QEMU has a temporary global variable (incoming_mode) to achieve
> > > this, but it's hard to understand and this hack bleeded into quite a few
> > > places that we could have avoided.  The lines in patch 2 touched may
> > > provide some idea.
> > >
> > > With a generic approach of setting migration parameters with cmdlines, we
> > > can remove this hack meanwhile QEMU should be able to keep the CPR behavior
> > > as before.  To CPR maintainers and reviewers: please have a closer look,
> > > even better if it can be smoke tested, to see if this works for Oracle's
> > > environment, TIA.
> > >
> > > The 1st patch implemented that new semantics.  It is straightforward: now
> > > we can setup any migration parameter using an extra line of:
> > >
> > >   -incoming config:key1=value1,key2=value2,...
> > 
> > Just merge my code mate.
> > https://lore.kernel.org/r/20251215220041.12657-25-farosas@suse.de
> 
> Heh, I apologize. I totally forgot it even if I seem to have read it..
> 
> The goal is differnent, though, which is to make it available before
> migration object initialization.  Say, I wished -global would work too but
> it doesn't, and it just can't.  So it's not "something good to have"
> anymore, but functionally required for either CPR and TAP in the future if
> we adopt this, they'll all read them early.
> 
> > 
> > A couple of differences from my patch:
> > 
> > - old-style keyval vs. json;
> > 
> > Isn't json preferred nowadays?
> 
> I didn't mention it in doc, but yeah this series also supports JSON due to
> the same qapi helper used.  I don't suggest json though for this entry (and
> that's also why I didn't mention it..) because 99% of migration parameters
> are scalars and I don't see why we should use JSON here.. the only one not
> scalar is block-bitmap-mapping but it's not intended to be used in
> -incoming at all (even if it'll work..).

99% of **todays** parameters are scalars.

The reason we are actively moving to CLI args which are modelled with QAPI
and expressable using JSON is because that future proofs the command line
for any kind of parameters we might need.

We should actively document and promote the use of JSON.

> > - using "config:"
> > 
> > This makes it non-uniform with uri and channels which don't have a
> > keyword in front of them. I guess I could live with it, but it seems
> > odd. I see that it makes parsing way easier.
> 
> Yeah, having some identifier would be nice.  I wished channels also have
> identifiers if we don't need to keep compatibility.

IMHO having a magic "config:" prefix is an anti-pattern because it
involves custom command line parsing logic.

Does "-incoming config" imply '-incoming defer' semantics or is it
independent ?  



> 
> > 
> > >
> > > So far only one such instance is allowed for simplicity, but it should be
> > > enough.  We still allow multiple -incoming for specifying channels, used
> > > together with one "-incoming config:*".
> > >
> > > Then parameters set this way will be visible almost anytime for QEMU, for
> > > example, during initialization of device backends (which is before
> > > migration object created).
> > >
> > > I posted this series majorly because I want to see if this will make a
> > > possible new user for the new "local" migration parameter proposed in
> > > Vladimir's series:
> > >
> > >   https://lore.kernel.org/r/20260522120534.77653-1-vsementsov@yandex-team.ru
> > >
> > > Especially, there're some context on this idea too in this email:
> > >
> > >   https://lore.kernel.org/all/ahdI7Vl5KraK566D@x1.local/
> > >
> > 
> > I like the idea overall.
> > 
> > > With this series, we should be able to drop "incoming-fds" TAP property
> > > from the other series, instead relying on the existing "local" parameters
> > > both in migration core and in TAP's property should suffice.
> > >
> > > One thing to mention is I didn't further make only-migratable into a
> > > migration parameter.  Logically it will also work now with only-migratable,
> > > but it then also means I'll need to convert it to a parameter, which will
> > > be mutable even after VM started.  It will change how only-migratable used
> > > to work, hence I skipped.
> > >
> > > After this, we also almost have no reason to use -global for migration
> > > parameters.  Capabilities are still not supported in -incoming cmdline,
> > > though.
> > >
> > 
> > Will we still merge capabilities and parameters? Then it would be a free
> > upgrade.
> 
> Yes!  /me waiting for your patches.
> 
> -- 
> Peter Xu
> 
> 

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Mark Cave-Ayland]

On 29/05/2026 15:15, Peter Xu wrote:

(cut)

>> I like the idea overall.
>>
>>> With this series, we should be able to drop "incoming-fds" TAP property
>>> from the other series, instead relying on the existing "local" parameters
>>> both in migration core and in TAP's property should suffice.
>>>
>>> One thing to mention is I didn't further make only-migratable into a
>>> migration parameter.  Logically it will also work now with only-migratable,
>>> but it then also means I'll need to convert it to a parameter, which will
>>> be mutable even after VM started.  It will change how only-migratable used
>>> to work, hence I skipped.
>>>
>>> After this, we also almost have no reason to use -global for migration
>>> parameters.  Capabilities are still not supported in -incoming cmdline,
>>> though.
>>>
>>
>> Will we still merge capabilities and parameters? Then it would be a free
>> upgrade.
> 
> Yes!  /me waiting for your patches.

Is this related to the series posted at 
https://patchew.org/QEMU/20251215220041.12657-1-farosas@suse.de/? What's 
the current status with this work?


ATB,

Mark.

Re: [PATCH RFC 1/2] migration/vl: Allow set parameters with -incoming config:*

[Mark Cave-Ayland]

On 02/06/2026 14:37, Fabiano Rosas wrote:

> Peter Xu <peterx@redhat.com> writes:
> 
>> On Fri, May 29, 2026 at 01:44:11PM -0300, Fabiano Rosas wrote:
>>> Peter Xu <peterx@redhat.com> writes:
>>>
>>>> On Thu, May 28, 2026 at 07:16:57PM -0300, Fabiano Rosas wrote:
>>>>> Peter Xu <peterx@redhat.com> writes:
>>>>>
>>>>>> QEMU doesn't yet have a good way to specify migration parameters so that
>>>>>> they can be available even during early stage of QEMU boots.  It is because
>>>>>> the migration object (who owns the migration parameters) will only be
>>>>>> created after PHASE_LATE_BACKENDS_CREATED.  It means anything before it
>>>>>> reading migration parameters is illegal.
>>>>>>
>>>>>> However, QEMU does have special use cases for such, namely only-migratable
>>>>>> flag, and cpr-transfer. Recently, we have one more possible user to read a
>>>>>> to-be-introduced new migration parameters during backend initialization
>>>>>> phase.  We can introduce yet another global variable (or per-device
>>>>>> parameter) to bypass this limitation, but we can also seek for a generic
>>>>>> solution that we can setup migration parameters very early, even during
>>>>>> backend initializations.
>>>>>>
>>>>>> See this discussion for more details on the context of the problem:
>>>>>>
>>>>>>    https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_r_ahdI7Vl5KraK566D-40x1.local&d=DwIBAg&c=s883GpUCOChKOHiocYtGcg&r=c23RpsaH4D2MKyD3EPJTDa0BAxz6tV8aUJqVSoytEiY&m=RMJscC71rPIaZcWE2xNWeuHeK0qmTkT4jnKXp0AVp4SEa7jlKxLNal50vCx6VIAo&s=MNLWQVZxg7cOInvRx6FinI_ACIiZdt8KJfho82JzKEg&e=
>>>>>>
>>>>>> This patch wants to take the latter approach.
>>>>>>
>>>>>> As a start, introduce a new way to specify migration parameters in QEMU
>>>>>> boot commandline, as proposed in the above discussion:
>>>>>>
>>>>>>    -incoming config:key1=value1,key2=value2,...
>>>>>>
>>>>>> When specified, QEMU will parse a string formatted MigrationParameters and
>>>>>> keep it.  When migration object is created, it will apply all the settings
>>>>>> as initial value.
>>>>>>
>>>>>> Since the application of boot parameters will be after object_new()
>>>>>> completes, it means it happens after all machine compat properties or
>>>>>> -global settings (which should be done during instance_post_init()).
>>>>>>
>>>>>> So far, it's still only a way to specify parameters.  All parameters are
>>>>>> not visible before migration object created, like before.  Any parameter
>>>>>> that needs to be visible during boot will need to opt-in this feature.
>>>>>> Follow up patches will switch the current users to use this model.
>>>>>>
>>>>>> Signed-off-by: Peter Xu <peterx@redhat.com>
>>>>>> ---
>>>>>>   include/migration/misc.h |  5 ++++
>>>>>>   migration/migration.c    | 54 ++++++++++++++++++++++++++++++++++++++++
>>>>>>   system/vl.c              |  7 ++++++
>>>>>>   qemu-options.hx          | 18 ++++++++++++--
>>>>>>   4 files changed, 82 insertions(+), 2 deletions(-)
>>>>>>
>>>>>> diff --git a/include/migration/misc.h b/include/migration/misc.h
>>>>>> index 3159a5e53c..aff79b1380 100644
>>>>>> --- a/include/migration/misc.h
>>>>>> +++ b/include/migration/misc.h
>>>>>> @@ -133,6 +133,11 @@ bool migrate_is_uri(const char *uri);
>>>>>>   /* Parse @uri and return @channel, returning true on success */
>>>>>>   bool migrate_uri_parse(const char *uri, MigrationChannel **channel,
>>>>>>                          Error **errp);
>>>>>> +/*
>>>>>> + * Parse @config_str in form of "config:key1=value1,..." to initialize
>>>>>> + * migration parameters.
>>>>>> + */
>>>>>> +bool migration_parameters_boot_parse(const char *config_str, Error **errp);
>>>>>>   
>>>>>
>>>>> 'boot' is not the right word. Maybe 'early'?
>>>>
>>>> I am ok to switch to "early".  Another option is.. "cmdline"?
>>>>
>>>
>>> Either is fine.
>>>
>>>>>
>>>>>>   /* migration/multifd-device-state.c */
>>>>>>   typedef struct SaveCompletePrecopyThreadData {
>>>>>> diff --git a/migration/migration.c b/migration/migration.c
>>>>>> index 074d3f2c69..d918be7a44 100644
>>>>>> --- a/migration/migration.c
>>>>>> +++ b/migration/migration.c
>>>>>> @@ -41,6 +41,7 @@
>>>>>>   #include "qapi/qapi-commands-migration.h"
>>>>>>   #include "qapi/qapi-events-migration.h"
>>>>>>   #include "qapi/qmp/qerror.h"
>>>>>> +#include "qapi/qobject-input-visitor.h"
>>>>>>   #include "qobject/qnull.h"
>>>>>>   #include "qemu/rcu.h"
>>>>>>   #include "postcopy-ram.h"
>>>>>> @@ -94,6 +95,9 @@ enum mig_rp_message_type {
>>>>>>   static MigrationState *current_migration;
>>>>>>   static MigrationIncomingState *current_incoming;
>>>>>>   
>>>>>> +/* Only used during boot, destroyed after migration object initialized */
>>>>>> +static MigrationParameters *mig_boot_params;
>>>>>> +
>>>>>>   static GSList *migration_blockers[MIG_MODE__MAX];
>>>>>>   
>>>>>>   static bool migration_object_check(MigrationState *ms, Error **errp);
>>>>>> @@ -102,6 +106,45 @@ static bool stop_return_path_thread_on_source(MigrationState *s);
>>>>>>   static void migration_release_dst_files(MigrationState *ms);
>>>>>>   static void migration_completion_end(MigrationState *s);
>>>>>>   
>>>>>> +bool migration_parameters_boot_parse(const char *config_str, Error **errp)
>>>>>> +{
>>>>>> +    Visitor *v;
>>>>>> +
>>>>>> +    if (mig_boot_params) {
>>>>>> +        error_setg(errp, "Only one -incoming config:* is allowed.");
>>>>>> +        return false;
>>>>>> +    }
>>>>>> +
>>>>>> +    v = qobject_input_visitor_new_str(config_str, NULL, errp);
>>>>>> +    if (!v) {
>>>>>> +        goto fail;
>>>>>> +    }
>>>>>> +
>>>>>> +    if (!visit_type_MigrationParameters(v, NULL, &mig_boot_params, errp)) {
>>>>>> +        goto fail;
>>>>>> +    }
>>>>>> +
>>>>>> +    visit_free(v);
>>>>>> +    return true;
>>>>>> +
>>>>>> +fail:
>>>>>> +    visit_free(v);
>>>>>> +    return false;
>>>>>> +}
>>>>>> +
>>>>>> +static void migration_parameters_boot_apply(void)
>>>>>> +{
>>>>>> +    if (mig_boot_params) {
>>>>>> +        /*
>>>>>> +         * This can fail, because qobject visitor doesn't do sanity check
>>>>>> +         * on values while parsing.  It's not too late; we're still in boot
>>>>>> +         * phase.
>>>>>> +         */
>>>>>> +        qmp_migrate_set_parameters(mig_boot_params, &error_abort);
>>>>>> +        g_clear_pointer(&mig_boot_params, qapi_free_MigrationParameters);
>>>>>> +    }
>>>>>> +}
>>>>>> +
>>>>>>   static void migration_downtime_start(MigrationState *s)
>>>>>>   {
>>>>>>       trace_vmstate_downtime_checkpoint("src-downtime-start");
>>>>>> @@ -322,6 +365,17 @@ void migration_object_init(void)
>>>>>>   
>>>>>>       current_incoming->exit_on_error = INMIGRATE_DEFAULT_EXIT_ON_ERROR;
>>>>>>   
>>>>>> +    /*
>>>>>> +     * Apply boot migration parameters in case the user specified some via
>>>>>> +     * command line "-incoming config:*". NOTE: this will overwrite machine
>>>>>> +     * type compat properties and -global settings!
>>>>>> +     */
>>>>>
>>>>> Hmm, but then it's exactly the same as -global migration.* ? Why do we
>>>>> need to extend -incoming?
>>>>>
>>>>> Also, I don't see how qmp_migrate_set_parameters can overwrite compat
>>>>> properties. They're not part of MigrationParameters.
>>>>
>>>> Some of this should have been mentioned in my other replies, let me know if
>>>> there's still concern over there after reading those.
>>>>
>>>
>>> The question about -global is non-sensical, forget about it.
>>>
>>>> In this case, after we set current_migration pointer compat properties are
>>>> applied, due to instance_post_init() happening within object_new().
>>>>
>>>
>>> Yes, but migration_properties has two distinct sets of Properties, one
>>> contains only compat properties that match machine.c properties and the
>>> other has properties that go into MigrationParameters. I was under the
>>> impression that we never tied compat to parameters.
>>>
>>> The only machine compat property that is also a migration parameter I
>>> see is zero-page-detection. Maybe we shouldn't have done that. Now we
>>> can't decouple MigrationParameters from -global. Unless, of course we
>>> declare that from now on compat needs to be done with a new property
>>> that's not part of MigrationParameters; and duplicate
>>> zero-page-detection.
>>
>> I always treated migration parameters to be a subset of what we can apply
>> to compat properties in the past.
>>
> 
> Good to know.
> 
>> For example, -global almost works the same as compat properties and also at
>> the same time, currently both done in device_post_init().
>>
>> In the future, if your series to convert caps to parameters work, then I
>> also want to add even more "parameters" (which used to be caps, like
>> postcopy preempt mode) into lists of machine compat properties, so e.g. we
>> can enable preempt mode by default.
>>
>> I hope that makes it useful, but if you still see some benefits of
>> splitting the two things (compat properties / parameters), let me know.  I
>> may not have fully caught what you're visioning.
>>
> 
> The problem I see is a long-standing one: MigrationParameters needs to
> be embedded into MigrationState because of -global (as both compat and
> debug feature). As long as that holds, we'll need workarounds such as
> having this extra early state.
> 
> But let's not make this a problem to be solved in this series, please
> disconsider this discussion.
> 
> Just to wrap my thoughts, for reference:
> 
> What I want is a migration-owned MigrationParameters that we can do
> anything with, decoupled from whatever is needed to expose compat
> properties. There's two ways to achieve this: One is to declare
> parameters are separate from compat and drop the debug usage (what I
> suggested in my previous email). The other is to make MigrationState
> stop being a QOM object and instead make MigrationParameters itself be a
> QOM object. Then we could reference its members directly from qdev.

Is the longer-term goal here to allow machine-versioned usage of all 
migration parameters supplied via -incoming via -global?

> 
>>>
>>>>>
>>>>>> +    migration_parameters_boot_apply();
>>>>>> +
>>>>>> +    /*
>>>>>> +     * The boot parameters should have been verified already, but leave it
>>>>>> +     * after applying boot parameters to do one check for everything.
>>>>>> +     */
>>>>>
>>>>> Will it get expensive? We do that for every QEMU start.
>>>>
>>>> Checking caps and parameters?  Why expensive?
>>>>
>>>
>>> Just because we're putting more migration code in QEMU's startup
>>> path. Most QEMU invocations don't care about migration. Eventually,
>>> checking every parameter twice might have an impact.
>>
>> I would expect most QEMU invocations should care about migration, but
>> irrelevant of that.. I never expected these checks to be expensive.
>>
> 
> Well, only a small subset of invocations will actually result in a
> migration. There's many uses of QEMU that don't require, or even cannot
> migrate.
> 
>> postcopy_ram_supported_by_host() might be the heaviest, but it's
>> conditional and I still don't see it an issue. We also don't run any loop.
>>
>> What made you worry on this?
>>
> 
> I don't know. The voices in my head.
> 
>>>
>>>> Here I should have mentioned we must do one shot check here, because
>>>> sometimes there're illegal mixtures of "-incoming config*" and default
>>>> values only after they merged together.  I'll update the comment for it,
>>>> IOW we must check here anyway.
>>>>
>>>
>>> Right.
>>>
>>>>>
>>>>>>       migration_object_check(current_migration, &error_fatal);
>>>>>>   
>>>>>>       ram_mig_init();
>>>>>> diff --git a/system/vl.c b/system/vl.c
>>>>>> index da36b2c6e1..49f5fa0c7b 100644
>>>>>> --- a/system/vl.c
>>>>>> +++ b/system/vl.c
>>>>>> @@ -1838,6 +1838,13 @@ static void incoming_option_parse(const char *str)
>>>>>>   
>>>>>>       if (!strcmp(str, "defer")) {
>>>>>>           channel = NULL;
>>>>>> +    } else if (!strncmp(str, "config:", 7)) {
>>>>>> +        /*
>>>>>> +         * This is not a channel setup, but configuration to incoming
>>>>>> +         * migration parameters to make them available during early boot.
>>>>>> +         */
>>>>>> +        migration_parameters_boot_parse(str + 7, &error_fatal);
>>>>>> +        return;
>>>>>>       } else if (migrate_is_uri(str)) {
>>>>>>           migrate_uri_parse(str, &channel, &error_fatal);
>>>>>>       } else {
>>>>>> diff --git a/qemu-options.hx b/qemu-options.hx
>>>>>> index 96ae41f787..1fc92a409a 100644
>>>>>> --- a/qemu-options.hx
>>>>>> +++ b/qemu-options.hx
>>>>>> @@ -5366,11 +5366,15 @@ DEF("incoming", HAS_ARG, QEMU_OPTION_incoming, \
>>>>>>       "-incoming <channel>\n" \
>>>>>>       "                accept incoming migration on the migration channel\n" \
>>>>>>       "-incoming defer\n" \
>>>>>> -    "                wait for the URI to be specified via migrate_incoming\n",
>>>>>> +    "                wait for the URI to be specified via migrate_incoming\n"
>>>>>> +    "-incoming config:key1=value1,key2=value2,...\n" \
>>>>>> +    "                specify migration parameters valid even during boot\n",
>>>>>>       QEMU_ARCH_ALL)
>>>>>>   SRST
>>>>>> +
>>>>>>   The -incoming option specifies the migration channel for an incoming
>>>>>> -migration.  It may be used multiple times to specify multiple
>>>>>> +migration, or can also be used to setup migration parameters for the
>>>>>> +incoming migration.  It may be used multiple times to specify multiple
>>>>>>   migration channel types.  The channel type is specified in <channel>,
>>>>>>   or is 'main' for all other forms of -incoming.  If multiple -incoming
>>>>>>   options are specified for a channel type, the last one takes precedence.
>>>>>> @@ -5411,6 +5415,16 @@ options are specified for a channel type, the last one takes precedence.
>>>>>>       Wait for the URI to be specified via migrate\_incoming. The monitor
>>>>>>       can be used to change settings (such as migration parameters) prior
>>>>>>       to issuing the migrate\_incoming to allow the migration to begin.
>>>>>> +
>>>>>> +``-incoming config:key1=value1[,key2=value2,...]``
>>>>>> +
>>>>>> +    Specify migration parameters in QEMU commandlines, so that these
>>>>>
>>>>> command line
>>>>
>>>> Fixed.
>>>>
>>>>>
>>>>>> +    parameters will be available even during very early boot of QEMU.
>>>>>> +    They will be applied properly after QEMU boots and when the migration
>>>>>
>>>>> The "applied properly" part is implementation detail.
>>>>
>>>> True, I simplified it:
>>>>
>>>> ``-incoming config:key1=value1[,key2=value2,...]``
>>>>
>>>>      Specify migration parameters in QEMU command line, so that these
>>>>      parameters will be available even during very early boot of QEMU.  It
>>>>      has similar effect as setting these parameters using QMP command
>>>>      ``migrate-set-parameters`` or HMP command ``migrate_set_parameter``.
>>>>
>>>
>>> ack.
>>>
>>>>>
>>>>>> +    core is initialized.  From that POV, it has similar effect as setting
>>>>>> +    these parameters using QMP command ``migrate-set-parameters`` or HMP
>>>>>> +    command ``migrate_set_parameter``.
>>>>>> +
>>>>>>   ERST
>>>>>>   
>>>>>>   DEF("only-migratable", 0, QEMU_OPTION_only_migratable, \


ATB,

Mark.

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Mark Cave-Ayland]

On 29/05/2026 15:15, Peter Xu wrote:

> On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote:
>> Peter Xu <peterx@redhat.com> writes:
>>
>>> CI: https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.com_peterx_qemu_-2D_pipelines_2560168907&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=c23RpsaH4D2MKyD3EPJTDa0BAxz6tV8aUJqVSoytEiY&m=UheDdoNwrmPPc3shvbHQ-dAHk7N6GGtWRAbXjZasaJCgV123Fp6foXXlMUC1mXjb&s=KXo-AcgkWswgIUOgetjfuqaSAGVBUC-c78vno6rnNV0&e=
>>>
>>> This series introduces a generic way to specify migration parameters that
>>> can be used even during the early boot phase of QEMU.
>>>
>>> One example use case that already existed is CPR-transfer / CPR-exec.
>>> currently QEMU has a temporary global variable (incoming_mode) to achieve
>>> this, but it's hard to understand and this hack bleeded into quite a few
>>> places that we could have avoided.  The lines in patch 2 touched may
>>> provide some idea.
>>>
>>> With a generic approach of setting migration parameters with cmdlines, we
>>> can remove this hack meanwhile QEMU should be able to keep the CPR behavior
>>> as before.  To CPR maintainers and reviewers: please have a closer look,
>>> even better if it can be smoke tested, to see if this works for Oracle's
>>> environment, TIA.
>>>
>>> The 1st patch implemented that new semantics.  It is straightforward: now
>>> we can setup any migration parameter using an extra line of:
>>>
>>>    -incoming config:key1=value1,key2=value2,...
>>
>> Just merge my code mate.
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_r_20251215220041.12657-2D25-2Dfarosas-40suse.de&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=c23RpsaH4D2MKyD3EPJTDa0BAxz6tV8aUJqVSoytEiY&m=UheDdoNwrmPPc3shvbHQ-dAHk7N6GGtWRAbXjZasaJCgV123Fp6foXXlMUC1mXjb&s=jaAeqTGqX39wQ3zMnICUZhIcIJl6a0lCALABkjC-HxI&e=
> 
> Heh, I apologize. I totally forgot it even if I seem to have read it..
> 
> The goal is differnent, though, which is to make it available before
> migration object initialization.  Say, I wished -global would work too but
> it doesn't, and it just can't.  So it's not "something good to have"
> anymore, but functionally required for either CPR and TAP in the future if
> we adopt this, they'll all read them early.

One of the reasons CPR requires a separate channel is because it needs 
to access the stream before the devices are created. Is this part of the 
same problem in that the migration object needs to be created much 
earlier than it is currently? And could doing this potentially remove 
the need for a separate CPR channel?

>> A couple of differences from my patch:
>>
>> - old-style keyval vs. json;
>>
>> Isn't json preferred nowadays?
> 
> I didn't mention it in doc, but yeah this series also supports JSON due to
> the same qapi helper used.  I don't suggest json though for this entry (and
> that's also why I didn't mention it..) because 99% of migration parameters
> are scalars and I don't see why we should use JSON here.. the only one not
> scalar is block-bitmap-mapping but it's not intended to be used in
> -incoming at all (even if it'll work..).
> 
>>
>> - using "config:"
>>
>> This makes it non-uniform with uri and channels which don't have a
>> keyword in front of them. I guess I could live with it, but it seems
>> odd. I see that it makes parsing way easier.
> 
> Yeah, having some identifier would be nice.  I wished channels also have
> identifiers if we don't need to keep compatibility.
> 
>>
>>>
>>> So far only one such instance is allowed for simplicity, but it should be
>>> enough.  We still allow multiple -incoming for specifying channels, used
>>> together with one "-incoming config:*".
>>>
>>> Then parameters set this way will be visible almost anytime for QEMU, for
>>> example, during initialization of device backends (which is before
>>> migration object created).
>>>
>>> I posted this series majorly because I want to see if this will make a
>>> possible new user for the new "local" migration parameter proposed in
>>> Vladimir's series:
>>>
>>>    https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_r_20260522120534.77653-2D1-2Dvsementsov-40yandex-2Dteam.ru&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=c23RpsaH4D2MKyD3EPJTDa0BAxz6tV8aUJqVSoytEiY&m=UheDdoNwrmPPc3shvbHQ-dAHk7N6GGtWRAbXjZasaJCgV123Fp6foXXlMUC1mXjb&s=T6LBhQkrn3e5gBGAQuTCshuzq04UbUaFAO1OaFnRAhY&e=
>>>
>>> Especially, there're some context on this idea too in this email:
>>>
>>>    https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_all_ahdI7Vl5KraK566D-40x1.local_&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=c23RpsaH4D2MKyD3EPJTDa0BAxz6tV8aUJqVSoytEiY&m=UheDdoNwrmPPc3shvbHQ-dAHk7N6GGtWRAbXjZasaJCgV123Fp6foXXlMUC1mXjb&s=Ix4yc95OzsoKMBcFrEc51cVrtKzUZKarzIBlN-IeYaI&e=
>>>
>>
>> I like the idea overall.
>>
>>> With this series, we should be able to drop "incoming-fds" TAP property
>>> from the other series, instead relying on the existing "local" parameters
>>> both in migration core and in TAP's property should suffice.
>>>
>>> One thing to mention is I didn't further make only-migratable into a
>>> migration parameter.  Logically it will also work now with only-migratable,
>>> but it then also means I'll need to convert it to a parameter, which will
>>> be mutable even after VM started.  It will change how only-migratable used
>>> to work, hence I skipped.
>>>
>>> After this, we also almost have no reason to use -global for migration
>>> parameters.  Capabilities are still not supported in -incoming cmdline,
>>> though.
>>>
>>
>> Will we still merge capabilities and parameters? Then it would be a free
>> upgrade.
> 
> Yes!  /me waiting for your patches.


ATB,

Mark.

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Peter Xu]

On Wed, Jun 03, 2026 at 10:48:54AM +0100, Daniel P. Berrangé wrote:
> On Fri, May 29, 2026 at 10:15:35AM -0400, Peter Xu wrote:
> > On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote:
> > > Peter Xu <peterx@redhat.com> writes:
> > > 
> > > > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907
> > > >
> > > > This series introduces a generic way to specify migration parameters that
> > > > can be used even during the early boot phase of QEMU.
> > > >
> > > > One example use case that already existed is CPR-transfer / CPR-exec.
> > > > currently QEMU has a temporary global variable (incoming_mode) to achieve
> > > > this, but it's hard to understand and this hack bleeded into quite a few
> > > > places that we could have avoided.  The lines in patch 2 touched may
> > > > provide some idea.
> > > >
> > > > With a generic approach of setting migration parameters with cmdlines, we
> > > > can remove this hack meanwhile QEMU should be able to keep the CPR behavior
> > > > as before.  To CPR maintainers and reviewers: please have a closer look,
> > > > even better if it can be smoke tested, to see if this works for Oracle's
> > > > environment, TIA.
> > > >
> > > > The 1st patch implemented that new semantics.  It is straightforward: now
> > > > we can setup any migration parameter using an extra line of:
> > > >
> > > >   -incoming config:key1=value1,key2=value2,...
> > > 
> > > Just merge my code mate.
> > > https://lore.kernel.org/r/20251215220041.12657-25-farosas@suse.de
> > 
> > Heh, I apologize. I totally forgot it even if I seem to have read it..
> > 
> > The goal is differnent, though, which is to make it available before
> > migration object initialization.  Say, I wished -global would work too but
> > it doesn't, and it just can't.  So it's not "something good to have"
> > anymore, but functionally required for either CPR and TAP in the future if
> > we adopt this, they'll all read them early.
> > 
> > > 
> > > A couple of differences from my patch:
> > > 
> > > - old-style keyval vs. json;
> > > 
> > > Isn't json preferred nowadays?
> > 
> > I didn't mention it in doc, but yeah this series also supports JSON due to
> > the same qapi helper used.  I don't suggest json though for this entry (and
> > that's also why I didn't mention it..) because 99% of migration parameters
> > are scalars and I don't see why we should use JSON here.. the only one not
> > scalar is block-bitmap-mapping but it's not intended to be used in
> > -incoming at all (even if it'll work..).
> 
> 99% of **todays** parameters are scalars.
> 
> The reason we are actively moving to CLI args which are modelled with QAPI
> and expressable using JSON is because that future proofs the command line
> for any kind of parameters we might need.
> 
> We should actively document and promote the use of JSON.

Ah OK, sure, will do when respin.

> 
> > > - using "config:"
> > > 
> > > This makes it non-uniform with uri and channels which don't have a
> > > keyword in front of them. I guess I could live with it, but it seems
> > > odd. I see that it makes parsing way easier.
> > 
> > Yeah, having some identifier would be nice.  I wished channels also have
> > identifiers if we don't need to keep compatibility.
> 
> IMHO having a magic "config:" prefix is an anti-pattern because it
> involves custom command line parsing logic.

Yes, it's not elegant.  I wished we had something like this though when
introducing the cpr channels; right now anything wasn't "defer" or URI
implies it's a "channel"..

I also wanted to avoid introducing new cmdline parameters, so migration
incoming cmdlines can stick with the same option.  If there's better
suggestion please shoot.

> 
> Does "-incoming config" imply '-incoming defer' semantics or is it
> independent ?  

They're independent.  Examples:

 a) "-incoming config:* -incoming tcp:*", setup parameters for TCP incoming
    migration without further deferral

 b) "-incoming config:*" only, setup global parameters for a possible
    upcoming outgoing migration

Thanks,

-- 
Peter Xu

Re: [PATCH RFC 1/2] migration/vl: Allow set parameters with -incoming config:*

[Peter Xu]

On Wed, Jun 03, 2026 at 04:36:37PM +0100, Mark Cave-Ayland wrote:
> Is the longer-term goal here to allow machine-versioned usage of all
> migration parameters supplied via -incoming via -global?

Nop.

Machine compat properties sticks with machines, they'll be applied properly
to migration objects at some point (needs to happen after machine object is
initialized).  It'll keep working, no plan to change that.

This series is only about exposing a new API to setup migration parameters
so that modules (like TAP as backends) can query at very early stage of
QEMU boot.

Thanks,

-- 
Peter Xu

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Peter Xu]

On Wed, Jun 03, 2026 at 04:41:31PM +0100, Mark Cave-Ayland wrote:
> One of the reasons CPR requires a separate channel is because it needs to
> access the stream before the devices are created.

Yes.

> Is this part of the same problem in that the migration object needs to be
> created much earlier than it is currently? And could doing this
> potentially remove the need for a separate CPR channel?

The problem is the migration object can't yet be created too early, due to
the fact it needs to apply compat properties.  That's why this series used
another approach to have a temporary early MigrationParameters object to
keep things until migration object is initialized (during which the temp
object will be destroyed).

Thanks,

-- 
Peter Xu

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Daniel P. Berrangé]

On Wed, Jun 03, 2026 at 11:50:53AM -0400, Peter Xu wrote:
> On Wed, Jun 03, 2026 at 10:48:54AM +0100, Daniel P. Berrangé wrote:
> > On Fri, May 29, 2026 at 10:15:35AM -0400, Peter Xu wrote:
> > > On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote:
> > > > Peter Xu <peterx@redhat.com> writes:
> > > > 
> > > > > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907
> > > > >
> > > > > This series introduces a generic way to specify migration parameters that
> > > > > can be used even during the early boot phase of QEMU.
> > > > >
> > > > > One example use case that already existed is CPR-transfer / CPR-exec.
> > > > > currently QEMU has a temporary global variable (incoming_mode) to achieve
> > > > > this, but it's hard to understand and this hack bleeded into quite a few
> > > > > places that we could have avoided.  The lines in patch 2 touched may
> > > > > provide some idea.
> > > > >
> > > > > With a generic approach of setting migration parameters with cmdlines, we
> > > > > can remove this hack meanwhile QEMU should be able to keep the CPR behavior
> > > > > as before.  To CPR maintainers and reviewers: please have a closer look,
> > > > > even better if it can be smoke tested, to see if this works for Oracle's
> > > > > environment, TIA.
> > > > >
> > > > > The 1st patch implemented that new semantics.  It is straightforward: now
> > > > > we can setup any migration parameter using an extra line of:
> > > > >
> > > > >   -incoming config:key1=value1,key2=value2,...

snip

> > > > - using "config:"
> > > > 
> > > > This makes it non-uniform with uri and channels which don't have a
> > > > keyword in front of them. I guess I could live with it, but it seems
> > > > odd. I see that it makes parsing way easier.
> > > 
> > > Yeah, having some identifier would be nice.  I wished channels also have
> > > identifiers if we don't need to keep compatibility.
> > 
> > IMHO having a magic "config:" prefix is an anti-pattern because it
> > involves custom command line parsing logic.
> 
> Yes, it's not elegant.  I wished we had something like this though when
> introducing the cpr channels; right now anything wasn't "defer" or URI
> implies it's a "channel"..
> 
> I also wanted to avoid introducing new cmdline parameters, so migration
> incoming cmdlines can stick with the same option.  If there's better
> suggestion please shoot.
> 
> > 
> > Does "-incoming config" imply '-incoming defer' semantics or is it
> > independent ?  
> 
> They're independent.  Examples:
> 
>  a) "-incoming config:* -incoming tcp:*", setup parameters for TCP incoming
>     migration without further deferral
> 
>  b) "-incoming config:*" only, setup global parameters for a possible
>     upcoming outgoing migration

In that case they should definitely be independent command line
options. The old "-incoming" design is already broken / limited
in the non-'defer' case because it is hardcoded to use URI syntax
which can't express all the address formats we accept in QAPI
syntax. Adding more special cases onto -incoming just makes the
bad situation even worse and is not a forward looking design.

If we need the ability to specify migration parameters on the
CLI, then as a starting point for the design, we should assume that
-incoming does not exist, and design a complete solution from scratch
that uses QAPI exclusively, both for addresses and configuration.

As discussed before, IMHO the "migrate" and "migrate-incoming"
commands need to accept both the address(s) and parameters/capabilities
as inline data items rather than relying on pre-configured global state
from the 'migrate-parameter' / 'migrate-capability' commands.

If we did that modelling for 'migrate-incoming' then that modelling of
command parmaeters could map directly to a new '-migrate-incoming'
command line argument that accepted exactly the same data model.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Fabiano Rosas]

Mark Cave-Ayland <mark.caveayland@nutanix.com> writes:

> On 29/05/2026 15:15, Peter Xu wrote:
>
> (cut)
>
>>> I like the idea overall.
>>>
>>>> With this series, we should be able to drop "incoming-fds" TAP property
>>>> from the other series, instead relying on the existing "local" parameters
>>>> both in migration core and in TAP's property should suffice.
>>>>
>>>> One thing to mention is I didn't further make only-migratable into a
>>>> migration parameter.  Logically it will also work now with only-migratable,
>>>> but it then also means I'll need to convert it to a parameter, which will
>>>> be mutable even after VM started.  It will change how only-migratable used
>>>> to work, hence I skipped.
>>>>
>>>> After this, we also almost have no reason to use -global for migration
>>>> parameters.  Capabilities are still not supported in -incoming cmdline,
>>>> though.
>>>>
>>>
>>> Will we still merge capabilities and parameters? Then it would be a free
>>> upgrade.
>> 
>> Yes!  /me waiting for your patches.
>
> Is this related to the series posted at 
> https://patchew.org/QEMU/20251215220041.12657-1-farosas@suse.de/? What's 
> the current status with this work?
>

Yes, the plan is still to merge parameters and capabilities into a
single 'options' term. This is largely about internal abstractions. I
plan to get back to that part of the work as soon as I find some cycles.

The further changes involving deprecations and adding the 'config'
parameter to migration commands have encountered some push-back, so I
don't think we'll get back to them at least before the above is all
finished. I still think those changes are worthwhile, but we need to
find consensus.

>
> ATB,
>
> Mark.

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Peter Xu]

On Wed, Jun 03, 2026 at 06:15:57PM +0100, Daniel P. Berrangé wrote:
> On Wed, Jun 03, 2026 at 11:50:53AM -0400, Peter Xu wrote:
> > On Wed, Jun 03, 2026 at 10:48:54AM +0100, Daniel P. Berrangé wrote:
> > > On Fri, May 29, 2026 at 10:15:35AM -0400, Peter Xu wrote:
> > > > On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote:
> > > > > Peter Xu <peterx@redhat.com> writes:
> > > > > 
> > > > > > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907
> > > > > >
> > > > > > This series introduces a generic way to specify migration parameters that
> > > > > > can be used even during the early boot phase of QEMU.
> > > > > >
> > > > > > One example use case that already existed is CPR-transfer / CPR-exec.
> > > > > > currently QEMU has a temporary global variable (incoming_mode) to achieve
> > > > > > this, but it's hard to understand and this hack bleeded into quite a few
> > > > > > places that we could have avoided.  The lines in patch 2 touched may
> > > > > > provide some idea.
> > > > > >
> > > > > > With a generic approach of setting migration parameters with cmdlines, we
> > > > > > can remove this hack meanwhile QEMU should be able to keep the CPR behavior
> > > > > > as before.  To CPR maintainers and reviewers: please have a closer look,
> > > > > > even better if it can be smoke tested, to see if this works for Oracle's
> > > > > > environment, TIA.
> > > > > >
> > > > > > The 1st patch implemented that new semantics.  It is straightforward: now
> > > > > > we can setup any migration parameter using an extra line of:
> > > > > >
> > > > > >   -incoming config:key1=value1,key2=value2,...
> 
> snip
> 
> > > > > - using "config:"
> > > > > 
> > > > > This makes it non-uniform with uri and channels which don't have a
> > > > > keyword in front of them. I guess I could live with it, but it seems
> > > > > odd. I see that it makes parsing way easier.
> > > > 
> > > > Yeah, having some identifier would be nice.  I wished channels also have
> > > > identifiers if we don't need to keep compatibility.
> > > 
> > > IMHO having a magic "config:" prefix is an anti-pattern because it
> > > involves custom command line parsing logic.
> > 
> > Yes, it's not elegant.  I wished we had something like this though when
> > introducing the cpr channels; right now anything wasn't "defer" or URI
> > implies it's a "channel"..
> > 
> > I also wanted to avoid introducing new cmdline parameters, so migration
> > incoming cmdlines can stick with the same option.  If there's better
> > suggestion please shoot.
> > 
> > > 
> > > Does "-incoming config" imply '-incoming defer' semantics or is it
> > > independent ?  
> > 
> > They're independent.  Examples:
> > 
> >  a) "-incoming config:* -incoming tcp:*", setup parameters for TCP incoming
> >     migration without further deferral
> > 
> >  b) "-incoming config:*" only, setup global parameters for a possible
> >     upcoming outgoing migration
> 
> In that case they should definitely be independent command line
> options. The old "-incoming" design is already broken / limited
> in the non-'defer' case because it is hardcoded to use URI syntax
> which can't express all the address formats we accept in QAPI
> syntax. Adding more special cases onto -incoming just makes the
> bad situation even worse and is not a forward looking design.
> 
> If we need the ability to specify migration parameters on the
> CLI, then as a starting point for the design, we should assume that
> -incoming does not exist, and design a complete solution from scratch
> that uses QAPI exclusively, both for addresses and configuration.

Do we want to obsolete -incoming?

> 
> As discussed before, IMHO the "migrate" and "migrate-incoming"
> commands need to accept both the address(s) and parameters/capabilities
> as inline data items rather than relying on pre-configured global state
> from the 'migrate-parameter' / 'migrate-capability' commands.

This is one example that "relying only on migrate/migrate-incoming to
specify parameters" won't work.

Essentially parameters like CPR's and the new "local" parameter wants to be
global so that it will be visible and will have an impact on how other QEMU
modules behave.  Here for TAP when "local" is globally on we may want to
initialize the device differently from the default operations.  That will
need to happen before any QMP commands.

> 
> If we did that modelling for 'migrate-incoming' then that modelling of
> command parmaeters could map directly to a new '-migrate-incoming'
> command line argument that accepted exactly the same data model.

This is true.  Then we will make this series to depend on Fabiano's
previous work (to be posted; per we talked yesterday).

I wanted to see if we can reduce the impact to minimum, -incoming is indeed
not well designed, but in real life we have control over what can happen
with URIs and the current "config:" won't conflict with any of them.

But if we strongly wish to obsolete -incoming completely, I'm OK too.

Thanks,

-- 
Peter Xu

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Fabiano Rosas]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Wed, Jun 03, 2026 at 11:50:53AM -0400, Peter Xu wrote:
>> On Wed, Jun 03, 2026 at 10:48:54AM +0100, Daniel P. Berrangé wrote:
>> > On Fri, May 29, 2026 at 10:15:35AM -0400, Peter Xu wrote:
>> > > On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote:
>> > > > Peter Xu <peterx@redhat.com> writes:
>> > > > 
>> > > > > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907
>> > > > >
>> > > > > This series introduces a generic way to specify migration parameters that
>> > > > > can be used even during the early boot phase of QEMU.
>> > > > >
>> > > > > One example use case that already existed is CPR-transfer / CPR-exec.
>> > > > > currently QEMU has a temporary global variable (incoming_mode) to achieve
>> > > > > this, but it's hard to understand and this hack bleeded into quite a few
>> > > > > places that we could have avoided.  The lines in patch 2 touched may
>> > > > > provide some idea.
>> > > > >
>> > > > > With a generic approach of setting migration parameters with cmdlines, we
>> > > > > can remove this hack meanwhile QEMU should be able to keep the CPR behavior
>> > > > > as before.  To CPR maintainers and reviewers: please have a closer look,
>> > > > > even better if it can be smoke tested, to see if this works for Oracle's
>> > > > > environment, TIA.
>> > > > >
>> > > > > The 1st patch implemented that new semantics.  It is straightforward: now
>> > > > > we can setup any migration parameter using an extra line of:
>> > > > >
>> > > > >   -incoming config:key1=value1,key2=value2,...
>
> snip
>
>> > > > - using "config:"
>> > > > 
>> > > > This makes it non-uniform with uri and channels which don't have a
>> > > > keyword in front of them. I guess I could live with it, but it seems
>> > > > odd. I see that it makes parsing way easier.
>> > > 
>> > > Yeah, having some identifier would be nice.  I wished channels also have
>> > > identifiers if we don't need to keep compatibility.
>> > 
>> > IMHO having a magic "config:" prefix is an anti-pattern because it
>> > involves custom command line parsing logic.
>> 
>> Yes, it's not elegant.  I wished we had something like this though when
>> introducing the cpr channels; right now anything wasn't "defer" or URI
>> implies it's a "channel"..
>> 
>> I also wanted to avoid introducing new cmdline parameters, so migration
>> incoming cmdlines can stick with the same option.  If there's better
>> suggestion please shoot.
>> 
>> > 
>> > Does "-incoming config" imply '-incoming defer' semantics or is it
>> > independent ?  
>> 
>> They're independent.  Examples:
>> 
>>  a) "-incoming config:* -incoming tcp:*", setup parameters for TCP incoming
>>     migration without further deferral
>> 
>>  b) "-incoming config:*" only, setup global parameters for a possible
>>     upcoming outgoing migration
>
> In that case they should definitely be independent command line
> options. The old "-incoming" design is already broken / limited
> in the non-'defer' case because it is hardcoded to use URI syntax
> which can't express all the address formats we accept in QAPI
> syntax. Adding more special cases onto -incoming just makes the
> bad situation even worse and is not a forward looking design.
>
> If we need the ability to specify migration parameters on the
> CLI, then as a starting point for the design, we should assume that
> -incoming does not exist, and design a complete solution from scratch
> that uses QAPI exclusively, both for addresses and configuration.
>
> As discussed before, IMHO the "migrate" and "migrate-incoming"
> commands need to accept both the address(s) and parameters/capabilities
> as inline data items rather than relying on pre-configured global state
> from the 'migrate-parameter' / 'migrate-capability' commands.
>
> If we did that modelling for 'migrate-incoming' then that modelling of
> command parmaeters could map directly to a new '-migrate-incoming'
> command line argument that accepted exactly the same data model.
>

Maybe a user creatable object would better fit this use case instead of
a new command. We could expose what is today MigrationParameters plus
the few compat options from migration_properties. It could work more or
less the same for the QMP migration commands, QMP set/get commands,
source and destination command lines, the compatibility use-case and the
debugging use-case.

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Fabiano Rosas]

Peter Xu <peterx@redhat.com> writes:

> On Wed, Jun 03, 2026 at 06:15:57PM +0100, Daniel P. Berrangé wrote:
>> On Wed, Jun 03, 2026 at 11:50:53AM -0400, Peter Xu wrote:
>> > On Wed, Jun 03, 2026 at 10:48:54AM +0100, Daniel P. Berrangé wrote:
>> > > On Fri, May 29, 2026 at 10:15:35AM -0400, Peter Xu wrote:
>> > > > On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote:
>> > > > > Peter Xu <peterx@redhat.com> writes:
>> > > > > 
>> > > > > > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907
>> > > > > >
>> > > > > > This series introduces a generic way to specify migration parameters that
>> > > > > > can be used even during the early boot phase of QEMU.
>> > > > > >
>> > > > > > One example use case that already existed is CPR-transfer / CPR-exec.
>> > > > > > currently QEMU has a temporary global variable (incoming_mode) to achieve
>> > > > > > this, but it's hard to understand and this hack bleeded into quite a few
>> > > > > > places that we could have avoided.  The lines in patch 2 touched may
>> > > > > > provide some idea.
>> > > > > >
>> > > > > > With a generic approach of setting migration parameters with cmdlines, we
>> > > > > > can remove this hack meanwhile QEMU should be able to keep the CPR behavior
>> > > > > > as before.  To CPR maintainers and reviewers: please have a closer look,
>> > > > > > even better if it can be smoke tested, to see if this works for Oracle's
>> > > > > > environment, TIA.
>> > > > > >
>> > > > > > The 1st patch implemented that new semantics.  It is straightforward: now
>> > > > > > we can setup any migration parameter using an extra line of:
>> > > > > >
>> > > > > >   -incoming config:key1=value1,key2=value2,...
>> 
>> snip
>> 
>> > > > > - using "config:"
>> > > > > 
>> > > > > This makes it non-uniform with uri and channels which don't have a
>> > > > > keyword in front of them. I guess I could live with it, but it seems
>> > > > > odd. I see that it makes parsing way easier.
>> > > > 
>> > > > Yeah, having some identifier would be nice.  I wished channels also have
>> > > > identifiers if we don't need to keep compatibility.
>> > > 
>> > > IMHO having a magic "config:" prefix is an anti-pattern because it
>> > > involves custom command line parsing logic.
>> > 
>> > Yes, it's not elegant.  I wished we had something like this though when
>> > introducing the cpr channels; right now anything wasn't "defer" or URI
>> > implies it's a "channel"..
>> > 
>> > I also wanted to avoid introducing new cmdline parameters, so migration
>> > incoming cmdlines can stick with the same option.  If there's better
>> > suggestion please shoot.
>> > 
>> > > 
>> > > Does "-incoming config" imply '-incoming defer' semantics or is it
>> > > independent ?  
>> > 
>> > They're independent.  Examples:
>> > 
>> >  a) "-incoming config:* -incoming tcp:*", setup parameters for TCP incoming
>> >     migration without further deferral
>> > 
>> >  b) "-incoming config:*" only, setup global parameters for a possible
>> >     upcoming outgoing migration
>> 
>> In that case they should definitely be independent command line
>> options. The old "-incoming" design is already broken / limited
>> in the non-'defer' case because it is hardcoded to use URI syntax
>> which can't express all the address formats we accept in QAPI
>> syntax. Adding more special cases onto -incoming just makes the
>> bad situation even worse and is not a forward looking design.
>> 
>> If we need the ability to specify migration parameters on the
>> CLI, then as a starting point for the design, we should assume that
>> -incoming does not exist, and design a complete solution from scratch
>> that uses QAPI exclusively, both for addresses and configuration.
>
> Do we want to obsolete -incoming?
>
>> 
>> As discussed before, IMHO the "migrate" and "migrate-incoming"
>> commands need to accept both the address(s) and parameters/capabilities
>> as inline data items rather than relying on pre-configured global state
>> from the 'migrate-parameter' / 'migrate-capability' commands.
>
> This is one example that "relying only on migrate/migrate-incoming to
> specify parameters" won't work.
>
> Essentially parameters like CPR's and the new "local" parameter wants to be
> global so that it will be visible and will have an impact on how other QEMU
> modules behave.  Here for TAP when "local" is globally on we may want to
> initialize the device differently from the default operations.  That will
> need to happen before any QMP commands.
>
>> 
>> If we did that modelling for 'migrate-incoming' then that modelling of
>> command parmaeters could map directly to a new '-migrate-incoming'
>> command line argument that accepted exactly the same data model.
>
> This is true.  Then we will make this series to depend on Fabiano's
> previous work (to be posted; per we talked yesterday).
>

I don't see the dependency. Take the 'config' out from this series and
pass a JSON into -incoming. It can be parsed into MigrationParameters
all the same.

> I wanted to see if we can reduce the impact to minimum, -incoming is indeed
> not well designed, but in real life we have control over what can happen
> with URIs and the current "config:" won't conflict with any of them.
>
> But if we strongly wish to obsolete -incoming completely, I'm OK too.
>
> Thanks,

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Daniel P. Berrangé]

On Wed, Jun 03, 2026 at 02:51:30PM -0300, Fabiano Rosas wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > In that case they should definitely be independent command line
> > options. The old "-incoming" design is already broken / limited
> > in the non-'defer' case because it is hardcoded to use URI syntax
> > which can't express all the address formats we accept in QAPI
> > syntax. Adding more special cases onto -incoming just makes the
> > bad situation even worse and is not a forward looking design.
> >
> > If we need the ability to specify migration parameters on the
> > CLI, then as a starting point for the design, we should assume that
> > -incoming does not exist, and design a complete solution from scratch
> > that uses QAPI exclusively, both for addresses and configuration.
> >
> > As discussed before, IMHO the "migrate" and "migrate-incoming"
> > commands need to accept both the address(s) and parameters/capabilities
> > as inline data items rather than relying on pre-configured global state
> > from the 'migrate-parameter' / 'migrate-capability' commands.
> >
> > If we did that modelling for 'migrate-incoming' then that modelling of
> > command parmaeters could map directly to a new '-migrate-incoming'
> > command line argument that accepted exactly the same data model.
> >
> 
> Maybe a user creatable object would better fit this use case instead of
> a new command. We could expose what is today MigrationParameters plus
> the few compat options from migration_properties. It could work more or
> less the same for the QMP migration commands, QMP set/get commands,
> source and destination command lines, the compatibility use-case and the
> debugging use-case.

If we can do something using "-object" that is useful both at command
line time and in QMP runtime, then that would be interesting too.

My general feeling wrt changes to the current command line is that
any suggestion should be desgined with a nod towards a future scenario
where QEMU is 100% configured with QMP. ie the full command line is

   qemu-system-x86_64 -qmp <address>

In this world, the current -incoming argument is already largely
unsatisfactory for anything other than "defer".  A model that uses
-object would fit in well, as would a hypothetical -migrate-incoming
that directly mapped to 'migrate-incoming' QMP including capabilities
and parameters.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Peter Xu]

On Wed, Jun 03, 2026 at 07:00:07PM +0100, Daniel P. Berrangé wrote:
> On Wed, Jun 03, 2026 at 02:51:30PM -0300, Fabiano Rosas wrote:
> > Daniel P. Berrangé <berrange@redhat.com> writes:
> > 
> > > In that case they should definitely be independent command line
> > > options. The old "-incoming" design is already broken / limited
> > > in the non-'defer' case because it is hardcoded to use URI syntax
> > > which can't express all the address formats we accept in QAPI
> > > syntax. Adding more special cases onto -incoming just makes the
> > > bad situation even worse and is not a forward looking design.
> > >
> > > If we need the ability to specify migration parameters on the
> > > CLI, then as a starting point for the design, we should assume that
> > > -incoming does not exist, and design a complete solution from scratch
> > > that uses QAPI exclusively, both for addresses and configuration.
> > >
> > > As discussed before, IMHO the "migrate" and "migrate-incoming"
> > > commands need to accept both the address(s) and parameters/capabilities
> > > as inline data items rather than relying on pre-configured global state
> > > from the 'migrate-parameter' / 'migrate-capability' commands.
> > >
> > > If we did that modelling for 'migrate-incoming' then that modelling of
> > > command parmaeters could map directly to a new '-migrate-incoming'
> > > command line argument that accepted exactly the same data model.
> > >
> > 
> > Maybe a user creatable object would better fit this use case instead of
> > a new command. We could expose what is today MigrationParameters plus
> > the few compat options from migration_properties. It could work more or
> > less the same for the QMP migration commands, QMP set/get commands,
> > source and destination command lines, the compatibility use-case and the
> > debugging use-case.
> 
> If we can do something using "-object" that is useful both at command
> line time and in QMP runtime, then that would be interesting too.
> 
> My general feeling wrt changes to the current command line is that
> any suggestion should be desgined with a nod towards a future scenario
> where QEMU is 100% configured with QMP. ie the full command line is
> 
>    qemu-system-x86_64 -qmp <address>
> 
> In this world, the current -incoming argument is already largely
> unsatisfactory for anything other than "defer".  A model that uses
> -object would fit in well, as would a hypothetical -migrate-incoming
> that directly mapped to 'migrate-incoming' QMP including capabilities
> and parameters.

I don't think even the current -incoming config:* is a blocker or add too
much complexity to the full-QMP-based solution.

When it comes, we can simply map all -incoming config:* (JSON or not) to
QMP command migrate-set-parameters, what we need is teaching that QMP
handler to apply parameters to the temporary MigrationParameters rather
than migration object when the latter hasn't been initialized.

I was talking to Fabiano and he suggested me to mention one more thing I
said on the list.  It's a matter of whether we should still invest time on
supporting migration caps/parameters in QMP migrate/migrate-incoming
commands.

Requirements like this (allow migration parameters to be accessible during
early stage of QEMU) is going towards the other direction of the that idea.
That means even if we put all configs (caps/params) into QMP command
migrate[-incoming], libvirt will still need to manage these global setup
and making sure when invoking migrate[-incoming] QMP commands they match
with the globals.  Say, if one start QEMU with -incoming config:local=on
but then invoke "migrate-incoming,local=off" it's illegal.

Considering that it looks like there're solid use cases that we want to
support (after CPR's "mode" parameter, now "local"), I want to discuss
again whether we want to still spend effort supporting "allow migrate QMP
command to specify capabilities/parameters".  Ultimately, we still seem to
need these global parameters.

Thanks,

-- 
Peter Xu

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Vladimir Sementsov-Ogievskiy]

On 03.06.26 21:46, Peter Xu wrote:
>> My general feeling wrt changes to the current command line is that
>> any suggestion should be desgined with a nod towards a future scenario
>> where QEMU is 100% configured with QMP. ie the full command line is
>>
>>     qemu-system-x86_64 -qmp <address>
>>

[..]

> 
> Requirements like this (allow migration parameters to be accessible during
> early stage of QEMU) is going towards the other direction of the that idea.
> That means even if we put all configs (caps/params) into QMP command
> migrate[-incoming], libvirt will still need to manage these global setup
> and making sure when invoking migrate[-incoming] QMP commands they match
> with the globals.  Say, if one start QEMU with -incomingconfig:local=on
> but then invoke "migrate-incoming,local=off" it's illegal.

Right.. And this show, that moving "local" from migrate-set-parameters to
commandline is not a clear solution for the whole problem.

We don't need cmdline argument. We need two things:

1. "local" must be set before initializing tap-device.

It not actually requires it being a cmdline parameter. Calling
migrate-set-parameters before netdev_add is also OK.

So, moving to cmdline solves this [1] point, but may be too restrictive.


2. "local" must not be changed after initializing tap-device.

And this one is not guaranteed anyway, with cmdline or with QMP.

---


So, in my series, if drop "incoming-fds" and rely on "local" instead, we
actually want "local" be immutable after first read in tap initialization
code.

Maybe, just implement this feature? So, in code it will look like:

set_migration_parameters(...) {

...

if "local" value is changing and "local" is immutable:
    fail

...

}


tap_initializaion(...) {

...

local = migrate_get_local_and_make_it_immutable(err_text="you can not change 'local' parameter value after TAP device initialization");

...


}

---

For user it should provide simple error messages

    "you can not change 'local' parameter value after TAP device initialization".

if user tries to change local when it's not allowed.


-- 
Best regards,
Vladimir

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Daniel P. Berrangé]

On Wed, Jun 03, 2026 at 02:46:52PM -0400, Peter Xu wrote:
> On Wed, Jun 03, 2026 at 07:00:07PM +0100, Daniel P. Berrangé wrote:
> > On Wed, Jun 03, 2026 at 02:51:30PM -0300, Fabiano Rosas wrote:
> > > Daniel P. Berrangé <berrange@redhat.com> writes:
> > > 
> > > > In that case they should definitely be independent command line
> > > > options. The old "-incoming" design is already broken / limited
> > > > in the non-'defer' case because it is hardcoded to use URI syntax
> > > > which can't express all the address formats we accept in QAPI
> > > > syntax. Adding more special cases onto -incoming just makes the
> > > > bad situation even worse and is not a forward looking design.
> > > >
> > > > If we need the ability to specify migration parameters on the
> > > > CLI, then as a starting point for the design, we should assume that
> > > > -incoming does not exist, and design a complete solution from scratch
> > > > that uses QAPI exclusively, both for addresses and configuration.
> > > >
> > > > As discussed before, IMHO the "migrate" and "migrate-incoming"
> > > > commands need to accept both the address(s) and parameters/capabilities
> > > > as inline data items rather than relying on pre-configured global state
> > > > from the 'migrate-parameter' / 'migrate-capability' commands.
> > > >
> > > > If we did that modelling for 'migrate-incoming' then that modelling of
> > > > command parmaeters could map directly to a new '-migrate-incoming'
> > > > command line argument that accepted exactly the same data model.
> > > >
> > > 
> > > Maybe a user creatable object would better fit this use case instead of
> > > a new command. We could expose what is today MigrationParameters plus
> > > the few compat options from migration_properties. It could work more or
> > > less the same for the QMP migration commands, QMP set/get commands,
> > > source and destination command lines, the compatibility use-case and the
> > > debugging use-case.
> > 
> > If we can do something using "-object" that is useful both at command
> > line time and in QMP runtime, then that would be interesting too.
> > 
> > My general feeling wrt changes to the current command line is that
> > any suggestion should be desgined with a nod towards a future scenario
> > where QEMU is 100% configured with QMP. ie the full command line is
> > 
> >    qemu-system-x86_64 -qmp <address>
> > 
> > In this world, the current -incoming argument is already largely
> > unsatisfactory for anything other than "defer".  A model that uses
> > -object would fit in well, as would a hypothetical -migrate-incoming
> > that directly mapped to 'migrate-incoming' QMP including capabilities
> > and parameters.
> 
> I don't think even the current -incoming config:* is a blocker or add too
> much complexity to the full-QMP-based solution.
> 
> When it comes, we can simply map all -incoming config:* (JSON or not) to
> QMP command migrate-set-parameters, what we need is teaching that QMP
> handler to apply parameters to the temporary MigrationParameters rather
> than migration object when the latter hasn't been initialized.
> 
> I was talking to Fabiano and he suggested me to mention one more thing I
> said on the list.  It's a matter of whether we should still invest time on
> supporting migration caps/parameters in QMP migrate/migrate-incoming
> commands.
> 
> Requirements like this (allow migration parameters to be accessible during
> early stage of QEMU) is going towards the other direction of the that idea.
> That means even if we put all configs (caps/params) into QMP command
> migrate[-incoming], libvirt will still need to manage these global setup
> and making sure when invoking migrate[-incoming] QMP commands they match
> with the globals.  Say, if one start QEMU with -incoming config:local=on
> but then invoke "migrate-incoming,local=off" it's illegal.
> 
> Considering that it looks like there're solid use cases that we want to
> support (after CPR's "mode" parameter, now "local"), I want to discuss
> again whether we want to still spend effort supporting "allow migrate QMP
> command to specify capabilities/parameters".  Ultimately, we still seem to
> need these global parameters.

This is saying that incoming migration is a multi phase/stage task.

There is a "prepare" phase which sets up QEMU ready to receive an
incoming migration.

There is a "running" phase where we are waiting for incoming connection
and/or handling the migration data stream.

The current "migrate_incoming" overloads both tasks into the same QMP
command such that they always happen at the same point in time.
"-incoming defer" was a crude hack to partially separate them, allowing
parameters/capabilities to be set before the real migrate_incoming
QMP command is invoked.

We can address that by separating the tasks explicitly with a
"migrate_prepare" command / -migrate-prepare CLI arg that sets all
the parameters/capabilities atomically, and a "migrate_run" command
that initiates the processing.

The "jobs" QAPI design has this explicit concept of different phases
that a job can be in, and provides commands for lifecycle handling.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH RFC 1/2] migration/vl: Allow set parameters with -incoming config:*

[Mark Cave-Ayland]

On 03/06/2026 16:55, Peter Xu wrote:

> On Wed, Jun 03, 2026 at 04:36:37PM +0100, Mark Cave-Ayland wrote:
>> Is the longer-term goal here to allow machine-versioned usage of all
>> migration parameters supplied via -incoming via -global?
> 
> Nop.
> 
> Machine compat properties sticks with machines, they'll be applied properly
> to migration objects at some point (needs to happen after machine object is
> initialized).  It'll keep working, no plan to change that.

What are the machine compat properties currently being used for in this 
case? Presumably something that isn't expressed through a capability?

> This series is only about exposing a new API to setup migration parameters
> so that modules (like TAP as backends) can query at very early stage of
> QEMU boot.

So shouldn't we just QOMify MigrationParameters so that our existing 
parser can be used without coming up with a custom solution? I feel I'm 
missing something obvious here since this was Fabiano's suggestion.


ATB,

Mark.

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Mark Cave-Ayland]

On 03/06/2026 16:59, Peter Xu wrote:

>> Is this part of the same problem in that the migration object needs to be
>> created much earlier than it is currently? And could doing this
>> potentially remove the need for a separate CPR channel?
> 
> The problem is the migration object can't yet be created too early, due to
> the fact it needs to apply compat properties.  That's why this series used
> another approach to have a temporary early MigrationParameters object to
> keep things until migration object is initialized (during which the temp
> object will be destroyed).

If the MigrationParameters are needed earlier, then presumably devices 
can only depend upon the value of migration capabilities, and not the 
value of migration compat props?

In that case could you not just build the migration compat props with 
.realized_set_allowed = true in a similar way to e.g. hw/nvram/xlnx-bbram.c?


ATB,

Mark.

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Mark Cave-Ayland]

On 03/06/2026 18:15, Daniel P. Berrangé wrote:

> On Wed, Jun 03, 2026 at 11:50:53AM -0400, Peter Xu wrote:
>> On Wed, Jun 03, 2026 at 10:48:54AM +0100, Daniel P. Berrangé wrote:
>>> On Fri, May 29, 2026 at 10:15:35AM -0400, Peter Xu wrote:
>>>> On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote:
>>>>> Peter Xu <peterx@redhat.com> writes:
>>>>>
>>>>>> CI: https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.com_peterx_qemu_-2D_pipelines_2560168907&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=c23RpsaH4D2MKyD3EPJTDa0BAxz6tV8aUJqVSoytEiY&m=kl69Otn4IxRhV6bx7NvgCZttOrO1SL4fZ-Qd3Jb-ODVtspTqSiQ6t1ujrrf-AB5J&s=nlkn4_NSgXkwTmC4JK_Le8z4ElYZ_cmlKRBCjJhTrIY&e=
>>>>>>
>>>>>> This series introduces a generic way to specify migration parameters that
>>>>>> can be used even during the early boot phase of QEMU.
>>>>>>
>>>>>> One example use case that already existed is CPR-transfer / CPR-exec.
>>>>>> currently QEMU has a temporary global variable (incoming_mode) to achieve
>>>>>> this, but it's hard to understand and this hack bleeded into quite a few
>>>>>> places that we could have avoided.  The lines in patch 2 touched may
>>>>>> provide some idea.
>>>>>>
>>>>>> With a generic approach of setting migration parameters with cmdlines, we
>>>>>> can remove this hack meanwhile QEMU should be able to keep the CPR behavior
>>>>>> as before.  To CPR maintainers and reviewers: please have a closer look,
>>>>>> even better if it can be smoke tested, to see if this works for Oracle's
>>>>>> environment, TIA.
>>>>>>
>>>>>> The 1st patch implemented that new semantics.  It is straightforward: now
>>>>>> we can setup any migration parameter using an extra line of:
>>>>>>
>>>>>>    -incoming config:key1=value1,key2=value2,...
> 
> snip
> 
>>>>> - using "config:"
>>>>>
>>>>> This makes it non-uniform with uri and channels which don't have a
>>>>> keyword in front of them. I guess I could live with it, but it seems
>>>>> odd. I see that it makes parsing way easier.
>>>>
>>>> Yeah, having some identifier would be nice.  I wished channels also have
>>>> identifiers if we don't need to keep compatibility.
>>>
>>> IMHO having a magic "config:" prefix is an anti-pattern because it
>>> involves custom command line parsing logic.
>>
>> Yes, it's not elegant.  I wished we had something like this though when
>> introducing the cpr channels; right now anything wasn't "defer" or URI
>> implies it's a "channel"..
>>
>> I also wanted to avoid introducing new cmdline parameters, so migration
>> incoming cmdlines can stick with the same option.  If there's better
>> suggestion please shoot.
>>
>>>
>>> Does "-incoming config" imply '-incoming defer' semantics or is it
>>> independent ?
>>
>> They're independent.  Examples:
>>
>>   a) "-incoming config:* -incoming tcp:*", setup parameters for TCP incoming
>>      migration without further deferral
>>
>>   b) "-incoming config:*" only, setup global parameters for a possible
>>      upcoming outgoing migration
> 
> In that case they should definitely be independent command line
> options. The old "-incoming" design is already broken / limited
> in the non-'defer' case because it is hardcoded to use URI syntax
> which can't express all the address formats we accept in QAPI
> syntax. Adding more special cases onto -incoming just makes the
> bad situation even worse and is not a forward looking design.
> 
> If we need the ability to specify migration parameters on the
> CLI, then as a starting point for the design, we should assume that
> -incoming does not exist, and design a complete solution from scratch
> that uses QAPI exclusively, both for addresses and configuration.

Yes, absolutely. There has been a lot of effort to standardise our 
parsing, so something that will accept the current (legacy?) -incoming 
command line along with a new JSON notation allowing the extra 
parameters feels like the right approach.

> As discussed before, IMHO the "migrate" and "migrate-incoming"
> commands need to accept both the address(s) and parameters/capabilities
> as inline data items rather than relying on pre-configured global state
> from the 'migrate-parameter' / 'migrate-capability' commands.
> 
> If we did that modelling for 'migrate-incoming' then that modelling of
> command parmaeters could map directly to a new '-migrate-incoming'
> command line argument that accepted exactly the same data model.
> 
> With regards,
> Daniel


ATB,

Mark.

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Peter Xu]

On Thu, Jun 04, 2026 at 11:00:17AM +0100, Mark Cave-Ayland wrote:
> On 03/06/2026 16:59, Peter Xu wrote:
> 
> > > Is this part of the same problem in that the migration object needs to be
> > > created much earlier than it is currently? And could doing this
> > > potentially remove the need for a separate CPR channel?
> > 
> > The problem is the migration object can't yet be created too early, due to
> > the fact it needs to apply compat properties.  That's why this series used
> > another approach to have a temporary early MigrationParameters object to
> > keep things until migration object is initialized (during which the temp
> > object will be destroyed).
> 
> If the MigrationParameters are needed earlier, then presumably devices can
> only depend upon the value of migration capabilities, and not the value of
> migration compat props?
> 
> In that case could you not just build the migration compat props with
> .realized_set_allowed = true in a similar way to e.g. hw/nvram/xlnx-bbram.c?

Could you elaborate?

A few contexts to provide:

Migration object (even if now a TYPE_DEVICE) doesn't use realize(), and it
will be TYPE_OBJECT soon, which is at least the current plan.  Direct use
of anything qdev specific won't work.

The other thing is, migration parameters by default should be allowed to be
set anytime after QEMU starts but before migration starts.  I don't yet
understand the reference of realized_set_allowed use case, and how that
idea can apply to migration.

Thanks,

-- 
Peter Xu

Re: [PATCH RFC 1/2] migration/vl: Allow set parameters with -incoming config:*

[Peter Xu]

On Thu, Jun 04, 2026 at 10:21:45AM +0100, Mark Cave-Ayland wrote:
> On 03/06/2026 16:55, Peter Xu wrote:
> 
> > On Wed, Jun 03, 2026 at 04:36:37PM +0100, Mark Cave-Ayland wrote:
> > > Is the longer-term goal here to allow machine-versioned usage of all
> > > migration parameters supplied via -incoming via -global?
> > 
> > Nop.
> > 
> > Machine compat properties sticks with machines, they'll be applied properly
> > to migration objects at some point (needs to happen after machine object is
> > initialized).  It'll keep working, no plan to change that.
> 
> What are the machine compat properties currently being used for in this
> case? Presumably something that isn't expressed through a capability?

Please refer to:

https://lore.kernel.org/qemu-devel/ahmiKxc0swrIJQVB@x1.local/

> 
> > This series is only about exposing a new API to setup migration parameters
> > so that modules (like TAP as backends) can query at very early stage of
> > QEMU boot.
> 
> So shouldn't we just QOMify MigrationParameters so that our existing parser
> can be used without coming up with a custom solution? I feel I'm missing
> something obvious here since this was Fabiano's suggestion.

IIUC this series almost does it, qobject_input_visitor_new_str() and then
visit with a follow up visit_type_MigrationParameters().

Do we explicitly need to make it a separate TYPE_OBJECT?  Is there any
benefit of it, and how that proposal differs from the current?

Thanks,

-- 
Peter Xu

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Peter Xu]

On Thu, Jun 04, 2026 at 01:00:10AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 03.06.26 21:46, Peter Xu wrote:
> > > My general feeling wrt changes to the current command line is that
> > > any suggestion should be desgined with a nod towards a future scenario
> > > where QEMU is 100% configured with QMP. ie the full command line is
> > > 
> > >     qemu-system-x86_64 -qmp <address>
> > > 
> 
> [..]
> 
> > 
> > Requirements like this (allow migration parameters to be accessible during
> > early stage of QEMU) is going towards the other direction of the that idea.
> > That means even if we put all configs (caps/params) into QMP command
> > migrate[-incoming], libvirt will still need to manage these global setup
> > and making sure when invoking migrate[-incoming] QMP commands they match
> > with the globals.  Say, if one start QEMU with -incomingconfig:local=on
> > but then invoke "migrate-incoming,local=off" it's illegal.
> 
> Right.. And this show, that moving "local" from migrate-set-parameters to
> commandline is not a clear solution for the whole problem.
> 
> We don't need cmdline argument. We need two things:
> 
> 1. "local" must be set before initializing tap-device.
> 
> It not actually requires it being a cmdline parameter. Calling
> migrate-set-parameters before netdev_add is also OK.

Ohh is it ok?  I thought it was not OK so we look at this.

Say, this is what I see on init TAP device when without hotplug:

qemu_create_late_backends() -> net_init_clients()

Such happens before migration object initialization.

> 
> So, moving to cmdline solves this [1] point, but may be too restrictive.
> 
> 
> 2. "local" must not be changed after initializing tap-device.
> 
> And this one is not guaranteed anyway, with cmdline or with QMP.
> 
> ---
> 
> 
> So, in my series, if drop "incoming-fds" and rely on "local" instead, we
> actually want "local" be immutable after first read in tap initialization
> code.
> 
> Maybe, just implement this feature? So, in code it will look like:
> 
> set_migration_parameters(...) {
> 
> ...
> 
> if "local" value is changing and "local" is immutable:
>    fail

We can't do that, can we?

Imagine we need to further migrate this VM to another host, where we need
to turn "local" off after this incoming migration.. we can forbid only
during incoming phase and re-enable the mutability, but it seems too much.

My understanding is such protection is fine but not strongly necessary.
IMHO we rely on a lot of things that admin needs to do right.  I hope this
isn't a major issue to offload that to admin to say the admin should always
do the right things.

We have a bunch of similar issues in QEMU IIUC, e.g. we have known issue
that some -device needs to be ordered in some way otherwise it'll stop
working.  We then need admin (or in this case libvirt) do the right thing
too.

> 
> ...
> 
> }
> 
> 
> tap_initializaion(...) {
> 
> ...
> 
> local = migrate_get_local_and_make_it_immutable(err_text="you can not change 'local' parameter value after TAP device initialization");
> 
> ...
> 
> 
> }
> 
> ---
> 
> For user it should provide simple error messages
> 
>    "you can not change 'local' parameter value after TAP device initialization".
> 
> if user tries to change local when it's not allowed.
> 
> 
> -- 
> Best regards,
> Vladimir
> 

-- 
Peter Xu

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Peter Xu]

On Thu, Jun 04, 2026 at 09:18:26AM +0100, Daniel P. Berrangé wrote:
> On Wed, Jun 03, 2026 at 02:46:52PM -0400, Peter Xu wrote:
> > On Wed, Jun 03, 2026 at 07:00:07PM +0100, Daniel P. Berrangé wrote:
> > > On Wed, Jun 03, 2026 at 02:51:30PM -0300, Fabiano Rosas wrote:
> > > > Daniel P. Berrangé <berrange@redhat.com> writes:
> > > > 
> > > > > In that case they should definitely be independent command line
> > > > > options. The old "-incoming" design is already broken / limited
> > > > > in the non-'defer' case because it is hardcoded to use URI syntax
> > > > > which can't express all the address formats we accept in QAPI
> > > > > syntax. Adding more special cases onto -incoming just makes the
> > > > > bad situation even worse and is not a forward looking design.
> > > > >
> > > > > If we need the ability to specify migration parameters on the
> > > > > CLI, then as a starting point for the design, we should assume that
> > > > > -incoming does not exist, and design a complete solution from scratch
> > > > > that uses QAPI exclusively, both for addresses and configuration.
> > > > >
> > > > > As discussed before, IMHO the "migrate" and "migrate-incoming"
> > > > > commands need to accept both the address(s) and parameters/capabilities
> > > > > as inline data items rather than relying on pre-configured global state
> > > > > from the 'migrate-parameter' / 'migrate-capability' commands.
> > > > >
> > > > > If we did that modelling for 'migrate-incoming' then that modelling of
> > > > > command parmaeters could map directly to a new '-migrate-incoming'
> > > > > command line argument that accepted exactly the same data model.
> > > > >
> > > > 
> > > > Maybe a user creatable object would better fit this use case instead of
> > > > a new command. We could expose what is today MigrationParameters plus
> > > > the few compat options from migration_properties. It could work more or
> > > > less the same for the QMP migration commands, QMP set/get commands,
> > > > source and destination command lines, the compatibility use-case and the
> > > > debugging use-case.
> > > 
> > > If we can do something using "-object" that is useful both at command
> > > line time and in QMP runtime, then that would be interesting too.
> > > 
> > > My general feeling wrt changes to the current command line is that
> > > any suggestion should be desgined with a nod towards a future scenario
> > > where QEMU is 100% configured with QMP. ie the full command line is
> > > 
> > >    qemu-system-x86_64 -qmp <address>
> > > 
> > > In this world, the current -incoming argument is already largely
> > > unsatisfactory for anything other than "defer".  A model that uses
> > > -object would fit in well, as would a hypothetical -migrate-incoming
> > > that directly mapped to 'migrate-incoming' QMP including capabilities
> > > and parameters.
> > 
> > I don't think even the current -incoming config:* is a blocker or add too
> > much complexity to the full-QMP-based solution.
> > 
> > When it comes, we can simply map all -incoming config:* (JSON or not) to
> > QMP command migrate-set-parameters, what we need is teaching that QMP
> > handler to apply parameters to the temporary MigrationParameters rather
> > than migration object when the latter hasn't been initialized.
> > 
> > I was talking to Fabiano and he suggested me to mention one more thing I
> > said on the list.  It's a matter of whether we should still invest time on
> > supporting migration caps/parameters in QMP migrate/migrate-incoming
> > commands.
> > 
> > Requirements like this (allow migration parameters to be accessible during
> > early stage of QEMU) is going towards the other direction of the that idea.
> > That means even if we put all configs (caps/params) into QMP command
> > migrate[-incoming], libvirt will still need to manage these global setup
> > and making sure when invoking migrate[-incoming] QMP commands they match
> > with the globals.  Say, if one start QEMU with -incoming config:local=on
> > but then invoke "migrate-incoming,local=off" it's illegal.
> > 
> > Considering that it looks like there're solid use cases that we want to
> > support (after CPR's "mode" parameter, now "local"), I want to discuss
> > again whether we want to still spend effort supporting "allow migrate QMP
> > command to specify capabilities/parameters".  Ultimately, we still seem to
> > need these global parameters.
> 
> This is saying that incoming migration is a multi phase/stage task.
> 
> There is a "prepare" phase which sets up QEMU ready to receive an
> incoming migration.
> 
> There is a "running" phase where we are waiting for incoming connection
> and/or handling the migration data stream.
> 
> The current "migrate_incoming" overloads both tasks into the same QMP
> command such that they always happen at the same point in time.
> "-incoming defer" was a crude hack to partially separate them, allowing
> parameters/capabilities to be set before the real migrate_incoming
> QMP command is invoked.
> 
> We can address that by separating the tasks explicitly with a
> "migrate_prepare" command / -migrate-prepare CLI arg that sets all
> the parameters/capabilities atomically, and a "migrate_run" command
> that initiates the processing.

I'm OK with this, or IMHO we can also stick with -incoming treating it as
the preparation phase.

For most of the time, I would slightly prefer keeping old things working
but building new things on top.  Obsolete should only happen if very needed
and properly justified.

For this one, I hope the QAPI helpers I used should introduce minimum
overhead even if we're reusing an old paramaeter.. we can still consider
introducing another parmeter besides -incoming, but it'll work similarly
like what this patch does, then.

> 
> The "jobs" QAPI design has this explicit concept of different phases
> that a job can be in, and provides commands for lifecycle handling.

We have some rich discussion before on Jobs:

https://lore.kernel.org/qemu-devel/878qg1uhbd.fsf_-_@pond.sub.org/#t

IIRC the consensus was it's only about the interfacing that can be shared or
not, which doesn't seem to be a huge mount of code that we can reuse; what
we can reuse is minimum and we may need to overload the interface when
migration joins the party.

We're definitely open to adopt anything Jobs did right.  We don't
necessarily need to switch to Jobs until further justified, which won't be
a trivial task.

Thanks,

-- 
Peter Xu

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Vladimir Sementsov-Ogievskiy]

On 04.06.26 21:01, Peter Xu wrote:
> On Thu, Jun 04, 2026 at 01:00:10AM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> On 03.06.26 21:46, Peter Xu wrote:
>>>> My general feeling wrt changes to the current command line is that
>>>> any suggestion should be desgined with a nod towards a future scenario
>>>> where QEMU is 100% configured with QMP. ie the full command line is
>>>>
>>>>      qemu-system-x86_64 -qmp <address>
>>>>
>>
>> [..]
>>
>>>
>>> Requirements like this (allow migration parameters to be accessible during
>>> early stage of QEMU) is going towards the other direction of the that idea.
>>> That means even if we put all configs (caps/params) into QMP command
>>> migrate[-incoming], libvirt will still need to manage these global setup
>>> and making sure when invoking migrate[-incoming] QMP commands they match
>>> with the globals.  Say, if one start QEMU with -incomingconfig:local=on
>>> but then invoke "migrate-incoming,local=off" it's illegal.
>>
>> Right.. And this show, that moving "local" from migrate-set-parameters to
>> commandline is not a clear solution for the whole problem.
>>
>> We don't need cmdline argument. We need two things:
>>
>> 1. "local" must be set before initializing tap-device.
>>
>> It not actually requires it being a cmdline parameter. Calling
>> migrate-set-parameters before netdev_add is also OK.
> 
> Ohh is it ok?  I thought it was not OK so we look at this.
> 
> Say, this is what I see on init TAP device when without hotplug:
> 
> qemu_create_late_backends() -> net_init_clients()
> 
> Such happens before migration object initialization.

Yes, it doesn't work for TAP devices added through commandline. But
if devices are added only throuhg QMP (isn't libvirt do so?), it
should work.

> 
>>
>> So, moving to cmdline solves this [1] point, but may be too restrictive.
>>
>>
>> 2. "local" must not be changed after initializing tap-device.
>>
>> And this one is not guaranteed anyway, with cmdline or with QMP.
>>
>> ---
>>
>>
>> So, in my series, if drop "incoming-fds" and rely on "local" instead, we
>> actually want "local" be immutable after first read in tap initialization
>> code.
>>
>> Maybe, just implement this feature? So, in code it will look like:
>>
>> set_migration_parameters(...) {
>>
>> ...
>>
>> if "local" value is changing and "local" is immutable:
>>     fail
> 
> We can't do that, can we?
> 
> Imagine we need to further migrate this VM to another host, where we need
> to turn "local" off after this incoming migration.. we can forbid only
> during incoming phase and re-enable the mutability, but it seems too much.

Agree, right. I missed further outgoing migration.

> 
> My understanding is such protection is fine but not strongly necessary.
> IMHO we rely on a lot of things that admin needs to do right.  I hope this
> isn't a major issue to offload that to admin to say the admin should always
> do the right things.
> 
> We have a bunch of similar issues in QEMU IIUC, e.g. we have known issue
> that some -device needs to be ordered in some way otherwise it'll stop
> working.  We then need admin (or in this case libvirt) do the right thing
> too.

So, possible way is:

1. net devices are only added by QMP commands
2. migration parameter "local" is set before adding devices

So, we can omit "incoming-fds" and rely on "local" && "support-local-migration".

And we can check in .pre_incoming of TAP device, does current value of "local"
equal to what it was at time of TAP initialization, and fail if it differs.

Still, having explicit incoming-fds looks safer, as it's more difficult to do
a mistake.

With incoming-fds=true, user is sure, that new target vm will not try
to connect/open resources, which are still used by running source vm,
regardless of migration configuration correctness.

For example, if user accidentally set "local" _after_ creating TAP device,
we can of-course check it in .pre_incoming and fail, but at the moment we
already break networking in running guest, by attaching to same backend on
target QEMU.

-- 
Best regards,
Vladimir

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Peter Xu]

On Fri, Jun 05, 2026 at 10:35:52AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 04.06.26 21:01, Peter Xu wrote:
> > On Thu, Jun 04, 2026 at 01:00:10AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > On 03.06.26 21:46, Peter Xu wrote:
> > > > > My general feeling wrt changes to the current command line is that
> > > > > any suggestion should be desgined with a nod towards a future scenario
> > > > > where QEMU is 100% configured with QMP. ie the full command line is
> > > > > 
> > > > >      qemu-system-x86_64 -qmp <address>
> > > > > 
> > > 
> > > [..]
> > > 
> > > > 
> > > > Requirements like this (allow migration parameters to be accessible during
> > > > early stage of QEMU) is going towards the other direction of the that idea.
> > > > That means even if we put all configs (caps/params) into QMP command
> > > > migrate[-incoming], libvirt will still need to manage these global setup
> > > > and making sure when invoking migrate[-incoming] QMP commands they match
> > > > with the globals.  Say, if one start QEMU with -incomingconfig:local=on
> > > > but then invoke "migrate-incoming,local=off" it's illegal.
> > > 
> > > Right.. And this show, that moving "local" from migrate-set-parameters to
> > > commandline is not a clear solution for the whole problem.
> > > 
> > > We don't need cmdline argument. We need two things:
> > > 
> > > 1. "local" must be set before initializing tap-device.
> > > 
> > > It not actually requires it being a cmdline parameter. Calling
> > > migrate-set-parameters before netdev_add is also OK.
> > 
> > Ohh is it ok?  I thought it was not OK so we look at this.
> > 
> > Say, this is what I see on init TAP device when without hotplug:
> > 
> > qemu_create_late_backends() -> net_init_clients()
> > 
> > Such happens before migration object initialization.
> 
> Yes, it doesn't work for TAP devices added through commandline. But
> if devices are added only throuhg QMP (isn't libvirt do so?), it
> should work.
> 
> > 
> > > 
> > > So, moving to cmdline solves this [1] point, but may be too restrictive.
> > > 
> > > 
> > > 2. "local" must not be changed after initializing tap-device.
> > > 
> > > And this one is not guaranteed anyway, with cmdline or with QMP.
> > > 
> > > ---
> > > 
> > > 
> > > So, in my series, if drop "incoming-fds" and rely on "local" instead, we
> > > actually want "local" be immutable after first read in tap initialization
> > > code.
> > > 
> > > Maybe, just implement this feature? So, in code it will look like:
> > > 
> > > set_migration_parameters(...) {
> > > 
> > > ...
> > > 
> > > if "local" value is changing and "local" is immutable:
> > >     fail
> > 
> > We can't do that, can we?
> > 
> > Imagine we need to further migrate this VM to another host, where we need
> > to turn "local" off after this incoming migration.. we can forbid only
> > during incoming phase and re-enable the mutability, but it seems too much.
> 
> Agree, right. I missed further outgoing migration.
> 
> > 
> > My understanding is such protection is fine but not strongly necessary.
> > IMHO we rely on a lot of things that admin needs to do right.  I hope this
> > isn't a major issue to offload that to admin to say the admin should always
> > do the right things.
> > 
> > We have a bunch of similar issues in QEMU IIUC, e.g. we have known issue
> > that some -device needs to be ordered in some way otherwise it'll stop
> > working.  We then need admin (or in this case libvirt) do the right thing
> > too.
> 
> So, possible way is:
> 
> 1. net devices are only added by QMP commands
> 2. migration parameter "local" is set before adding devices
> 
> So, we can omit "incoming-fds" and rely on "local" && "support-local-migration".
> 
> And we can check in .pre_incoming of TAP device, does current value of "local"
> equal to what it was at time of TAP initialization, and fail if it differs.
> 
> Still, having explicit incoming-fds looks safer, as it's more difficult to do
> a mistake.

If this will be the only outlier, then maybe yes, an extra option in a
special device backend isn't much of an issue.

The question is if there'll be more things relying on "local=on".  We may
not want to keep introducing per-device flags even if all of them can play
the same role as migration's "local".

> 
> With incoming-fds=true, user is sure, that new target vm will not try
> to connect/open resources, which are still used by running source vm,
> regardless of migration configuration correctness.
> 
> For example, if user accidentally set "local" _after_ creating TAP device,
> we can of-course check it in .pre_incoming and fail, but at the moment we
> already break networking in running guest, by attaching to same backend on
> target QEMU.
> 
> -- 
> Best regards,
> Vladimir
> 

-- 
Peter Xu

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Vladimir Sementsov-Ogievskiy]

On 05.06.26 17:08, Peter Xu wrote:
> On Fri, Jun 05, 2026 at 10:35:52AM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> On 04.06.26 21:01, Peter Xu wrote:
>>> On Thu, Jun 04, 2026 at 01:00:10AM +0300, Vladimir Sementsov-Ogievskiy wrote:
>>>> On 03.06.26 21:46, Peter Xu wrote:
>>>>>> My general feeling wrt changes to the current command line is that
>>>>>> any suggestion should be desgined with a nod towards a future scenario
>>>>>> where QEMU is 100% configured with QMP. ie the full command line is
>>>>>>
>>>>>>       qemu-system-x86_64 -qmp <address>
>>>>>>
>>>> [..]
>>>>
>>>>> Requirements like this (allow migration parameters to be accessible during
>>>>> early stage of QEMU) is going towards the other direction of the that idea.
>>>>> That means even if we put all configs (caps/params) into QMP command
>>>>> migrate[-incoming], libvirt will still need to manage these global setup
>>>>> and making sure when invoking migrate[-incoming] QMP commands they match
>>>>> with the globals.  Say, if one start QEMU with -incomingconfig:local=on
>>>>> but then invoke "migrate-incoming,local=off" it's illegal.
>>>> Right.. And this show, that moving "local" from migrate-set-parameters to
>>>> commandline is not a clear solution for the whole problem.
>>>>
>>>> We don't need cmdline argument. We need two things:
>>>>
>>>> 1. "local" must be set before initializing tap-device.
>>>>
>>>> It not actually requires it being a cmdline parameter. Calling
>>>> migrate-set-parameters before netdev_add is also OK.
>>> Ohh is it ok?  I thought it was not OK so we look at this.
>>>
>>> Say, this is what I see on init TAP device when without hotplug:
>>>
>>> qemu_create_late_backends() -> net_init_clients()
>>>
>>> Such happens before migration object initialization.
>> Yes, it doesn't work for TAP devices added through commandline. But
>> if devices are added only throuhg QMP (isn't libvirt do so?), it
>> should work.
>>
>>>> So, moving to cmdline solves this [1] point, but may be too restrictive.
>>>>
>>>>
>>>> 2. "local" must not be changed after initializing tap-device.
>>>>
>>>> And this one is not guaranteed anyway, with cmdline or with QMP.
>>>>
>>>> ---
>>>>
>>>>
>>>> So, in my series, if drop "incoming-fds" and rely on "local" instead, we
>>>> actually want "local" be immutable after first read in tap initialization
>>>> code.
>>>>
>>>> Maybe, just implement this feature? So, in code it will look like:
>>>>
>>>> set_migration_parameters(...) {
>>>>
>>>> ...
>>>>
>>>> if "local" value is changing and "local" is immutable:
>>>>      fail
>>> We can't do that, can we?
>>>
>>> Imagine we need to further migrate this VM to another host, where we need
>>> to turn "local" off after this incoming migration.. we can forbid only
>>> during incoming phase and re-enable the mutability, but it seems too much.
>> Agree, right. I missed further outgoing migration.
>>
>>> My understanding is such protection is fine but not strongly necessary.
>>> IMHO we rely on a lot of things that admin needs to do right.  I hope this
>>> isn't a major issue to offload that to admin to say the admin should always
>>> do the right things.
>>>
>>> We have a bunch of similar issues in QEMU IIUC, e.g. we have known issue
>>> that some -device needs to be ordered in some way otherwise it'll stop
>>> working.  We then need admin (or in this case libvirt) do the right thing
>>> too.
>> So, possible way is:
>>
>> 1. net devices are only added by QMP commands
>> 2. migration parameter "local" is set before adding devices
>>
>> So, we can omit "incoming-fds" and rely on "local" && "support-local-migration".
>>
>> And we can check in .pre_incoming of TAP device, does current value of "local"
>> equal to what it was at time of TAP initialization, and fail if it differs.
>>
>> Still, having explicit incoming-fds looks safer, as it's more difficult to do
>> a mistake.
> If this will be the only outlier, then maybe yes, an extra option in a
> special device backend isn't much of an issue.
> 
> The question is if there'll be more things relying on "local=on".  We may
> not want to keep introducing per-device flags even if all of them can play
> the same role as migration's "local".

Agree.

Hmm, so, may be, I just go this way for my series? It will just work with
these two restrictions above, which is OK for me. And no conflict with
new "-incoming" design.

-- 
Best regards,
Vladimir

Re: [PATCH RFC 0/2] migration/vl: new -incoming config:* for early migration parameters

[Peter Xu]

On Fri, Jun 05, 2026 at 05:37:03PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> Hmm, so, may be, I just go this way for my series? It will just work with
> these two restrictions above, which is OK for me. And no conflict with
> new "-incoming" design.

Sounds good.  IIUC that's also our original plan to make it experimental
and see if that can lands earlier, so no worry on whatever happens with
this series so far, feel free to go ahead.

Thanks,

-- 
Peter Xu