precreate phase

precreate phase

[Steven Sistare]

Peter, Fabiano,

I have a nice solution that allows dest qemu configuration with cpr-transfer.
I define a new qemu initialization phase called 'precreate' which occurs
before most backends or devices have been created.  The only exception
is monitor devices and the qtest device and their chardevs, which are created.
I untangled dependencies so that that migration_object_init can be called
before this phase.  The monitor accepts commands during this phase and can
receive migration configuration commands.  qemu starts listening on the
normal migration URI during this phase (which can come from either the
qemu command line or from a migrate_incoming command) so the user can
issue query-migrate to get the socket-address for dynamically allocated
port numbers.

qemu enters the phase if qemu is started with the -precreate option.
The phase ends when the user sends a precreate-exit command.
Qemu then calls cpr_load_state and reads from the normal migration connection
as in the existing design, including the HUP.

(This sounds like the preconfig phase, but that occurs too late, after
backends have been created.)

I implemented this scheme for another reason but abandoned it, so I need
a few days to rebase it and test.  It is not small, and requires approvals
from additional maintainers.  Let me know if you think this is the right
solution, and I will revive it.

Preview:

   0725d70 vl: precreate phase
   edd2dee net: cleanup for precreate phase
   4733c00 accel: encapsulate search state
   6d26ea4 accel: accel preinit function
   518e737 accel: split configure_accelerators
   8ef936b accel: delete accelerator and machine props
   b5c2a84 accel: set accelerator and machine props earlier
   56890a0 migration: init early
   76506b2 monitor: chardev name
   e85111d qom: get properties
   8356f7e qemu-option: filtered foreach
   fb6cc98 qemu-options: pass object to filter
   bdfec1b monitor: connect in precreate
   8f2f292 qtest: connect in precreate
   629cb50 migration: connect URI early
   0ae530e migration: allow commands during precreate and preconfig

  accel/accel-system.c            |   2 -
  accel/kvm/kvm-all.c             |  58 +++++---
  accel/xen/xen-all.c             |  11 +-
  hmp-commands.hx                 |  20 +++
  include/migration/misc.h        |   2 +
  include/monitor/hmp.h           |   1 +
  include/monitor/monitor.h       |   1 +
  include/qapi/visitor.h          |   1 +
  include/qemu/accel.h            |   1 +
  include/qemu/option.h           |   5 +
  include/qom/object_interfaces.h |   2 +
  include/sysemu/sysemu.h         |   3 +
  migration/migration.c           |  31 +++++
  monitor/hmp-cmds.c              |   8 ++
  monitor/monitor.c               |  21 +++
  net/net.c                       |   4 +-
  qapi/migration.json             |  16 ++-
  qapi/misc.json                  |  23 +++-
  qemu-options.hx                 |  13 ++
  qom/object_interfaces.c         |  27 ++--
  system/vl.c                     | 291 +++++++++++++++++++++++++++++++---------
  target/i386/nvmm/nvmm-all.c     |  10 +-
  target/i386/whpx/whpx-all.c     |  14 +-
  util/qemu-option.c              |  25 ++++
  24 files changed, 471 insertions(+), 119 deletions(-)

Re: precreate phase

[Fabiano Rosas]

Steven Sistare <steven.sistare@oracle.com> writes:

> Peter, Fabiano,
>
> I have a nice solution that allows dest qemu configuration with cpr-transfer.
> I define a new qemu initialization phase called 'precreate' which occurs
> before most backends or devices have been created.  The only exception
> is monitor devices and the qtest device and their chardevs, which are created.
> I untangled dependencies so that that migration_object_init can be called
> before this phase.  The monitor accepts commands during this phase and can
> receive migration configuration commands.  qemu starts listening on the
> normal migration URI during this phase (which can come from either the
> qemu command line or from a migrate_incoming command) so the user can
> issue query-migrate to get the socket-address for dynamically allocated
> port numbers.
>
> qemu enters the phase if qemu is started with the -precreate option.
> The phase ends when the user sends a precreate-exit command.
> Qemu then calls cpr_load_state and reads from the normal migration connection
> as in the existing design, including the HUP.
>
> (This sounds like the preconfig phase, but that occurs too late, after
> backends have been created.)
>
> I implemented this scheme for another reason but abandoned it, so I need
> a few days to rebase it and test.  It is not small, and requires approvals
> from additional maintainers.  Let me know if you think this is the right
> solution, and I will revive it.

Seems reasonable to me, given the requirements we're working with. Was
there a branch we could skim somewhere?

>
> Preview:
>
>    0725d70 vl: precreate phase
>    edd2dee net: cleanup for precreate phase
>    4733c00 accel: encapsulate search state
>    6d26ea4 accel: accel preinit function
>    518e737 accel: split configure_accelerators
>    8ef936b accel: delete accelerator and machine props
>    b5c2a84 accel: set accelerator and machine props earlier
>    56890a0 migration: init early
>    76506b2 monitor: chardev name
>    e85111d qom: get properties
>    8356f7e qemu-option: filtered foreach
>    fb6cc98 qemu-options: pass object to filter
>    bdfec1b monitor: connect in precreate
>    8f2f292 qtest: connect in precreate
>    629cb50 migration: connect URI early
>    0ae530e migration: allow commands during precreate and preconfig
>
>   accel/accel-system.c            |   2 -
>   accel/kvm/kvm-all.c             |  58 +++++---
>   accel/xen/xen-all.c             |  11 +-
>   hmp-commands.hx                 |  20 +++
>   include/migration/misc.h        |   2 +
>   include/monitor/hmp.h           |   1 +
>   include/monitor/monitor.h       |   1 +
>   include/qapi/visitor.h          |   1 +
>   include/qemu/accel.h            |   1 +
>   include/qemu/option.h           |   5 +
>   include/qom/object_interfaces.h |   2 +
>   include/sysemu/sysemu.h         |   3 +
>   migration/migration.c           |  31 +++++
>   monitor/hmp-cmds.c              |   8 ++
>   monitor/monitor.c               |  21 +++
>   net/net.c                       |   4 +-
>   qapi/migration.json             |  16 ++-
>   qapi/misc.json                  |  23 +++-
>   qemu-options.hx                 |  13 ++
>   qom/object_interfaces.c         |  27 ++--
>   system/vl.c                     | 291 +++++++++++++++++++++++++++++++---------
>   target/i386/nvmm/nvmm-all.c     |  10 +-
>   target/i386/whpx/whpx-all.c     |  14 +-
>   util/qemu-option.c              |  25 ++++
>   24 files changed, 471 insertions(+), 119 deletions(-)

Re: precreate phase

[Peter Xu]

On Thu, Oct 10, 2024 at 05:51:16PM -0300, Fabiano Rosas wrote:
> Steven Sistare <steven.sistare@oracle.com> writes:
> 
> > Peter, Fabiano,
> >
> > I have a nice solution that allows dest qemu configuration with cpr-transfer.
> > I define a new qemu initialization phase called 'precreate' which occurs
> > before most backends or devices have been created.  The only exception
> > is monitor devices and the qtest device and their chardevs, which are created.
> > I untangled dependencies so that that migration_object_init can be called
> > before this phase.  The monitor accepts commands during this phase and can
> > receive migration configuration commands.  qemu starts listening on the
> > normal migration URI during this phase (which can come from either the
> > qemu command line or from a migrate_incoming command) so the user can
> > issue query-migrate to get the socket-address for dynamically allocated
> > port numbers.
> >
> > qemu enters the phase if qemu is started with the -precreate option.
> > The phase ends when the user sends a precreate-exit command.

Perhaps x-exit-precreate, just to match x-exit-preconfig?

> > Qemu then calls cpr_load_state and reads from the normal migration connection
> > as in the existing design, including the HUP.

I didn't fully get this part - can the cpr-uri channel completely avoided
then?  Do you mean the HUP can be avoided with it (alone with -cpr-uri dest
qemu cmdline)?

> >
> > (This sounds like the preconfig phase, but that occurs too late, after
> > backends have been created.)
> >
> > I implemented this scheme for another reason but abandoned it, so I need
> > a few days to rebase it and test.  It is not small, and requires approvals
> > from additional maintainers.  Let me know if you think this is the right
> > solution, and I will revive it.
> 
> Seems reasonable to me, given the requirements we're working with. Was
> there a branch we could skim somewhere?

Yes a tree can be clearer, with an example usage.

This design looks even cleaner to me (but maybe misunderstood some part),
it may slow down the merge because we need broader reviews.  So it's just
that it might have higher chance miss 9.2.

> 
> >
> > Preview:
> >
> >    0725d70 vl: precreate phase
> >    edd2dee net: cleanup for precreate phase
> >    4733c00 accel: encapsulate search state
> >    6d26ea4 accel: accel preinit function
> >    518e737 accel: split configure_accelerators
> >    8ef936b accel: delete accelerator and machine props
> >    b5c2a84 accel: set accelerator and machine props earlier
> >    56890a0 migration: init early
> >    76506b2 monitor: chardev name
> >    e85111d qom: get properties
> >    8356f7e qemu-option: filtered foreach
> >    fb6cc98 qemu-options: pass object to filter
> >    bdfec1b monitor: connect in precreate
> >    8f2f292 qtest: connect in precreate
> >    629cb50 migration: connect URI early
> >    0ae530e migration: allow commands during precreate and preconfig
> >
> >   accel/accel-system.c            |   2 -
> >   accel/kvm/kvm-all.c             |  58 +++++---
> >   accel/xen/xen-all.c             |  11 +-
> >   hmp-commands.hx                 |  20 +++
> >   include/migration/misc.h        |   2 +
> >   include/monitor/hmp.h           |   1 +
> >   include/monitor/monitor.h       |   1 +
> >   include/qapi/visitor.h          |   1 +
> >   include/qemu/accel.h            |   1 +
> >   include/qemu/option.h           |   5 +
> >   include/qom/object_interfaces.h |   2 +
> >   include/sysemu/sysemu.h         |   3 +
> >   migration/migration.c           |  31 +++++
> >   monitor/hmp-cmds.c              |   8 ++
> >   monitor/monitor.c               |  21 +++
> >   net/net.c                       |   4 +-
> >   qapi/migration.json             |  16 ++-
> >   qapi/misc.json                  |  23 +++-
> >   qemu-options.hx                 |  13 ++
> >   qom/object_interfaces.c         |  27 ++--
> >   system/vl.c                     | 291 +++++++++++++++++++++++++++++++---------
> >   target/i386/nvmm/nvmm-all.c     |  10 +-
> >   target/i386/whpx/whpx-all.c     |  14 +-
> >   util/qemu-option.c              |  25 ++++
> >   24 files changed, 471 insertions(+), 119 deletions(-)
> 

-- 
Peter Xu

Re: precreate phase

[Paolo Bonzini]

On 10/10/24 22:07, Steven Sistare wrote:
> Peter, Fabiano,
> 
> I have a nice solution that allows dest qemu configuration with cpr- 
> transfer.
> I define a new qemu initialization phase called 'precreate' which occurs
> before most backends or devices have been created.  The only exception
> is monitor devices and the qtest device and their chardevs, which are 
> created.
> I untangled dependencies so that that migration_object_init can be called
> before this phase.  The monitor accepts commands during this phase and can
> receive migration configuration commands.  qemu starts listening on the
> normal migration URI during this phase (which can come from either the
> qemu command line or from a migrate_incoming command) so the user can
> issue query-migrate to get the socket-address for dynamically allocated
> port numbers.

Can you provide an example of the commands?

Moving migration_object_init() earlier sounds like a good idea anyway!

Paolo

> qemu enters the phase if qemu is started with the -precreate option.
> The phase ends when the user sends a precreate-exit command.
> Qemu then calls cpr_load_state and reads from the normal migration 
> connection
> as in the existing design, including the HUP.
> 
> (This sounds like the preconfig phase, but that occurs too late, after
> backends have been created.)
> 
> I implemented this scheme for another reason but abandoned it, so I need
> a few days to rebase it and test.  It is not small, and requires approvals
> from additional maintainers.  Let me know if you think this is the right
> solution, and I will revive it.
> 
> Preview:
> 
>    0725d70 vl: precreate phase
>    edd2dee net: cleanup for precreate phase
>    4733c00 accel: encapsulate search state
>    6d26ea4 accel: accel preinit function
>    518e737 accel: split configure_accelerators
>    8ef936b accel: delete accelerator and machine props
>    b5c2a84 accel: set accelerator and machine props earlier
>    56890a0 migration: init early
>    76506b2 monitor: chardev name
>    e85111d qom: get properties
>    8356f7e qemu-option: filtered foreach
>    fb6cc98 qemu-options: pass object to filter
>    bdfec1b monitor: connect in precreate
>    8f2f292 qtest: connect in precreate
>    629cb50 migration: connect URI early
>    0ae530e migration: allow commands during precreate and preconfig
> 
>   accel/accel-system.c            |   2 -
>   accel/kvm/kvm-all.c             |  58 +++++---
>   accel/xen/xen-all.c             |  11 +-
>   hmp-commands.hx                 |  20 +++
>   include/migration/misc.h        |   2 +
>   include/monitor/hmp.h           |   1 +
>   include/monitor/monitor.h       |   1 +
>   include/qapi/visitor.h          |   1 +
>   include/qemu/accel.h            |   1 +
>   include/qemu/option.h           |   5 +
>   include/qom/object_interfaces.h |   2 +
>   include/sysemu/sysemu.h         |   3 +
>   migration/migration.c           |  31 +++++
>   monitor/hmp-cmds.c              |   8 ++
>   monitor/monitor.c               |  21 +++
>   net/net.c                       |   4 +-
>   qapi/migration.json             |  16 ++-
>   qapi/misc.json                  |  23 +++-
>   qemu-options.hx                 |  13 ++
>   qom/object_interfaces.c         |  27 ++--
>   system/vl.c                     | 291 ++++++++++++++++++++++++++++++ 
> +---------
>   target/i386/nvmm/nvmm-all.c     |  10 +-
>   target/i386/whpx/whpx-all.c     |  14 +-
>   util/qemu-option.c              |  25 ++++
>   24 files changed, 471 insertions(+), 119 deletions(-)
> 
> 
> 
> 

Re: precreate phase

[Peter Xu]

On Thu, Oct 10, 2024 at 11:19:15PM +0200, Paolo Bonzini wrote:
> Moving migration_object_init() earlier sounds like a good idea anyway!

I take the last sentence back of my other reply - I believe I
underestimated the potential reviewers of the upcoming precreate
patchset.. :)

-- 
Peter Xu