Re: [Qemu-devel] [PATCH v3 00/29] postcopy+vhost-user/shared ram

["Michael S. Tsirkin" <mst@redhat.com>]

On Tue, Feb 27, 2018 at 08:05:25PM +0000, Dr. David Alan Gilbert wrote:
> * Michael S. Tsirkin (mst@redhat.com) wrote:
> > On Fri, Feb 16, 2018 at 01:15:56PM +0000, Dr. David Alan Gilbert (git) wrote:
> > > From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > > 
> > > Hi,
> > >   This is the first non-RFC version of this patch set that
> > > enables postcopy migration with shared memory to a vhost user process.
> > > It's based off current head.
> > > 
> > > I've tested with vhost-user-bridge and a modified dpdk; both very
> > > lightly.
> > > 
> > > Compared to v2 we're now using the just-merged reworks to the vhost
> > > code (suggested by Igor), so that the huge page region merging is now a lot simpler
> > > in this series. The handshake between the client and the qemu for the
> > > set-mem-table is now a bit more complex to resolve a previous race where
> > > the client would start sending requests to the qemu prior to the qemu
> > > being ready to accept them.
> > > 
> > > Dave
> > 
> > From vhost-user POV this seems mostly fine to me.
> 
> OK, great - it would be nice to get this merged in the upcoming release
> (Hint: Anyone else please review!)
> 
> > I would like to have dependency of specific messages on the
> > protocol features documented, and the order of messages
> > documented a bit more explicitly.
> 
> Something like the following? (appropriately merged in with the
> individual commits):
> 
> diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt
> index 4bf7d8ef99..7841812766 100644
> --- a/docs/interop/vhost-user.txt
> +++ b/docs/interop/vhost-user.txt
> @@ -461,7 +461,7 @@ Master message types
>        for each memory mapped region. The size and ordering of the fds matches
>        the number and ordering of memory regions.
>  
> -      When postcopy-listening has been received, SET_MEM_TABLE replies with
> +      When VHOST_USER_POSTCOPY_LISTEN has been received, SET_MEM_TABLE replies with
>        the bases of the memory mapped regions to the master.  It must have mmap'd
>        the regions but not yet accessed them and should not yet generate a userfault
>        event. Note NEED_REPLY_MASK is not set in this case.
> @@ -687,7 +687,8 @@ Master message types
>        Master payload: N/A
>        Slave payload: userfault fd + u64
>  
> -      Master advises slave that a migration with postcopy enabled is underway,
> +      When VHOST_USER_PROTOCOL_F_PAGEFAULT is supported, the
> +      master advises slave that a migration with postcopy enabled is underway,
>        the slave must open a userfaultfd for later use.
>        Note that at this stage the migration is still in precopy mode.
>  
> @@ -696,6 +697,8 @@ Master message types
>        Master payload: N/A
>  
>        Master advises slave that a transition to postcopy mode has happened.
> +      This is always sent sometime after a VHOST_USER_POSTCOPY_ADVISE, and
> +      thus only when VHOST_USER_PROTOCOL_F_PAGEFAULT is supported.
>  
>   * VHOST_USER_POSTCOPY_END
>        Id: 28
> @@ -704,6 +707,8 @@ Master message types
>        Master advises that postcopy migration has now completed.  The
>        slave must disable the userfaultfd. The response is an acknowledgement
>        only.
> +      This message is sent at the end of the migration, after
> +      VHOST_USER_POSTCOPY_LISTEN was previously sent.

And maybe mention VHOST_USER_PROTOCOL_F_PAGEFAULT here too.

>  Slave message types
>  -------------------
> 
> Dave
> 
> > 
> > 
> > 
> > > Dr. David Alan Gilbert (29):
> > >   migrate: Update ram_block_discard_range for shared
> > >   qemu_ram_block_host_offset
> > >   postcopy: use UFFDIO_ZEROPAGE only when available
> > >   postcopy: Add notifier chain
> > >   postcopy: Add vhost-user flag for postcopy and check it
> > >   vhost-user: Add 'VHOST_USER_POSTCOPY_ADVISE' message
> > >   libvhost-user: Support sending fds back to qemu
> > >   libvhost-user: Open userfaultfd
> > >   postcopy: Allow registering of fd handler
> > >   vhost+postcopy: Register shared ufd with postcopy
> > >   vhost+postcopy: Transmit 'listen' to client
> > >   postcopy+vhost-user: Split set_mem_table for postcopy
> > >   migration/ram: ramblock_recv_bitmap_test_byte_offset
> > >   libvhost-user+postcopy: Register new regions with the ufd
> > >   vhost+postcopy: Send address back to qemu
> > >   vhost+postcopy: Stash RAMBlock and offset
> > >   vhost+postcopy: Send requests to source for shared pages
> > >   vhost+postcopy: Resolve client address
> > >   postcopy: wake shared
> > >   postcopy: postcopy_notify_shared_wake
> > >   vhost+postcopy: Add vhost waker
> > >   vhost+postcopy: Call wakeups
> > >   libvhost-user: mprotect & madvises for postcopy
> > >   vhost-user: Add VHOST_USER_POSTCOPY_END message
> > >   vhost+postcopy: Wire up POSTCOPY_END notify
> > >   vhost: Huge page align and merge
> > >   postcopy: Allow shared memory
> > >   libvhost-user: Claim support for postcopy
> > >   postcopy shared docs
> > > 
> > >  contrib/libvhost-user/libvhost-user.c | 303 ++++++++++++++++++++++++-
> > >  contrib/libvhost-user/libvhost-user.h |   8 +
> > >  docs/devel/migration.rst              |  41 ++++
> > >  docs/interop/vhost-user.txt           |  42 ++++
> > >  exec.c                                |  85 +++++--
> > >  hw/virtio/trace-events                |  16 +-
> > >  hw/virtio/vhost-user.c                | 411 +++++++++++++++++++++++++++++++++-
> > >  hw/virtio/vhost.c                     |  66 +++++-
> > >  include/exec/cpu-common.h             |   4 +
> > >  migration/migration.c                 |   6 +
> > >  migration/migration.h                 |   4 +
> > >  migration/postcopy-ram.c              | 350 +++++++++++++++++++++++------
> > >  migration/postcopy-ram.h              |  69 ++++++
> > >  migration/ram.c                       |   5 +
> > >  migration/ram.h                       |   1 +
> > >  migration/savevm.c                    |  13 ++
> > >  migration/trace-events                |   6 +
> > >  trace-events                          |   3 +-
> > >  vl.c                                  |   2 +
> > >  19 files changed, 1337 insertions(+), 98 deletions(-)
> > > 
> > > -- 
> > > 2.14.3
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK