Re: [Qemu-devel] Migration ToDo list (a.k.a. Rant)

["Denis V. Lunev" <den-lists@virtuozzo.com>]

On 05/04/2016 02:20 PM, Juan Quintela wrote:
> Hi
>
> I am lots of times asked about what is the ToDo list for migration, that
> was on my head, and random notes over my desk, so, trying some
> organization (Yes, I would put this in the wiki).
>
>
> - migration thread on reception
>    would make trivial to do other things while receiving, and would make
>    postcopy easier also (I was going to put much easier, but postcopy is
>    never easy).
>
> - migration capabilities and parameters
>    this is a mess.  Not, is worse than that.  I don't know who is to
>    blame here, but something needs to be done:
>
>       void qmp_migrate_set_parameters(bool has_compress_level,
>                                  int64_t compress_level,
>                                  bool has_compress_threads,
>                                  int64_t compress_threads,
>                                  bool has_decompress_threads,
>                                  int64_t decompress_threads,
>                                  bool has_x_cpu_throttle_initial,
>                                  int64_t x_cpu_throttle_initial,
>                                  bool has_x_cpu_throttle_increment,
>                                  int64_t x_cpu_throttle_increment,
>                                  bool has_multifd_threads,
>                                  int64_t multifd_threads,
>                                  Error **errp)
>
>
>
>      Can we move this to an array of structs, please, pretty please?
>      I think that for this one, the blame is on qmp
>
>     but we can continue:
>
>     migrate
>     migrate_cancel
>     migrate_incoming
>     migrate_start_postcopy
>
>        Not a lot to do until here
>
>     migrate_set_capability
>        Minor nickpit, if it only allow booleans, "migrate_set_capability x-multifd",
>        should be an equivalent of "migrate_set_capability x-multifd on"
>
>     migrate_set_cache_size
>     migrate_set_downtime
>     migrate_set_speed
>        This three should be claimed obsolete, deprecated, whatever, and
>        make it on top of next one
>
>     migrate_set_parameter
>
>     Now to read the migration information:
>
>       migrate_capabilities
>         good
>       migrate_parameters
>         good
>       migrate_cache_size
>         good, but we are missing migrate_speed and migrate_downtime, see
>         why I want it be inside migrate_set_parameters
>
>       migrate
>         now, this is ..... weird?  We put here lots of information, and
>         this is basically the only way to put information out.  To make
>         things more interesting, the values change meaning during
>         migration, and the fields it shows change also over time.
>
> - info migrate
>    This deserves its own item.  Lets see a typical output
>
> (qemu)info migrate
>
> capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-multifd: on
>
>     Aha, we have the capabilities, but not the parameters.  This is
>     historical, I know, but don't belong here.
>
> Migration status: completed
>     ok
> total time: 1621 milliseconds
>     ok
> downtime: 208 milliseconds
>     ok
> setup: 9 milliseconds
>     ok
>
> transferred ram: 609708 kbytes
>     kilo bytes, not pages
>
> throughput: 27.64 mbps
>     but we measure bandwidth is megabytes by second
>     previous one was kylobytes
>
> remaining ram: 0 kbytes
>
> total ram: 2106180 kbytes
>     this amount don't change.  I can understand why it was here.
>
> duplicate: 452528 pages
>     name is historical.  It really means pages filled with the same
>     characeter.  Althought in practical effects it means zero pages
>
> skipped: 0 pages
>     Even I don't remember what this means.
> normal: 151064 pages
>     This is normal pages that we have sent, i.e. pages that are not zero
>     pages nor skipped pages.  Notice that we have put here pages, not
>     bytes, not kilobytes, but pages.
>
> normal bytes: 604256 kbytes
>     Don't worry, we put for you the same number as kilobytes.
>
> dirty sync count: 11
>     Number of iterations over the full ram.  Yes, I know, we are very,
>     very bad at naming.
>
> And we still have more optional information that appears if we are doing
> block migration, xbzrle, compression, rdma, etc, etc.
>
> We need to decide some units also internal.  Some things are in bytes,
> some are in kilobytes, some are in pages.  Some are in host pages, or
> guest pages, or who knows :-(
>
>
> - Block migration (the migration/block.c one).  This is the bastard
>    child of migration.  Much less tested, we should make a decision
>    about letting it live or deprecating it.  Things needed from memory:
>       - functions should return the same values than ram.c
>         some functions don't have "exact" values, and return 1 when there
>         are more than one block dirty, etc, etc
>       - if we continue maintaing it, allowing it to have _some_ shared
>         devices and some non shared ones, insntead of everything?
>
> - RDMA: Another step child
>
>    This is really, really weird.  We don't use the normal infrastructure
>    for RDMA, we use the ram_control_* stuff.  We should really move to
>    use the normal stuff here.
>
> - autoconverge code:  This could be used outside of migration (i.e. just
>    to slow down a guess).  We should really do some measurement here to
>    see how useful it is for migration.  If the guest is using lots of
>    memory dirtying, we end having to throttle the guest 90% or so :-(
>
> - xbzrle.  We only have one cache, we should decide how to work with
>    this for multithread/compression.
>
> - When we do migration, we have spaguetti code to decide if:
>    * it is a zero page
>    * it is a duplicated page
>    * it is a xbzrle page
>    * it is a compressed page
>    And as the code is written, it is not trivial to add new "options".  I
>    think that we should "re-think" what combinations are allowed an which
>    ones make nosense.
>
> - savevm and migration: they use two different paths for not really good
>    reason.  We should really abstract this to a single code path.
>    We always forget the savevm one when we do changes.
>
> - error handling.  Every function should return an error.  Every
>    function should return an error.
>
> - qemu_get_buffer() don't give one error if there is nothing to read,
>    sniff.
>
> - Multipage support: Welcome to the XXI century.  Now almost all
>    architectures have HugePages.  And other have different sized pages
>    (in PPC is not strange that page size of host and guest differ).  We
>    have work to do here.  For starters, sending Huge pages as one chunk
>    will make TransparentHugePages happier.
>
> - Bitmaps.  Related with previous one.  We should really be better about
>    walking them and about synchronising them between qemu/kernel.
>
> - COLO: We need to integrate it.
>
> I will continue the rant at some other point O:-)  Just now I need to
> left for the bar.
>
> Thanks for your attention, Juan.
>
> PD.  I just looked while I wrote this to the channel code from Daniel, a
> step on the right direction.
>
>
Let me add too.

- snapshot management (savevm/loadvm) via QMP interface

Den