Re: [Qemu-devel] [PATCH v3 4/5] migration: Append JSON description of migration stream

[Eric Blake <eblake@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 3052 bytes --]

On 12/26/2014 07:42 AM, Alexander Graf wrote:
> One of the annoyances of the current migration format is the fact that
> it's not self-describing. In fact, it's not properly describing at all.
> Some code randomly scattered throughout QEMU elaborates roughly how to
> read and write a stream of bytes.
> 
> We discussed an idea during KVM Forum 2013 to add a JSON description of
> the migration protocol itself to the migration stream. This patch
> adds a section after the VM_END migration end marker that contains
> description data on what the device sections of the stream are composed of.
> 
> With an additional external program this allows us to decipher the
> contents of any migration stream and hopefully make migration bugs easier
> to track down.

Hmm.  The new format IS self-describing, but ONLY because JSON is
technically possible to parse in reverse.  That is, you cannot reliably
look for end-of-stream marker from the beginning of the stream and
reliably get to the JSON description at the end every single time
(because if you don't already know how to interpret the stream, how can
you find end-of-stream), but you CAN reliably look to see if the stream
ends in a JSON object, and then scan backwards for the matching { that
opens the object.

I'm still wondering if you ought to throw in any additional tricks to
make finding the start of the JSON object much easier, such as a
subsection near the beginning of the migration stream, and/or a final
offset description at the tail of the file that says where the JSON
object starts (no additional help for a pipeline, which still has to
store things in memory to loc.  Even doing something like:

stream EOS [ { description... }, offset-of-description ]

and using a JSON array rather than a JSON object as the description
would make it much faster to find the start of the JSON object without
having to scan backwards for matching { (although sticking the offset at
the tail doesn't help the issue that when receiving the migration stream
over a pipeline, you still have to reserve memory for the entire stream
in order to find that offset).

> 
> Signed-off-by: Alexander Graf <agraf@suse.de>
> 
> ---
> 
> v2 -> v3:
> 
>   - Dont compress objects with subsections
>   - Destroy QJSON object
> ---
>  hw/pci/pci.c                  |   2 +-
>  hw/scsi/spapr_vscsi.c         |   2 +-
>  hw/virtio/virtio.c            |   2 +-
>  include/migration/migration.h |   1 +
>  include/migration/vmstate.h   |   3 +-
>  migration/vmstate.c           | 186 ++++++++++++++++++++++++++++++++++++++++--
>  savevm.c                      |  54 ++++++++++--
>  tests/test-vmstate.c          |   6 +-
>  8 files changed, 237 insertions(+), 19 deletions(-)
> 

At this point, I haven't actually reviewed the patch (I'm more
interested in seeing the JSON it generates), but want to make sure we're
getting an optimal design.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]