From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53171) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y8WUR-0001mH-UX for qemu-devel@nongnu.org; Tue, 06 Jan 2015 10:56:17 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Y8WUM-0007Sl-RW for qemu-devel@nongnu.org; Tue, 06 Jan 2015 10:56:15 -0500 Received: from mx1.redhat.com ([209.132.183.28]:50402) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Y8WUM-0007Sd-KH for qemu-devel@nongnu.org; Tue, 06 Jan 2015 10:56:10 -0500 Message-ID: <54AC0597.8020009@redhat.com> Date: Tue, 06 Jan 2015 08:56:07 -0700 From: Eric Blake MIME-Version: 1.0 References: <1419604968-87437-1-git-send-email-agraf@suse.de> <1419604968-87437-5-git-send-email-agraf@suse.de> In-Reply-To: <1419604968-87437-5-git-send-email-agraf@suse.de> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="PvrcWq7fipJ6U42OPHChpXMXMCcknCl9P" Subject: Re: [Qemu-devel] [PATCH v3 4/5] migration: Append JSON description of migration stream List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexander Graf , qemu-devel@nongnu.org Cc: amit.shah@redhat.com, pbonzini@redhat.com, afaerber@suse.de, quintela@redhat.com This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --PvrcWq7fipJ6U42OPHChpXMXMCcknCl9P Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 12/26/2014 07:42 AM, Alexander Graf wrote: > One of the annoyances of the current migration format is the fact that > it's not self-describing. In fact, it's not properly describing at all.= > Some code randomly scattered throughout QEMU elaborates roughly how to > read and write a stream of bytes. >=20 > We discussed an idea during KVM Forum 2013 to add a JSON description of= > the migration protocol itself to the migration stream. This patch > adds a section after the VM_END migration end marker that contains > description data on what the device sections of the stream are composed= of. >=20 > With an additional external program this allows us to decipher the > contents of any migration stream and hopefully make migration bugs easi= er > to track down. Hmm. The new format IS self-describing, but ONLY because JSON is technically possible to parse in reverse. That is, you cannot reliably look for end-of-stream marker from the beginning of the stream and reliably get to the JSON description at the end every single time (because if you don't already know how to interpret the stream, how can you find end-of-stream), but you CAN reliably look to see if the stream ends in a JSON object, and then scan backwards for the matching { that opens the object. I'm still wondering if you ought to throw in any additional tricks to make finding the start of the JSON object much easier, such as a subsection near the beginning of the migration stream, and/or a final offset description at the tail of the file that says where the JSON object starts (no additional help for a pipeline, which still has to store things in memory to loc. Even doing something like: stream EOS [ { description... }, offset-of-description ] and using a JSON array rather than a JSON object as the description would make it much faster to find the start of the JSON object without having to scan backwards for matching { (although sticking the offset at the tail doesn't help the issue that when receiving the migration stream over a pipeline, you still have to reserve memory for the entire stream in order to find that offset). >=20 > Signed-off-by: Alexander Graf >=20 > --- >=20 > v2 -> v3: >=20 > - Dont compress objects with subsections > - Destroy QJSON object > --- > hw/pci/pci.c | 2 +- > hw/scsi/spapr_vscsi.c | 2 +- > hw/virtio/virtio.c | 2 +- > include/migration/migration.h | 1 + > include/migration/vmstate.h | 3 +- > migration/vmstate.c | 186 ++++++++++++++++++++++++++++++++++= ++++++-- > savevm.c | 54 ++++++++++-- > tests/test-vmstate.c | 6 +- > 8 files changed, 237 insertions(+), 19 deletions(-) >=20 At this point, I haven't actually reviewed the patch (I'm more interested in seeing the JSON it generates), but want to make sure we're getting an optimal design. --=20 Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --PvrcWq7fipJ6U42OPHChpXMXMCcknCl9P Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCAAGBQJUrAWXAAoJEKeha0olJ0NqFuEH/ijZUjblMR6hArD5KFY2Aafp mf+H/TYq4LMrGTbuAaUZteQBU0KzGWBFUUhs587OVEkR6gdxYnU6hAdbpqaVLzWg g8/sR4oSV7/8TOy2lNxIboCs8dK5FtdBfzfmMyHeu+3H56e4PaYUmrE949rg7YSf cB2Tiu/jCi/6MJenqC+Me59tGTjp8m+uA0l3ihSatJl57ewLxHcGYxToHJ+eVsUa tlzRyTNLNIM4FTSA1oyfVCcaE57K0hhbqjp8d8fQoDLFhUYOaEg5SxK5mQ7MMSP2 FrotuU9O6VQr7Dnul9DBbRmaWpanSYBJZ5F3jbL+aZQh5oJcBq4E7TmijvlvgM0= =C0Sz -----END PGP SIGNATURE----- --PvrcWq7fipJ6U42OPHChpXMXMCcknCl9P--