* [Qemu-devel] [PATCH v3 1/5] QJSON: Add JSON writer
2014-12-26 14:42 [Qemu-devel] [PATCH v3 0/5] Migration Deciphering aid Alexander Graf
@ 2014-12-26 14:42 ` Alexander Graf
2015-01-06 15:41 ` Eric Blake
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 2/5] " Alexander Graf
` (4 subsequent siblings)
5 siblings, 1 reply; 19+ messages in thread
From: Alexander Graf @ 2014-12-26 14:42 UTC (permalink / raw)
To: qemu-devel; +Cc: amit.shah, pbonzini, afaerber, quintela
To support programmatic JSON assembly while keeping the code that generates it
readable, this patch introduces a simple JSON writer. It emits JSON serially
into a buffer in memory.
The nice thing about this writer is its simplicity and low memory overhead.
Unlike the QMP JSON writer, this one does not need to spawn QObjects for every
element it wants to represent.
This is a prerequisite for the migration stream format description generator.
Signed-off-by: Alexander Graf <agraf@suse.de>
---
Makefile.objs | 1 +
include/qjson.h | 28 +++++++++++++++++
qjson.c | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 126 insertions(+)
create mode 100644 include/qjson.h
create mode 100644 qjson.c
diff --git a/Makefile.objs b/Makefile.objs
index abeb902..28999d3 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -51,6 +51,7 @@ common-obj-$(CONFIG_LINUX) += fsdev/
common-obj-y += migration/
common-obj-y += qemu-char.o #aio.o
common-obj-y += page_cache.o
+common-obj-y += qjson.o
common-obj-$(CONFIG_SPICE) += spice-qemu-char.o
diff --git a/include/qjson.h b/include/qjson.h
new file mode 100644
index 0000000..8f8c145
--- /dev/null
+++ b/include/qjson.h
@@ -0,0 +1,28 @@
+/*
+ * QEMU JSON writer
+ *
+ * Copyright Alexander Graf
+ *
+ * Authors:
+ * Alexander Graf <agraf@suse.de
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+#ifndef QEMU_QJSON_H
+#define QEMU_QJSON_H
+
+typedef struct QJSON QJSON;
+
+QJSON *qjson_new(void);
+void json_prop_str(QJSON *json, const char *name, const char *str);
+void json_prop_int(QJSON *json, const char *name, int64_t val);
+void json_end_array(QJSON *json);
+void json_start_array(QJSON *json, const char *name);
+void json_end_object(QJSON *json);
+void json_start_object(QJSON *json, const char *name);
+const char *qjson_get_str(QJSON *json);
+void qjson_finish(QJSON *json);
+
+#endif /* QEMU_QJSON_H */
diff --git a/qjson.c b/qjson.c
new file mode 100644
index 0000000..7a7cd72
--- /dev/null
+++ b/qjson.c
@@ -0,0 +1,97 @@
+/*
+ * QEMU JSON writer
+ *
+ * Copyright Alexander Graf
+ *
+ * Authors:
+ * Alexander Graf <agraf@suse.de
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#include <qapi/qmp/qstring.h>
+#include <stdbool.h>
+#include <glib.h>
+#include <qjson.h>
+
+struct QJSON {
+ QString *str;
+ bool omit_comma;
+ unsigned long self_size_offset;
+};
+
+static void json_emit_element(QJSON *json, const char *name)
+{
+ /* Check whether we need to print a , before an element */
+ if (json->omit_comma) {
+ json->omit_comma = false;
+ } else {
+ qstring_append(json->str, ", ");
+ }
+
+ if (name) {
+ qstring_append(json->str, "\"");
+ qstring_append(json->str, name);
+ qstring_append(json->str, "\" : ");
+ }
+}
+
+void json_start_object(QJSON *json, const char *name)
+{
+ json_emit_element(json, name);
+ qstring_append(json->str, "{ ");
+ json->omit_comma = true;
+}
+
+void json_end_object(QJSON *json)
+{
+ qstring_append(json->str, " }");
+ json->omit_comma = false;
+}
+
+void json_start_array(QJSON *json, const char *name)
+{
+ json_emit_element(json, name);
+ qstring_append(json->str, "[ ");
+ json->omit_comma = true;
+}
+
+void json_end_array(QJSON *json)
+{
+ qstring_append(json->str, " ]");
+ json->omit_comma = false;
+}
+
+void json_prop_int(QJSON *json, const char *name, int64_t val)
+{
+ json_emit_element(json, name);
+ qstring_append_int(json->str, val);
+}
+
+void json_prop_str(QJSON *json, const char *name, const char *str)
+{
+ json_emit_element(json, name);
+ qstring_append_chr(json->str, '"');
+ qstring_append(json->str, str);
+ qstring_append_chr(json->str, '"');
+}
+
+const char *qjson_get_str(QJSON *json)
+{
+ return qstring_get_str(json->str);
+}
+
+QJSON *qjson_new(void)
+{
+ QJSON *json = g_new(QJSON, 1);
+ json->str = qstring_from_str("{ ");
+ json->omit_comma = true;
+ return json;
+}
+
+void qjson_finish(QJSON *json)
+{
+ json_end_object(json);
+}
--
1.7.12.4
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [Qemu-devel] [PATCH v3 1/5] QJSON: Add JSON writer
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 1/5] QJSON: Add JSON writer Alexander Graf
@ 2015-01-06 15:41 ` Eric Blake
2015-01-06 21:39 ` Alexander Graf
0 siblings, 1 reply; 19+ messages in thread
From: Eric Blake @ 2015-01-06 15:41 UTC (permalink / raw)
To: Alexander Graf, qemu-devel; +Cc: amit.shah, pbonzini, afaerber, quintela
[-- Attachment #1: Type: text/plain, Size: 1506 bytes --]
On 12/26/2014 07:42 AM, Alexander Graf wrote:
> To support programmatic JSON assembly while keeping the code that generates it
> readable, this patch introduces a simple JSON writer. It emits JSON serially
> into a buffer in memory.
>
> The nice thing about this writer is its simplicity and low memory overhead.
> Unlike the QMP JSON writer, this one does not need to spawn QObjects for every
> element it wants to represent.
>
> This is a prerequisite for the migration stream format description generator.
>
> Signed-off-by: Alexander Graf <agraf@suse.de>
> ---
> Makefile.objs | 1 +
> include/qjson.h | 28 +++++++++++++++++
> qjson.c | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 126 insertions(+)
> create mode 100644 include/qjson.h
> create mode 100644 qjson.c
> +struct QJSON {
> + QString *str;
> + bool omit_comma;
> + unsigned long self_size_offset;
Would size_t be smarter for this field?
> +}
> +
> +const char *qjson_get_str(QJSON *json)
> +{
> + return qstring_get_str(json->str);
> +}
> +
> +QJSON *qjson_new(void)
> +{
> + QJSON *json = g_new(QJSON, 1);
> + json->str = qstring_from_str("{ ");
> + json->omit_comma = true;
> + return json;
If I'm not mistaken, this creates an open-ended object, as in "{ ...".
Should qjson_get_str add the closing }?
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [Qemu-devel] [PATCH v3 1/5] QJSON: Add JSON writer
2015-01-06 15:41 ` Eric Blake
@ 2015-01-06 21:39 ` Alexander Graf
0 siblings, 0 replies; 19+ messages in thread
From: Alexander Graf @ 2015-01-06 21:39 UTC (permalink / raw)
To: Eric Blake, qemu-devel; +Cc: amit.shah, pbonzini, afaerber, quintela
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 06.01.15 16:41, Eric Blake wrote:
> On 12/26/2014 07:42 AM, Alexander Graf wrote:
>> To support programmatic JSON assembly while keeping the code that
>> generates it readable, this patch introduces a simple JSON
>> writer. It emits JSON serially into a buffer in memory.
>>
>> The nice thing about this writer is its simplicity and low memory
>> overhead. Unlike the QMP JSON writer, this one does not need to
>> spawn QObjects for every element it wants to represent.
>>
>> This is a prerequisite for the migration stream format
>> description generator.
>>
>> Signed-off-by: Alexander Graf <agraf@suse.de> --- Makefile.objs
>> | 1 + include/qjson.h | 28 +++++++++++++++++ qjson.c |
>> 97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3
>> files changed, 126 insertions(+) create mode 100644
>> include/qjson.h create mode 100644 qjson.c
>
>> +struct QJSON { + QString *str; + bool omit_comma; +
>> unsigned long self_size_offset;
>
> Would size_t be smarter for this field?
Turns out the smartest thing to do is to remove the field - it was
unused :).
Btw, thanks a lot for the review!
Alex
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org
iQIcBAEBAgAGBQJUrFYOAAoJECszeR4D/txgCFgP/0r29Xbr/5Ly0MJsP2TN9VcK
NKSL31izigBmQ60Z6cf8sTzqPwZ0mgrFEUBelqz102dWu2DXe+DHa0Uel15/Gxd2
ZR8RVtY86mIvSfdARGShF+pzcUdzMUKlliYbagQglb9zHnnRkYR0VtXy6G1CQczq
lzQDZT0OW9m75+/PPC2V1/g8aWYZbgODI0w+CKEFQM8znhq3qVqiFcRx3HuLHwch
8FiuLYen4XzGjOZDj2oLyU8Aescwkdot4AkfAgf63TJmN2zyhXZQy6BFuJOfoYJK
5no4k7xNvjHX83fx9cT0RdQ6fZzfF/NcRKNlMdJmoi+0dDe87mzahR4Yufs0X+ir
xPIRKXuQWSSJgroboHVQOAlg0IYNdYn5Km3JK8Jx9yBnl6Me0Ow5P6oI0ZAeorWF
vzf395oFq6m29uaMUtmyMfk8dL023xJYIbYr2euoP0gipVvWAKKnBpWc4nWVrq3y
6SnlMbzyA5mb4I0iHGyaQp4WGvFFA/uwW8k+Xz5KpMJgQQoIrtvDaUw0E3f+YHGC
x2/a3FLmxhDAYIBfKlSBWpejG5OuIT6Af5t5xvDO45B+izGVm3WPctEYVkkYTSeq
jaqDpkdBZY8HxG+oYnJzH8wxPSELLM61PqYZyrDTMPdzepthepYgXCSWTFmBMZD7
OhJSkVksn4Khk0x7l1gy
=5f/z
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 19+ messages in thread
* [Qemu-devel] [PATCH v3 2/5] QJSON: Add JSON writer
2014-12-26 14:42 [Qemu-devel] [PATCH v3 0/5] Migration Deciphering aid Alexander Graf
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 1/5] QJSON: Add JSON writer Alexander Graf
@ 2014-12-26 14:42 ` Alexander Graf
2015-01-06 15:44 ` Eric Blake
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 3/5] qemu-file: Add fast ftell code path Alexander Graf
` (3 subsequent siblings)
5 siblings, 1 reply; 19+ messages in thread
From: Alexander Graf @ 2014-12-26 14:42 UTC (permalink / raw)
To: qemu-devel; +Cc: amit.shah, pbonzini, afaerber, quintela
To support programmatic JSON assembly while keeping the code that generates it
readable, this patch introduces a simple JSON writer. It emits JSON serially
into a buffer in memory.
The nice thing about this writer is its simplicity and low memory overhead.
Unlike the QMP JSON writer, this one does not need to spawn QObjects for every
element it wants to represent.
This is a prerequisite for the migration stream format description generator.
Signed-off-by: Alexander Graf <agraf@suse.de>
---
v2 -> v3:
- QOMify the QJSON object, makes for easier destruction
---
include/qjson.h | 3 ++-
qjson.c | 39 ++++++++++++++++++++++++++++++++++++---
2 files changed, 38 insertions(+), 4 deletions(-)
diff --git a/include/qjson.h b/include/qjson.h
index 8f8c145..7c54fdf 100644
--- a/include/qjson.h
+++ b/include/qjson.h
@@ -4,7 +4,7 @@
* Copyright Alexander Graf
*
* Authors:
- * Alexander Graf <agraf@suse.de
+ * Alexander Graf <agraf@suse.de>
*
* This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
* See the COPYING.LIB file in the top-level directory.
@@ -13,6 +13,7 @@
#ifndef QEMU_QJSON_H
#define QEMU_QJSON_H
+#define TYPE_QJSON "QJSON"
typedef struct QJSON QJSON;
QJSON *qjson_new(void);
diff --git a/qjson.c b/qjson.c
index 7a7cd72..8d911d0 100644
--- a/qjson.c
+++ b/qjson.c
@@ -15,8 +15,11 @@
#include <stdbool.h>
#include <glib.h>
#include <qjson.h>
+#include <qemu/module.h>
+#include <qom/object.h>
struct QJSON {
+ Object obj;
QString *str;
bool omit_comma;
unsigned long self_size_offset;
@@ -85,9 +88,7 @@ const char *qjson_get_str(QJSON *json)
QJSON *qjson_new(void)
{
- QJSON *json = g_new(QJSON, 1);
- json->str = qstring_from_str("{ ");
- json->omit_comma = true;
+ QJSON *json = (QJSON *)object_new(TYPE_QJSON);
return json;
}
@@ -95,3 +96,35 @@ void qjson_finish(QJSON *json)
{
json_end_object(json);
}
+
+static void qjson_initfn(Object *obj)
+{
+ QJSON *json = (QJSON *)object_dynamic_cast(obj, TYPE_QJSON);
+ assert(json);
+
+ json->str = qstring_from_str("{ ");
+ json->omit_comma = true;
+}
+
+static void qjson_finalizefn(Object *obj)
+{
+ QJSON *json = (QJSON *)object_dynamic_cast(obj, TYPE_QJSON);
+
+ assert(json);
+ qobject_decref(QOBJECT(json->str));
+}
+
+static const TypeInfo qjson_type_info = {
+ .name = TYPE_QJSON,
+ .parent = TYPE_OBJECT,
+ .instance_size = sizeof(QJSON),
+ .instance_init = qjson_initfn,
+ .instance_finalize = qjson_finalizefn,
+};
+
+static void qjson_register_types(void)
+{
+ type_register_static(&qjson_type_info);
+}
+
+type_init(qjson_register_types)
--
1.7.12.4
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [Qemu-devel] [PATCH v3 2/5] QJSON: Add JSON writer
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 2/5] " Alexander Graf
@ 2015-01-06 15:44 ` Eric Blake
2015-01-06 21:16 ` Alexander Graf
0 siblings, 1 reply; 19+ messages in thread
From: Eric Blake @ 2015-01-06 15:44 UTC (permalink / raw)
To: Alexander Graf, qemu-devel; +Cc: amit.shah, pbonzini, afaerber, quintela
[-- Attachment #1: Type: text/plain, Size: 1973 bytes --]
On 12/26/2014 07:42 AM, Alexander Graf wrote:
> To support programmatic JSON assembly while keeping the code that generates it
> readable, this patch introduces a simple JSON writer. It emits JSON serially
> into a buffer in memory.
>
> The nice thing about this writer is its simplicity and low memory overhead.
> Unlike the QMP JSON writer, this one does not need to spawn QObjects for every
> element it wants to represent.
>
> This is a prerequisite for the migration stream format description generator.
>
> Signed-off-by: Alexander Graf <agraf@suse.de>
>
> ---
>
> v2 -> v3:
>
> - QOMify the QJSON object, makes for easier destruction
> ---
> include/qjson.h | 3 ++-
> qjson.c | 39 ++++++++++++++++++++++++++++++++++++---
> 2 files changed, 38 insertions(+), 4 deletions(-)
>
> diff --git a/include/qjson.h b/include/qjson.h
> index 8f8c145..7c54fdf 100644
> --- a/include/qjson.h
> +++ b/include/qjson.h
> @@ -4,7 +4,7 @@
> * Copyright Alexander Graf
> *
> * Authors:
> - * Alexander Graf <agraf@suse.de
> + * Alexander Graf <agraf@suse.de>
Umm, this should be squashed into 1/5.
> @@ -85,9 +88,7 @@ const char *qjson_get_str(QJSON *json)
>
> QJSON *qjson_new(void)
> {
> - QJSON *json = g_new(QJSON, 1);
> - json->str = qstring_from_str("{ ");
> - json->omit_comma = true;
> + QJSON *json = (QJSON *)object_new(TYPE_QJSON);
> return json;
This undoes the dangling object that I complained about on patch 1;
maybe there's some more squashing to do?...
> +static void qjson_initfn(Object *obj)
> +{
> + QJSON *json = (QJSON *)object_dynamic_cast(obj, TYPE_QJSON);
> + assert(json);
> +
> + json->str = qstring_from_str("{ ");
> + json->omit_comma = true;
...or is it still an incomplete object, just now in a different location?
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [Qemu-devel] [PATCH v3 2/5] QJSON: Add JSON writer
2015-01-06 15:44 ` Eric Blake
@ 2015-01-06 21:16 ` Alexander Graf
0 siblings, 0 replies; 19+ messages in thread
From: Alexander Graf @ 2015-01-06 21:16 UTC (permalink / raw)
To: Eric Blake, qemu-devel; +Cc: amit.shah, pbonzini, afaerber, quintela
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 06.01.15 16:44, Eric Blake wrote:
> On 12/26/2014 07:42 AM, Alexander Graf wrote:
>> To support programmatic JSON assembly while keeping the code that
>> generates it readable, this patch introduces a simple JSON
>> writer. It emits JSON serially into a buffer in memory.
>>
>> The nice thing about this writer is its simplicity and low memory
>> overhead. Unlike the QMP JSON writer, this one does not need to
>> spawn QObjects for every element it wants to represent.
>>
>> This is a prerequisite for the migration stream format
>> description generator.
>>
>> Signed-off-by: Alexander Graf <agraf@suse.de>
>>
>> ---
>>
>> v2 -> v3:
>>
>> - QOMify the QJSON object, makes for easier destruction ---
>> include/qjson.h | 3 ++- qjson.c | 39
>> ++++++++++++++++++++++++++++++++++++--- 2 files changed, 38
>> insertions(+), 4 deletions(-)
>>
>> diff --git a/include/qjson.h b/include/qjson.h index
>> 8f8c145..7c54fdf 100644 --- a/include/qjson.h +++
>> b/include/qjson.h @@ -4,7 +4,7 @@ * Copyright Alexander Graf * *
>> Authors: - * Alexander Graf <agraf@suse.de + * Alexander Graf
>> <agraf@suse.de>
>
> Umm, this should be squashed into 1/5.
Honestly, I think the whole patch should've gotten squashed into 1/5.
I'm not quite sure what went wrong here :).
>
>> @@ -85,9 +88,7 @@ const char *qjson_get_str(QJSON *json)
>>
>> QJSON *qjson_new(void) { - QJSON *json = g_new(QJSON, 1); -
>> json->str = qstring_from_str("{ "); - json->omit_comma =
>> true; + QJSON *json = (QJSON *)object_new(TYPE_QJSON); return
>> json;
>
> This undoes the dangling object that I complained about on patch
> 1; maybe there's some more squashing to do?...
>
>
>> +static void qjson_initfn(Object *obj) +{ + QJSON *json =
>> (QJSON *)object_dynamic_cast(obj, TYPE_QJSON); +
>> assert(json); + + json->str = qstring_from_str("{ "); +
>> json->omit_comma = true;
>
> ...or is it still an incomplete object, just now in a different
> location?
The idea is that you call finish and only then receive the string. I
don't think doing that from get_str would be too much of an
improvement - it'd still be wrong if we're in the middle of assembling
an object.
Alex
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org
iQIcBAEBAgAGBQJUrFC2AAoJECszeR4D/txgFUoP/A79amlLyEBoD7/G1jHttwS2
sYKst8ZM+v//dB5Xe+DzU53XgflAkpUoz27zSveluD1A3YQ3zpFYraujaewSn++m
sJZS/7bS9WND0vhKdGj41iJXdjxSlHqcdrsKRqYR4MaaTjH8eVoabECS+CcdK0xc
R+x08gSUI22+tCKPnudRMm3mlVqJ5O+1KbTYShQQlP5go1p6UN32LONgZ4u9oJPL
uQ1vkkrHleHa+RoW4HzRCSO+LQuY/y8CMV1ufS50pILSoX5C/jDdOzqvpc3fR9Ba
X1dWVATsF8gk+Hu4sNtWAqiPSsSGg/V0CnBASRYwt/skoNtXoZ++/UWr8B+F/IKc
h2UCdO64TlbbDtGYGn/4ZQuUIsGLPvm0yN8J1LshVt85wWJaNe2dw8hirxjzCFoB
A6YnueXbaxkHlQkVXL0u2E/EUnmYlRl8JauFBxVDuw6MQecQtc7MIFS0zr8XyKDt
IxspJpOKGXAuU7oK+1kmy2aOlMJCVwgdthCv9qDaQl39pTpyF38Jf8kOAFWSkz+C
bQqq4EZrI4vMlHcwY4a06YnFH7Ek8Q7YZdFj/W2CIJKsIMh0k79mFjm4r54JrIUF
UTXoR/DuZd/zRGLBnj8PoBsfiXk3ZGJCtAyrlns2T32sEbiJmL3lqggiyKzh3+Dd
S9NeyhQvJ/xvYPQ89N7p
=Zod1
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 19+ messages in thread
* [Qemu-devel] [PATCH v3 3/5] qemu-file: Add fast ftell code path
2014-12-26 14:42 [Qemu-devel] [PATCH v3 0/5] Migration Deciphering aid Alexander Graf
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 1/5] QJSON: Add JSON writer Alexander Graf
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 2/5] " Alexander Graf
@ 2014-12-26 14:42 ` Alexander Graf
2015-01-06 15:46 ` Eric Blake
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 4/5] migration: Append JSON description of migration stream Alexander Graf
` (2 subsequent siblings)
5 siblings, 1 reply; 19+ messages in thread
From: Alexander Graf @ 2014-12-26 14:42 UTC (permalink / raw)
To: qemu-devel; +Cc: amit.shah, pbonzini, afaerber, quintela
For ftell we flush the output buffer to ensure that we don't have anything
lingering in our internal buffers. This is a very safe thing to do.
However, with the dynamic size measurement that the dynamic vmstate
description will bring this would turn out quite slow.
Instead, we can fast path this specific measurement and just take the
internal buffers into account when telling the kernel our position.
I'm sure I overlooked some corner cases where this doesn't work, so
instead of tuning the safe, existing version, this patch adds a fast
variant of ftell that gets used by the dynamic vmstate description code
which isn't critical when it fails.
Signed-off-by: Alexander Graf <agraf@suse.de>
---
v2 -> v3:
- improve ftell_fast, now works with bdrv too
---
include/migration/qemu-file.h | 1 +
migration/qemu-file.c | 16 ++++++++++++++++
2 files changed, 17 insertions(+)
diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
index 401676b..6b6772b 100644
--- a/include/migration/qemu-file.h
+++ b/include/migration/qemu-file.h
@@ -112,6 +112,7 @@ QEMUFile *qemu_bufopen(const char *mode, QEMUSizedBuffer *input);
int qemu_get_fd(QEMUFile *f);
int qemu_fclose(QEMUFile *f);
int64_t qemu_ftell(QEMUFile *f);
+int64_t qemu_ftell_fast(QEMUFile *f);
void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, int size);
void qemu_put_byte(QEMUFile *f, int v);
/*
diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index d2d4007..cdfec56 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -440,6 +440,22 @@ int qemu_get_byte(QEMUFile *f)
return result;
}
+int64_t qemu_ftell_fast(QEMUFile *f)
+{
+ int64_t ret = f->pos;
+ int i;
+
+ if (f->ops->writev_buffer) {
+ for (i = 0; i < f->iovcnt; i++) {
+ ret += f->iov[i].iov_len;
+ }
+ } else {
+ ret += f->buf_index;
+ }
+
+ return ret;
+}
+
int64_t qemu_ftell(QEMUFile *f)
{
qemu_fflush(f);
--
1.7.12.4
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [Qemu-devel] [PATCH v3 3/5] qemu-file: Add fast ftell code path
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 3/5] qemu-file: Add fast ftell code path Alexander Graf
@ 2015-01-06 15:46 ` Eric Blake
0 siblings, 0 replies; 19+ messages in thread
From: Eric Blake @ 2015-01-06 15:46 UTC (permalink / raw)
To: Alexander Graf, qemu-devel; +Cc: amit.shah, pbonzini, afaerber, quintela
[-- Attachment #1: Type: text/plain, Size: 1174 bytes --]
On 12/26/2014 07:42 AM, Alexander Graf wrote:
> For ftell we flush the output buffer to ensure that we don't have anything
> lingering in our internal buffers. This is a very safe thing to do.
>
> However, with the dynamic size measurement that the dynamic vmstate
> description will bring this would turn out quite slow.
>
> Instead, we can fast path this specific measurement and just take the
> internal buffers into account when telling the kernel our position.
>
> I'm sure I overlooked some corner cases where this doesn't work, so
> instead of tuning the safe, existing version, this patch adds a fast
> variant of ftell that gets used by the dynamic vmstate description code
> which isn't critical when it fails.
>
> Signed-off-by: Alexander Graf <agraf@suse.de>
>
> ---
>
> v2 -> v3:
>
> - improve ftell_fast, now works with bdrv too
> ---
> include/migration/qemu-file.h | 1 +
> migration/qemu-file.c | 16 ++++++++++++++++
> 2 files changed, 17 insertions(+)
Reviewed-by: Eric Blake <eblake@redhat.com>
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* [Qemu-devel] [PATCH v3 4/5] migration: Append JSON description of migration stream
2014-12-26 14:42 [Qemu-devel] [PATCH v3 0/5] Migration Deciphering aid Alexander Graf
` (2 preceding siblings ...)
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 3/5] qemu-file: Add fast ftell code path Alexander Graf
@ 2014-12-26 14:42 ` Alexander Graf
2015-01-06 15:56 ` Eric Blake
2015-01-20 10:30 ` Amit Shah
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 5/5] Add migration stream analyzation script Alexander Graf
2015-01-20 10:31 ` [Qemu-devel] [PATCH v3 0/5] Migration Deciphering aid Amit Shah
5 siblings, 2 replies; 19+ messages in thread
From: Alexander Graf @ 2014-12-26 14:42 UTC (permalink / raw)
To: qemu-devel; +Cc: amit.shah, pbonzini, afaerber, quintela
One of the annoyances of the current migration format is the fact that
it's not self-describing. In fact, it's not properly describing at all.
Some code randomly scattered throughout QEMU elaborates roughly how to
read and write a stream of bytes.
We discussed an idea during KVM Forum 2013 to add a JSON description of
the migration protocol itself to the migration stream. This patch
adds a section after the VM_END migration end marker that contains
description data on what the device sections of the stream are composed of.
With an additional external program this allows us to decipher the
contents of any migration stream and hopefully make migration bugs easier
to track down.
Signed-off-by: Alexander Graf <agraf@suse.de>
---
v2 -> v3:
- Dont compress objects with subsections
- Destroy QJSON object
---
hw/pci/pci.c | 2 +-
hw/scsi/spapr_vscsi.c | 2 +-
hw/virtio/virtio.c | 2 +-
include/migration/migration.h | 1 +
include/migration/vmstate.h | 3 +-
migration/vmstate.c | 186 ++++++++++++++++++++++++++++++++++++++++--
savevm.c | 54 ++++++++++--
tests/test-vmstate.c | 6 +-
8 files changed, 237 insertions(+), 19 deletions(-)
diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index 371699c..38d0b7b 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -512,7 +512,7 @@ void pci_device_save(PCIDevice *s, QEMUFile *f)
* This makes us compatible with old devices
* which never set or clear this bit. */
s->config[PCI_STATUS] &= ~PCI_STATUS_INTERRUPT;
- vmstate_save_state(f, pci_get_vmstate(s), s);
+ vmstate_save_state(f, pci_get_vmstate(s), s, NULL);
/* Restore the interrupt status bit. */
pci_update_irq_status(s);
}
diff --git a/hw/scsi/spapr_vscsi.c b/hw/scsi/spapr_vscsi.c
index 20b20f0..3639235 100644
--- a/hw/scsi/spapr_vscsi.c
+++ b/hw/scsi/spapr_vscsi.c
@@ -630,7 +630,7 @@ static void vscsi_save_request(QEMUFile *f, SCSIRequest *sreq)
vscsi_req *req = sreq->hba_private;
assert(req->active);
- vmstate_save_state(f, &vmstate_spapr_vscsi_req, req);
+ vmstate_save_state(f, &vmstate_spapr_vscsi_req, req, NULL);
DPRINTF("VSCSI: saving tag=%u, current desc#%d, offset=%x\n",
req->qtag, req->cur_desc_num, req->cur_desc_offset);
diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index 013979a..d735343 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -955,7 +955,7 @@ void virtio_save(VirtIODevice *vdev, QEMUFile *f)
}
/* Subsections */
- vmstate_save_state(f, &vmstate_virtio, vdev);
+ vmstate_save_state(f, &vmstate_virtio, vdev, NULL);
}
int virtio_set_features(VirtIODevice *vdev, uint32_t val)
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 3cb5ba8..f37348b 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -33,6 +33,7 @@
#define QEMU_VM_SECTION_END 0x03
#define QEMU_VM_SECTION_FULL 0x04
#define QEMU_VM_SUBSECTION 0x05
+#define QEMU_VM_VMDESCRIPTION 0x06
struct MigrationParams {
bool blk;
diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index e45fc49..1ed5311 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -29,6 +29,7 @@
#ifndef CONFIG_USER_ONLY
#include <migration/qemu-file.h>
#endif
+#include <qjson.h>
typedef void SaveStateHandler(QEMUFile *f, void *opaque);
typedef int LoadStateHandler(QEMUFile *f, void *opaque, int version_id);
@@ -779,7 +780,7 @@ extern const VMStateInfo vmstate_info_bitmap;
int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
void *opaque, int version_id);
void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
- void *opaque);
+ void *opaque, QJSON *vmdesc);
int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
const VMStateDescription *vmsd,
diff --git a/migration/vmstate.c b/migration/vmstate.c
index 3dde574..2a91f1f 100644
--- a/migration/vmstate.c
+++ b/migration/vmstate.c
@@ -4,9 +4,10 @@
#include "migration/vmstate.h"
#include "qemu/bitops.h"
#include "trace.h"
+#include "qjson.h"
static void vmstate_subsection_save(QEMUFile *f, const VMStateDescription *vmsd,
- void *opaque);
+ void *opaque, QJSON *vmdesc);
static int vmstate_subsection_load(QEMUFile *f, const VMStateDescription *vmsd,
void *opaque);
@@ -138,32 +139,181 @@ int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
return 0;
}
+static int vmfield_name_num(VMStateField *start, VMStateField *search)
+{
+ VMStateField *field;
+ int found = 0;
+
+ for (field = start; field->name; field++) {
+ if (!strcmp(field->name, search->name)) {
+ if (field == search) {
+ return found;
+ }
+ found++;
+ }
+ }
+
+ return -1;
+}
+
+static bool vmfield_name_is_unique(VMStateField *start, VMStateField *search)
+{
+ VMStateField *field;
+ int found = 0;
+
+ for (field = start; field->name; field++) {
+ if (!strcmp(field->name, search->name)) {
+ found++;
+ /* name found more than once, so it's not unique */
+ if (found > 1) {
+ return false;
+ }
+ }
+ }
+
+ return true;
+}
+
+static const char *vmfield_get_type_name(VMStateField *field)
+{
+ const char *type = "unknown";
+
+ if (field->flags & VMS_STRUCT) {
+ type = "struct";
+ } else if (field->info->name) {
+ type = field->info->name;
+ }
+
+ return type;
+}
+
+static bool vmsd_can_compress(VMStateField *field)
+{
+ if (field->field_exists) {
+ /* Dynamically existing fields mess up compression */
+ return false;
+ }
+
+ if (field->flags & VMS_STRUCT) {
+ VMStateField *sfield = field->vmsd->fields;
+ while (sfield->name) {
+ if (!vmsd_can_compress(sfield)) {
+ /* Child elements can't compress, so can't we */
+ return false;
+ }
+ sfield++;
+ }
+
+ if (field->vmsd->subsections) {
+ /* Subsections may come and go, better don't compress */
+ return false;
+ }
+ }
+
+ return true;
+}
+
+static void vmsd_desc_field_start(const VMStateDescription *vmsd, QJSON *vmdesc,
+ VMStateField *field, int i, int max)
+{
+ char *name, *old_name;
+ bool is_array = max > 1;
+ bool can_compress = vmsd_can_compress(field);
+
+ if (!vmdesc) {
+ return;
+ }
+
+ name = g_strdup(field->name);
+
+ /* Field name is not unique, need to make it unique */
+ if (!vmfield_name_is_unique(vmsd->fields, field)) {
+ int num = vmfield_name_num(vmsd->fields, field);
+ old_name = name;
+ name = g_strdup_printf("%s[%d]", name, num);
+ g_free(old_name);
+ }
+
+ json_start_object(vmdesc, NULL);
+ json_prop_str(vmdesc, "name", name);
+ if (is_array) {
+ if (can_compress) {
+ json_prop_int(vmdesc, "array_len", max);
+ } else {
+ json_prop_int(vmdesc, "index", i);
+ }
+ }
+ json_prop_str(vmdesc, "type", vmfield_get_type_name(field));
+
+ if (field->flags & VMS_STRUCT) {
+ json_start_object(vmdesc, "struct");
+ }
+
+ g_free(name);
+}
+
+static void vmsd_desc_field_end(const VMStateDescription *vmsd, QJSON *vmdesc,
+ VMStateField *field, size_t size, int i)
+{
+ if (!vmdesc) {
+ return;
+ }
+
+ if (field->flags & VMS_STRUCT) {
+ /* We printed a struct in between, close its child object */
+ json_end_object(vmdesc);
+ }
+
+ json_prop_int(vmdesc, "size", size);
+ json_end_object(vmdesc);
+}
+
void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
- void *opaque)
+ void *opaque, QJSON *vmdesc)
{
VMStateField *field = vmsd->fields;
if (vmsd->pre_save) {
vmsd->pre_save(opaque);
}
+
+ if (vmdesc) {
+ json_prop_str(vmdesc, "vmsd_name", vmsd->name);
+ json_prop_int(vmdesc, "version", vmsd->version_id);
+ json_start_array(vmdesc, "fields");
+ }
+
while (field->name) {
if (!field->field_exists ||
field->field_exists(opaque, vmsd->version_id)) {
void *base_addr = vmstate_base_addr(opaque, field, false);
int i, n_elems = vmstate_n_elems(opaque, field);
int size = vmstate_size(opaque, field);
+ int64_t old_offset, written_bytes;
+ QJSON *vmdesc_loop = vmdesc;
for (i = 0; i < n_elems; i++) {
void *addr = base_addr + size * i;
+ vmsd_desc_field_start(vmsd, vmdesc_loop, field, i, n_elems);
+ old_offset = qemu_ftell_fast(f);
+
if (field->flags & VMS_ARRAY_OF_POINTER) {
addr = *(void **)addr;
}
if (field->flags & VMS_STRUCT) {
- vmstate_save_state(f, field->vmsd, addr);
+ vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
} else {
field->info->put(f, addr, size);
}
+
+ written_bytes = qemu_ftell_fast(f) - old_offset;
+ vmsd_desc_field_end(vmsd, vmdesc_loop, field, written_bytes, i);
+
+ /* Compressed arrays only care about the first element */
+ if (vmdesc_loop && vmsd_can_compress(field)) {
+ vmdesc_loop = NULL;
+ }
}
} else {
if (field->flags & VMS_MUST_EXIST) {
@@ -174,7 +324,12 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
}
field++;
}
- vmstate_subsection_save(f, vmsd, opaque);
+
+ if (vmdesc) {
+ json_end_array(vmdesc);
+ }
+
+ vmstate_subsection_save(f, vmsd, opaque, vmdesc);
}
static const VMStateDescription *
@@ -231,24 +386,43 @@ static int vmstate_subsection_load(QEMUFile *f, const VMStateDescription *vmsd,
}
static void vmstate_subsection_save(QEMUFile *f, const VMStateDescription *vmsd,
- void *opaque)
+ void *opaque, QJSON *vmdesc)
{
const VMStateSubsection *sub = vmsd->subsections;
+ bool subsection_found = false;
while (sub && sub->needed) {
if (sub->needed(opaque)) {
const VMStateDescription *vmsd = sub->vmsd;
uint8_t len;
+ if (vmdesc) {
+ /* Only create subsection array when we have any */
+ if (!subsection_found) {
+ json_start_array(vmdesc, "subsections");
+ subsection_found = true;
+ }
+
+ json_start_object(vmdesc, NULL);
+ }
+
qemu_put_byte(f, QEMU_VM_SUBSECTION);
len = strlen(vmsd->name);
qemu_put_byte(f, len);
qemu_put_buffer(f, (uint8_t *)vmsd->name, len);
qemu_put_be32(f, vmsd->version_id);
- vmstate_save_state(f, vmsd, opaque);
+ vmstate_save_state(f, vmsd, opaque, vmdesc);
+
+ if (vmdesc) {
+ json_end_object(vmdesc);
+ }
}
sub++;
}
+
+ if (vmdesc && subsection_found) {
+ json_end_array(vmdesc);
+ }
}
/* bool */
diff --git a/savevm.c b/savevm.c
index 08ec678..48e3e5c 100644
--- a/savevm.c
+++ b/savevm.c
@@ -572,14 +572,34 @@ static int vmstate_load(QEMUFile *f, SaveStateEntry *se, int version_id)
return vmstate_load_state(f, se->vmsd, se->opaque, version_id);
}
-static void vmstate_save(QEMUFile *f, SaveStateEntry *se)
+static void vmstate_save_old_style(QEMUFile *f, SaveStateEntry *se, QJSON *vmdesc)
+{
+ int64_t old_offset, size;
+
+ old_offset = qemu_ftell_fast(f);
+ se->ops->save_state(f, se->opaque);
+ size = qemu_ftell_fast(f) - old_offset;
+
+ if (vmdesc) {
+ json_prop_int(vmdesc, "size", size);
+ json_start_array(vmdesc, "fields");
+ json_start_object(vmdesc, NULL);
+ json_prop_str(vmdesc, "name", "data");
+ json_prop_int(vmdesc, "size", size);
+ json_prop_str(vmdesc, "type", "buffer");
+ json_end_object(vmdesc);
+ json_end_array(vmdesc);
+ }
+}
+
+static void vmstate_save(QEMUFile *f, SaveStateEntry *se, QJSON *vmdesc)
{
trace_vmstate_save(se->idstr, se->vmsd ? se->vmsd->name : "(old)");
- if (!se->vmsd) { /* Old style */
- se->ops->save_state(f, se->opaque);
+ if (!se->vmsd) {
+ vmstate_save_old_style(f, se, vmdesc);
return;
}
- vmstate_save_state(f, se->vmsd, se->opaque);
+ vmstate_save_state(f, se->vmsd, se->opaque, vmdesc);
}
bool qemu_savevm_state_blocked(Error **errp)
@@ -692,6 +712,8 @@ int qemu_savevm_state_iterate(QEMUFile *f)
void qemu_savevm_state_complete(QEMUFile *f)
{
+ QJSON *vmdesc;
+ int vmdesc_len;
SaveStateEntry *se;
int ret;
@@ -721,6 +743,9 @@ void qemu_savevm_state_complete(QEMUFile *f)
}
}
+ vmdesc = qjson_new();
+ json_prop_int(vmdesc, "page_size", TARGET_PAGE_SIZE);
+ json_start_array(vmdesc, "devices");
QTAILQ_FOREACH(se, &savevm_handlers, entry) {
int len;
@@ -728,6 +753,11 @@ void qemu_savevm_state_complete(QEMUFile *f)
continue;
}
trace_savevm_section_start(se->idstr, se->section_id);
+
+ json_start_object(vmdesc, NULL);
+ json_prop_str(vmdesc, "name", se->idstr);
+ json_prop_int(vmdesc, "instance_id", se->instance_id);
+
/* Section type */
qemu_put_byte(f, QEMU_VM_SECTION_FULL);
qemu_put_be32(f, se->section_id);
@@ -740,11 +770,23 @@ void qemu_savevm_state_complete(QEMUFile *f)
qemu_put_be32(f, se->instance_id);
qemu_put_be32(f, se->version_id);
- vmstate_save(f, se);
+ vmstate_save(f, se, vmdesc);
+
+ json_end_object(vmdesc);
trace_savevm_section_end(se->idstr, se->section_id);
}
qemu_put_byte(f, QEMU_VM_EOF);
+
+ json_end_array(vmdesc);
+ qjson_finish(vmdesc);
+ vmdesc_len = strlen(qjson_get_str(vmdesc));
+
+ qemu_put_byte(f, QEMU_VM_VMDESCRIPTION);
+ qemu_put_be32(f, vmdesc_len);
+ qemu_put_buffer(f, (uint8_t *)qjson_get_str(vmdesc), vmdesc_len);
+ object_unref(OBJECT(vmdesc));
+
qemu_fflush(f);
}
@@ -843,7 +885,7 @@ static int qemu_save_device_state(QEMUFile *f)
qemu_put_be32(f, se->instance_id);
qemu_put_be32(f, se->version_id);
- vmstate_save(f, se);
+ vmstate_save(f, se, NULL);
}
qemu_put_byte(f, QEMU_VM_EOF);
diff --git a/tests/test-vmstate.c b/tests/test-vmstate.c
index 5e0fd13..55cb93b 100644
--- a/tests/test-vmstate.c
+++ b/tests/test-vmstate.c
@@ -95,7 +95,7 @@ static void save_vmstate(const VMStateDescription *desc, void *obj)
QEMUFile *f = open_test_file(true);
/* Save file with vmstate */
- vmstate_save_state(f, desc, obj);
+ vmstate_save_state(f, desc, obj, NULL);
qemu_put_byte(f, QEMU_VM_EOF);
g_assert(!qemu_file_get_error(f));
qemu_fclose(f);
@@ -404,7 +404,7 @@ static void test_save_noskip(void)
QEMUFile *fsave = qemu_bufopen("w", NULL);
TestStruct obj = { .a = 1, .b = 2, .c = 3, .d = 4, .e = 5, .f = 6,
.skip_c_e = false };
- vmstate_save_state(fsave, &vmstate_skipping, &obj);
+ vmstate_save_state(fsave, &vmstate_skipping, &obj, NULL);
g_assert(!qemu_file_get_error(fsave));
uint8_t expected[] = {
@@ -424,7 +424,7 @@ static void test_save_skip(void)
QEMUFile *fsave = qemu_bufopen("w", NULL);
TestStruct obj = { .a = 1, .b = 2, .c = 3, .d = 4, .e = 5, .f = 6,
.skip_c_e = true };
- vmstate_save_state(fsave, &vmstate_skipping, &obj);
+ vmstate_save_state(fsave, &vmstate_skipping, &obj, NULL);
g_assert(!qemu_file_get_error(fsave));
uint8_t expected[] = {
--
1.7.12.4
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [Qemu-devel] [PATCH v3 4/5] migration: Append JSON description of migration stream
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 4/5] migration: Append JSON description of migration stream Alexander Graf
@ 2015-01-06 15:56 ` Eric Blake
2015-01-06 21:25 ` Alexander Graf
2015-01-20 10:30 ` Amit Shah
1 sibling, 1 reply; 19+ messages in thread
From: Eric Blake @ 2015-01-06 15:56 UTC (permalink / raw)
To: Alexander Graf, qemu-devel; +Cc: amit.shah, pbonzini, afaerber, quintela
[-- Attachment #1: Type: text/plain, Size: 3052 bytes --]
On 12/26/2014 07:42 AM, Alexander Graf wrote:
> One of the annoyances of the current migration format is the fact that
> it's not self-describing. In fact, it's not properly describing at all.
> Some code randomly scattered throughout QEMU elaborates roughly how to
> read and write a stream of bytes.
>
> We discussed an idea during KVM Forum 2013 to add a JSON description of
> the migration protocol itself to the migration stream. This patch
> adds a section after the VM_END migration end marker that contains
> description data on what the device sections of the stream are composed of.
>
> With an additional external program this allows us to decipher the
> contents of any migration stream and hopefully make migration bugs easier
> to track down.
Hmm. The new format IS self-describing, but ONLY because JSON is
technically possible to parse in reverse. That is, you cannot reliably
look for end-of-stream marker from the beginning of the stream and
reliably get to the JSON description at the end every single time
(because if you don't already know how to interpret the stream, how can
you find end-of-stream), but you CAN reliably look to see if the stream
ends in a JSON object, and then scan backwards for the matching { that
opens the object.
I'm still wondering if you ought to throw in any additional tricks to
make finding the start of the JSON object much easier, such as a
subsection near the beginning of the migration stream, and/or a final
offset description at the tail of the file that says where the JSON
object starts (no additional help for a pipeline, which still has to
store things in memory to loc. Even doing something like:
stream EOS [ { description... }, offset-of-description ]
and using a JSON array rather than a JSON object as the description
would make it much faster to find the start of the JSON object without
having to scan backwards for matching { (although sticking the offset at
the tail doesn't help the issue that when receiving the migration stream
over a pipeline, you still have to reserve memory for the entire stream
in order to find that offset).
>
> Signed-off-by: Alexander Graf <agraf@suse.de>
>
> ---
>
> v2 -> v3:
>
> - Dont compress objects with subsections
> - Destroy QJSON object
> ---
> hw/pci/pci.c | 2 +-
> hw/scsi/spapr_vscsi.c | 2 +-
> hw/virtio/virtio.c | 2 +-
> include/migration/migration.h | 1 +
> include/migration/vmstate.h | 3 +-
> migration/vmstate.c | 186 ++++++++++++++++++++++++++++++++++++++++--
> savevm.c | 54 ++++++++++--
> tests/test-vmstate.c | 6 +-
> 8 files changed, 237 insertions(+), 19 deletions(-)
>
At this point, I haven't actually reviewed the patch (I'm more
interested in seeing the JSON it generates), but want to make sure we're
getting an optimal design.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [Qemu-devel] [PATCH v3 4/5] migration: Append JSON description of migration stream
2015-01-06 15:56 ` Eric Blake
@ 2015-01-06 21:25 ` Alexander Graf
0 siblings, 0 replies; 19+ messages in thread
From: Alexander Graf @ 2015-01-06 21:25 UTC (permalink / raw)
To: Eric Blake, qemu-devel; +Cc: amit.shah, pbonzini, afaerber, quintela
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 06.01.15 16:56, Eric Blake wrote:
> On 12/26/2014 07:42 AM, Alexander Graf wrote:
>> One of the annoyances of the current migration format is the fact
>> that it's not self-describing. In fact, it's not properly
>> describing at all. Some code randomly scattered throughout QEMU
>> elaborates roughly how to read and write a stream of bytes.
>>
>> We discussed an idea during KVM Forum 2013 to add a JSON
>> description of the migration protocol itself to the migration
>> stream. This patch adds a section after the VM_END migration end
>> marker that contains description data on what the device sections
>> of the stream are composed of.
>>
>> With an additional external program this allows us to decipher
>> the contents of any migration stream and hopefully make migration
>> bugs easier to track down.
>
> Hmm. The new format IS self-describing, but ONLY because JSON is
> technically possible to parse in reverse. That is, you cannot
> reliably look for end-of-stream marker from the beginning of the
> stream and reliably get to the JSON description at the end every
> single time (because if you don't already know how to interpret the
> stream, how can you find end-of-stream), but you CAN reliably look
> to see if the stream ends in a JSON object, and then scan backwards
> for the matching { that opens the object.
>
> I'm still wondering if you ought to throw in any additional tricks
> to make finding the start of the JSON object much easier, such as
> a subsection near the beginning of the migration stream, and/or a
> final offset description at the tail of the file that says where
> the JSON object starts (no additional help for a pipeline, which
> still has to store things in memory to loc. Even doing something
> like:
>
> stream EOS [ { description... }, offset-of-description ]
>
> and using a JSON array rather than a JSON object as the
> description would make it much faster to find the start of the JSON
> object without having to scan backwards for matching { (although
> sticking the offset at the tail doesn't help the issue that when
> receiving the migration stream over a pipeline, you still have to
> reserve memory for the entire stream in order to find that
> offset).
Yeah, I've pondered ideas similar to these around myself. The problem
is that I don't think they're much more reliable than what we have
now. You know that our JSON data is always ASCII, so if you simply
scan backwards for the EOF marker, you've basically solved the problem
if you know the end of the file.
If you don't know the end of the file (read: pipe or stream), then
there's really only so much we can do.
The problem with writing a section anywhere before the device sections
which we can't parse otherwise is that at that point in time we don't
know the location of our JSON stream, as the migration stream hasn't
been written yet.
So in a nutshell, we're safe with seekable files I think. If you can
come up with a solution that works nicely for pipes, please let me
know :).
>
>>
>> Signed-off-by: Alexander Graf <agraf@suse.de>
>>
>> ---
>>
>> v2 -> v3:
>>
>> - Dont compress objects with subsections - Destroy QJSON object
>> --- hw/pci/pci.c | 2 +- hw/scsi/spapr_vscsi.c
>> | 2 +- hw/virtio/virtio.c | 2 +-
>> include/migration/migration.h | 1 + include/migration/vmstate.h
>> | 3 +- migration/vmstate.c | 186
>> ++++++++++++++++++++++++++++++++++++++++-- savevm.c
>> | 54 ++++++++++-- tests/test-vmstate.c | 6 +- 8 files
>> changed, 237 insertions(+), 19 deletions(-)
>>
>
> At this point, I haven't actually reviewed the patch (I'm more
> interested in seeing the JSON it generates), but want to make sure
> we're getting an optimal design.
Sure, the python tool in 5/5 can dump the JSON as well :). Just run it
on any guest's migration file you like.
Alex
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org
iQIcBAEBAgAGBQJUrFKwAAoJECszeR4D/txgiyEP/3Es3GwAivBoHKgkqnW1TVuf
9nk1+yPHEP7jwQc3z8hpbLZR9lk1H0Pin1rHW67ICvv1+iD/0ShavYml+/kCumtz
/lroJGjZ5nb80mdTsnvWIXqra3s6SYF3F5wxuCHb8QInbu2kEi5rx1MT+np2cIGh
BtfuoLf2dGhb4Rkfmtxsjn8Bo8MFfoyXjNZt7ugFqvhXNP05y4jQg1FmkFnOT9l4
KSsvsI6L3Q7K/jwmjnMS+GghMcUrL2KiKYreSHpruYYMuiKrW6d2NzJYv5OoisyO
gIfw/ZaLlK/uuD8sTuJ7NlU1CdDi+OtOSbrtNz2frA+P5bybfFrrBNfODisB1Q/g
zH9BTo3b5JgecHfkzGMsYa16e+kbK6LPCnw0bpc5EmELaAM3f/Za0rKksmwhcJZV
lzTLrUWMCEZqoZlRrPvuJ5ntBryWmZgyMHNflkaX4l8FsFnYjevW5QNHKYPeCpOV
wam7GiWw2gxz7XBVfRpESqimi9c4slthIrfP/schJrtRtxUQyaHUsEa8s8O/FZEU
utCuy7jAKaiShYgTyiplQ1Gx8UFgpIlET5ToHUQWnau6adVJNOFqnAhoM+Rr5wWV
LRgCyepS2RP/vNyTY091M50fo1N73lFXLuL4EkimSq8CxwdvVs+7vgmeIfZnno30
VRr6Fz6+PnB/AWkK+FS+
=mViD
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] [PATCH v3 4/5] migration: Append JSON description of migration stream
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 4/5] migration: Append JSON description of migration stream Alexander Graf
2015-01-06 15:56 ` Eric Blake
@ 2015-01-20 10:30 ` Amit Shah
1 sibling, 0 replies; 19+ messages in thread
From: Amit Shah @ 2015-01-20 10:30 UTC (permalink / raw)
To: Alexander Graf; +Cc: pbonzini, quintela, qemu-devel, afaerber
On (Fri) 26 Dec 2014 [15:42:47], Alexander Graf wrote:
> One of the annoyances of the current migration format is the fact that
> it's not self-describing. In fact, it's not properly describing at all.
> Some code randomly scattered throughout QEMU elaborates roughly how to
> read and write a stream of bytes.
>
> We discussed an idea during KVM Forum 2013 to add a JSON description of
> the migration protocol itself to the migration stream. This patch
> adds a section after the VM_END migration end marker that contains
> description data on what the device sections of the stream are composed of.
Can you add a note saying why this is safe for migrations to qemu
versions that don't expect this new json data?
Amit
^ permalink raw reply [flat|nested] 19+ messages in thread
* [Qemu-devel] [PATCH v3 5/5] Add migration stream analyzation script
2014-12-26 14:42 [Qemu-devel] [PATCH v3 0/5] Migration Deciphering aid Alexander Graf
` (3 preceding siblings ...)
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 4/5] migration: Append JSON description of migration stream Alexander Graf
@ 2014-12-26 14:42 ` Alexander Graf
2015-01-06 16:05 ` Eric Blake
2015-01-20 10:31 ` [Qemu-devel] [PATCH v3 0/5] Migration Deciphering aid Amit Shah
5 siblings, 1 reply; 19+ messages in thread
From: Alexander Graf @ 2014-12-26 14:42 UTC (permalink / raw)
To: qemu-devel; +Cc: amit.shah, pbonzini, afaerber, quintela
This patch adds a python tool to the scripts directory that can read
a dumped migration stream if it contains the JSON description of the
device states. I constructs a human readable JSON stream out of it.
It's very simple to use:
$ qemu-system-x86_64
(qemu) migrate "exec:cat > mig"
$ ./scripts/analyze_migration.py -f mig
Signed-off-by: Alexander Graf <agraf@suse.de>
---
v1 -> v2:
- Remove support for multiple vmsd versions
- Add support for HTAB
- Add support for array_len
- Move to new naming schema for fields
- Use dynamic page size
v2 -> v3:
- Add tell function to MigrationFile
- Report where subsections were not found
- Print ram sections with size
- Remove foreign desc file
- Add memory dump support (-m option)
- Add -x option to extract all sections into files
---
scripts/analyze-migration.py | 592 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 592 insertions(+)
create mode 100755 scripts/analyze-migration.py
diff --git a/scripts/analyze-migration.py b/scripts/analyze-migration.py
new file mode 100755
index 0000000..3363172
--- /dev/null
+++ b/scripts/analyze-migration.py
@@ -0,0 +1,592 @@
+#!/usr/bin/env python
+#
+# Migration Stream Analyzer
+#
+# Copyright (c) 2013 Alexander Graf <agraf@suse.de>
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, see <http://www.gnu.org/licenses/>.
+
+import numpy as np
+import json
+import os
+import argparse
+import collections
+import pprint
+
+def mkdir_p(path):
+ try:
+ os.makedirs(path)
+ except OSError:
+ pass
+
+class MigrationFile(object):
+ def __init__(self, filename):
+ self.filename = filename
+ self.file = open(self.filename, "rb")
+
+ def read64(self):
+ return np.asscalar(np.fromfile(self.file, count=1, dtype='>i8')[0])
+
+ def read32(self):
+ return np.asscalar(np.fromfile(self.file, count=1, dtype='>i4')[0])
+
+ def read16(self):
+ return np.asscalar(np.fromfile(self.file, count=1, dtype='>i2')[0])
+
+ def read8(self):
+ return np.asscalar(np.fromfile(self.file, count=1, dtype='>i1')[0])
+
+ def readstr(self, len = None):
+ if len is None:
+ len = self.read8()
+ if len == 0:
+ return ""
+ return np.fromfile(self.file, count=1, dtype=('S%d' % len))[0]
+
+ def readvar(self, size = None):
+ if size is None:
+ size = self.read8()
+ if size == 0:
+ return ""
+ value = self.file.read(size)
+ if len(value) != size:
+ raise Exception("Unexpected end of %s at 0x%x" % (self.filename, self.file.tell()))
+ return value
+
+ def tell(self):
+ return self.file.tell()
+
+ # The VMSD description is at the end of the file, after EOF. Look for
+ # the last NULL byte, then for the beginning brace of JSON.
+ def read_migration_debug_json(self):
+ QEMU_VM_VMDESCRIPTION = 0x06
+
+ # Remember the offset in the file when we started
+ entrypos = self.file.tell()
+
+ # Read the last 10MB
+ self.file.seek(0, os.SEEK_END)
+ endpos = self.file.tell()
+ self.file.seek(max(-endpos, -10 * 1024 * 1024), os.SEEK_END)
+ datapos = self.file.tell()
+ data = self.file.read()
+ # The full file read closed the file as well, reopen it
+ self.file = open(self.filename, "rb")
+
+ # Find the last NULL byte, then the first brace after that. This should
+ # be the beginning of our JSON data.
+ nulpos = data.rfind("\0")
+ jsonpos = data.find("{", nulpos)
+
+ # Check backwards from there and see whether we guessed right
+ self.file.seek(datapos + jsonpos - 5, 0)
+ if self.read8() != QEMU_VM_VMDESCRIPTION:
+ raise Exception("No Debug Migration device found")
+
+ jsonlen = self.read32()
+
+ # Seek back to where we were at the beginning
+ self.file.seek(entrypos, 0)
+
+ return data[jsonpos:jsonpos + jsonlen]
+
+ def close(self):
+ self.file.close()
+
+class RamSection(object):
+ RAM_SAVE_FLAG_COMPRESS = 0x02
+ RAM_SAVE_FLAG_MEM_SIZE = 0x04
+ RAM_SAVE_FLAG_PAGE = 0x08
+ RAM_SAVE_FLAG_EOS = 0x10
+ RAM_SAVE_FLAG_CONTINUE = 0x20
+ RAM_SAVE_FLAG_XBZRLE = 0x40
+ RAM_SAVE_FLAG_HOOK = 0x80
+
+ def __init__(self, file, version_id, ramargs, section_key):
+ if version_id != 4:
+ raise Exception("Unknown RAM version %d" % version_id)
+
+ self.file = file
+ self.section_key = section_key
+ self.TARGET_PAGE_SIZE = ramargs['page_size']
+ self.dump_memory = ramargs['dump_memory']
+ self.write_memory = ramargs['write_memory']
+ self.sizeinfo = collections.OrderedDict()
+ self.data = collections.OrderedDict()
+ self.data['section sizes'] = self.sizeinfo
+ self.name = ''
+ if self.write_memory:
+ self.files = { }
+ if self.dump_memory:
+ self.memory = collections.OrderedDict()
+ self.data['memory'] = self.memory
+
+ def __repr__(self):
+ return self.data.__repr__()
+
+ def __str__(self):
+ return self.data.__str__()
+
+ def getDict(self):
+ return self.data
+
+ def read(self):
+ # Read all RAM sections
+ while True:
+ addr = self.file.read64()
+ flags = addr & (self.TARGET_PAGE_SIZE - 1)
+ addr &= ~(self.TARGET_PAGE_SIZE - 1)
+
+ if flags & self.RAM_SAVE_FLAG_MEM_SIZE:
+ while True:
+ namelen = self.file.read8()
+ # We assume that no RAM chunk is big enough to ever
+ # hit the first byte of the address, so when we see
+ # a zero here we know it has to be an address, not the
+ # length of the next block.
+ if namelen == 0:
+ self.file.file.seek(-1, 1)
+ break
+ self.name = self.file.readstr(len = namelen)
+ len = self.file.read64()
+ self.sizeinfo[self.name] = '0x%016x' % len
+ if self.write_memory:
+ print self.name
+ mkdir_p('./' + os.path.dirname(self.name))
+ f = open('./' + self.name, "wb")
+ f.truncate(0)
+ f.truncate(len)
+ self.files[self.name] = f
+ flags &= ~self.RAM_SAVE_FLAG_MEM_SIZE
+
+ if flags & self.RAM_SAVE_FLAG_COMPRESS:
+ if flags & self.RAM_SAVE_FLAG_CONTINUE:
+ flags &= ~self.RAM_SAVE_FLAG_CONTINUE
+ else:
+ self.name = self.file.readstr()
+ fill_char = self.file.read8()
+ # The page in question is filled with fill_char now
+ if self.write_memory and fill_char != 0:
+ self.files[self.name].seek(addr, os.SEEK_SET)
+ self.files[self.name].write(chr(fill_char) * self.TARGET_PAGE_SIZE)
+ if self.dump_memory:
+ self.memory['%s (0x%016x)' % (self.name, addr)] = 'Filled with 0x%02x' % fill_char
+ flags &= ~self.RAM_SAVE_FLAG_COMPRESS
+ elif flags & self.RAM_SAVE_FLAG_PAGE:
+ if flags & self.RAM_SAVE_FLAG_CONTINUE:
+ flags &= ~self.RAM_SAVE_FLAG_CONTINUE
+ else:
+ self.name = self.file.readstr()
+
+ if self.write_memory or self.dump_memory:
+ data = self.file.readvar(size = self.TARGET_PAGE_SIZE)
+ else: # Just skip RAM data
+ self.file.file.seek(self.TARGET_PAGE_SIZE, 1)
+
+ if self.write_memory:
+ self.files[self.name].seek(addr, os.SEEK_SET)
+ self.files[self.name].write(data)
+ if self.dump_memory:
+ hexdata = " ".join("{0:02x}".format(ord(c)) for c in data)
+ self.memory['%s (0x%016x)' % (self.name, addr)] = hexdata
+
+ flags &= ~self.RAM_SAVE_FLAG_PAGE
+ elif flags & self.RAM_SAVE_FLAG_XBZRLE:
+ raise Exception("XBZRLE RAM compression is not supported yet")
+ elif flags & self.RAM_SAVE_FLAG_HOOK:
+ raise Exception("RAM hooks don't make sense with files")
+
+ # End of RAM section
+ if flags & self.RAM_SAVE_FLAG_EOS:
+ break
+
+ if flags != 0:
+ raise Exception("Unknown RAM flags: %x" % flags)
+
+ def __del__(self):
+ if self.write_memory:
+ for key in self.files:
+ self.files[key].close()
+
+
+class HTABSection(object):
+ HASH_PTE_SIZE_64 = 16
+
+ def __init__(self, file, version_id, device, section_key):
+ if version_id != 1:
+ raise Exception("Unknown HTAB version %d" % version_id)
+
+ self.file = file
+ self.section_key = section_key
+
+ def read(self):
+
+ header = self.file.read32()
+
+ if (header > 0):
+ # First section, just the hash shift
+ return
+
+ # Read until end marker
+ while True:
+ index = self.file.read32()
+ n_valid = self.file.read16()
+ n_invalid = self.file.read16()
+
+ if index == 0 and n_valid == 0 and n_invalid == 0:
+ break
+
+ self.file.readvar(n_valid * HASH_PTE_SIZE_64)
+
+ def getDict(self):
+ return ""
+
+class VMSDFieldGeneric(object):
+ def __init__(self, desc, file):
+ self.file = file
+ self.desc = desc
+ self.data = ""
+
+ def __repr__(self):
+ return str(self.__str__())
+
+ def __str__(self):
+ return " ".join("{0:02x}".format(ord(c)) for c in self.data)
+
+ def getDict(self):
+ return self.__str__()
+
+ def read(self):
+ size = int(self.desc['size'])
+ self.data = self.file.readvar(size)
+ return self.data
+
+class VMSDFieldInt(VMSDFieldGeneric):
+ def __init__(self, desc, file):
+ super(VMSDFieldInt, self).__init__(desc, file)
+ self.size = int(desc['size'])
+ self.format = '0x%%0%dx' % (self.size * 2)
+ self.sdtype = '>i%d' % self.size
+ self.udtype = '>u%d' % self.size
+
+ def __repr__(self):
+ if self.data < 0:
+ return ('%s (%d)' % ((self.format % self.udata), self.data))
+ else:
+ return self.format % self.data
+
+ def __str__(self):
+ return self.__repr__()
+
+ def getDict(self):
+ return self.__str__()
+
+ def read(self):
+ super(VMSDFieldInt, self).read()
+ self.sdata = np.fromstring(self.data, count=1, dtype=(self.sdtype))[0]
+ self.udata = np.fromstring(self.data, count=1, dtype=(self.udtype))[0]
+ self.data = self.sdata
+ return self.data
+
+class VMSDFieldUInt(VMSDFieldInt):
+ def __init__(self, desc, file):
+ super(VMSDFieldUInt, self).__init__(desc, file)
+
+ def read(self):
+ super(VMSDFieldUInt, self).read()
+ self.data = self.udata
+ return self.data
+
+class VMSDFieldIntLE(VMSDFieldInt):
+ def __init__(self, desc, file):
+ super(VMSDFieldIntLE, self).__init__(desc, file)
+ self.dtype = '<i%d' % self.size
+
+class VMSDFieldBool(VMSDFieldGeneric):
+ def __init__(self, desc, file):
+ super(VMSDFieldBool, self).__init__(desc, file)
+
+ def __repr__(self):
+ return self.data.__repr__()
+
+ def __str__(self):
+ return self.data.__str__()
+
+ def getDict(self):
+ return self.data
+
+ def read(self):
+ super(VMSDFieldBool, self).read()
+ if self.data[0] == 0:
+ self.data = False
+ else:
+ self.data = True
+ return self.data
+
+class VMSDFieldStruct(VMSDFieldGeneric):
+ QEMU_VM_SUBSECTION = 0x05
+
+ def __init__(self, desc, file):
+ super(VMSDFieldStruct, self).__init__(desc, file)
+ self.data = collections.OrderedDict()
+
+ # When we see compressed array elements, unfold them here
+ new_fields = []
+ for field in self.desc['struct']['fields']:
+ if not 'array_len' in field:
+ new_fields.append(field)
+ continue
+ array_len = field.pop('array_len')
+ field['index'] = 0
+ new_fields.append(field)
+ for i in xrange(1, array_len):
+ c = field.copy()
+ c['index'] = i
+ new_fields.append(c)
+
+ self.desc['struct']['fields'] = new_fields
+
+ def __repr__(self):
+ return self.data.__repr__()
+
+ def __str__(self):
+ return self.data.__str__()
+
+ def read(self):
+ for field in self.desc['struct']['fields']:
+ try:
+ reader = vmsd_field_readers[field['type']]
+ except:
+ reader = VMSDFieldGeneric
+
+ field['data'] = reader(field, self.file)
+ field['data'].read()
+
+ if 'index' in field:
+ if field['name'] not in self.data:
+ self.data[field['name']] = []
+ a = self.data[field['name']]
+ if len(a) != int(field['index']):
+ raise Exception("internal index of data field unmatched (%d/%d)" % (len(a), int(field['index'])))
+ a.append(field['data'])
+ else:
+ self.data[field['name']] = field['data']
+
+ if 'subsections' in self.desc['struct']:
+ for subsection in self.desc['struct']['subsections']:
+ if self.file.read8() != self.QEMU_VM_SUBSECTION:
+ raise Exception("Subsection %s not found at offset %x" % ( subsection['vmsd_name'], self.file.tell()))
+ name = self.file.readstr()
+ version_id = self.file.read32()
+ self.data[name] = VMSDSection(self.file, version_id, subsection, (name, 0))
+ self.data[name].read()
+
+ def getDictItem(self, value):
+ # Strings would fall into the array category, treat
+ # them specially
+ if value.__class__ is ''.__class__:
+ return value
+
+ try:
+ return self.getDictOrderedDict(value)
+ except:
+ try:
+ return self.getDictArray(value)
+ except:
+ try:
+ return value.getDict()
+ except:
+ return value
+
+ def getDictArray(self, array):
+ r = []
+ for value in array:
+ r.append(self.getDictItem(value))
+ return r
+
+ def getDictOrderedDict(self, dict):
+ r = collections.OrderedDict()
+ for (key, value) in dict.items():
+ r[key] = self.getDictItem(value)
+ return r
+
+ def getDict(self):
+ return self.getDictOrderedDict(self.data)
+
+vmsd_field_readers = {
+ "bool" : VMSDFieldBool,
+ "int8" : VMSDFieldInt,
+ "int16" : VMSDFieldInt,
+ "int32" : VMSDFieldInt,
+ "int32 equal" : VMSDFieldInt,
+ "int32 le" : VMSDFieldIntLE,
+ "int64" : VMSDFieldInt,
+ "uint8" : VMSDFieldUInt,
+ "uint16" : VMSDFieldUInt,
+ "uint32" : VMSDFieldUInt,
+ "uint32 equal" : VMSDFieldUInt,
+ "uint64" : VMSDFieldUInt,
+ "int64 equal" : VMSDFieldInt,
+ "uint8 equal" : VMSDFieldInt,
+ "uint16 equal" : VMSDFieldInt,
+ "float64" : VMSDFieldGeneric,
+ "timer" : VMSDFieldGeneric,
+ "buffer" : VMSDFieldGeneric,
+ "unused_buffer" : VMSDFieldGeneric,
+ "bitmap" : VMSDFieldGeneric,
+ "struct" : VMSDFieldStruct,
+ "unknown" : VMSDFieldGeneric,
+}
+
+class VMSDSection(VMSDFieldStruct):
+ def __init__(self, file, version_id, device, section_key):
+ self.file = file
+ self.data = ""
+ self.vmsd_name = ""
+ self.section_key = section_key
+ desc = device
+ if 'vmsd_name' in device:
+ self.vmsd_name = device['vmsd_name']
+
+ # A section really is nothing but a FieldStruct :)
+ super(VMSDSection, self).__init__({ 'struct' : desc }, file)
+
+###############################################################################
+
+class MigrationDump(object):
+ QEMU_VM_FILE_MAGIC = 0x5145564d
+ QEMU_VM_FILE_VERSION = 0x00000003
+ QEMU_VM_EOF = 0x00
+ QEMU_VM_SECTION_START = 0x01
+ QEMU_VM_SECTION_PART = 0x02
+ QEMU_VM_SECTION_END = 0x03
+ QEMU_VM_SECTION_FULL = 0x04
+ QEMU_VM_SUBSECTION = 0x05
+ QEMU_VM_VMDESCRIPTION = 0x06
+
+ def __init__(self, filename):
+ self.section_classes = { ( 'ram', 0 ) : [ RamSection, None ],
+ ( 'spapr/htab', 0) : ( HTABSection, None ) }
+ self.filename = filename
+ self.vmsd_desc = None
+
+ def read(self, desc_only = False, dump_memory = False, write_memory = False):
+ # Read in the whole file
+ file = MigrationFile(self.filename)
+
+ # File magic
+ data = file.read32()
+ if data != self.QEMU_VM_FILE_MAGIC:
+ raise Exception("Invalid file magic %x" % data)
+
+ # Version (has to be v3)
+ data = file.read32()
+ if data != self.QEMU_VM_FILE_VERSION:
+ raise Exception("Invalid version number %d" % data)
+
+ self.load_vmsd_json(file)
+
+ # Read sections
+ self.sections = collections.OrderedDict()
+
+ if desc_only:
+ return
+
+ ramargs = {}
+ ramargs['page_size'] = self.vmsd_desc['page_size']
+ ramargs['dump_memory'] = dump_memory
+ ramargs['write_memory'] = write_memory
+ self.section_classes[('ram',0)][1] = ramargs
+
+ while True:
+ section_type = file.read8()
+ if section_type == self.QEMU_VM_EOF:
+ break
+ elif section_type == self.QEMU_VM_SECTION_START or section_type == self.QEMU_VM_SECTION_FULL:
+ section_id = file.read32()
+ name = file.readstr()
+ instance_id = file.read32()
+ version_id = file.read32()
+ section_key = (name, instance_id)
+ classdesc = self.section_classes[section_key]
+ section = classdesc[0](file, version_id, classdesc[1], section_key)
+ self.sections[section_id] = section
+ section.read()
+ elif section_type == self.QEMU_VM_SECTION_PART or section_type == self.QEMU_VM_SECTION_END:
+ section_id = file.read32()
+ self.sections[section_id].read()
+ else:
+ raise Exception("Unknown section type: %d" % section_type)
+ file.close()
+
+ def load_vmsd_json(self, file):
+ vmsd_json = file.read_migration_debug_json()
+ self.vmsd_desc = json.loads(vmsd_json, object_pairs_hook=collections.OrderedDict)
+ for device in self.vmsd_desc['devices']:
+ key = (device['name'], device['instance_id'])
+ value = ( VMSDSection, device )
+ self.section_classes[key] = value
+
+ def getDict(self):
+ r = collections.OrderedDict()
+ for (key, value) in self.sections.items():
+ key = "%s (%d)" % ( value.section_key[0], key )
+ r[key] = value.getDict()
+ return r
+
+###############################################################################
+
+class JSONEncoder(json.JSONEncoder):
+ def default(self, o):
+ if isinstance(o, VMSDFieldGeneric):
+ return str(o)
+ return json.JSONEncoder.default(self, o)
+
+parser = argparse.ArgumentParser()
+parser.add_argument("-f", "--file", help='migration dump to read from', required=True)
+parser.add_argument("-m", "--memory", help='dump RAM contents as well', action='store_true')
+parser.add_argument("-d", "--dump", help='what to dump ("state" or "desc")', default='state')
+parser.add_argument("-x", "--extract", help='extract contents into individual files', action='store_true')
+args = parser.parse_args()
+
+jsonenc = JSONEncoder(indent=4, separators=(',', ': '))
+
+if args.extract:
+ dump = MigrationDump(args.file)
+
+ dump.read(desc_only = True)
+ print "desc.json"
+ f = open("desc.json", "wb")
+ f.truncate()
+ f.write(jsonenc.encode(dump.vmsd_desc))
+ f.close()
+
+ dump.read(write_memory = True)
+ dict = dump.getDict()
+ print "state.json"
+ f = open("state.json", "wb")
+ f.truncate()
+ f.write(jsonenc.encode(dict))
+ f.close()
+elif args.dump == "state":
+ dump = MigrationDump(args.file)
+ dump.read(dump_memory = args.memory)
+ dict = dump.getDict()
+ print jsonenc.encode(dict)
+elif args.dump == "desc":
+ dump = MigrationDump(args.file)
+ dump.read(desc_only = True)
+ print jsonenc.encode(dump.vmsd_desc)
+else:
+ raise Exception("Please specify either -x, -d state or -d dump")
--
1.7.12.4
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [Qemu-devel] [PATCH v3 5/5] Add migration stream analyzation script
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 5/5] Add migration stream analyzation script Alexander Graf
@ 2015-01-06 16:05 ` Eric Blake
2015-01-06 21:29 ` Alexander Graf
0 siblings, 1 reply; 19+ messages in thread
From: Eric Blake @ 2015-01-06 16:05 UTC (permalink / raw)
To: Alexander Graf, qemu-devel; +Cc: amit.shah, pbonzini, afaerber, quintela
[-- Attachment #1: Type: text/plain, Size: 2460 bytes --]
On 12/26/2014 07:42 AM, Alexander Graf wrote:
> This patch adds a python tool to the scripts directory that can read
> a dumped migration stream if it contains the JSON description of the
> device states. I constructs a human readable JSON stream out of it.
>
> It's very simple to use:
>
> $ qemu-system-x86_64
> (qemu) migrate "exec:cat > mig"
> $ ./scripts/analyze_migration.py -f mig
>
> Signed-off-by: Alexander Graf <agraf@suse.de>
>
> ---
> diff --git a/scripts/analyze-migration.py b/scripts/analyze-migration.py
> new file mode 100755
> index 0000000..3363172
> --- /dev/null
> +++ b/scripts/analyze-migration.py
> @@ -0,0 +1,592 @@
> +#!/usr/bin/env python
> +#
> +# Migration Stream Analyzer
> +#
> +# Copyright (c) 2013 Alexander Graf <agraf@suse.de>
Started in 2013, but you may want to add 2014 and 2015 by the time this
patch is polished.
> +
> + # The VMSD description is at the end of the file, after EOF. Look for
> + # the last NULL byte, then for the beginning brace of JSON.
Hmm, you picked up on an optimization that I missed in my ramblings
about 4/5 :) Since a well-formed JSON description contains no NUL bytes
or control characters, a pipeline read of a migration stream can just
continually look for every '\6' marker byte followed by '{' as the
potential start of an object, and reset that search every time a '\0' or
another '\6' byte is seen; eventually, this will result in just the tail
of the file being seen as the JSON object (assuming that we never add
any other tail metadata - or that all other tail metadata is added as
backwards-compatible extensions to the JSON). It STILL might be worth
the efficiency gain of stashing an offset in the file somewhere (as it
is faster to seek to an offset than it is to scan for particular
patterns), but at least your scan works forwards rather than in reverse.
> +
> + # Find the last NULL byte, then the first brace after that. This should
> + # be the beginning of our JSON data.
> + nulpos = data.rfind("\0")
> + jsonpos = data.find("{", nulpos)
Are you sure that there will NEVER be a '{' somewhere between the last
NUL and the end-of-stream marker? Shouldn't you really be searching for
the last QEMU_VM_VMDESCRIPTION ('\6') magic byte, rather than NUL?
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [Qemu-devel] [PATCH v3 5/5] Add migration stream analyzation script
2015-01-06 16:05 ` Eric Blake
@ 2015-01-06 21:29 ` Alexander Graf
0 siblings, 0 replies; 19+ messages in thread
From: Alexander Graf @ 2015-01-06 21:29 UTC (permalink / raw)
To: Eric Blake, qemu-devel; +Cc: amit.shah, pbonzini, afaerber, quintela
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 06.01.15 17:05, Eric Blake wrote:
> On 12/26/2014 07:42 AM, Alexander Graf wrote:
>> This patch adds a python tool to the scripts directory that can
>> read a dumped migration stream if it contains the JSON
>> description of the device states. I constructs a human readable
>> JSON stream out of it.
>>
>> It's very simple to use:
>>
>> $ qemu-system-x86_64 (qemu) migrate "exec:cat > mig" $
>> ./scripts/analyze_migration.py -f mig
>>
>> Signed-off-by: Alexander Graf <agraf@suse.de>
>>
>> ---
>
>> diff --git a/scripts/analyze-migration.py
>> b/scripts/analyze-migration.py new file mode 100755 index
>> 0000000..3363172 --- /dev/null +++
>> b/scripts/analyze-migration.py @@ -0,0 +1,592 @@ +#!/usr/bin/env
>> python +# +# Migration Stream Analyzer +# +# Copyright (c) 2013
>> Alexander Graf <agraf@suse.de>
>
> Started in 2013, but you may want to add 2014 and 2015 by the time
> this patch is polished.
Oops :).
>
>> + + # The VMSD description is at the end of the file, after
>> EOF. Look for + # the last NULL byte, then for the beginning
>> brace of JSON.
>
> Hmm, you picked up on an optimization that I missed in my
> ramblings about 4/5 :) Since a well-formed JSON description
> contains no NUL bytes or control characters, a pipeline read of a
> migration stream can just continually look for every '\6' marker
> byte followed by '{' as the potential start of an object, and reset
> that search every time a '\0' or another '\6' byte is seen;
> eventually, this will result in just the tail of the file being
> seen as the JSON object (assuming that we never add any other tail
> metadata - or that all other tail metadata is added as
> backwards-compatible extensions to the JSON). It STILL might be
> worth the efficiency gain of stashing an offset in the file
> somewhere (as it is faster to seek to an offset than it is to scan
> for particular patterns), but at least your scan works forwards
> rather than in reverse.
Have you tried to run the script yet? Full search, assembly and
analysis of a normal sized migration stream take less than a second -
and that's with in-memory assembly of all the device objects and
properties (which takes the bulk of the time).
I don't think we have an efficiency problem :).
>
>> + + # Find the last NULL byte, then the first brace after
>> that. This should + # be the beginning of our JSON data. +
>> nulpos = data.rfind("\0") + jsonpos = data.find("{",
>> nulpos)
>
> Are you sure that there will NEVER be a '{' somewhere between the
> last NUL and the end-of-stream marker? Shouldn't you really be
> searching for the last QEMU_VM_VMDESCRIPTION ('\6') magic byte,
> rather than NUL?
NUL is the EOF marker, so I'm definitely hoping for it :). If we
decide to add more data after EOF (which we really shouldn't I think),
we can always adjust the script again IMHO.
Alex
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org
iQIcBAEBAgAGBQJUrFO0AAoJECszeR4D/txg2wUQAIp8R22LiQzqQPrqs5VMiUbE
kOLvieg0fRRHmgQ/SE7aHc/s4sC88EEKsOBintijgcNPkt8Lu65xm71JvI9660BM
9Y/ASYIliNrBKr28ThszK074xAKsvkjSBULIPH6hMUk9e6P8p5SjnKckerWWc7a/
YK7xa+1uxTaK/3+JSzV4ZKFVwbIePrEXqGxYHFcKVqmNs/OSqNwkIkErFJQCyd8D
edqU/zQ4A4v+ZlJl4HZkZqXCBwUGJd6kF0ZUDmF4a7lsAELbv+MTxmVy8enBmH8p
KngnDkI67tpQmlxwSYu9tBW/25qcTIiZOigGT/VShDN6cRQfbw+h3lC3klt/GKA7
ghu464ept6/xI5FQpMkdkPQ742c6lhkuQAUTVqlorVtTsc9ZF2ONLl36Y/xPNmZB
R2BLDq3zDijWe6spWrfoMmc2pmtohHwyBNDtjVbI3nAkhdPddHdR2SzaCUf0B4Nj
tMckMDCucGWYMGTXGMElkw9SZE8kwExPme43nSsGRaHdUVLv24kIv3lNhUlIlpBq
Ljg9CMsHV6RhDySIvucfDueOpmgnyc7KGadXVEFr1pfEJrvg/SjhGDdX+ilmQeRg
iCk0f0/DyJWhPskGDeKBMm9DiJ0QZ3j+Lnw+ZYnOEazyr5tOy0kTuhmPx1eDK9yg
q1YJ7JGn7zVDwoBVWiXK
=wERA
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] [PATCH v3 0/5] Migration Deciphering aid
2014-12-26 14:42 [Qemu-devel] [PATCH v3 0/5] Migration Deciphering aid Alexander Graf
` (4 preceding siblings ...)
2014-12-26 14:42 ` [Qemu-devel] [PATCH v3 5/5] Add migration stream analyzation script Alexander Graf
@ 2015-01-20 10:31 ` Amit Shah
2015-01-20 10:54 ` Alexander Graf
5 siblings, 1 reply; 19+ messages in thread
From: Amit Shah @ 2015-01-20 10:31 UTC (permalink / raw)
To: Alexander Graf; +Cc: pbonzini, quintela, qemu-devel, afaerber
On (Fri) 26 Dec 2014 [15:42:43], Alexander Graf wrote:
> Migration is a black hole to most people. One of the biggest reasons for
> this is that its protocol is a secret, undocumented sauce of code rolling
> around random parts of the QEMU code base.
>
> But what if we simply exposed the description of how the format looks like
> alongside the actual migration stream? This is what this patch set does.
>
> It adds a new section that comes after the end of stream marker (so that it
> doesn't slow down migration) that contains a JSON description of the device
> state description.
>
> Along with this patch set also comes a python script that can read said JSON
> from a migration dump and decipher the device state and ram contents of the
> migration dump using it.
>
> With this, you can now fully examine all glorious details that go over the
> wire when virtual machine state gets dumped, such as during live migration.
>
> We discussed the approach taken here during KVM Forum 2013. Originally, my idea
> was to include a special device that contains the JSON data which can be enabled
> on demand. Anthony suggested however to just always include the description data
> after the end marker which I think is a great idea.
>
> Example decoded migration: http://csgraf.de/mig/mig.txt
> Example migration description: http://csgraf.de/mig/mig.desc.txt
> Presentation: https://www.youtube.com/watch?v=iq1x40Qsrew
> Slides: https://www.dropbox.com/s/otp2pk2n3g087zp/Live%20Migration.pdf
Nice to finally see this!
I guess you have a v4 coming soon?
Amit
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [Qemu-devel] [PATCH v3 0/5] Migration Deciphering aid
2015-01-20 10:31 ` [Qemu-devel] [PATCH v3 0/5] Migration Deciphering aid Amit Shah
@ 2015-01-20 10:54 ` Alexander Graf
2015-01-21 6:05 ` Amit Shah
0 siblings, 1 reply; 19+ messages in thread
From: Alexander Graf @ 2015-01-20 10:54 UTC (permalink / raw)
To: Amit Shah; +Cc: pbonzini, quintela, qemu-devel, afaerber
On 20.01.15 11:31, Amit Shah wrote:
> On (Fri) 26 Dec 2014 [15:42:43], Alexander Graf wrote:
>> Migration is a black hole to most people. One of the biggest reasons for
>> this is that its protocol is a secret, undocumented sauce of code rolling
>> around random parts of the QEMU code base.
>>
>> But what if we simply exposed the description of how the format looks like
>> alongside the actual migration stream? This is what this patch set does.
>>
>> It adds a new section that comes after the end of stream marker (so that it
>> doesn't slow down migration) that contains a JSON description of the device
>> state description.
>>
>> Along with this patch set also comes a python script that can read said JSON
>> from a migration dump and decipher the device state and ram contents of the
>> migration dump using it.
>>
>> With this, you can now fully examine all glorious details that go over the
>> wire when virtual machine state gets dumped, such as during live migration.
>>
>> We discussed the approach taken here during KVM Forum 2013. Originally, my idea
>> was to include a special device that contains the JSON data which can be enabled
>> on demand. Anthony suggested however to just always include the description data
>> after the end marker which I think is a great idea.
>>
>> Example decoded migration: http://csgraf.de/mig/mig.txt
>> Example migration description: http://csgraf.de/mig/mig.desc.txt
>> Presentation: https://www.youtube.com/watch?v=iq1x40Qsrew
>> Slides: https://www.dropbox.com/s/otp2pk2n3g087zp/Live%20Migration.pdf
>
> Nice to finally see this!
>
> I guess you have a v4 coming soon?
Yeah, I was just waiting on a bit more review :)
Alex
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] [PATCH v3 0/5] Migration Deciphering aid
2015-01-20 10:54 ` Alexander Graf
@ 2015-01-21 6:05 ` Amit Shah
0 siblings, 0 replies; 19+ messages in thread
From: Amit Shah @ 2015-01-21 6:05 UTC (permalink / raw)
To: Alexander Graf; +Cc: pbonzini, quintela, qemu-devel, afaerber
On (Tue) 20 Jan 2015 [11:54:50], Alexander Graf wrote:
>
>
> On 20.01.15 11:31, Amit Shah wrote:
> > On (Fri) 26 Dec 2014 [15:42:43], Alexander Graf wrote:
> >> Migration is a black hole to most people. One of the biggest reasons for
> >> this is that its protocol is a secret, undocumented sauce of code rolling
> >> around random parts of the QEMU code base.
> >>
> >> But what if we simply exposed the description of how the format looks like
> >> alongside the actual migration stream? This is what this patch set does.
> >>
> >> It adds a new section that comes after the end of stream marker (so that it
> >> doesn't slow down migration) that contains a JSON description of the device
> >> state description.
> >>
> >> Along with this patch set also comes a python script that can read said JSON
> >> from a migration dump and decipher the device state and ram contents of the
> >> migration dump using it.
> >>
> >> With this, you can now fully examine all glorious details that go over the
> >> wire when virtual machine state gets dumped, such as during live migration.
> >>
> >> We discussed the approach taken here during KVM Forum 2013. Originally, my idea
> >> was to include a special device that contains the JSON data which can be enabled
> >> on demand. Anthony suggested however to just always include the description data
> >> after the end marker which I think is a great idea.
> >>
> >> Example decoded migration: http://csgraf.de/mig/mig.txt
> >> Example migration description: http://csgraf.de/mig/mig.desc.txt
> >> Presentation: https://www.youtube.com/watch?v=iq1x40Qsrew
> >> Slides: https://www.dropbox.com/s/otp2pk2n3g087zp/Live%20Migration.pdf
> >
> > Nice to finally see this!
> >
> > I guess you have a v4 coming soon?
>
> Yeah, I was just waiting on a bit more review :)
I have no problem with the series :-)
In fact I forgot to note - the vmstate checker can use the json output
introduced here :)
Amit
^ permalink raw reply [flat|nested] 19+ messages in thread