From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58003) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZXeQ7-0008Q3-Kq for qemu-devel@nongnu.org; Thu, 03 Sep 2015 19:59:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZXeQ3-00087b-Gm for qemu-devel@nongnu.org; Thu, 03 Sep 2015 19:59:55 -0400 Received: from e17.ny.us.ibm.com ([129.33.205.207]:43743) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZXeQ3-00087V-Bu for qemu-devel@nongnu.org; Thu, 03 Sep 2015 19:59:51 -0400 Received: from /spool/local by e17.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 3 Sep 2015 19:59:49 -0400 Received: from b01cxnp22033.gho.pok.ibm.com (b01cxnp22033.gho.pok.ibm.com [9.57.198.23]) by d01dlp01.pok.ibm.com (Postfix) with ESMTP id A0E7238C8046 for ; Thu, 3 Sep 2015 19:59:47 -0400 (EDT) Received: from d01av01.pok.ibm.com (d01av01.pok.ibm.com [9.56.224.215]) by b01cxnp22033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id t83NxlwT58720432 for ; Thu, 3 Sep 2015 23:59:47 GMT Received: from d01av01.pok.ibm.com (localhost [127.0.0.1]) by d01av01.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id t83Nxlru013466 for ; Thu, 3 Sep 2015 19:59:47 -0400 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable From: Michael Roth In-Reply-To: <1441290623-13631-31-git-send-email-armbru@redhat.com> References: <1441290623-13631-1-git-send-email-armbru@redhat.com> <1441290623-13631-31-git-send-email-armbru@redhat.com> Message-ID: <20150903235941.10296.72342@loki> Date: Thu, 03 Sep 2015 18:59:41 -0500 Subject: Re: [Qemu-devel] [PATCH RFC v4 30/32] qapi: New QMP command query-schema for QMP schema introspection List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Markus Armbruster , qemu-devel@nongnu.org Quoting Markus Armbruster (2015-09-03 09:30:21) > qapi/introspect.json defines the introspection schema. It's designed > for QMP introspection, but should do for similar uses, such as QGA. > = > The introspection schema does not reflect all the rules and > restrictions that apply to QAPI schemata. A valid QAPI schema has an > introspection value conforming to the introspection schema, but the > converse is not true. > = > Introspection lowers away a number of schema details, and makes > implicit things explicit: > = > * The built-in types are declared with their JSON type. > = > TODO Should we map all the integer types to just int? > = > * Implicit type definitions are made explicit, and given > auto-generated names: > = > - Array types, named by appending "List" to the name of their > element type, like in generated C. > = > - The enumeration types implicitly defined by simple union types, > named by appending "Kind" to the name of their simple union type, > like in generated C. > = > - Types that don't occur in generated C. Their names start with ':' > so they don't clash with the user's names. > = > * All type references are by name. > = > * The struct and union types are generalized into an object type. > = > * Base types are flattened. > = > * Commands take a single argument and return a single result. > = > Dictionary argument or list result is an implicit type definition. > = > The empty object type is used when a command takes no arguments or > produces no results. > = > The argument is always of object type, but the introspection schema > doesn't reflect that. > = > The 'gen': false directive is omitted as implementation detail. > = > The 'success-response' directive is omitted as well for now, even > though it's not an implementation detail, because it's not used by > QMP. > = > * Events carry a single data value. > = > Implicit type definition and empty object type use, just like for > commands. > = > The value is of object type, but the introspection schema doesn't > reflect that. > = > * Types not used by commands or events are omitted. > = > Indirect use counts as use. > = > * Optional members have a default, which can only be null right now > = > Instead of a mandatory "optional" flag, we have an optional default. > No default means mandatory, default null means optional without > default value. Non-null is available for optional with default > (possible future extension). > = > * Clients should *not* look up types by name, because type names are > not ABI. Look up the command or event you're interested in, then > follow the references. > = > TODO Should we hide the type names to eliminate the temptation? > = > New generator scripts/qapi-introspect.py computes an introspection > value for its input, and generates a C variable holding it. > = > It can generate awfully long lines. Marked TODO. > = > A new test-qmp-input-visitor test case feeds its result for both > tests/qapi-schema/qapi-schema-test.json and qapi-schema.json to a > QmpInputVisitor to verify it actually conforms to the schema. > = > New QMP command query-schema takes its return value from that > variable. Its reply is some 85KiBytes for me right now. > = > If this turns out to be too much, we have a couple of options: > = > * We can use shorter names in the JSON. Not the QMP style. > = > * Optionally return the sub-schema for commands and events given as > arguments. > = > Right now qmp_query_schema() sends the string literal computed by > qmp-introspect.py. To compute sub-schema at run time, we'd have to > duplicate parts of qapi-introspect.py in C. Unattractive. > = > * Let clients cache the output of query-schema. > = > It changes only on QEMU upgrades, i.e. rarely. Provide a command > query-schema-hash. Clients can have a cache indexed by hash, and > re-query the schema only when they don't have it cached. Even > simpler: put the hash in the QMP greeting. > = > Signed-off-by: Markus Armbruster > --- > .gitignore | 1 + > Makefile | 9 +- > Makefile.objs | 4 +- > docs/qapi-code-gen.txt | 226 ++++++++++++++++++= +- > monitor.c | 15 ++ > qapi-schema.json | 3 + > qapi/introspect.json | 271 ++++++++++++++++++= ++++++ > qmp-commands.hx | 17 ++ > scripts/qapi-introspect.py | 174 +++++++++++++++ > scripts/qapi.py | 12 +- > tests/.gitignore | 1 + > tests/Makefile | 10 +- > tests/qapi-schema/alternate-good.out | 1 + > tests/qapi-schema/args-member-array.out | 1 + > tests/qapi-schema/comments.out | 1 + > tests/qapi-schema/empty.out | 1 + > tests/qapi-schema/enum-empty.out | 1 + > tests/qapi-schema/event-case.out | 1 + > tests/qapi-schema/flat-union-reverse-define.out | 1 + > tests/qapi-schema/ident-with-escape.out | 1 + > tests/qapi-schema/include-relpath.out | 1 + > tests/qapi-schema/include-repetition.out | 1 + > tests/qapi-schema/include-simple.out | 1 + > tests/qapi-schema/indented-expr.out | 1 + > tests/qapi-schema/qapi-schema-test.out | 1 + > tests/qapi-schema/returns-int.out | 1 + > tests/test-qmp-input-strict.c | 55 +++++ > 27 files changed, 801 insertions(+), 11 deletions(-) > create mode 100644 qapi/introspect.json > create mode 100644 scripts/qapi-introspect.py > = > diff --git a/qapi-schema.json b/qapi-schema.json > index e91e3b9..af19810 100644 > --- a/qapi-schema.json > +++ b/qapi-schema.json > @@ -14,6 +14,9 @@ > # Tracing commands > { 'include': 'qapi/trace.json' } > = > +# QAPI introspection > +{ 'include': 'qapi/introspect.json' } > + > ## > # @LostTickPolicy: > # > diff --git a/qapi/introspect.json b/qapi/introspect.json > new file mode 100644 > index 0000000..171baba > --- /dev/null > +++ b/qapi/introspect.json > @@ -0,0 +1,271 @@ > +# -*- Mode: Python -*- > +# > +# QAPI/QMP introspection > +# > +# Copyright (C) 2015 Red Hat, Inc. > +# > +# Authors: > +# Markus Armbruster > +# > +# This work is licensed under the terms of the GNU GPL, version 2 or lat= er. > +# See the COPYING file in the top-level directory. > + > +## > +# @query-schema > +# > +# Command query-schema exposes the QMP wire ABI as an array of > +# SchemaInfo. This lets QMP clients figure out what commands and > +# events are available in this QEMU, and their parameters and results. > +# > +# Returns: array of @SchemaInfo, where each element describes an > +# entity in the ABI: command, event, type, ... > +# > +# Note: the QAPI schema is also used to help define *internal* > +# interfaces, by defining QAPI types. These are not part of the QMP > +# wire ABI, and therefore not returned by this command. Maybe add something like: "These internal interfaces may place additional restrictions on the values of individual fields, so users should reference the QAPI schema to avoid unexpected behavior resulting from invalid field values." > diff --git a/qmp-commands.hx b/qmp-commands.hx > index b06d74c..6232ca0 100644 > --- a/qmp-commands.hx > +++ b/qmp-commands.hx > @@ -2168,6 +2168,23 @@ EQMP > }, > = > SQMP > +query-schema > +------------ > + > +Return the QMP wire schema. The returned value is a json-array of > +named schema entities. Entities are commands, events and various > +types. See docs/qapi-code-gen.txt for information on their structure > +and intended use. We should probably duplicate any notes from QAPI schema description here as well, or at least direct users to reference them. > + > +EQMP > + > + { > + .name =3D "query-schema", > + .args_type =3D "", > + .mhandler.cmd_new =3D qmp_query_schema, > + }, > + > +SQMP > query-chardev > ------------- > = > diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py > new file mode 100644 > index 0000000..4bcffc4 > --- /dev/null > +++ b/scripts/qapi-introspect.py > @@ -0,0 +1,174 @@ > +# > +# QAPI introspection generator > +# > +# Copyright (C) 2015 Red Hat, Inc. > +# > +# Authors: > +# Markus Armbruster > +# > +# This work is licensed under the terms of the GNU GPL, version 2. > +# See the COPYING file in the top-level directory. > + > +from qapi import * > +import string > + > +# Caveman's json.dumps() replacement (we're stuck at 2.4) > +# TODO try to use json.dumps() once we get unstuck > +def to_json(obj, level=3D0): > + if obj =3D=3D None: > + ret =3D 'null' > + elif isinstance(obj, str): > + ret =3D '"' + obj.replace('"', r'\"') + '"' > + elif isinstance(obj, list): > + elts =3D [to_json(elt, level + 1) > + for elt in obj] > + ret =3D '[' + ', '.join(elts) + ']' > + elif isinstance(obj, dict): > + elts =3D ['"%s": %s' % (key, to_json(obj[key], level + 1)) > + for key in sorted(obj.keys())] > + ret =3D '{' + ', '.join(elts) + '}' > + else: > + assert False # not implemented > + if level =3D=3D 1: > + ret =3D '\n' + ret > + return ret > + > +def to_c_string(string): > + return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"' > + > +class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor): > + def __init__(self): > + self.defn =3D None > + self.decl =3D None > + self.schema =3D None > + self.jsons =3D None > + self.used_types =3D None > + > + def visit_begin(self, schema): > + self.schema =3D schema > + self.jsons =3D [] > + self.used_types =3D [] > + return QAPISchemaType # don't visit types for now > + > + def visit_end(self): > + # visit the types that are actually used > + for typ in self.used_types: > + typ.visit(self) > + self.jsons.sort() > + # generate C > + # TODO can generate awfully long lines Not sure if this is planned for this series, but a multi-line representation in the generated files would make things a bit easier to review/check. If we went a little further and made them pretty-printed it might make some of the docs referencing the output a bit more clear as well. Not a huge deal, but don't think it would take much code. Would suggest a JSON library or something but that would probably mangle the ordering, which wouldn't help with readability. > + name =3D prefix + 'qmp_schema_json' > + self.decl =3D mcgen(''' > +extern const char %(c_name)s[]; > +''', > + c_name=3Dc_name(name)) > + lines =3D to_json(self.jsons).split('\n') > + c_string =3D '\n '.join([to_c_string(line) for line in lines]) > + self.defn =3D mcgen(''' > +const char %(c_name)s[] =3D %(c_string)s; > +''', > + c_name=3Dc_name(name), > + c_string=3Dc_string) > + self.schema =3D None > + self.jsons =3D None > + self.used_types =3D None