qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [Qemu-devel] [PATCH v4 00/26] Add infrastructure for QIDL-based device serialization
@ 2012-10-12 21:10 Michael Roth
  2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 01/26] qapi: qapi-visit.py -> qapi_visit.py so we can import Michael Roth
                   ` (26 more replies)
  0 siblings, 27 replies; 40+ messages in thread
From: Michael Roth @ 2012-10-12 21:10 UTC (permalink / raw)
  To: qemu-devel; +Cc: kwolf, peter.maydell, aliguori, blauwirbel, pbonzini

These patches are based are origin/master, and can also be obtained from:

git://github.com/mdroth/qemu.git qidl-base-v4

Changes since v3:

 - Added documentation for array_capacity/array_size/embedded QAPI
   directives and improved existing docs. (Paolo)
     > qapi: Improve existing docs and document annotated QAPI types
 - Fix potential for unbalanced structures with visit_type_carray() in
   similar manner to type_list()/type_struct(). (Paolo)
 - Some comments in qapi_visit.py to clarify carray handling. (Paolo)
 - Avoid redundant re-definitions in qidl.h for -DQIDL_GEN (Paolo)
 - Added additional unit tests to exercise QAPI handling of
   array_capacity/array_size/embedded values for 'annotated' QAPI-defined
   types.
 - Added a to_json() method to OrderedDict() so that ordering is retained
   in the schemas that QIDL generates.
     > qapi: ordereddict, add to_json() method
 - *.qidl.schema QAPI schema files are now generated along-side *.qidl.c
   files when --enable-debug configure option is specified.
 - 2 additional patches to fix pre-existing QAPI issues triggered in testing:
     > qapi: QmpInputVisitor, don't re-allocate memory in start_struct
     > qapi: fix potential segfault for visit_type_size()

Changes since v2:

 - Documentations fix-ups and clarifications (Eric)
 - Moved annotations in front of variable names and documented this as
   the default (Paolo, Kevin)
 - Switched to q_* prefix instead of q<Capital> (Blue, Paolo, Anthony)
 - Switched to GPLv2+ licensing in QIDL lexer/parser
 - Minor clean-ups in unit tests

Changes since v1:

 - Simplified declaration format for QIDL-fied structures (Anthony, Blue)
 - Documentations fix-ups and clarifications (Eric, Peter)
 - Reduced build-time impact of QIDL by scanning for a QIDL_ENABLED()
   directive in .c files before running them through the
   the preprocessor and QIDL parser, and using a Makefile-set cflag
   to avoid declaring QIDL-related code/structures for files that don't
   include the directive.
 - Moved lexer functionality into a standalone lexer class, simplified
   interface to avoid the need to track offsets into the token stream.
 - Fixed an issue when deserializing a static array of structs using the
   new visit_type_carray() interfaces
 - Included a fix for a qom-fuse bug caused by multiple threads contending
   for QMP responses
 - Added brief descriptions of annotations to qidl.h
 - Minor clean-ups to the QIDL parser code

Changes since rfc v2:

 - Parser/Codegen fix-ups for cases encountered converting piix ide and usb.
 - Fixed license headers.
 - Stricter arg-checking for QIDL macros when passing to codegen.
 - Renamed QAPI visit_*_array interfaces to visit_*_carray to clarify that
   these are serialization routines for single-dimension C arrays.

These patches add infrastructure and unit tests for QIDL, which provides
a serialization framework for QEMU device structures by generating visitor
routines for device structs based on simple field annotations. Examples of
how this is done are included in patch 17, but, for brevity, a sample struct
such as this:

    typedef struct SerialDevice {
        SysBusDevice parent;

        uint8_t thr;            /* transmit holding register */
        uint8_t lsr;            /* line status register */
        uint8_t ier;            /* interrupt enable register */

        int int_pending;        /* whether we have a pending queued interrupt */
        CharDriverState *chr;   /* backend */
    } SerialDevice;

can now be made serializable with the following changes:

    typedef struct SerialDevice SerialDevice;

    QIDL_DECLARE(SerialDevice) {
        SysBusDevice parent;

        uint8_t thr;              /* transmit holding register */
        uint8_t lsr;              /* line status register */
        uint8_t ier;              /* interrupt enable register */

        int q_derived int_pending; /* whether we have a pending queued interrupt */
        CharDriverState q_immutable *chr; /* backend */
    };

To make use of generated visitor code, and .c file need only call the
QIDL_ENABLE() somewhere in the code body, which will then give it access to
visitor routines for any QIDL-ified device structures in that file, or included
from other files. These routines can then be used for
serialization/deserialization of the device state in a manner suitable for tasks
such as introspection/testing (generally via a r/w 'state' QOM property) or
migration.

The overall goal is to expose all migrateable device state in this manner so
that we can decouple serialization duties from savevm/VMState and convert them
into a stable, code-agnostic wire protocol, relying instead on intermediate
translation routines to handle the work of massaging serialized state into a
format suitable for the wire and vice-versa.

The following WIP branch contains the first set of QIDL conversions:

https://github.com/mdroth/qemu/commits/qidl

So far i440fx, pcibus, cirrus_vga, uhci, rtc, and isa/piix ide have been
converted, and I'm hoping to have most common PC devices converted over
within the next few weeks.

Please review. Comments/suggestions are very welcome.

 Makefile                                       |   26 +-
 QMP/qom-fuse                                   |   39 +-
 configure                                      |    1 +
 docs/qapi-code-gen.txt                         |  251 ++++++++++++-
 docs/qidl.txt                                  |  347 +++++++++++++++++
 hw/qdev-properties.h                           |  151 ++++++++
 hw/qdev.h                                      |  126 +------
 module.h                                       |    2 +
 qapi/Makefile.objs                             |    1 +
 qapi/misc-qapi-visit.c                         |   14 +
 qapi/qapi-visit-core.c                         |   31 +-
 qapi/qapi-visit-core.h                         |   11 +
 qapi/qmp-input-visitor.c                       |   32 +-
 qapi/qmp-output-visitor.c                      |   19 +
 qidl.h                                         |  105 ++++++
 rules.mak                                      |   25 +-
 scripts/lexer.py                               |  306 +++++++++++++++
 scripts/ordereddict.py                         |   51 +++
 scripts/qapi-visit.py                          |  364 ------------------
 scripts/qapi.py                                |   16 +-
 scripts/{qapi-commands.py => qapi_commands.py} |    8 +-
 scripts/{qapi-types.py => qapi_types.py}       |    2 +-
 scripts/qapi_visit.py                          |  481 ++++++++++++++++++++++++
 scripts/qidl.py                                |  290 ++++++++++++++
 scripts/qidl_parser.py                         |  269 +++++++++++++
 tests/Makefile                                 |   20 +-
 tests/test-qidl-included.h                     |   31 ++
 tests/test-qidl-linked.c                       |   93 +++++
 tests/test-qidl-linked.h                       |   18 +
 tests/test-qidl.c                              |  376 ++++++++++++++++++
 tests/test-qmp-input-visitor.c                 |    2 +-
 vl.c                                           |    1 +
 32 files changed, 2965 insertions(+), 544 deletions(-)

^ permalink raw reply	[flat|nested] 40+ messages in thread
end of thread, other threads:[~2012-10-19  9:01 UTC | newest]

Thread overview: 40+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-10-12 21:10 [Qemu-devel] [PATCH v4 00/26] Add infrastructure for QIDL-based device serialization Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 01/26] qapi: qapi-visit.py -> qapi_visit.py so we can import Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 02/26] qapi: qapi-types.py -> qapi_types.py Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 03/26] qapi: qapi-commands.py -> qapi_commands.py Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 04/26] qapi: qapi_visit.py, make code useable as module Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 05/26] qapi: qapi_visit.py, support arrays and complex qapi definitions Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 06/26] qapi: qapi_visit.py, support generating static functions Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 07/26] qapi: qapi_visit.py, support for visiting non-pointer/embedded structs Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 08/26] qapi: add visitor interfaces for C arrays Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 09/26] qapi: QmpOutputVisitor, implement array handling Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 10/26] qapi: QmpInputVisitor, " Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 11/26] qapi: QmpInputVisitor, don't re-allocate memory in start_struct Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 12/26] qapi: fix potential segfault for visit_type_size() Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 13/26] qapi: ordereddict, add to_json() method Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 14/26] qapi: qapi.py, make json parser more robust Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 15/26] qapi: add open-coded visitor for struct tm types Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 16/26] qapi: Improve existing docs and document annotated QAPI types Michael Roth
2012-10-12 21:10 ` [Qemu-devel] [PATCH v4 17/26] qom-fuse: force single-threaded mode to avoid QMP races Michael Roth
2012-10-12 21:11 ` [Qemu-devel] [PATCH v4 18/26] qom-fuse: workaround for truncated properties > 4096 Michael Roth
2012-10-12 21:11 ` [Qemu-devel] [PATCH v4 19/26] module additions for schema registration Michael Roth
2012-10-12 21:11 ` [Qemu-devel] [PATCH v4 20/26] qdev: move Property-related declarations to qdev-properties.h Michael Roth
2012-10-12 21:11 ` [Qemu-devel] [PATCH v4 21/26] qidl: add documentation Michael Roth
2012-10-12 21:11 ` [Qemu-devel] [PATCH v4 22/26] qidl: add lexer library (based on QC parser) Michael Roth
2012-10-16  7:26   ` Paolo Bonzini
2012-10-12 21:11 ` [Qemu-devel] [PATCH v4 23/26] qidl: add C parser " Michael Roth
2012-10-12 21:11 ` [Qemu-devel] [PATCH v4 24/26] qidl: add QAPI-based code generator Michael Roth
2012-10-15  8:12   ` Paolo Bonzini
2012-10-15 13:08     ` Paolo Bonzini
2012-10-15 16:35       ` Michael Roth
2012-10-15 19:37         ` Michael Roth
2012-10-16  7:20         ` Paolo Bonzini
2012-10-19  3:06           ` Michael Roth
2012-10-19  9:01             ` Paolo Bonzini
2012-10-12 21:11 ` [Qemu-devel] [PATCH v4 25/26] qidl: qidl.h, definitions for qidl annotations Michael Roth
2012-10-12 21:11 ` [Qemu-devel] [PATCH v4 26/26] qidl: unit tests and build infrastructure Michael Roth
2012-10-15 10:05   ` Paolo Bonzini
2012-10-15 16:37     ` Michael Roth
2012-10-16  7:21       ` Paolo Bonzini
2012-10-19  3:12         ` Michael Roth
2012-10-15  8:09 ` [Qemu-devel] [PATCH v4 00/26] Add infrastructure for QIDL-based device serialization Paolo Bonzini

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).