qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [Qemu-devel] [PATCH v2] Add infrastructure for QIDL-based device serialization
@ 2012-09-21 14:07 Michael Roth
  2012-09-21 14:07 ` [Qemu-devel] [PATCH 01/22] qapi: qapi-visit.py -> qapi_visit.py so we can import Michael Roth
                   ` (22 more replies)
  0 siblings, 23 replies; 40+ messages in thread
From: Michael Roth @ 2012-09-21 14:07 UTC (permalink / raw)
  To: qemu-devel; +Cc: blauwirbel, peter.maydell, aliguori, eblake

These patches are based are origin/master, and can also be obtained from:

git://github.com/mdroth/qemu.git qidl-base-v2

Changes since v1:

 - Simplified declaration format for QIDL-fied structures (Anthony, Blue)
 - Documentations fix-ups and clarifications (Eric, Peter)
 - Reduced build-time impact of QIDL by scanning for a QIDL_ENABLED()
   directive in .c files before running them through the
   the preprocessor and QIDL parser, and using a Makefile-set cflag
   to avoid declaring QIDL-related code/structures for files that don't
   include the directive.
 - Moved lexer functionality into a standalone lexer class, simplified
   interface to avoid the need to track offsets into the token stream.
 - Fixed an issue when deserializing a static array of structs using the
   new visit_type_carray() interfaces
 - Included a fix for a qom-fuse bug caused by multiple threads contending
   for QMP responses
 - Added brief descriptions of annotations to qidl.h
 - Minor clean-ups to the QIDL parser code

Changes since rfc v2:

 - Parser/Codegen fix-ups for cases encountered converting piix ide and usb.
 - Fixed license headers.
 - Stricter arg-checking for QIDL macros when passing to codegen.
 - Renamed QAPI visit_*_array interfaces to visit_*_carray to clarify that
   these are serialization routines for single-dimension C arrays.

These patches add infrastructure and unit tests for QIDL, which provides
a serialization framework for QEMU device structures by generating visitor
routines for device structs based on simple field annotations. Examples of
how this is done are included in patch 17, but, for brevity, a sample struct
such as this:

    typedef struct SerialDevice {
        SysBusDevice parent;

        uint8_t thr;            /* transmit holding register */
        uint8_t lsr;            /* line status register */
        uint8_t ier;            /* interrupt enable register */

        int int_pending;        /* whether we have a pending queued interrupt */
        CharDriverState *chr;   /* backend */
    } SerialDevice;

can now be made serializable with the following changes:

    typedef struct SerialDevice SerialDevice;

    QIDL_DECLARE(SerialDevice) {
        SysBusDevice parent;

        uint8_t thr;              /* transmit holding register */
        uint8_t lsr;              /* line status register */
        uint8_t ier;              /* interrupt enable register */

        int int_pending qDerived; /* whether we have a pending queued interrupt */
        CharDriverState *chr qImmutable; /* backend */
    };

To make use of generated visitor code, and .c file need only call the
QIDL_ENABLE() somewhere in the code body, which will then give it access to
visitor routines for any QIDL-ified device structures in that file, or included
from other files. These routines can then be used for
serialization/deserialization of the device state in a manner suitable for tasks
such as introspection/testing (generally via a r/w 'state' QOM property) or
migration.

The overall goal is to expose all migrateable device state in this manner so
that we can decouple serialization duties from savevm/VMState and convert them
into a stable, code-agnostic wire protocol, relying instead on intermediate
translation routines to handle the work of massaging serialized state into a
format suitable for the wire and vice-versa.

The following WIP branch contains the first set of QIDL conversions:

https://github.com/mdroth/qemu/commits/qidl

So far i440fx, pcibus, cirrus_vga, uhci, rtc, and isa/piix ide have been
converted, and I'm hoping to have most common PC devices converted over
within the next few weeks.

Please review. Comments/suggestions are very welcome.

 Makefile                                       |   26 +-
 QMP/qom-fuse                                   |   39 ++-
 docs/qidl.txt                                  |  347 +++++++++++++++++++
 hw/qdev-properties.h                           |  151 ++++++++
 hw/qdev.h                                      |  126 +------
 module.h                                       |    2 +
 qapi/Makefile.objs                             |    1 +
 qapi/misc-qapi-visit.c                         |   14 +
 qapi/qapi-visit-core.c                         |   25 ++
 qapi/qapi-visit-core.h                         |   11 +
 qapi/qmp-input-visitor.c                       |   32 +-
 qapi/qmp-output-visitor.c                      |   20 ++
 qidl.h                                         |  113 ++++++
 rules.mak                                      |   20 +-
 scripts/lexer.py                               |  306 ++++++++++++++++
 scripts/qapi-visit.py                          |  364 -------------------
 scripts/qapi.py                                |   10 +-
 scripts/{qapi-commands.py => qapi_commands.py} |    8 +-
 scripts/{qapi-types.py => qapi_types.py}       |    2 +-
 scripts/qapi_visit.py                          |  443 ++++++++++++++++++++++++
 scripts/qidl.py                                |  281 +++++++++++++++
 scripts/qidl_parser.py                         |  262 ++++++++++++++
 tests/Makefile                                 |   20 +-
 tests/test-qidl-included.h                     |   31 ++
 tests/test-qidl-linked.c                       |   93 +++++
 tests/test-qidl-linked.h                       |   18 +
 tests/test-qidl.c                              |  187 ++++++++++
 vl.c                                           |    1 +
 28 files changed, 2425 insertions(+), 528 deletions(-)

^ permalink raw reply	[flat|nested] 40+ messages in thread
* [Qemu-devel] [RFC v2] Use QEMU IDL for device serialization/introspection
@ 2012-07-24 17:20 Michael Roth
  2012-07-24 17:20 ` [Qemu-devel] [PATCH 09/22] qapi: QmpOutputVisitor, implement array handling Michael Roth
  0 siblings, 1 reply; 40+ messages in thread
From: Michael Roth @ 2012-07-24 17:20 UTC (permalink / raw)
  To: qemu-devel
  Cc: aliguori, quintela, owasserm, yamahata, pbonzini, akong, afaerber

These patches are based on origin/master, and can also be obtained from:

git://github.com/mdroth/qemu.git qidl-rfc2

= CHANGES SINCE RFC V1 =

 - QIDL'd cirrus_vga as a more interesting example
 - add support for QIDL-defined properties
 - pass all files through preprocessor before parsing QIDL annotations to
   gracefully handle #define's/#if's/etc
 - use code-injection/macros to make generated visitor/property code
   available to each compilation unit
 - drop schema generation, instead, expose the QIDL-generated, binary-specific
   schemas via QOM to make it easier to analyze migration compatibility given
   2 versions of QEMU
 - drop VMSTATE code generation, focus on exposing enough state that VMSTATE
   can be eventually be layered on top using qom_path + json_path to access
   all required fields
 - wrap annotations in QIDL(...) to better support arguments and namespacing
 - seperate parser (qidl_parser.py) from code-generation (qidl.py)
 - fix various parsing corner cases:
    - handling of embedded, non-pointer structs and nested struct declarations
    - handling of unsigned/union/int types
    - support expressions in QIDL() parameters and field declarations (to
      be evaluated in the context of the generated visitor function)

= OVERVIEW =

The goal of these patches is to explore how we can leverage an IDL to improve
device serialization/migration.

Patches 1-17 are all infrastructure, and the initial commits for the QIDL
parser/code generator/documentation.

Patches 18+ are our first users: the unit tests, RTC, and cirrus_vga.

After annotating a QOM device with QIDL you can up with something
this (using QOM-FUSE as an example):

    mdroth@loki:~/w/qom$ ls -l qidl/schemas/
    total 0
    drwxr-xr-x 2 mdroth mdroth 4096 Dec 31  1969 CirrusVGAState
    drwxr-xr-x 2 mdroth mdroth 4096 Dec 31  1969 ISACirrusVGAState
    drwxr-xr-x 2 mdroth mdroth 4096 Dec 31  1969 PCICirrusVGAState
    drwxr-xr-x 2 mdroth mdroth 4096 Dec 31  1969 PCIDevice
    drwxr-xr-x 2 mdroth mdroth 4096 Dec 31  1969 RTCState
    -rw-r--r-- 1 mdroth mdroth 4096 Dec 31  1969 type
    drwxr-xr-x 2 mdroth mdroth 4096 Dec 31  1969 VGACommonState

    mdroth@loki:~/w/qom$ ls -l machine/unattached/device\[8\]/
    total 0
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 addr
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 command_serr_enable
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 legacy-addr
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 legacy-command_serr_enable
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 legacy-multifunction
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 legacy-romfile
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 multifunction
    lrwxr-xr-x 2 mdroth mdroth   4096 Dec 31  1969 parent_bus -> ../../../machine/i440fx/pci.0
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 rombar
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 romfile
    -rw-r--r-- 1 mdroth mdroth 175236 Dec 31  1969 state
    lrwxr-xr-x 2 mdroth mdroth   4096 Dec 31  1969 state_schema -> ../../../qidl/schemas/PCICirrusVGAState
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 type

    mdroth@loki:~/w/qom$ cat machine/unattached/device\[8\]/state
    {u'cirrus_vga': {u'cirrus_blt_srcpitch': 0, u'cirrus_blt_width': 0, \
    u'cirrus_blt_bgcol': 0, u'last_hw_cursor_size': 0, \
    u'real_vram_size': 4194304, u'cirrus_blt_pixelwidth': 0, \
    u'cirrus_blt_mode': 0, u'last_hw_cursor_y_start': 0, \
    u'cirrus_blt_srcaddr': 0, u'cirrus_srccounter': 0, \
    u'cirrus_hidden_palette': [0, 0, 0, 0, 0, ...

At this point I'd like to push to get QIDL in place to expose guest-visible
device state via qom-get so it can be used for pre/post-migration
analysis/verification of device state. I think this is useful in-and-of
itself, and should be fairly low risk.

With the infrastructure in place we can then look at leveraging it for
vmstate and/or a new protocol, and also to figure out a nice way to add
capabilities-based transformations for post-serialize/pre-deserialize to
improve migration compatibility for any wire protocol (QEMUFile/vmstate
or otherwise) we layer on top. VMState at least can be done incrementally, so
we can begin looking at this immediately.

= General/Future Plans =

This is all very much open to discussion, and I only have a general idea of
how we can leverage this to improve migration compatibility/support. That
said:

With everything in place, we'd now have a means to serialize device state into
QObjects (or whatever). We can then perform whatever transformations/mark-up
we need to do (based on capabilities negotation centering around per-device
capabilities, for example), and send the state over the wire for migration.

The wire protocol is then simply a matter of how we encode the transformed
QObject. So a BER protocol could be implemented by creating something analagous
to the JSON-encoding facilities in qjson.c. Or, we could just keep using JSON,
perhaps with compression on top.

Eventually we can extend this approach to send device properties and encode the
actual composition tree in such a way that we can create machine machines on
the target side and avoid the need to duplicate the command-line invocation,
though that will take some substantial re-architecting/removal of the various
factory interfaces and further QOMification.

I'm not planning on incorporating memory migration into this, but it may be
possible to extend this approach to events/data blocks as well.

^ permalink raw reply	[flat|nested] 40+ messages in thread
end of thread, other threads:[~2012-09-26 15:13 UTC | newest]

Thread overview: 40+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-09-21 14:07 [Qemu-devel] [PATCH v2] Add infrastructure for QIDL-based device serialization Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 01/22] qapi: qapi-visit.py -> qapi_visit.py so we can import Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 02/22] qapi: qapi-types.py -> qapi_types.py Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 03/22] qapi: qapi-commands.py -> qapi_commands.py Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 04/22] qapi: qapi_visit.py, make code useable as module Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 05/22] qapi: qapi_visit.py, support arrays and complex qapi definitions Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 06/22] qapi: qapi_visit.py, support generating static functions Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 07/22] qapi: qapi_visit.py, support for visiting non-pointer/embedded structs Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 08/22] qapi: add visitor interfaces for C arrays Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 09/22] qapi: QmpOutputVisitor, implement array handling Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 10/22] qapi: QmpInputVisitor, " Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 11/22] qapi: qapi.py, make json parser more robust Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 12/22] qapi: add open-coded visitor for struct tm types Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 13/22] qom-fuse: force single-threaded mode to avoid QMP races Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 14/22] qom-fuse: workaround for truncated properties > 4096 Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 15/22] module additions for schema registration Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 16/22] qdev: move Property-related declarations to qdev-properties.h Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 17/22] qidl: add documentation Michael Roth
2012-09-21 23:16   ` Eric Blake
2012-09-21 14:07 ` [Qemu-devel] [PATCH 18/22] qidl: add lexer library (based on QC parser) Michael Roth
2012-09-21 23:18   ` Eric Blake
2012-09-21 23:52     ` Michael Roth
2012-09-25 21:09       ` Anthony Liguori
2012-09-21 14:07 ` [Qemu-devel] [PATCH 19/22] qidl: add C parser " Michael Roth
2012-09-21 23:19   ` Eric Blake
2012-09-21 14:07 ` [Qemu-devel] [PATCH 20/22] qidl: add QAPI-based code generator Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 21/22] qidl: qidl.h, definitions for qidl annotations Michael Roth
2012-09-21 14:07 ` [Qemu-devel] [PATCH 22/22] qidl: unit tests and build infrastructure Michael Roth
2012-09-21 15:57 ` [Qemu-devel] [PATCH v2] Add infrastructure for QIDL-based device serialization Paolo Bonzini
2012-09-21 16:24   ` Michael Roth
2012-09-22 14:33     ` Blue Swirl
2012-09-24 18:14       ` Michael Roth
2012-09-25  6:37         ` Paolo Bonzini
2012-09-25 15:45           ` Michael Roth
2012-09-25 21:12             ` Anthony Liguori
2012-09-26  9:57               ` Paolo Bonzini
2012-09-26 10:20               ` Kevin Wolf
2012-09-26 10:33                 ` Paolo Bonzini
2012-09-26 15:12                   ` Michael Roth
  -- strict thread matches above, loose matches on Subject: below --
2012-07-24 17:20 [Qemu-devel] [RFC v2] Use QEMU IDL for device serialization/introspection Michael Roth
2012-07-24 17:20 ` [Qemu-devel] [PATCH 09/22] qapi: QmpOutputVisitor, implement array handling Michael Roth

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).