qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [Qemu-devel] [RFC v2 00/22] QMP: out-of-band (OOB) execution support
@ 2017-09-29  3:38 Peter Xu
  2017-09-29  3:38 ` [Qemu-devel] [RFC v2 01/22] char-io: fix possible race on IOWatchPoll Peter Xu
                   ` (23 more replies)
  0 siblings, 24 replies; 48+ messages in thread
From: Peter Xu @ 2017-09-29  3:38 UTC (permalink / raw)
  To: qemu-devel
  Cc: Paolo Bonzini, Daniel P . Berrange, Stefan Hajnoczi, Fam Zheng,
	Juan Quintela, mdroth, peterx, Eric Blake, Laurent Vivier,
	Marc-André Lureau, Markus Armbruster,
	Dr . David Alan Gilbert

This is RFC version 2 of monitor OOB series.  It is based on following
series:

  [PATCH v4 0/5] iothread: allow to create internal iothreads
  (queued in Stefan's tree, pre-requisite for iothread usage)
  [PATCH v2 0/4] chardev: support non-default gcontext
  (queued in Paolo's tree, fix chardev reconnect issue)

Big changes from version 1:

* Use IOThread for monitor IO thread.

* Request queue switched from global queue to per-monitor queue, with
  basic flow control (currently 8 outstanding requests per monitor).

* Moved all documents into a single patch.

* Start to use "capabilities" field in QMP greeting message, to
  declare OOB support on server side.

* Added per-monitor response queue, so finally it looks like this:

               queue/kick              queue/kick
     JSON Parser ======> QMP Dispatcher =====> Responder
         /|\ |     (3)       /|\  |      (4)      | /|\
      (1) |  | (2)            |   |               |  |
          |  |                |  \|/           (6)|  |(5)
          |  |            main thread             |  |
          |  |                                    |  |
          |  +--------> monitor IO thread <-------+  |
          +-----------/                   \----------+

v2 changelog:
- use 10-char hash for git commit ID [Eric]
- instead of changing qstring_get_str(), add new
  qstring_get_try_str(), and rename qobject_get_str() to
  qobject_get_try_str() to follow the naming rule [Eric]
- add one more patch to let object_property_get_str() leverage the new
  qobject_get_try_str(). [Eric]
- introduce JSONMessageEmitFunc in the JSON parser patch [Eric]
- use per-monitor request queue, rather than a global queue
- add queue flow control, when full (current queue depth: 8), send
  event to notify client (two new patches added for this)
- stop monitor thread first before destroying monitor objects [Dan]
- document that client should drop responses with unknown IDs.
- move all the documents into a single standalone patch.
- use iothread for the monitor io thread.
- respond in IO thread, rather than the dispatcher
- new patch: in QMP greeting message, add 'oob' field in
  'capabilities', so that client can explicitly know whether server
  supports oob

TODO:
- qmp qtest for oob

Please review.  Thanks,

========== below is original RFC cover letter ===============

This series was born from this one:

  https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html

The design comes from Markus, and also the whole-bunch-of discussions
in previous thread.  My heartful thanks to Markus, Daniel, Dave,
Stefan, etc. on discussing the topic (...again!), providing shiny
ideas and suggestions.  Finally we got such a solution that seems to
satisfy everyone.

I re-started the versioning since this series is totally different
from previous one.  Now it's version 1.

In case new reviewers come along the way without reading previous
discussions, I will try to do a summary on what this is all about.

What is OOB execution?
======================

It's the shortcut of Out-Of-Band execution, its name is given by
Markus.  It's a way to quickly execute a QMP request.  Say, originally
QMP is going throw these steps:

      JSON Parser --> QMP Dispatcher --> Respond
          /|\    (2)                (3)     |
       (1) |                               \|/ (4)
           +---------  main thread  --------+

The requests are executed by the so-called QMP-dispatcher after the
JSON is parsed.  If OOB is on, we run the command directly in the
parser and quickly returns.

Yeah I know in current code the parser calls dispatcher directly
(please see handle_qmp_command()).  However it's not true again after
this series (parser will has its own IO thread, and dispatcher will
still be run in main thread).  So this OOB does brings something
different.

There are more details on why OOB and the difference/relationship
between OOB, async QMP, block/general jobs, etc.. but IMHO that's
slightly out of topic (and believe me, it's not easy for me to
summarize that).  For more information, please refers to [1].

Summary ends here.

Some Implementation Details
===========================

Again, I mentioned that the old QMP workflow is this:

      JSON Parser --> QMP Dispatcher --> Respond
          /|\    (2)                (3)     |
       (1) |                               \|/ (4)
           +---------  main thread  --------+

What this series does is, firstly:

      JSON Parser     QMP Dispatcher --> Respond
          /|\ |           /|\       (4)     |
           |  | (2)        | (3)            |  (5)
       (1) |  +----->      |               \|/
           +---------  main thread  <-------+

And further:

               queue/kick
     JSON Parser ======> QMP Dispatcher --> Respond
         /|\ |     (3)       /|\        (4)    |
      (1) |  | (2)            |                |  (5)
          | \|/               |               \|/
        IO thread         main thread  <-------+

Then it introduced the "allow-oob" parameter in QAPI schema to define
commands, and "run-oob" flag to let oob-allowed command to run in the
parser.

The last patch enables this for "migrate-incoming" command.

Please review.  Thanks.

[1] https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html

Peter Xu (22):
  char-io: fix possible race on IOWatchPoll
  qobject: introduce qstring_get_try_str()
  qobject: introduce qobject_get_try_str()
  qobject: let object_property_get_str() use new API
  monitor: move skip_flush into monitor_data_init
  qjson: add "opaque" field to JSONMessageParser
  monitor: move the cur_mon hack deeper for QMP
  monitor: unify global init
  monitor: create monitor dedicate iothread
  monitor: allow to use IO thread for parsing
  monitor: introduce monitor_qmp_respond()
  monitor: let mon_list be tail queue
  monitor: separate QMP parser and dispatcher
  qmp: add new event "request-dropped"
  monitor: send event when request queue full
  monitor: enable IO thread for (qmp & !mux) typed
  qapi: introduce new cmd option "allow-oob"
  qmp: support out-of-band (oob) execution
  qmp: let migrate-incoming allow out-of-band
  qmp: isolate responses into io thread
  qmp: introduce QMPCapability
  docs: update QMP documents for OOB commands

 chardev/char-io.c                |  16 +-
 docs/devel/qapi-code-gen.txt     |  51 ++++-
 docs/interop/qmp-spec.txt        |  24 ++-
 include/monitor/monitor.h        |   2 +-
 include/qapi/qmp/dispatch.h      |   2 +
 include/qapi/qmp/json-streamer.h |  10 +-
 include/qapi/qmp/qstring.h       |   2 +
 monitor.c                        | 441 ++++++++++++++++++++++++++++++++-------
 qapi-schema.json                 |  48 +++++
 qapi/introspect.json             |   6 +-
 qapi/migration.json              |   3 +-
 qapi/qmp-dispatch.c              |  34 +++
 qga/main.c                       |   5 +-
 qobject/json-streamer.c          |   6 +-
 qobject/qjson.c                  |   5 +-
 qobject/qstring.c                |  21 ++
 qom/object.c                     |   9 +-
 scripts/qapi-commands.py         |  19 +-
 scripts/qapi-introspect.py       |  10 +-
 scripts/qapi.py                  |  15 +-
 scripts/qapi2texi.py             |   2 +-
 tests/libqtest.c                 |   5 +-
 tests/qapi-schema/test-qapi.py   |   2 +-
 tests/qmp-test.c                 |   2 +-
 trace-events                     |   2 +
 vl.c                             |   3 +-
 26 files changed, 630 insertions(+), 115 deletions(-)

-- 
2.13.5

^ permalink raw reply	[flat|nested] 48+ messages in thread

end of thread, other threads:[~2017-10-23  6:08 UTC | newest]

Thread overview: 48+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-09-29  3:38 [Qemu-devel] [RFC v2 00/22] QMP: out-of-band (OOB) execution support Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 01/22] char-io: fix possible race on IOWatchPoll Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 02/22] qobject: introduce qstring_get_try_str() Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 03/22] qobject: introduce qobject_get_try_str() Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 04/22] qobject: let object_property_get_str() use new API Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 05/22] monitor: move skip_flush into monitor_data_init Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 06/22] qjson: add "opaque" field to JSONMessageParser Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 07/22] monitor: move the cur_mon hack deeper for QMP Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 08/22] monitor: unify global init Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 09/22] monitor: create monitor dedicate iothread Peter Xu
2017-10-12 12:29   ` Stefan Hajnoczi
2017-10-16  7:16     ` Peter Xu
2017-10-18 15:32       ` Stefan Hajnoczi
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 10/22] monitor: allow to use IO thread for parsing Peter Xu
2017-10-12 12:35   ` Stefan Hajnoczi
2017-10-16  7:37     ` Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 11/22] monitor: introduce monitor_qmp_respond() Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 12/22] monitor: let mon_list be tail queue Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 13/22] monitor: separate QMP parser and dispatcher Peter Xu
2017-10-12 12:50   ` Stefan Hajnoczi
2017-10-16  7:50     ` Peter Xu
2017-10-18 15:31       ` Stefan Hajnoczi
2017-10-19  6:36         ` Peter Xu
2017-10-19 13:13           ` Stefan Hajnoczi
2017-10-20  9:19             ` Paolo Bonzini
2017-10-23  6:07               ` Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 14/22] qmp: add new event "request-dropped" Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 15/22] monitor: send event when request queue full Peter Xu
2017-10-12 12:56   ` Stefan Hajnoczi
2017-10-16  8:11     ` Peter Xu
2017-10-18 15:28       ` Stefan Hajnoczi
2017-10-19  7:16         ` Peter Xu
2017-10-19 13:11           ` Stefan Hajnoczi
2017-10-20  4:26             ` Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 16/22] monitor: enable IO thread for (qmp & !mux) typed Peter Xu
2017-10-12 12:57   ` Stefan Hajnoczi
2017-10-16  8:16     ` Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 17/22] qapi: introduce new cmd option "allow-oob" Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 18/22] qmp: support out-of-band (oob) execution Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 19/22] qmp: let migrate-incoming allow out-of-band Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 20/22] qmp: isolate responses into io thread Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 21/22] qmp: introduce QMPCapability Peter Xu
2017-09-29  3:38 ` [Qemu-devel] [RFC v2 22/22] docs: update QMP documents for OOB commands Peter Xu
2017-09-29  3:58 ` [Qemu-devel] [RFC v2 00/22] QMP: out-of-band (OOB) execution support no-reply
2017-09-29  4:14   ` Peter Xu
2017-09-29 19:03     ` Eric Blake
2017-09-30  0:28       ` Peter Xu
2017-09-29  4:20 ` no-reply

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).