Summary of Re: Making QEMU easier for management tools and applications

[Markus Armbruster <armbru@redhat.com>]

This thread has accumulated more than a hundred messages over seven
weeks.  We need a summary.

The conversation has gone in several directions.  In this message, I'll
cover just the one I consider most important: machine-friendly initial
configuration.  I'll do the rest later, if I have stamina left.

= QMP is fine for machines, CLI is not =

We have two major external host interfaces: the CLI and QMP.  Compare:

QMP
* Purpose: control at run-time
* Commands with argument and return value
* Events with argument value
* Simple type system
* Based on QAPI
* Regular, well-defined syntax: QAPI schema on top of JSON
* Documentation is required; actual documentation is of mixed quality
* Introspectable, except for a few mostly QOM-related holes where we
  bypass QAPI
* Machine-friendly, complemented by human-only HMP

CLI
* Purpose: initial configuration
* Options with argument
* Based on a crazy zoo of QemuOpts (with and without dotted keys), QAPI
  (only a few recent options), ad hoc parsers
* Option argument syntax is mostly (variations of) key=value,...
* QemuOpts' "type system" is "list of key-value pairs, where value can
  be string, bool, (unsigned) integer, or size", optionally restricted
  to known keys with known value types.
* QAPI type system is the same as in QMP
* Introspection is completely inadequate: misses options, incorrect
  option names, misses option arguments partly or completely
* Configuration files with INI-like syntax, completely inadequate: can't
  do most options
* Not machine-friendly
* Maintaining it is a pain, evolving it is worse

The inadequacy of the CLI has become a serious issue.

= Ways to provide machine-friendly initial configuration =

Two ways to provide machine-friendly initial configuration on par with
QMP have been proposed:

1. Extend QMP

   Machines use the CLI only to configure a QMP socket.  The remainder
   of the CLI becomes human-only, with much relaxed compatibility rules.

2. QAPIfy the CLI

   Provide a machine-friendly CLI based on QAPI and JSON.  The current
   CLI becomes human-only, with much relaxed compatibility rules.

   Aside: I looked into cleaning up the human-only CLI at the same time,
   but the need to maintain compatibility until the transition to the
   machine-friendly CLI is complete makes this hard.  It needs to be
   cleaned up, though.  More on that below.

To extend QMP, we wrap QMP commands around the internal initial
configuration interfaces.  These QMP commands take arguments, but don't
return anything.  Many of them will only make sense during initial
configuration.  We'll want to express that in the schema, and enforce it
in the QMP core.  Others will behave differently, e.g. cold plug during
initial configuration, hot plug once the guest runs.

Configuration files are just QMP commands from the initial configuration
subset read from a file instead of a chardev.  JSON is a poor choice for
configuration files, and QMP's verbosity makes it poorer.  We'll want
more suitable concrete syntax for configuration files.

To improve the CLI, we wrap QAPI-based CLI options around the internal
initial configuration interfaces.  We'll want some infrastructure to
generate CLI option boilerplate, just like we generate QMP command
boilerplate.

Configuration files are just CLI options read from a file instead of
argv[].  Again, we'll want a more suitable concrete syntax there.

With an improved CLI, I'd expect machines to use configuration files so
they don't have to mess with shell quoting.  With extended QMP, they'd
perhaps rather reuse their existing QMP code to send the configuration
down a socket.  Less efficient, but I doubt it'll matter.

In both cases, the internal configuration interfaces need to be
converted from QemuOpts to QAPI types.

The one clear advantage of extending QMP is the ability to mix query
commands with initial configuration.

This is also a clear disadvantage: we need to make it work.  Existing
QMP commands may rely on initial configuration to be complete.  We may
have to allow only QMP commands we carefully checked.

How important is the advantage?

= Cleaning up the human-only CLI =

In both cases, the existing CLI remains in a human-only role.  That's a
truckload of messy code to maintain.  Not good.  Can we replace it by
thin wrappers around the machine-friendly alternative, ideally
incrementally?  Can we generate a useful part of these wrappers?

If we QAPIfy the CLI, the QAPI generator already knows the CLI, and only
has to be taught the general human-friendly key=value,... syntax, plus a
way to specify exceptions.

If we extend QMP, we'll additionally need some of the CLI QAPIfication
infrastructure for this.  Hmm.