[PULL 0/6] QAPI patches patches for 2025-02-26

[PULL 0/6] QAPI patches patches for 2025-02-26

[Markus Armbruster]

The following changes since commit b69801dd6b1eb4d107f7c2f643adf0a4e3ec9124:

  Merge tag 'for_upstream' of https://git.kernel.org/pub/scm/virt/kvm/mst/qemu into staging (2025-02-22 05:06:39 +0800)

are available in the Git repository at:

  https://repo.or.cz/qemu/armbru.git tags/pull-qapi-2025-02-26

for you to fetch changes up to 4bdf4e66233a7c476b934a1c4776ef7d136ff867:

  qapi: pluggable backend code generators (2025-02-26 11:12:46 +0100)

----------------------------------------------------------------
QAPI patches patches for 2025-02-26

----------------------------------------------------------------
Daniel P. Berrangé (1):
      qapi: pluggable backend code generators

John Snow (4):
      qapi: update pylintrc config
      python: add qapi static analysis tests
      docs/qapidoc: support header-less freeform sections
      docs/qapidoc: remove example section support

Roman Penyaev (1):
      qapi/char.json: minor doc rewording for `hub` device

 docs/sphinx/qapidoc.py          | 11 ++----
 qapi/char.json                  |  2 +-
 python/tests/qapi-flake8.sh     |  2 ++
 python/tests/qapi-isort.sh      |  2 ++
 python/tests/qapi-mypy.sh       |  2 ++
 python/tests/qapi-pylint.sh     |  4 +++
 scripts/qapi/backend.py         | 63 ++++++++++++++++++++++++++++++++
 scripts/qapi/main.py            | 80 +++++++++++++++++++++++------------------
 scripts/qapi/pylintrc           |  1 +
 tests/qapi-schema/doc-good.json |  4 +++
 tests/qapi-schema/doc-good.out  |  3 ++
 11 files changed, 130 insertions(+), 44 deletions(-)
 create mode 100755 python/tests/qapi-flake8.sh
 create mode 100755 python/tests/qapi-isort.sh
 create mode 100755 python/tests/qapi-mypy.sh
 create mode 100755 python/tests/qapi-pylint.sh
 create mode 100644 scripts/qapi/backend.py

-- 
2.48.1

[PULL 1/6] qapi/char.json: minor doc rewording for `hub` device

[Markus Armbruster]

From: Roman Penyaev <r.peniaev@gmail.com>

Refine documentation for the hub device, specify the maximum.

Signed-off-by: Roman Penyaev <r.peniaev@gmail.com>
Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Cc: qemu-devel@nongnu.org
Message-ID: <20250219090607.559887-1-r.peniaev@gmail.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 qapi/char.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/qapi/char.json b/qapi/char.json
index f02b66c06b..dde2f9538f 100644
--- a/qapi/char.json
+++ b/qapi/char.json
@@ -337,7 +337,7 @@
 #
 # Configuration info for hub chardevs.
 #
-# @chardevs: List of chardev IDs, which should be added to this hub
+# @chardevs: IDs to be added to this hub (maximum 4 devices).
 #
 # Since: 10.0
 ##
-- 
2.48.1

[PULL 2/6] qapi: update pylintrc config

[Markus Armbruster]

From: John Snow <jsnow@redhat.com>

If you've got a newer pylint, it'll whine about positional arguments
separately from the regular ones. Update the configuration to ignore
both categories of warning.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250224033741.222749-2-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi/pylintrc | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index c028a1f9f5..d24eece741 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -17,6 +17,7 @@ disable=consider-using-f-string,
         too-many-arguments,
         too-many-branches,
         too-many-instance-attributes,
+        too-many-positional-arguments,
         too-many-statements,
         useless-option-value,
 
-- 
2.48.1

[PULL 3/6] python: add qapi static analysis tests

[Markus Armbruster]

From: John Snow <jsnow@redhat.com>

Update the python tests to also check qapi. No idea why I didn't do this
before. I guess I was counting on moving it under python/ and then just
forgot after that was NACKed. Oops, this turns out to be really easy.

flake8, isort and mypy use the tool configuration from the existing
python directory. pylint continues to use the special configuration
located in scripts/qapi/ - that configuration is more permissive. If we
wish to unify the two configurations, that's a separate series and a
discussion for a later date.

As a result of this patch, one would be able to run any of the following
tests locally from the qemu.git/python directory and have it cover the
scripts/qapi/ module as well. All of the following options run the
python tests, static analysis tests, and linter checks; but with
different combinations of dependencies and interpreters.

- "make check-minreqs" Run tests specifically under our oldest supported
  Python and our oldest supported dependencies. This is the test that
  runs on GitLab as "check-python-minreqs". This helps ensure we do not
  regress support on older platforms accidentally.

- "make check-tox" Runs the tests under the newest supported
  dependencies, but under each supported version of Python in turn. At
  time of writing, this is Python 3.8 to 3.13 inclusive. This test helps
  catch bleeding-edge problems before they become problems for developer
  workstations. This is the GitLab test "check-python-tox" and is an
  optionally run, may-fail test due to the unpredictable nature of new
  dependencies being released into the ecosystem that may cause
  regressions.

- "make check-dev" Runs the tests under the newest supported
  dependencies using whatever version of Python the user happens to have
  installed. This is a quick convenience check that does not map to any
  particular GitLab test.

(Note! check-dev may be busted on Fedora 41 and bleeding edge versions
of setuptools. That's unrelated to this patch and I'll address it
separately and soon. Thank you for your patience, --mgmt)

Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250224033741.222749-3-jsnow@redhat.com>
Acked-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 python/tests/qapi-flake8.sh | 2 ++
 python/tests/qapi-isort.sh  | 2 ++
 python/tests/qapi-mypy.sh   | 2 ++
 python/tests/qapi-pylint.sh | 4 ++++
 4 files changed, 10 insertions(+)
 create mode 100755 python/tests/qapi-flake8.sh
 create mode 100755 python/tests/qapi-isort.sh
 create mode 100755 python/tests/qapi-mypy.sh
 create mode 100755 python/tests/qapi-pylint.sh

diff --git a/python/tests/qapi-flake8.sh b/python/tests/qapi-flake8.sh
new file mode 100755
index 0000000000..51916a9019
--- /dev/null
+++ b/python/tests/qapi-flake8.sh
@@ -0,0 +1,2 @@
+#!/bin/sh -e
+python3 -m flake8 ../scripts/qapi/
diff --git a/python/tests/qapi-isort.sh b/python/tests/qapi-isort.sh
new file mode 100755
index 0000000000..60ed5eeb34
--- /dev/null
+++ b/python/tests/qapi-isort.sh
@@ -0,0 +1,2 @@
+#!/bin/sh -e
+python3 -m isort --sp . -c ../scripts/qapi/
diff --git a/python/tests/qapi-mypy.sh b/python/tests/qapi-mypy.sh
new file mode 100755
index 0000000000..377b29b873
--- /dev/null
+++ b/python/tests/qapi-mypy.sh
@@ -0,0 +1,2 @@
+#!/bin/sh -e
+python3 -m mypy ../scripts/qapi
diff --git a/python/tests/qapi-pylint.sh b/python/tests/qapi-pylint.sh
new file mode 100755
index 0000000000..d4869578e5
--- /dev/null
+++ b/python/tests/qapi-pylint.sh
@@ -0,0 +1,4 @@
+#!/bin/sh -e
+SETUPTOOLS_USE_DISTUTILS=stdlib python3 -m pylint \
+                                --rcfile=../scripts/qapi/pylintrc \
+                                ../scripts/qapi/
-- 
2.48.1

[PULL 4/6] docs/qapidoc: support header-less freeform sections

[Markus Armbruster]

From: John Snow <jsnow@redhat.com>

The code as written crashes when a free-form documentation block doesn't
start with a heading or subheading, for example:

| ##
| # Just text, no heading.
| ##

The code will attempt to use the `node` variable uninitialized. To fix,
create a generic block to insert the doc text into.

(This patch also removes a lingering pylint warning in the QAPIDoc
implementation that prevents getting a clean baseline to use for
forthcoming additions.)

Fixes: 43e0d14ee09a (docs/sphinx: fix extra stuff in TOC after freeform QMP sections)
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250224033741.222749-5-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Test updated to cover this]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 docs/sphinx/qapidoc.py          | 2 ++
 tests/qapi-schema/doc-good.json | 4 ++++
 tests/qapi-schema/doc-good.out  | 3 +++
 3 files changed, 9 insertions(+)

diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index 5f96b46270..5a4d7388b2 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -421,6 +421,8 @@ def freeform(self, doc):
             node = self._start_new_heading(heading, len(leader))
             if text == '':
                 return
+        else:
+            node = nodes.container()
 
         self._parse_text_into_node(text, node)
         self._cur_doc = None
diff --git a/tests/qapi-schema/doc-good.json b/tests/qapi-schema/doc-good.json
index f64bf38d85..0a4f139f83 100644
--- a/tests/qapi-schema/doc-good.json
+++ b/tests/qapi-schema/doc-good.json
@@ -11,6 +11,10 @@
 # = Section
 ##
 
+##
+# Just text, no heading.
+##
+
 ##
 # == Subsection
 #
diff --git a/tests/qapi-schema/doc-good.out b/tests/qapi-schema/doc-good.out
index ec277be91e..0a9da3efde 100644
--- a/tests/qapi-schema/doc-good.out
+++ b/tests/qapi-schema/doc-good.out
@@ -56,6 +56,9 @@ event EVT_BOXED Object
 doc freeform
     body=
 = Section
+doc freeform
+    body=
+Just text, no heading.
 doc freeform
     body=
 == Subsection
-- 
2.48.1

[PULL 5/6] docs/qapidoc: remove example section support

[Markus Armbruster]

From: John Snow <jsnow@redhat.com>

Since commit 3c5f6114 (qapi: remove "Example" doc section), Example
sections no longer exist, so this support in qapidoc is now dead code.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250224033741.222749-7-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 docs/sphinx/qapidoc.py | 9 +--------
 1 file changed, 1 insertion(+), 8 deletions(-)

diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index 5a4d7388b2..61997fd21a 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -254,10 +254,6 @@ def _nodes_for_features(self, doc):
         section += dlnode
         return [section]
 
-    def _nodes_for_example(self, exampletext):
-        """Return list of doctree nodes for a code example snippet"""
-        return [nodes.literal_block(exampletext, exampletext)]
-
     def _nodes_for_sections(self, doc):
         """Return list of doctree nodes for additional sections"""
         nodelist = []
@@ -275,10 +271,7 @@ def _nodes_for_sections(self, doc):
                 continue
 
             snode = self._make_section(section.tag)
-            if section.tag.startswith('Example'):
-                snode += self._nodes_for_example(dedent(section.text))
-            else:
-                self._parse_text_into_node(dedent(section.text), snode)
+            self._parse_text_into_node(dedent(section.text), snode)
             nodelist.append(snode)
         return nodelist
 
-- 
2.48.1

[PULL 6/6] qapi: pluggable backend code generators

[Markus Armbruster]

From: Daniel P. Berrangé <berrange@redhat.com>

The 'qapi.backend.QAPIBackend' class defines an API contract for code
generators. The current generator is put into a new class
'qapi.backend.QAPICBackend' and made to be the default impl.

A custom generator can be requested using the '-k' arg which takes a
fully qualified python class name

   qapi-gen.py -B the.python.module.QAPIMyBackend

This allows out of tree code to use the QAPI generator infrastructure
to create new language bindings for QAPI schemas. This has the caveat
that the QAPI generator APIs are not guaranteed stable, so consumers
of this feature may have to update their code to be compatible with
future QEMU releases.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20250224182030.2089959-1-berrange@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Error checking and messages tweaked]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
---
 scripts/qapi/backend.py | 63 ++++++++++++++++++++++++++++++++
 scripts/qapi/main.py    | 80 +++++++++++++++++++++++------------------
 2 files changed, 108 insertions(+), 35 deletions(-)
 create mode 100644 scripts/qapi/backend.py

diff --git a/scripts/qapi/backend.py b/scripts/qapi/backend.py
new file mode 100644
index 0000000000..14e60aa67a
--- /dev/null
+++ b/scripts/qapi/backend.py
@@ -0,0 +1,63 @@
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+from abc import ABC, abstractmethod
+
+from .commands import gen_commands
+from .events import gen_events
+from .features import gen_features
+from .introspect import gen_introspect
+from .schema import QAPISchema
+from .types import gen_types
+from .visit import gen_visit
+
+
+class QAPIBackend(ABC):
+
+    @abstractmethod
+    def generate(self,
+                 schema: QAPISchema,
+                 output_dir: str,
+                 prefix: str,
+                 unmask: bool,
+                 builtins: bool,
+                 gen_tracing: bool) -> None:
+        """
+        Generate code for the given schema into the target directory.
+
+        :param schema: The primary QAPI schema object.
+        :param output_dir: The output directory to store generated code.
+        :param prefix: Optional C-code prefix for symbol names.
+        :param unmask: Expose non-ABI names through introspection?
+        :param builtins: Generate code for built-in types?
+
+        :raise QAPIError: On failures.
+        """
+
+
+class QAPICBackend(QAPIBackend):
+
+    def generate(self,
+                 schema: QAPISchema,
+                 output_dir: str,
+                 prefix: str,
+                 unmask: bool,
+                 builtins: bool,
+                 gen_tracing: bool) -> None:
+        """
+        Generate C code for the given schema into the target directory.
+
+        :param schema_file: The primary QAPI schema file.
+        :param output_dir: The output directory to store generated code.
+        :param prefix: Optional C-code prefix for symbol names.
+        :param unmask: Expose non-ABI names through introspection?
+        :param builtins: Generate code for built-in types?
+
+        :raise QAPIError: On failures.
+        """
+        gen_types(schema, output_dir, prefix, builtins)
+        gen_features(schema, output_dir, prefix)
+        gen_visit(schema, output_dir, prefix, builtins)
+        gen_commands(schema, output_dir, prefix, gen_tracing)
+        gen_events(schema, output_dir, prefix)
+        gen_introspect(schema, output_dir, prefix, unmask)
diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py
index 324081b9fc..ff42b43cd6 100644
--- a/scripts/qapi/main.py
+++ b/scripts/qapi/main.py
@@ -8,18 +8,14 @@
 """
 
 import argparse
+from importlib import import_module
 import sys
 from typing import Optional
 
-from .commands import gen_commands
+from .backend import QAPIBackend, QAPICBackend
 from .common import must_match
 from .error import QAPIError
-from .events import gen_events
-from .features import gen_features
-from .introspect import gen_introspect
 from .schema import QAPISchema
-from .types import gen_types
-from .visit import gen_visit
 
 
 def invalid_prefix_char(prefix: str) -> Optional[str]:
@@ -29,32 +25,42 @@ def invalid_prefix_char(prefix: str) -> Optional[str]:
     return None
 
 
-def generate(schema_file: str,
-             output_dir: str,
-             prefix: str,
-             unmask: bool = False,
-             builtins: bool = False,
-             gen_tracing: bool = False) -> None:
-    """
-    Generate C code for the given schema into the target directory.
+def create_backend(path: str) -> QAPIBackend:
+    if path is None:
+        return QAPICBackend()
 
-    :param schema_file: The primary QAPI schema file.
-    :param output_dir: The output directory to store generated code.
-    :param prefix: Optional C-code prefix for symbol names.
-    :param unmask: Expose non-ABI names through introspection?
-    :param builtins: Generate code for built-in types?
+    module_path, dot, class_name = path.rpartition('.')
+    if not dot:
+        print(f"argument of -B must be of the form MODULE.CLASS",
+              file=sys.stderr)
+        sys.exit(1)
 
-    :raise QAPIError: On failures.
-    """
-    assert invalid_prefix_char(prefix) is None
+    try:
+        mod = import_module(module_path)
+    except Exception as ex:
+        print(f"unable to import '{module_path}': {ex}", file=sys.stderr)
+        sys.exit(1)
 
-    schema = QAPISchema(schema_file)
-    gen_types(schema, output_dir, prefix, builtins)
-    gen_features(schema, output_dir, prefix)
-    gen_visit(schema, output_dir, prefix, builtins)
-    gen_commands(schema, output_dir, prefix, gen_tracing)
-    gen_events(schema, output_dir, prefix)
-    gen_introspect(schema, output_dir, prefix, unmask)
+    try:
+        klass = getattr(mod, class_name)
+    except AttributeError:
+        print(f"module '{module_path}' has no class '{class_name}'",
+              file=sys.stderr)
+        sys.exit(1)
+
+    try:
+        backend = klass()
+    except Exception as ex:
+        print(f"backend '{path}' cannot be instantiated: {ex}",
+              file=sys.stderr)
+        sys.exit(1)
+
+    if not isinstance(backend, QAPIBackend):
+        print(f"backend '{path}' must be an instance of QAPIBackend",
+              file=sys.stderr)
+        sys.exit(1)
+
+    return backend
 
 
 def main() -> int:
@@ -77,6 +83,8 @@ def main() -> int:
     parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
                         dest='unmask',
                         help="expose non-ABI names in introspection")
+    parser.add_argument('-B', '--backend', default=None,
+                        help="Python module name for code generator")
 
     # Option --suppress-tracing exists so we can avoid solving build system
     # problems.  TODO Drop it when we no longer need it.
@@ -93,12 +101,14 @@ def main() -> int:
         return 1
 
     try:
-        generate(args.schema,
-                 output_dir=args.output_dir,
-                 prefix=args.prefix,
-                 unmask=args.unmask,
-                 builtins=args.builtins,
-                 gen_tracing=not args.suppress_tracing)
+        schema = QAPISchema(args.schema)
+        backend = create_backend(args.backend)
+        backend.generate(schema,
+                         output_dir=args.output_dir,
+                         prefix=args.prefix,
+                         unmask=args.unmask,
+                         builtins=args.builtins,
+                         gen_tracing=not args.suppress_tracing)
     except QAPIError as err:
         print(err, file=sys.stderr)
         return 1
-- 
2.48.1

Re: [PULL 3/6] python: add qapi static analysis tests

[Stefan Hajnoczi]

Hi John,
Please take a look at this CI failure:
https://gitlab.com/qemu-project/qemu/-/jobs/9284725716#L150

If you cannot reproduce it locally there is a chance that other pull
requests on the staging branch caused the errors. If that's the case,
please wait for the next update to qemu.git/master and rebase your
patch.

Thanks,
Stefan

Re: [PULL 3/6] python: add qapi static analysis tests

[Markus Armbruster]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> Hi John,
> Please take a look at this CI failure:
> https://gitlab.com/qemu-project/qemu/-/jobs/9284725716#L150
>
> If you cannot reproduce it locally there is a chance that other pull
> requests on the staging branch caused the errors. If that's the case,
> please wait for the next update to qemu.git/master and rebase your
> patch.

I'm having difficulties running CI locally, but I think I see what's
wrong.  I'll submit v2 without this patch, and submit a fixed version of
the patch with my next pyll request.

Thanks!