[PATCH 00/12] qapi: add formal "intro" section

[PATCH 00/12] qapi: add formal "intro" section

[John Snow]

Hiya, this is a series that explores a potential syntax for a
designated "Intro" section. Markus knows why I want this, but for
everyone else: a designated "Introduction" section is useful for the
desired "inliner" feature for the new QAPI doc system. Commits explain
a bit more. This is prep work and doesn't really change anything
tangibly except source code syntax for the QAPI docs.

It is designed so that this conversion can happen incrementally with
no actual difference to the rendered manuals, so each QAPI module can
be converted one at a time for easier review and merging in an
arbitrary order.

This series demonstrates conversion of just four modules; if I'm given
a thumbs up, I will convert the rest of QAPI, one module (or
maintainer stanza) per patch like how I handled adding
cross-references.

John Snow (12):
  tests/qapi: generate output in source order
  qapi/docs: remove unused QAPIDoc subsection members
  qapi/docs: make remaining subsection members "private"
  qapi/docs: add "Intro" section
  qapi/docs: adjust stub member insertion algorithm
  qapi/docs: remove implicit Plain section
  qapi/docs: add "Intro" section parsing
  qapi/docs: Add rendering for INTRO sections
  qapi: convert intro sections for accelerator.json
  qapi: convert intro sections for acpi-hest.json
  qapi: convert intro sections for acpi.json
  qapi: convert intro sections for audio.json

 docs/devel/qapi-code-gen.rst            |  17 +-
 docs/sphinx/qapidoc.py                  |  52 +++---
 qapi/accelerator.json                   |  24 +--
 qapi/acpi-hest.json                     |   5 +-
 qapi/acpi.json                          |  23 +--
 qapi/audio.json                         | 112 ++++---------
 .editorconfig                           |   1 +
 scripts/qapi/parser.py                  | 204 ++++++++++++++----------
 tests/qapi-schema/doc-good.out          |  30 ++--
 tests/qapi-schema/doc-missing-colon.err |   2 +-
 tests/qapi-schema/test-qapi.py          |  15 +-
 11 files changed, 239 insertions(+), 246 deletions(-)

-- 
2.53.0

[PATCH 01/12] tests/qapi: generate output in source order

[John Snow]

Rewrite the test doc generator to produce output in source order instead
of arbitrarily by section name.

This patch removes our last use of the "body" field, which has an effect
on how the first plaintext section of each test documented is printed:
we now print "section=Plain" followed by the section text instead of
"body=[...]".

This patch is motivated by a desire to move the QAPIDoc API away from
named fields for specific sections in a bid to force all users to simply
iterate through all_sections in order, instead - and to remove the named
subsections.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qapi-schema/doc-good.out | 26 +++++++++++++-------------
 tests/qapi-schema/test-qapi.py | 15 ++++++++-------
 2 files changed, 21 insertions(+), 20 deletions(-)

diff --git a/tests/qapi-schema/doc-good.out b/tests/qapi-schema/doc-good.out
index 04a55072646..e2be6f96bbf 100644
--- a/tests/qapi-schema/doc-good.out
+++ b/tests/qapi-schema/doc-good.out
@@ -54,15 +54,15 @@ event EVT_BOXED Object
     boxed=True
     feature feat3
 doc freeform
-    body=
+    section=Plain
 *******
 Section
 *******
 doc freeform
-    body=
+    section=Plain
 Just text, no heading.
 doc freeform
-    body=
+    section=Plain
 Subsection
 ==========
 
@@ -106,7 +106,7 @@ Examples:
 - *verbatim*
 - {braces}
 doc symbol=Enum
-    body=
+    section=Plain
 
     arg=one
 The _one_ {and only}, description on the same line
@@ -119,13 +119,13 @@ a member feature
     section=Plain
 @two is undocumented
 doc symbol=Base
-    body=
+    section=Plain
 
     arg=base1
  description starts on a new line,
  minimally indented
 doc symbol=Variant1
-    body=
+    section=Plain
 A paragraph
 
 Another paragraph
@@ -138,15 +138,15 @@ a feature
     feature=member-feat
 a member feature
 doc symbol=Variant2
-    body=
+    section=Plain
 
 doc symbol=Object
-    body=
+    section=Plain
 
     feature=union-feat1
 a feature
 doc symbol=Alternate
-    body=
+    section=Plain
 
     arg=i
 description starts on the same line
@@ -157,11 +157,11 @@ description starts on the same line
     feature=alt-feat
 a feature
 doc freeform
-    body=
+    section=Plain
 Another subsection
 ==================
 doc symbol=cmd
-    body=
+    section=Plain
 
     arg=arg1
     description starts on a new line,
@@ -210,7 +210,7 @@ Note::
     section=Since
 2.10
 doc symbol=cmd-boxed
-    body=
+    section=Plain
 If you're bored enough to read this, go see a video of boxed cats
     feature=cmd-feat1
 a feature
@@ -223,7 +223,7 @@ another feature
 
    <- ... has no title ...
 doc symbol=EVT_BOXED
-    body=
+    section=Plain
 
     feature=feat3
 a feature
diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
index cf7fb8a6df5..27885147b4d 100755
--- a/tests/qapi-schema/test-qapi.py
+++ b/tests/qapi-schema/test-qapi.py
@@ -19,6 +19,7 @@
 from io import StringIO
 
 from qapi.error import QAPIError
+from qapi.parser import QAPIDoc
 from qapi.schema import QAPISchema, QAPISchemaVisitor
 
 
@@ -116,13 +117,13 @@ def test_frontend(fname):
             print('doc symbol=%s' % doc.symbol)
         else:
             print('doc freeform')
-        print('    body=\n%s' % doc.body.text)
-        for arg, section in doc.args.items():
-            print('    arg=%s\n%s' % (arg, section.text))
-        for feat, section in doc.features.items():
-            print('    feature=%s\n%s' % (feat, section.text))
-        for section in doc.sections:
-            print('    section=%s\n%s' % (section.kind, section.text))
+        for section in doc.all_sections:
+            if section.kind == QAPIDoc.Kind.MEMBER:
+                print('    arg=%s\n%s' % (section.name, section.text))
+            elif section.kind == QAPIDoc.Kind.FEATURE:
+                print('    feature=%s\n%s' % (section.name, section.text))
+            else:
+                print('    section=%s\n%s' % (section.kind, section.text))
 
 
 def open_test_result(dir_name, file_name, update):
-- 
2.53.0

[PATCH 02/12] qapi/docs: remove unused QAPIDoc subsection members

[John Snow]

All users of the QAPIDoc object should be iterating over all_sections
and not grabbing arbitrary subsections, if possible. Remove the 'body'
and 'sections' members, as they are no longer used.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/parser.py | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index c3cf33904ef..b33edbba74f 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -732,8 +732,6 @@ def __init__(self, info: QAPISourceInfo, symbol: Optional[str] = None):
         self.all_sections: List[QAPIDoc.Section] = [
             QAPIDoc.Section(info, QAPIDoc.Kind.PLAIN)
         ]
-        # the body section
-        self.body: Optional[QAPIDoc.Section] = self.all_sections[0]
         # dicts mapping parameter/feature names to their description
         self.args: Dict[str, QAPIDoc.ArgSection] = {}
         self.features: Dict[str, QAPIDoc.ArgSection] = {}
@@ -742,8 +740,6 @@ def __init__(self, info: QAPISourceInfo, symbol: Optional[str] = None):
         self.errors: Optional[QAPIDoc.Section] = None
         # "Since" section
         self.since: Optional[QAPIDoc.Section] = None
-        # sections other than .body, .args, .features
-        self.sections: List[QAPIDoc.Section] = []
 
     def end(self) -> None:
         for section in self.all_sections:
@@ -766,7 +762,6 @@ def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
 
         # start new section
         section = self.Section(info, kind)
-        self.sections.append(section)
         self.all_sections.append(section)
 
     def new_tagged_section(
@@ -790,7 +785,6 @@ def new_tagged_section(
                 raise QAPISemError(
                     info, "duplicated '%s' section" % kind)
             self.since = section
-        self.sections.append(section)
         self.all_sections.append(section)
 
     def _new_description(
-- 
2.53.0

[PATCH 03/12] qapi/docs: make remaining subsection members "private"

[John Snow]

These fields are used to provide error checking and internal logistics
and should not be used by a user of the library to directly access
documentation sections, so make them private.

The "since" field alone is left public, as the qapidoc generator does
use this field to pull that section out of the regular flow of the
document.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/parser.py | 54 +++++++++++++++++++++++-------------------
 1 file changed, 29 insertions(+), 25 deletions(-)

diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index b33edbba74f..da4756a7424 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -555,7 +555,7 @@ def get_doc(self) -> 'QAPIDoc':
                     break
                 # Non-blank line, first of a section
                 if line == 'Features:':
-                    if doc.features:
+                    if doc.has_features:
                         raise QAPIParseError(
                             self, "duplicated 'Features:' line")
                     self.accept(False)
@@ -570,7 +570,7 @@ def get_doc(self) -> 'QAPIDoc':
                         if text:
                             doc.append_line(text)
                         line = self.get_doc_indented(doc)
-                    if not doc.features:
+                    if not doc.has_features:
                         raise QAPIParseError(
                             self, 'feature descriptions expected')
                     no_more_args = True
@@ -733,14 +733,18 @@ def __init__(self, info: QAPISourceInfo, symbol: Optional[str] = None):
             QAPIDoc.Section(info, QAPIDoc.Kind.PLAIN)
         ]
         # dicts mapping parameter/feature names to their description
-        self.args: Dict[str, QAPIDoc.ArgSection] = {}
-        self.features: Dict[str, QAPIDoc.ArgSection] = {}
+        self._args: Dict[str, QAPIDoc.ArgSection] = {}
+        self._features: Dict[str, QAPIDoc.ArgSection] = {}
         # a command's "Returns" and "Errors" section
-        self.returns: Optional[QAPIDoc.Section] = None
-        self.errors: Optional[QAPIDoc.Section] = None
+        self._returns: Optional[QAPIDoc.Section] = None
+        self._errors: Optional[QAPIDoc.Section] = None
         # "Since" section
         self.since: Optional[QAPIDoc.Section] = None
 
+    @property
+    def has_features(self) -> bool:
+        return bool(self._features)
+
     def end(self) -> None:
         for section in self.all_sections:
             section.text = section.text.strip('\n')
@@ -771,15 +775,15 @@ def new_tagged_section(
     ) -> None:
         section = self.Section(info, kind)
         if kind == QAPIDoc.Kind.RETURNS:
-            if self.returns:
+            if self._returns:
                 raise QAPISemError(
                     info, "duplicated '%s' section" % kind)
-            self.returns = section
+            self._returns = section
         elif kind == QAPIDoc.Kind.ERRORS:
-            if self.errors:
+            if self._errors:
                 raise QAPISemError(
                     info, "duplicated '%s' section" % kind)
-            self.errors = section
+            self._errors = section
         elif kind == QAPIDoc.Kind.SINCE:
             if self.since:
                 raise QAPISemError(
@@ -803,16 +807,16 @@ def _new_description(
         desc[name] = section
 
     def new_argument(self, info: QAPISourceInfo, name: str) -> None:
-        self._new_description(info, name, QAPIDoc.Kind.MEMBER, self.args)
+        self._new_description(info, name, QAPIDoc.Kind.MEMBER, self._args)
 
     def new_feature(self, info: QAPISourceInfo, name: str) -> None:
-        self._new_description(info, name, QAPIDoc.Kind.FEATURE, self.features)
+        self._new_description(info, name, QAPIDoc.Kind.FEATURE, self._features)
 
     def append_line(self, line: str) -> None:
         self.all_sections[-1].append_line(line)
 
     def connect_member(self, member: 'QAPISchemaMember') -> None:
-        if member.name not in self.args:
+        if member.name not in self._args:
             assert member.info
             if self.symbol not in member.info.pragma.documentation_exceptions:
                 raise QAPISemError(member.info,
@@ -823,7 +827,7 @@ def connect_member(self, member: 'QAPISchemaMember') -> None:
 
             section = QAPIDoc.ArgSection(
                 self.info, QAPIDoc.Kind.MEMBER, member.name)
-            self.args[member.name] = section
+            self._args[member.name] = section
 
             # Determine where to insert stub doc - it should go at the
             # end of the members section(s), if any. Note that index 0
@@ -835,14 +839,14 @@ def connect_member(self, member: 'QAPISchemaMember') -> None:
                     index += 1
             self.all_sections.insert(index, section)
 
-        self.args[member.name].connect(member)
+        self._args[member.name].connect(member)
 
     def connect_feature(self, feature: 'QAPISchemaFeature') -> None:
-        if feature.name not in self.features:
+        if feature.name not in self._features:
             raise QAPISemError(feature.info,
                                "feature '%s' lacks documentation"
                                % feature.name)
-        self.features[feature.name].connect(feature)
+        self._features[feature.name].connect(feature)
 
     def ensure_returns(self, info: QAPISourceInfo) -> None:
 
@@ -883,18 +887,18 @@ def _insert_near_kind(
 
     def check_expr(self, expr: QAPIExpression) -> None:
         if 'command' in expr:
-            if self.returns and 'returns' not in expr:
+            if self._returns and 'returns' not in expr:
                 raise QAPISemError(
-                    self.returns.info,
+                    self._returns.info,
                     "'Returns' section, but command doesn't return anything")
         else:
-            if self.returns:
+            if self._returns:
                 raise QAPISemError(
-                    self.returns.info,
+                    self._returns.info,
                     "'Returns' section is only valid for commands")
-            if self.errors:
+            if self._errors:
                 raise QAPISemError(
-                    self.errors.info,
+                    self._errors.info,
                     "'Errors' section is only valid for commands")
 
     def check(self) -> None:
@@ -914,5 +918,5 @@ def check_args_section(
                         "do" if len(bogus) > 1 else "does"
                     ))
 
-        check_args_section(self.args, 'member')
-        check_args_section(self.features, 'feature')
+        check_args_section(self._args, 'member')
+        check_args_section(self._features, 'feature')
-- 
2.53.0

[PATCH 04/12] qapi/docs: add "Intro" section

[John Snow]

This patch adds an "Intro" doc section, intended to eventually replace
the "Plain" doc section alongside a forthcoming "Details" section.

For now, this section is not actually instantiated or used, but
subsequent patches will slowly convert the leading introductory sections
of QAPIDoc documentation to use the new Intro section.

A main motivation of this series of changes is to more explicitly
delineate the "Introductory" documentation for each QAPI definition for
the sake of the inliner. When inlining members, examples, and details
from multiple QAPIDoc sections, we will want to omit the "Introductory"
text from inlined definitions while keeping notes, caution boxes,
examples, and so on.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/sphinx/qapidoc.py | 2 +-
 scripts/qapi/parser.py | 5 +++--
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index c2f09bac16c..1f7c15b7075 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -368,7 +368,7 @@ def visit_sections(self, ent: QAPISchemaDefinition) -> None:
         for i, section in enumerate(sections):
             section.text = self.reformat_arobase(section.text)
 
-            if section.kind == QAPIDoc.Kind.PLAIN:
+            if section.kind.name in ("PLAIN", "INTRO"):
                 self.visit_paragraph(section)
             elif section.kind == QAPIDoc.Kind.MEMBER:
                 assert isinstance(section, QAPIDoc.ArgSection)
diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index da4756a7424..97e7dacb0fd 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -631,7 +631,7 @@ def get_doc(self) -> 'QAPIDoc':
                     line = self.get_doc_indented(doc)
                     no_more_args = True
                 else:
-                    # plain paragraph
+                    # plain paragraph(s)
                     doc.ensure_untagged_section(self.info)
                     doc.append_line(line)
                     line = self.get_doc_paragraph(doc)
@@ -681,6 +681,7 @@ class Kind(enum.Enum):
         ERRORS = 4
         SINCE = 5
         TODO = 6
+        INTRO = 7
 
         @staticmethod
         def from_string(kind: str) -> 'QAPIDoc.Kind':
@@ -748,7 +749,7 @@ def has_features(self) -> bool:
     def end(self) -> None:
         for section in self.all_sections:
             section.text = section.text.strip('\n')
-            if section.kind != QAPIDoc.Kind.PLAIN and section.text == '':
+            if not (section.kind.name in ("INTRO", "PLAIN") or section.text):
                 raise QAPISemError(
                     section.info, "text required after '%s:'" % section.kind)
 
-- 
2.53.0

[PATCH 05/12] qapi/docs: adjust stub member insertion algorithm

[John Snow]

A forthcoming patch removes the implicit PLAIN section that always
starts a QAPIDoc section list. Further future changes begin converting
"PLAIN" sections to "INTRO" sections. To accommodate this, the insertion
algorithm that places stub and dummy members must be adjusted to cope.

This algorithm can handle zero-or-more PLAIN *or* INTRO sections at the
beginning of a QAPIDoc object.

Since we have three places that need to insert stub members, take the
opportunity to unify and deduplicate this code.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/sphinx/qapidoc.py | 36 ++++++++---------
 scripts/qapi/parser.py | 90 +++++++++++++++++++++++++++---------------
 2 files changed, 75 insertions(+), 51 deletions(-)

diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index 1f7c15b7075..70ab9cdc214 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -349,30 +349,32 @@ def _get_target(
                 )
 
     def visit_sections(self, ent: QAPISchemaDefinition) -> None:
+        # Generate a placeholder right after the member section(s) which
+        # will be used to generate documentation for "The members of..."
+        # pointers in the rendered document.
+        # This is a temporary hack until the inliner is merged.
+        if ent.doc:
+            ent.doc.append_member_stub(
+                QAPIDoc.ArgSection(
+                    ent.doc.info, QAPIDoc.Kind.MEMBER, "q_dummy"
+                )
+            )
+
         sections = ent.doc.all_sections if ent.doc else []
 
-        # Determine the index location at which we should generate
-        # documentation for "The members of ..." pointers. This should
-        # go at the end of the members section(s) if any. Note that
-        # index 0 is assumed to be a plain intro section, even if it is
-        # empty; and that a members section if present will always
-        # immediately follow the opening PLAIN section.
-        gen_index = 1
-        if len(sections) > 1:
-            while sections[gen_index].kind == QAPIDoc.Kind.MEMBER:
-                gen_index += 1
-                if gen_index >= len(sections):
-                    break
-
         # Add sections in source order:
-        for i, section in enumerate(sections):
+        for section in sections:
             section.text = self.reformat_arobase(section.text)
 
             if section.kind.name in ("PLAIN", "INTRO"):
                 self.visit_paragraph(section)
             elif section.kind == QAPIDoc.Kind.MEMBER:
                 assert isinstance(section, QAPIDoc.ArgSection)
-                self.visit_member(section)
+                if section.name == "q_dummy":
+                    # Generate "The members of ..." entries if necessary
+                    self._insert_member_pointer(ent)
+                else:
+                    self.visit_member(section)
             elif section.kind == QAPIDoc.Kind.FEATURE:
                 assert isinstance(section, QAPIDoc.ArgSection)
                 self.visit_feature(section)
@@ -386,10 +388,6 @@ def visit_sections(self, ent: QAPISchemaDefinition) -> None:
             else:
                 assert False
 
-            # Generate "The members of ..." entries if necessary:
-            if i == gen_index - 1:
-                self._insert_member_pointer(ent)
-
         self.ensure_blank_line()
 
     # Transmogrification core methods
diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 97e7dacb0fd..b21796b3e80 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -816,6 +816,58 @@ def new_feature(self, info: QAPISourceInfo, name: str) -> None:
     def append_line(self, line: str) -> None:
         self.all_sections[-1].append_line(line)
 
+    def _insert_near_kind(
+        self,
+        kind: 'QAPIDoc.Kind',
+        new_sect: 'QAPIDoc.Section',
+        after: bool = False,
+    ) -> bool:
+        """Insert or append a new doc section at a specific point."""
+        for idx, sect in enumerate(reversed(self.all_sections)):
+            if sect.kind == kind:
+                pos = len(self.all_sections) - idx - 1
+                if after:
+                    pos += 1
+                self.all_sections.insert(pos, new_sect)
+                return True
+        return False
+
+    def _insert_after_intro(
+        self,
+        section: 'QAPIDoc.Section',
+    ) -> None:
+        """
+        Insert a section immediately after the intro section.
+
+        While we convert PLAIN sections to INTRO sections, all
+        contiguous INTRO/PLAIN sections at the start of a QAPIDoc
+        section list are treated as "the intro".
+
+        Once INTRO conversion is complete, this helper will no longer be
+        needed and ``_insert_near_kind(QAPIDoc.Kind.INTRO, ...)`` will
+        be sufficient.
+        """
+        index = 0
+        for index, ref_section in enumerate(self.all_sections):
+            if ref_section.kind.name in ("PLAIN", "INTRO"):
+                continue
+            break
+        else:
+            index += 1
+
+        self.all_sections.insert(index, section)
+
+    def append_member_stub(self, stub: 'QAPIDoc.Section') -> None:
+
+        """
+        Append a stub section after any Member sections.
+        """
+        if self._insert_near_kind(QAPIDoc.Kind.MEMBER, stub, True):
+            return
+
+        # No MEMBER sections present. Insert after INTRO/PLAIN sections.
+        self._insert_after_intro(stub)
+
     def connect_member(self, member: 'QAPISchemaMember') -> None:
         if member.name not in self._args:
             assert member.info
@@ -825,20 +877,10 @@ def connect_member(self, member: 'QAPISchemaMember') -> None:
                                    % (member.role, member.name))
             # Insert stub documentation section for missing member docs.
             # TODO: drop when undocumented members are outlawed
-
-            section = QAPIDoc.ArgSection(
+            stub_section = QAPIDoc.ArgSection(
                 self.info, QAPIDoc.Kind.MEMBER, member.name)
-            self._args[member.name] = section
-
-            # Determine where to insert stub doc - it should go at the
-            # end of the members section(s), if any. Note that index 0
-            # is assumed to be an untagged intro section, even if it is
-            # empty.
-            index = 1
-            if len(self.all_sections) > 1:
-                while self.all_sections[index].kind == QAPIDoc.Kind.MEMBER:
-                    index += 1
-            self.all_sections.insert(index, section)
+            self._args[member.name] = stub_section
+            self.append_member_stub(stub_section)
 
         self._args[member.name].connect(member)
 
@@ -851,27 +893,13 @@ def connect_feature(self, feature: 'QAPISchemaFeature') -> None:
 
     def ensure_returns(self, info: QAPISourceInfo) -> None:
 
-        def _insert_near_kind(
-            kind: QAPIDoc.Kind,
-            new_sect: QAPIDoc.Section,
-            after: bool = False,
-        ) -> bool:
-            for idx, sect in enumerate(reversed(self.all_sections)):
-                if sect.kind == kind:
-                    pos = len(self.all_sections) - idx - 1
-                    if after:
-                        pos += 1
-                    self.all_sections.insert(pos, new_sect)
-                    return True
-            return False
-
         if any(s.kind == QAPIDoc.Kind.RETURNS for s in self.all_sections):
             return
 
         # Stub "Returns" section for undocumented returns value
         stub = QAPIDoc.Section(info, QAPIDoc.Kind.RETURNS)
 
-        if any(_insert_near_kind(kind, stub, after) for kind, after in (
+        if any(self._insert_near_kind(kind, stub, after) for kind, after in (
                 # 1. If arguments, right after those.
                 (QAPIDoc.Kind.MEMBER, True),
                 # 2. Elif errors, right *before* those.
@@ -881,10 +909,8 @@ def _insert_near_kind(
         )):
             return
 
-        # Otherwise, it should go right after the intro. The intro
-        # is always the first section and is always present (even
-        # when empty), so we can insert directly at index=1 blindly.
-        self.all_sections.insert(1, stub)
+        # Otherwise, it should go right after the intro.
+        self._insert_after_intro(stub)
 
     def check_expr(self, expr: QAPIExpression) -> None:
         if 'command' in expr:
-- 
2.53.0

[PATCH 06/12] qapi/docs: remove implicit Plain section

[John Snow]

Prior to this patch, we always instantiate an empty Plain
section. Removing this allows us to gradually phase out the "Plain"
section in favor of "Intro" and "Details" sections while keeping "Plain"
around for the interim churn during the series - meaning that we don't
actually know at __init__ time which type of section we'll have first.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/parser.py         |  4 +---
 tests/qapi-schema/doc-good.out | 14 --------------
 2 files changed, 1 insertion(+), 17 deletions(-)

diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index b21796b3e80..6612f471bb8 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -730,9 +730,7 @@ def __init__(self, info: QAPISourceInfo, symbol: Optional[str] = None):
         # definition doc's symbol, None for free-form doc
         self.symbol: Optional[str] = symbol
         # the sections in textual order
-        self.all_sections: List[QAPIDoc.Section] = [
-            QAPIDoc.Section(info, QAPIDoc.Kind.PLAIN)
-        ]
+        self.all_sections: List[QAPIDoc.Section] = []
         # dicts mapping parameter/feature names to their description
         self._args: Dict[str, QAPIDoc.ArgSection] = {}
         self._features: Dict[str, QAPIDoc.ArgSection] = {}
diff --git a/tests/qapi-schema/doc-good.out b/tests/qapi-schema/doc-good.out
index e2be6f96bbf..6fcc8175cfe 100644
--- a/tests/qapi-schema/doc-good.out
+++ b/tests/qapi-schema/doc-good.out
@@ -106,8 +106,6 @@ Examples:
 - *verbatim*
 - {braces}
 doc symbol=Enum
-    section=Plain
-
     arg=one
 The _one_ {and only}, description on the same line
     arg=two
@@ -119,8 +117,6 @@ a member feature
     section=Plain
 @two is undocumented
 doc symbol=Base
-    section=Plain
-
     arg=base1
  description starts on a new line,
  minimally indented
@@ -138,16 +134,10 @@ a feature
     feature=member-feat
 a member feature
 doc symbol=Variant2
-    section=Plain
-
 doc symbol=Object
-    section=Plain
-
     feature=union-feat1
 a feature
 doc symbol=Alternate
-    section=Plain
-
     arg=i
 description starts on the same line
     remainder indented the same
@@ -161,8 +151,6 @@ doc freeform
 Another subsection
 ==================
 doc symbol=cmd
-    section=Plain
-
     arg=arg1
     description starts on a new line,
     indented
@@ -223,7 +211,5 @@ another feature
 
    <- ... has no title ...
 doc symbol=EVT_BOXED
-    section=Plain
-
     feature=feat3
 a feature
-- 
2.53.0

[PATCH 07/12] qapi/docs: add "Intro" section parsing

[John Snow]

Add parsing for explicit Intro section syntax.

A side effect of this patch is that we will always create an empty Intro
section, similar to how we used to have an empty Plain section. The
tests are adjusted accordingly, rendered document output does not change
at all.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/devel/qapi-code-gen.rst            | 17 +++++----
 scripts/qapi/parser.py                  | 47 ++++++++++++++++++-------
 tests/qapi-schema/doc-good.out          | 18 ++++++++++
 tests/qapi-schema/doc-missing-colon.err |  2 +-
 4 files changed, 64 insertions(+), 20 deletions(-)

diff --git a/docs/devel/qapi-code-gen.rst b/docs/devel/qapi-code-gen.rst
index 3a632b4a648..a8175934d52 100644
--- a/docs/devel/qapi-code-gen.rst
+++ b/docs/devel/qapi-code-gen.rst
@@ -862,7 +862,7 @@ documentation comment.
 If the documentation comment starts like ::
 
     ##
-    # @SYMBOL:
+    # @SYMBOL: [...]
 
 it documents the definition of SYMBOL, else it's free-form
 documentation.
@@ -984,11 +984,11 @@ definition it documents.
 When documentation is required (see pragma_ 'doc-required'), every
 definition must have documentation.
 
-Definition documentation starts with a line naming the definition,
-followed by an optional overview, a description of each argument (for
-commands and events), member (for structs and unions), branch (for
-alternates), or value (for enums), a description of each feature (if
-any), and finally optional tagged sections.
+Definition documentation starts with a description naming the definition
+with an optional overview, a description of each argument (for commands
+and events), member (for structs and unions), branch (for alternates),
+or value (for enums), a description of each feature (if any), and
+finally optional tagged sections.
 
 Descriptions start with '\@name:'.  The description text must be
 indented like this::
@@ -996,6 +996,11 @@ indented like this::
  # @name: Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed
  #     do eiusmod tempor incididunt ut labore et dolore magna aliqua.
 
+Definition descriptions are special: the optional introductory overview
+describing the definition will not be inlined when referenced by other
+definitions (such as when using 'base' to include members from another
+definition), while other descriptions and tagged sections will be.
+
 .. FIXME The parser accepts these things in almost any order.
 
 .. FIXME union branches should be described, too.
diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index 6612f471bb8..c23fd26aaa7 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -24,6 +24,7 @@
     Match,
     Optional,
     Set,
+    Tuple,
     Union,
 )
 
@@ -524,6 +525,30 @@ def get_doc_paragraph(self, doc: 'QAPIDoc') -> Optional[str]:
                 return line
             doc.append_line(line)
 
+    def _get_doc_intro(
+        self,
+        line: str,
+        info: QAPISourceInfo
+    ) -> Tuple['QAPIDoc', Optional[str]]:
+        match = self._match_at_name_colon(line)
+        if not match:
+            raise QAPIParseError(self, "@name must end with ':'")
+
+        # Invalid names are not checked here, but the name
+        # provided *must* match the following definition,
+        # which *is* validated in expr.py.
+        symbol = match.group(1)
+        if not symbol:
+            raise QAPIParseError(self, "name required after '@'")
+        doc = QAPIDoc(info, symbol)
+
+        doc.ensure_untagged_section(info, QAPIDoc.Kind.INTRO)
+        text = line[match.end():]
+        if text:
+            doc.append_line(text)
+
+        return doc, self.get_doc_indented(doc)
+
     def get_doc(self) -> 'QAPIDoc':
         if self.val != '##':
             raise QAPIParseError(
@@ -532,18 +557,9 @@ def get_doc(self) -> 'QAPIDoc':
         self.accept(False)
         line = self.get_doc_line()
         if line is not None and line.startswith('@'):
+
             # Definition documentation
-            if not line.endswith(':'):
-                raise QAPIParseError(self, "line should end with ':'")
-            # Invalid names are not checked here, but the name
-            # provided *must* match the following definition,
-            # which *is* validated in expr.py.
-            symbol = line[1:-1]
-            if not symbol:
-                raise QAPIParseError(self, "name required after '@'")
-            doc = QAPIDoc(info, symbol)
-            self.accept(False)
-            line = self.get_doc_line()
+            doc, line = self._get_doc_intro(line, info)
             no_more_args = False
 
             while line is not None:
@@ -751,8 +767,13 @@ def end(self) -> None:
                 raise QAPISemError(
                     section.info, "text required after '%s:'" % section.kind)
 
-    def ensure_untagged_section(self, info: QAPISourceInfo) -> None:
-        kind = QAPIDoc.Kind.PLAIN
+    def ensure_untagged_section(
+        self,
+        info: QAPISourceInfo,
+        kind: Optional['QAPIDoc.Kind'] = None,
+    ) -> None:
+        if kind is None:
+            kind = QAPIDoc.Kind.PLAIN
 
         if self.all_sections and self.all_sections[-1].kind == kind:
             # extend current section
diff --git a/tests/qapi-schema/doc-good.out b/tests/qapi-schema/doc-good.out
index 6fcc8175cfe..bc89853765f 100644
--- a/tests/qapi-schema/doc-good.out
+++ b/tests/qapi-schema/doc-good.out
@@ -106,6 +106,8 @@ Examples:
 - *verbatim*
 - {braces}
 doc symbol=Enum
+    section=Intro
+
     arg=one
 The _one_ {and only}, description on the same line
     arg=two
@@ -117,10 +119,14 @@ a member feature
     section=Plain
 @two is undocumented
 doc symbol=Base
+    section=Intro
+
     arg=base1
  description starts on a new line,
  minimally indented
 doc symbol=Variant1
+    section=Intro
+
     section=Plain
 A paragraph
 
@@ -134,10 +140,16 @@ a feature
     feature=member-feat
 a member feature
 doc symbol=Variant2
+    section=Intro
+
 doc symbol=Object
+    section=Intro
+
     feature=union-feat1
 a feature
 doc symbol=Alternate
+    section=Intro
+
     arg=i
 description starts on the same line
     remainder indented the same
@@ -151,6 +163,8 @@ doc freeform
 Another subsection
 ==================
 doc symbol=cmd
+    section=Intro
+
     arg=arg1
     description starts on a new line,
     indented
@@ -198,6 +212,8 @@ Note::
     section=Since
 2.10
 doc symbol=cmd-boxed
+    section=Intro
+
     section=Plain
 If you're bored enough to read this, go see a video of boxed cats
     feature=cmd-feat1
@@ -211,5 +227,7 @@ another feature
 
    <- ... has no title ...
 doc symbol=EVT_BOXED
+    section=Intro
+
     feature=feat3
 a feature
diff --git a/tests/qapi-schema/doc-missing-colon.err b/tests/qapi-schema/doc-missing-colon.err
index cbcea007153..bd5862b30f3 100644
--- a/tests/qapi-schema/doc-missing-colon.err
+++ b/tests/qapi-schema/doc-missing-colon.err
@@ -1 +1 @@
-doc-missing-colon.json:4:1: line should end with ':'
+doc-missing-colon.json:4:1: @name must end with ':'
-- 
2.53.0

[PATCH 08/12] qapi/docs: Add rendering for INTRO sections

[John Snow]

The only real difference here is that we need to dedent all but the
first line so that it renders correctly. We don't need to do this for
members and features because they are always rendered as part of a field
list directive which expects indented lines - Undecorated plaintext
doesn't, so we chop the indent off.

This does not reflow the text or mess with the source info in any way,
so "blame" for error messages should be unchanged.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/sphinx/qapidoc.py | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index 70ab9cdc214..6b8e4ecd76a 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -35,6 +35,7 @@
 from pathlib import Path
 import re
 import sys
+import textwrap
 from typing import TYPE_CHECKING
 
 from docutils import nodes
@@ -150,8 +151,14 @@ def add_lines(
         self,
         content: str,
         info: QAPISourceInfo,
+        dedent: bool = False,
     ) -> None:
         lines = content.splitlines(True)
+
+        if dedent:
+            txt = "".join(lines[1:])
+            lines[1:] = textwrap.dedent(txt).splitlines(True)
+
         for i, line in enumerate(lines):
             self.add_line_raw(line, info.fname, info.line + i)
 
@@ -223,13 +230,14 @@ def reformat_arobase(text: str) -> str:
 
     # Transmogrification helpers
 
-    def visit_paragraph(self, section: QAPIDoc.Section) -> None:
+    def visit_text(self, section: QAPIDoc.Section) -> None:
         # Squelch empty paragraphs.
         if not section.text:
             return
 
+        dedent = bool(section.kind == QAPIDoc.Kind.INTRO)
         self.ensure_blank_line()
-        self.add_lines(section.text, section.info)
+        self.add_lines(section.text, section.info, dedent)
         self.ensure_blank_line()
 
     def visit_member(self, section: QAPIDoc.ArgSection) -> None:
@@ -367,7 +375,7 @@ def visit_sections(self, ent: QAPISchemaDefinition) -> None:
             section.text = self.reformat_arobase(section.text)
 
             if section.kind.name in ("PLAIN", "INTRO"):
-                self.visit_paragraph(section)
+                self.visit_text(section)
             elif section.kind == QAPIDoc.Kind.MEMBER:
                 assert isinstance(section, QAPIDoc.ArgSection)
                 if section.name == "q_dummy":
-- 
2.53.0

[PATCH 09/12] qapi: convert intro sections for accelerator.json

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 qapi/accelerator.json | 24 ++++++------------------
 1 file changed, 6 insertions(+), 18 deletions(-)

diff --git a/qapi/accelerator.json b/qapi/accelerator.json
index 0cf5e0f9d94..71047538ee1 100644
--- a/qapi/accelerator.json
+++ b/qapi/accelerator.json
@@ -12,9 +12,7 @@
 { 'include': 'common.json' }
 
 ##
-# @KvmInfo:
-#
-# Information about support for KVM acceleration
+# @KvmInfo: Information about support for KVM acceleration
 #
 # @enabled: true if KVM acceleration is active
 #
@@ -25,9 +23,7 @@
 { 'struct': 'KvmInfo', 'data': {'enabled': 'bool', 'present': 'bool'} }
 
 ##
-# @query-kvm:
-#
-# Return information about KVM acceleration
+# @query-kvm: Return information about KVM acceleration
 #
 # Since: 0.14
 #
@@ -39,9 +35,7 @@
 { 'command': 'query-kvm', 'returns': 'KvmInfo' }
 
 ##
-# @x-accel-stats:
-#
-# Query accelerator statistics
+# @x-accel-stats: Query accelerator statistics
 #
 # Features:
 #
@@ -56,9 +50,7 @@
   'features': [ 'unstable' ] }
 
 ##
-# @Accelerator:
-#
-# Information about support for MSHV acceleration
+# @Accelerator: Information about support for MSHV acceleration
 #
 # @hvf: Apple Hypervisor.framework
 #
@@ -81,9 +73,7 @@
 { 'enum': 'Accelerator', 'data': ['hvf', 'kvm', 'mshv', 'nvmm', 'qtest', 'tcg', 'whpx', 'xen'] }
 
 ##
-# @AcceleratorInfo:
-#
-# Information about support for various accelerators
+# @AcceleratorInfo: Information about support for various accelerators
 #
 # @enabled: the accelerator that is in use
 #
@@ -95,9 +85,7 @@
 { 'struct': 'AcceleratorInfo', 'data': {'enabled': 'Accelerator', 'present': ['Accelerator']} }
 
 ##
-# @query-accelerators:
-#
-# Return information about accelerators
+# @query-accelerators: Return information about accelerators
 #
 # Returns: @AcceleratorInfo
 #
-- 
2.53.0

[PATCH 10/12] qapi: convert intro sections for acpi-hest.json

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 qapi/acpi-hest.json | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/qapi/acpi-hest.json b/qapi/acpi-hest.json
index a01f1dee095..6a48e80082e 100644
--- a/qapi/acpi-hest.json
+++ b/qapi/acpi-hest.json
@@ -12,9 +12,8 @@
 ##
 
 ##
-# @inject-ghes-v2-error:
-#
-# Inject an error with additional ACPI 6.1 GHESv2 error information
+# @inject-ghes-v2-error: Inject an error with additional ACPI 6.1
+#     GHESv2 error information
 #
 # @cper: contains a base64 encoded string with raw data for a single
 #     CPER record with Generic Error Status Block, Generic Error Data
-- 
2.53.0

[PATCH 11/12] qapi: convert intro sections for acpi.json

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 qapi/acpi.json | 23 ++++++++---------------
 1 file changed, 8 insertions(+), 15 deletions(-)

diff --git a/qapi/acpi.json b/qapi/acpi.json
index 906b3687a55..ea6ad2cd45f 100644
--- a/qapi/acpi.json
+++ b/qapi/acpi.json
@@ -12,9 +12,8 @@
 ##
 
 ##
-# @AcpiTableOptions:
-#
-# Specify an ACPI table on the command line to load.
+# @AcpiTableOptions: Specify an ACPI table on the
+#     command line to load.
 #
 # At most one of @file and @data can be specified.  The list of files
 # specified by any one of them is loaded and concatenated in order.
@@ -80,11 +79,9 @@
 { 'enum': 'ACPISlotType', 'data': [ 'DIMM', 'CPU' ] }
 
 ##
-# @ACPIOSTInfo:
-#
-# OSPM Status Indication for a device.  For description of possible
-# values of @source and @status fields see "_OST (OSPM Status
-# Indication)" chapter of ACPI5.0 spec.
+# @ACPIOSTInfo: OSPM Status Indication for a device.  For description
+#     of possible values of @source and @status fields see "_OST (OSPM
+#     Status Indication)" chapter of ACPI5.0 spec.
 #
 # @device: device ID associated with slot
 #
@@ -106,10 +103,8 @@
               'status': 'int' } }
 
 ##
-# @query-acpi-ospm-status:
-#
-# Return a list of `ACPIOSTInfo` for devices that support status
-# reporting via ACPI _OST method.
+# @query-acpi-ospm-status: Return a list of `ACPIOSTInfo` for devices
+#     that support status reporting via ACPI _OST method.
 #
 # Since: 2.1
 #
@@ -125,9 +120,7 @@
 { 'command': 'query-acpi-ospm-status', 'returns': ['ACPIOSTInfo'] }
 
 ##
-# @ACPI_DEVICE_OST:
-#
-# Emitted when guest executes ACPI _OST method.
+# @ACPI_DEVICE_OST: Emitted when guest executes ACPI _OST method.
 #
 # @info: OSPM Status Indication
 #
-- 
2.53.0

[PATCH 12/12] qapi: convert intro sections for audio.json

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 qapi/audio.json | 112 ++++++++++++++----------------------------------
 .editorconfig   |   1 +
 2 files changed, 33 insertions(+), 80 deletions(-)

diff --git a/qapi/audio.json b/qapi/audio.json
index 2df87b97101..62d64eabf1c 100644
--- a/qapi/audio.json
+++ b/qapi/audio.json
@@ -13,10 +13,8 @@
 ##
 
 ##
-# @AudiodevPerDirectionOptions:
-#
-# General audio backend options that are used for both playback and
-# recording.
+# @AudiodevPerDirectionOptions: General audio backend options that are
+#     used for both playback and recording.
 #
 # @mixing-engine: use QEMU's mixing engine to mix all streams inside
 #     QEMU and convert audio formats when not supported by the
@@ -52,9 +50,7 @@
     '*buffer-length':  'uint32' } }
 
 ##
-# @AudiodevGenericOptions:
-#
-# Generic driver-specific options.
+# @AudiodevGenericOptions: Generic driver-specific options.
 #
 # @in: options of the capture stream
 #
@@ -68,9 +64,7 @@
     '*out': 'AudiodevPerDirectionOptions' } }
 
 ##
-# @AudiodevDBusOptions:
-#
-# Options of the D-Bus audio backend.
+# @AudiodevDBusOptions: Options of the D-Bus audio backend.
 #
 # @in: options of the capture stream
 #
@@ -88,10 +82,8 @@
     '*nsamples': 'uint32'} }
 
 ##
-# @AudiodevAlsaPerDirectionOptions:
-#
-# Options of the ALSA backend that are used for both playback and
-# recording.
+# @AudiodevAlsaPerDirectionOptions: Options of the ALSA backend that
+#     are used for both playback and recording.
 #
 # @dev: the name of the ALSA device to use (default 'default')
 #
@@ -110,9 +102,7 @@
     '*try-poll':      'bool' } }
 
 ##
-# @AudiodevAlsaOptions:
-#
-# Options of the ALSA audio backend.
+# @AudiodevAlsaOptions: Options of the ALSA audio backend.
 #
 # @in: options of the capture stream
 #
@@ -129,9 +119,7 @@
     '*threshold': 'uint32' } }
 
 ##
-# @AudiodevSndioOptions:
-#
-# Options of the sndio audio backend.
+# @AudiodevSndioOptions: Options of the sndio audio backend.
 #
 # @in: options of the capture stream
 #
@@ -151,10 +139,8 @@
     '*latency':   'uint32'} }
 
 ##
-# @AudiodevCoreaudioPerDirectionOptions:
-#
-# Options of the Core Audio backend that are used for both playback
-# and recording.
+# @AudiodevCoreaudioPerDirectionOptions: Options of the Core Audio
+#     backend that are used for both playback and recording.
 #
 # @buffer-count: number of buffers
 #
@@ -166,9 +152,7 @@
     '*buffer-count': 'uint32' } }
 
 ##
-# @AudiodevCoreaudioOptions:
-#
-# Options of the coreaudio audio backend.
+# @AudiodevCoreaudioOptions: Options of the coreaudio audio backend.
 #
 # @in: options of the capture stream
 #
@@ -182,9 +166,7 @@
     '*out': 'AudiodevCoreaudioPerDirectionOptions' } }
 
 ##
-# @AudiodevDsoundOptions:
-#
-# Options of the DirectSound audio backend.
+# @AudiodevDsoundOptions: Options of the DirectSound audio backend.
 #
 # @in: options of the capture stream
 #
@@ -202,10 +184,8 @@
     '*latency': 'uint32' } }
 
 ##
-# @AudiodevJackPerDirectionOptions:
-#
-# Options of the JACK backend that are used for both playback and
-# recording.
+# @AudiodevJackPerDirectionOptions: Options of the JACK backend that
+#     are used for both playback and recording.
 #
 # @server-name: select from among several possible concurrent server
 #     instances (default: environment variable $JACK_DEFAULT_SERVER if
@@ -236,9 +216,7 @@
     '*exact-name':    'bool' } }
 
 ##
-# @AudiodevJackOptions:
-#
-# Options of the JACK audio backend.
+# @AudiodevJackOptions: Options of the JACK audio backend.
 #
 # @in: options of the capture stream
 #
@@ -252,10 +230,8 @@
     '*out': 'AudiodevJackPerDirectionOptions' } }
 
 ##
-# @AudiodevOssPerDirectionOptions:
-#
-# Options of the OSS backend that are used for both playback and
-# recording.
+# @AudiodevOssPerDirectionOptions: Options of the OSS backend that are
+#     used for both playback and recording.
 #
 # @dev: file name of the OSS device (default '/dev/dsp')
 #
@@ -274,9 +250,7 @@
     '*try-poll':     'bool' } }
 
 ##
-# @AudiodevOssOptions:
-#
-# Options of the OSS audio backend.
+# @AudiodevOssOptions: Options of the OSS audio backend.
 #
 # @in: options of the capture stream
 #
@@ -304,10 +278,8 @@
     '*dsp-policy': 'uint32' } }
 
 ##
-# @AudiodevPaPerDirectionOptions:
-#
-# Options of the Pulseaudio backend that are used for both playback
-# and recording.
+# @AudiodevPaPerDirectionOptions: Options of the Pulseaudio backend
+#     that are used for both playback and recording.
 #
 # @name: name of the sink/source to use
 #
@@ -329,9 +301,7 @@
     '*latency': 'uint32' } }
 
 ##
-# @AudiodevPaOptions:
-#
-# Options of the PulseAudio audio backend.
+# @AudiodevPaOptions: Options of the PulseAudio audio backend.
 #
 # @in: options of the capture stream
 #
@@ -348,10 +318,8 @@
     '*server': 'str' } }
 
 ##
-# @AudiodevPipewirePerDirectionOptions:
-#
-# Options of the PipeWire backend that are used for both playback and
-# recording.
+# @AudiodevPipewirePerDirectionOptions: Options of the PipeWire
+#     backend that are used for both playback and recording.
 #
 # @name: name of the sink/source to use
 #
@@ -373,9 +341,7 @@
     '*latency': 'uint32' } }
 
 ##
-# @AudiodevPipewireOptions:
-#
-# Options of the PipeWire audio backend.
+# @AudiodevPipewireOptions: Options of the PipeWire audio backend.
 #
 # @in: options of the capture stream
 #
@@ -389,10 +355,8 @@
     '*out':    'AudiodevPipewirePerDirectionOptions' } }
 
 ##
-# @AudiodevSdlPerDirectionOptions:
-#
-# Options of the SDL audio backend that are used for both playback and
-# recording.
+# @AudiodevSdlPerDirectionOptions: Options of the SDL audio backend
+#     that are used for both playback and recording.
 #
 # @buffer-count: number of buffers (default 4)
 #
@@ -404,9 +368,7 @@
     '*buffer-count': 'uint32' } }
 
 ##
-# @AudiodevSdlOptions:
-#
-# Options of the SDL audio backend.
+# @AudiodevSdlOptions: Options of the SDL audio backend.
 #
 # @in: options of the recording stream
 #
@@ -420,9 +382,7 @@
     '*out': 'AudiodevSdlPerDirectionOptions' } }
 
 ##
-# @AudiodevWavOptions:
-#
-# Options of the wav audio backend.
+# @AudiodevWavOptions: Options of the wav audio backend.
 #
 # @in: options of the capture stream
 #
@@ -439,9 +399,7 @@
     '*path': 'str' } }
 
 ##
-# @AudioFormat:
-#
-# An enumeration of possible audio formats.
+# @AudioFormat: An enumeration of possible audio formats.
 #
 # @u8: unsigned 8 bit integer
 #
@@ -463,9 +421,7 @@
   'data': [ 'u8', 's8', 'u16', 's16', 'u32', 's32', 'f32' ] }
 
 ##
-# @AudiodevDriver:
-#
-# An enumeration of possible audio backend drivers.
+# @AudiodevDriver: An enumeration of possible audio backend drivers.
 #
 # @jack: JACK audio backend (since 5.1)
 #
@@ -487,9 +443,7 @@
             'wav' ] }
 
 ##
-# @Audiodev:
-#
-# Options of an audio backend.
+# @Audiodev: Options of an audio backend.
 #
 # @id: identifier of the backend
 #
@@ -533,9 +487,7 @@
     'wav':       'AudiodevWavOptions' } }
 
 ##
-# @query-audiodevs:
-#
-# Return information about audiodev configuration
+# @query-audiodevs: Return information about audiodev configuration
 #
 # Since: 8.0
 ##
diff --git a/.editorconfig b/.editorconfig
index 258d41ab485..dcf8f6ded6d 100644
--- a/.editorconfig
+++ b/.editorconfig
@@ -47,6 +47,7 @@ emacs_mode = glsl
 [*.json]
 indent_style = space
 emacs_mode = python
+max_line_length = 70
 
 # by default follow QEMU's style
 [*.pl]
-- 
2.53.0

Re: [PATCH 00/12] qapi: add formal "intro" section

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Hiya, this is a series that explores a potential syntax for a
> designated "Intro" section. Markus knows why I want this, but for
> everyone else: a designated "Introduction" section is useful for the
> desired "inliner" feature for the new QAPI doc system. Commits explain
> a bit more. This is prep work and doesn't really change anything
> tangibly except source code syntax for the QAPI docs.

Let me elaborate a bit.

1. Why "intro"?

Generated documentation often refers to types like this:

    Command announce-self (Since: 4.0)

       Trigger generation of broadcast RARP frames to update network
       switches.  This can be useful when network bonds fail-over the
       active slave.

       Arguments:
          * The members of "AnnounceParameters".

       Example::

          -> { "execute": "announce-self",
               "arguments": {
                   "initial": 50, "max": 550, "rounds": 10, "step": 50,
                   "interfaces": ["vn2", "vn3"], "id": "bob" } }
          <- { "return": {} }

The arguments are hidden behind link "AnnounceParameters".  Following
leads to:

    Object AnnounceParameters (Since: 4.0)

       Parameters for self-announce timers

       Members:
          * initial ("int") -- Initial delay (in ms) before sending
            the first GARP/RARP announcement

          * max ("int") -- Maximum delay (in ms) between GARP/RARP
            announcement packets

          * rounds ("int") -- Number of self-announcement attempts

          * step ("int") -- Delay increase (in ms) after each self-
            announcement attempt

          * interfaces ("[string]", *optional*) -- An optional list of
            interface names, which restricts the announcement to the
            listed interfaces.  (Since 4.1)

          * id ("string", *optional*) -- A name to be used to identify
            an instance of announce-timers and to allow it to modified
            later.  Not for use as part of the migration parameters.
            (Since 4.1)

This is bad UX.  Especially when you have to chase multiple links.

John's doc inliner inlines AnnounceParameters documentation into
announce-self documentation.

However, we don't want to inline "Parameters for self-announce timers".

Definitions typically start with a short description of the thing being
defined.  John calls this "intro".  We don't want to inline these.

Syntactically, a doc comment begins with optional plain paragraphs.
These end when something else begins: argument or member description,
tagged section such as "Returns:", ...

Most of the time, these initial paragraphs are exactly the "intro".  But
not always.

Even without the inliner, we often need to know where "intro" ends.  In
the example above, "Arguments:" is auto-generated right after the
"intro", and for that, the generator needs to know where "intro" ends.

In his "qapi: enforce section ordering" series, John proposed syntax to
explicitly end "intro": a line "Details:".  He had to add it to roughly
one in 25 definition docs.

I fear adding "Details:" when needed is easily forgotten when it's so
rarely needed.  And I don't even trust myself to catch it patch review.
So we looked for clearer syntax.

This series implements one: instead of letting intro end when something
else starts, use indentation like we do for descriptions and tagged
sections.

Let's have a look at the current doc source for announce-self:

    ##
    # @announce-self:
    #
    # Trigger generation of broadcast RARP frames to update network
    # switches.  This can be useful when network bonds fail-over the
    # active slave.
    #
    # TODO: This line is a hack to separate the example from the body
    #
    # .. qmp-example::
    #
    #     -> { "execute": "announce-self",
    #          "arguments": {
    #              "initial": 50, "max": 550, "rounds": 10, "step": 50,
    #              "interfaces": ["vn2", "vn3"], "id": "bob" } }
    #     <- { "return": {} }
    #
    # Since: 4.0
    ##

"Intro" ends when something else starts, namely the TODO: section.  We
need this hack so the doc generator inserts the "Arguments:" part where
we want it.

John's "qapi: enforce section ordering" series replaces the TODO hack
with a "Details:" line.

In new syntax, this instead becomes:

    ##
    # @announce-self: Trigger generation of broadcast RARP frames to
    #     update network switches.  This can be useful when network bonds
    #     fail-over the active slave.
    #
    # .. qmp-example::
    #
    #     -> { "execute": "announce-self",
    #          "arguments": {
    #              "initial": 50, "max": 550, "rounds": 10, "step": 50,
    #              "interfaces": ["vn2", "vn3"], "id": "bob" } }
    #     <- { "return": {} }
    #
    # Since: 4.0
    ##

Now "intro" is indented, and its end is obvious.  Mistakes seem
unlikely.

We could instead do:

    ##
    # @announce-self:
    #     Trigger generation of broadcast RARP frames to update network
    #     switches.  This can be useful when network bonds fail-over the
    #     active slave.
    #
    # .. qmp-example::
    #
    #     -> { "execute": "announce-self",
    #          "arguments": {
    #              "initial": 50, "max": 550, "rounds": 10, "step": 50,
    #              "interfaces": ["vn2", "vn3"], "id": "bob" } }
    #     <- { "return": {} }
    #
    # Since: 4.0
    ##

I think I like this a bit better.

> It is designed so that this conversion can happen incrementally with
> no actual difference to the rendered manuals, so each QAPI module can
> be converted one at a time for easier review and merging in an
> arbitrary order.

Why not wholesale conversion?  As long as generated output stays the
same.  It does in this series, except for harmless line breaks.

> This series demonstrates conversion of just four modules; if I'm given
> a thumbs up, I will convert the rest of QAPI, one module (or
> maintainer stanza) per patch like how I handled adding
> cross-references.

Re: [PATCH 00/12] qapi: add formal "intro" section

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 6750 bytes --]

On Fri, Apr 24, 2026, 8:50 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > Hiya, this is a series that explores a potential syntax for a
> > designated "Intro" section. Markus knows why I want this, but for
> > everyone else: a designated "Introduction" section is useful for the
> > desired "inliner" feature for the new QAPI doc system. Commits explain
> > a bit more. This is prep work and doesn't really change anything
> > tangibly except source code syntax for the QAPI docs.
>
> Let me elaborate a bit.
>
> 1. Why "intro"?
>
> Generated documentation often refers to types like this:
>
>     Command announce-self (Since: 4.0)
>
>        Trigger generation of broadcast RARP frames to update network
>        switches.  This can be useful when network bonds fail-over the
>        active slave.
>
>        Arguments:
>           * The members of "AnnounceParameters".
>
>        Example::
>
>           -> { "execute": "announce-self",
>                "arguments": {
>                    "initial": 50, "max": 550, "rounds": 10, "step": 50,
>                    "interfaces": ["vn2", "vn3"], "id": "bob" } }
>           <- { "return": {} }
>
> The arguments are hidden behind link "AnnounceParameters".  Following
> leads to:
>
>     Object AnnounceParameters (Since: 4.0)
>
>        Parameters for self-announce timers
>
>        Members:
>           * initial ("int") -- Initial delay (in ms) before sending
>             the first GARP/RARP announcement
>
>           * max ("int") -- Maximum delay (in ms) between GARP/RARP
>             announcement packets
>
>           * rounds ("int") -- Number of self-announcement attempts
>
>           * step ("int") -- Delay increase (in ms) after each self-
>             announcement attempt
>
>           * interfaces ("[string]", *optional*) -- An optional list of
>             interface names, which restricts the announcement to the
>             listed interfaces.  (Since 4.1)
>
>           * id ("string", *optional*) -- A name to be used to identify
>             an instance of announce-timers and to allow it to modified
>             later.  Not for use as part of the migration parameters.
>             (Since 4.1)
>
> This is bad UX.  Especially when you have to chase multiple links.
>
> John's doc inliner inlines AnnounceParameters documentation into
> announce-self documentation.
>
> However, we don't want to inline "Parameters for self-announce timers".
>
> Definitions typically start with a short description of the thing being
> defined.  John calls this "intro".  We don't want to inline these.
>
> Syntactically, a doc comment begins with optional plain paragraphs.
> These end when something else begins: argument or member description,
> tagged section such as "Returns:", ...
>
> Most of the time, these initial paragraphs are exactly the "intro".  But
> not always.
>
> Even without the inliner, we often need to know where "intro" ends.  In
> the example above, "Arguments:" is auto-generated right after the
> "intro", and for that, the generator needs to know where "intro" ends.
>
> In his "qapi: enforce section ordering" series, John proposed syntax to
> explicitly end "intro": a line "Details:".  He had to add it to roughly
> one in 25 definition docs.
>
> I fear adding "Details:" when needed is easily forgotten when it's so
> rarely needed.  And I don't even trust myself to catch it patch review.
> So we looked for clearer syntax.
>
> This series implements one: instead of letting intro end when something
> else starts, use indentation like we do for descriptions and tagged
> sections.
>
> Let's have a look at the current doc source for announce-self:
>
>     ##
>     # @announce-self:
>     #
>     # Trigger generation of broadcast RARP frames to update network
>     # switches.  This can be useful when network bonds fail-over the
>     # active slave.
>     #
>     # TODO: This line is a hack to separate the example from the body
>     #
>     # .. qmp-example::
>     #
>     #     -> { "execute": "announce-self",
>     #          "arguments": {
>     #              "initial": 50, "max": 550, "rounds": 10, "step": 50,
>     #              "interfaces": ["vn2", "vn3"], "id": "bob" } }
>     #     <- { "return": {} }
>     #
>     # Since: 4.0
>     ##
>
> "Intro" ends when something else starts, namely the TODO: section.  We
> need this hack so the doc generator inserts the "Arguments:" part where
> we want it.
>
> John's "qapi: enforce section ordering" series replaces the TODO hack
> with a "Details:" line.
>
> In new syntax, this instead becomes:
>
>     ##
>     # @announce-self: Trigger generation of broadcast RARP frames to
>     #     update network switches.  This can be useful when network bonds
>     #     fail-over the active slave.
>     #
>     # .. qmp-example::
>     #
>     #     -> { "execute": "announce-self",
>     #          "arguments": {
>     #              "initial": 50, "max": 550, "rounds": 10, "step": 50,
>     #              "interfaces": ["vn2", "vn3"], "id": "bob" } }
>     #     <- { "return": {} }
>     #
>     # Since: 4.0
>     ##
>
> Now "intro" is indented, and its end is obvious.  Mistakes seem
> unlikely.
>
> We could instead do:
>
>     ##
>     # @announce-self:
>     #     Trigger generation of broadcast RARP frames to update network
>     #     switches.  This can be useful when network bonds fail-over the
>     #     active slave.
>     #
>     # .. qmp-example::
>     #
>     #     -> { "execute": "announce-self",
>     #          "arguments": {
>     #              "initial": 50, "max": 550, "rounds": 10, "step": 50,
>     #              "interfaces": ["vn2", "vn3"], "id": "bob" } }
>     #     <- { "return": {} }
>     #
>     # Since: 4.0
>     ##
>
> I think I like this a bit better.
>
> > It is designed so that this conversion can happen incrementally with
> > no actual difference to the rendered manuals, so each QAPI module can
> > be converted one at a time for easier review and merging in an
> > arbitrary order.
>
> Why not wholesale conversion?  As long as generated output stays the
> same.  It does in this series, except for harmless line breaks.
>

When we last spoke, we weren't convinced this was the right approach and we
both wanted to see it. So I did enough to get a taste.


> > This series demonstrates conversion of just four modules; if I'm given
> > a thumbs up, I will convert the rest of QAPI, one module (or
> > maintainer stanza) per patch like how I handled adding
> > cross-references.
>
>

[-- Attachment #2: Type: text/html, Size: 9048 bytes --]

Re: [PATCH 00/12] qapi: add formal "intro" section

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Fri, Apr 24, 2026, 8:50 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > Hiya, this is a series that explores a potential syntax for a
>> > designated "Intro" section. Markus knows why I want this, but for
>> > everyone else: a designated "Introduction" section is useful for the
>> > desired "inliner" feature for the new QAPI doc system. Commits explain
>> > a bit more. This is prep work and doesn't really change anything
>> > tangibly except source code syntax for the QAPI docs.

[...]

>> Why not wholesale conversion?  As long as generated output stays the
>> same.  It does in this series, except for harmless line breaks.
>>
>
> When we last spoke, we weren't convinced this was the right approach and we
> both wanted to see it. So I did enough to get a taste.

Yes, and it makes sense.

>> > This series demonstrates conversion of just four modules; if I'm given
>> > a thumbs up, I will convert the rest of QAPI, one module (or
>> > maintainer stanza) per patch like how I handled adding
>> > cross-references.

Re: [PATCH 00/12] qapi: add formal "intro" section

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2366 bytes --]

On Thu, Apr 23, 2026, 6:00 PM John Snow <jsnow@redhat.com> wrote:

> Hiya, this is a series that explores a potential syntax for a
> designated "Intro" section. Markus knows why I want this, but for
> everyone else: a designated "Introduction" section is useful for the
> desired "inliner" feature for the new QAPI doc system. Commits explain
> a bit more. This is prep work and doesn't really change anything
> tangibly except source code syntax for the QAPI docs.
>
> It is designed so that this conversion can happen incrementally with
> no actual difference to the rendered manuals, so each QAPI module can
> be converted one at a time for easier review and merging in an
> arbitrary order.
>
> This series demonstrates conversion of just four modules; if I'm given
> a thumbs up, I will convert the rest of QAPI, one module (or
> maintainer stanza) per patch like how I handled adding
> cross-references.
>
> John Snow (12):
>   tests/qapi: generate output in source order
>   qapi/docs: remove unused QAPIDoc subsection members
>   qapi/docs: make remaining subsection members "private"
>   qapi/docs: add "Intro" section
>   qapi/docs: adjust stub member insertion algorithm
>   qapi/docs: remove implicit Plain section
>   qapi/docs: add "Intro" section parsing
>   qapi/docs: Add rendering for INTRO sections
>   qapi: convert intro sections for accelerator.json
>   qapi: convert intro sections for acpi-hest.json
>   qapi: convert intro sections for acpi.json
>   qapi: convert intro sections for audio.json
>
>  docs/devel/qapi-code-gen.rst            |  17 +-
>  docs/sphinx/qapidoc.py                  |  52 +++---
>  qapi/accelerator.json                   |  24 +--
>  qapi/acpi-hest.json                     |   5 +-
>  qapi/acpi.json                          |  23 +--
>  qapi/audio.json                         | 112 ++++---------
>  .editorconfig                           |   1 +
>

Whoops, that snuck in by accident. Ignore! I needed some help doing the
line wraps in qapi :)

 scripts/qapi/parser.py                  | 204 ++++++++++++++----------
>  tests/qapi-schema/doc-good.out          |  30 ++--
>  tests/qapi-schema/doc-missing-colon.err |   2 +-
>  tests/qapi-schema/test-qapi.py          |  15 +-
>  11 files changed, 239 insertions(+), 246 deletions(-)
>
> --
> 2.53.0
>
>
>

[-- Attachment #2: Type: text/html, Size: 3217 bytes --]

Re: [PATCH 00/12] qapi: add formal "intro" section

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Thu, Apr 23, 2026, 6:00 PM John Snow <jsnow@redhat.com> wrote:
>
>> Hiya, this is a series that explores a potential syntax for a
>> designated "Intro" section. Markus knows why I want this, but for
>> everyone else: a designated "Introduction" section is useful for the
>> desired "inliner" feature for the new QAPI doc system. Commits explain
>> a bit more. This is prep work and doesn't really change anything
>> tangibly except source code syntax for the QAPI docs.
>>
>> It is designed so that this conversion can happen incrementally with
>> no actual difference to the rendered manuals, so each QAPI module can
>> be converted one at a time for easier review and merging in an
>> arbitrary order.
>>
>> This series demonstrates conversion of just four modules; if I'm given
>> a thumbs up, I will convert the rest of QAPI, one module (or
>> maintainer stanza) per patch like how I handled adding
>> cross-references.
>>
>> John Snow (12):
>>   tests/qapi: generate output in source order
>>   qapi/docs: remove unused QAPIDoc subsection members
>>   qapi/docs: make remaining subsection members "private"
>>   qapi/docs: add "Intro" section
>>   qapi/docs: adjust stub member insertion algorithm
>>   qapi/docs: remove implicit Plain section
>>   qapi/docs: add "Intro" section parsing
>>   qapi/docs: Add rendering for INTRO sections
>>   qapi: convert intro sections for accelerator.json
>>   qapi: convert intro sections for acpi-hest.json
>>   qapi: convert intro sections for acpi.json
>>   qapi: convert intro sections for audio.json
>>
>>  docs/devel/qapi-code-gen.rst            |  17 +-
>>  docs/sphinx/qapidoc.py                  |  52 +++---
>>  qapi/accelerator.json                   |  24 +--
>>  qapi/acpi-hest.json                     |   5 +-
>>  qapi/acpi.json                          |  23 +--
>>  qapi/audio.json                         | 112 ++++---------
>>  .editorconfig                           |   1 +
>>
>
> Whoops, that snuck in by accident. Ignore! I needed some help doing the
> line wraps in qapi :)

Would it make sense as a separate patch?

>>  scripts/qapi/parser.py                  | 204 ++++++++++++++----------
>>  tests/qapi-schema/doc-good.out          |  30 ++--
>>  tests/qapi-schema/doc-missing-colon.err |   2 +-
>>  tests/qapi-schema/test-qapi.py          |  15 +-
>>  11 files changed, 239 insertions(+), 246 deletions(-)
>>
>> --
>> 2.53.0
>>
>>
>>