[PATCH 00/11] docs/qapi: enable new guest-agent and storage-daemon docs

[PATCH 00/11] docs/qapi: enable new guest-agent and storage-daemon docs

[John Snow]

Add namespaces, turn on QGA and QSD.

John Snow (11):
  docs/qapi_domain: isolate TYPE_CHECKING imports
  docs/qapi-domain: always store fully qualified name in signode
  docs/qapi_domain: add namespace support to FQN
  docs/qapi-domain: add :namespace: override option
  docs/qapi-domain: add qapi:namespace directive
  docs/qapidoc: add :namespace: option to qapi-doc directive
  docs/qapi_domain: add namespace support to cross-references
  docs/qapi-domain: add namespaced index support
  docs: add QAPI namespace "QMP" to qemu-qmp-ref
  docs: disambiguate references in qapi-domain.rst
  docs: enable transmogrifier for QSD and QGA

 docs/conf.py                                 |   7 +
 docs/devel/qapi-domain.rst                   |  70 ++++-
 docs/interop/qemu-ga-ref.rst                 |   2 +
 docs/interop/qemu-qmp-ref.rst                |   1 +
 docs/interop/qemu-storage-daemon-qmp-ref.rst |   2 +
 docs/sphinx/qapi_domain.py                   | 297 +++++++++++++------
 docs/sphinx/qapidoc.py                       |  12 +
 qapi/qapi-schema.json                        |   2 +-
 qga/qapi-schema.json                         |   3 +
 storage-daemon/qapi/qapi-schema.json         |   8 +
 10 files changed, 303 insertions(+), 101 deletions(-)

-- 
2.48.1

[PATCH 01/11] docs/qapi_domain: isolate TYPE_CHECKING imports

[John Snow]

When using the annotations feature, type hints do not need to be
imported at runtime, only at type check time. Move type-check-only
imports into a conditional to reduce the number of imports needed at
runtime.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/sphinx/qapi_domain.py | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/docs/sphinx/qapi_domain.py b/docs/sphinx/qapi_domain.py
index 7ff618d8cda..d52e7df7bc7 100644
--- a/docs/sphinx/qapi_domain.py
+++ b/docs/sphinx/qapi_domain.py
@@ -9,15 +9,9 @@
 
 from typing import (
     TYPE_CHECKING,
-    AbstractSet,
-    Any,
-    Dict,
-    Iterable,
     List,
     NamedTuple,
-    Optional,
     Tuple,
-    Union,
     cast,
 )
 
@@ -34,7 +28,6 @@
     SpaceNode,
 )
 from sphinx import addnodes
-from sphinx.addnodes import desc_signature, pending_xref
 from sphinx.directives import ObjectDescription
 from sphinx.domains import (
     Domain,
@@ -49,13 +42,24 @@
 
 
 if TYPE_CHECKING:
+    from typing import (
+        AbstractSet,
+        Any,
+        Dict,
+        Iterable,
+        Optional,
+        Union,
+    )
+
     from docutils.nodes import Element, Node
 
+    from sphinx.addnodes import desc_signature, pending_xref
     from sphinx.application import Sphinx
     from sphinx.builders import Builder
     from sphinx.environment import BuildEnvironment
     from sphinx.util.typing import OptionSpec
 
+
 logger = logging.getLogger(__name__)
 
 
-- 
2.48.1

[PATCH 02/11] docs/qapi-domain: always store fully qualified name in signode

[John Snow]

Currently, only the definition name is stored in the tree metadata; but
the node property is confusingly called "fullname". Rectify this by
always storing the FQN in the tree metadata.

... While we're here, re-organize the code in preparation for namespace
support to make it a bit easier to add additional components of the
FQN. With this change, there is now extremely little code left that's
taken directly from the Python domain :)

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/sphinx/qapi_domain.py | 64 +++++++++++++++++++++++++-------------
 1 file changed, 42 insertions(+), 22 deletions(-)

diff --git a/docs/sphinx/qapi_domain.py b/docs/sphinx/qapi_domain.py
index d52e7df7bc7..6b23fc73ef2 100644
--- a/docs/sphinx/qapi_domain.py
+++ b/docs/sphinx/qapi_domain.py
@@ -178,6 +178,25 @@ def get_index_text(self, name: Signature) -> Tuple[str, str]:
         # NB: this is used for the global index, not the QAPI index.
         return ("single", f"{name} (QMP {self.objtype})")
 
+    def _get_context(self) -> str:
+        modname = self.options.get(
+            "module", self.env.ref_context.get("qapi:module", "")
+        )
+        assert isinstance(modname, str)
+        return modname
+
+    def _get_fqn(self, name: Signature) -> str:
+        modname = self._get_context()
+
+        # If we're documenting a module, don't include the module as
+        # part of the FQN; we ARE the module!
+        if self.objtype == "module":
+            modname = ""
+
+        if modname:
+            name = f"{modname}.{name}"
+        return name
+
     def add_target_and_index(
         self, name: Signature, sig: str, signode: desc_signature
     ) -> None:
@@ -187,14 +206,8 @@ def add_target_and_index(
 
         assert self.objtype
 
-        # If we're documenting a module, don't include the module as
-        # part of the FQN.
-        modname = ""
-        if self.objtype != "module":
-            modname = self.options.get(
-                "module", self.env.ref_context.get("qapi:module")
-            )
-        fullname = (modname + "." if modname else "") + name
+        if not (fullname := signode.get("fullname", "")):
+            fullname = self._get_fqn(name)
 
         node_id = make_id(
             self.env, self.state.document, self.objtype, fullname
@@ -213,18 +226,21 @@ def add_target_and_index(
                     (arity, indextext, node_id, "", None)
                 )
 
+    @staticmethod
+    def split_fqn(name: str) -> Tuple[str, str]:
+        if "." in name:
+            module, name = name.split(".")
+        else:
+            module = ""
+
+        return (module, name)
+
     def _object_hierarchy_parts(
         self, sig_node: desc_signature
     ) -> Tuple[str, ...]:
         if "fullname" not in sig_node:
             return ()
-        modname = sig_node.get("module")
-        fullname = sig_node["fullname"]
-
-        if modname:
-            return (modname, *fullname.split("."))
-
-        return tuple(fullname.split("."))
+        return self.split_fqn(sig_node["fullname"])
 
     def _toc_entry_name(self, sig_node: desc_signature) -> str:
         # This controls the name in the TOC and on the sidebar.
@@ -235,13 +251,19 @@ def _toc_entry_name(self, sig_node: desc_signature) -> str:
             return ""
 
         config = self.env.app.config
-        *parents, name = toc_parts
+        modname, name = toc_parts
+
         if config.toc_object_entries_show_parents == "domain":
-            return sig_node.get("fullname", name)
+            ret = name
+            if modname and modname != self.env.ref_context.get(
+                "qapi:module", ""
+            ):
+                ret = f"{modname}.{name}"
+            return ret
         if config.toc_object_entries_show_parents == "hide":
             return name
         if config.toc_object_entries_show_parents == "all":
-            return ".".join(parents + [name])
+            return sig_node.get("fullname", name)
         return ""
 
 
@@ -312,11 +334,9 @@ def handle_signature(self, sig: str, signode: desc_signature) -> Signature:
         As such, the only argument here is "sig", which is just the QAPI
         definition name.
         """
-        modname = self.options.get(
-            "module", self.env.ref_context.get("qapi:module")
-        )
+        modname = self._get_context()
 
-        signode["fullname"] = sig
+        signode["fullname"] = self._get_fqn(sig)
         signode["module"] = modname
         sig_prefix = self.get_signature_prefix()
         if sig_prefix:
-- 
2.48.1

[PATCH 03/11] docs/qapi_domain: add namespace support to FQN

[John Snow]

This patch adds a namespace component to the "Fully Qualified Name", in
the form of "domain:module.name". As there are no namespace directives
or options yet, this component will simply be empty as of this patch.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/sphinx/qapi_domain.py | 35 +++++++++++++++++++++++++++--------
 1 file changed, 27 insertions(+), 8 deletions(-)

diff --git a/docs/sphinx/qapi_domain.py b/docs/sphinx/qapi_domain.py
index 6b23fc73ef2..48a082d489a 100644
--- a/docs/sphinx/qapi_domain.py
+++ b/docs/sphinx/qapi_domain.py
@@ -178,15 +178,18 @@ def get_index_text(self, name: Signature) -> Tuple[str, str]:
         # NB: this is used for the global index, not the QAPI index.
         return ("single", f"{name} (QMP {self.objtype})")
 
-    def _get_context(self) -> str:
+    def _get_context(self) -> Tuple[str, str]:
+        namespace = self.options.get(
+            "namespace", self.env.ref_context.get("qapi:namespace", "")
+        )
         modname = self.options.get(
             "module", self.env.ref_context.get("qapi:module", "")
         )
-        assert isinstance(modname, str)
-        return modname
+
+        return namespace, modname
 
     def _get_fqn(self, name: Signature) -> str:
-        modname = self._get_context()
+        namespace, modname = self._get_context()
 
         # If we're documenting a module, don't include the module as
         # part of the FQN; we ARE the module!
@@ -195,6 +198,8 @@ def _get_fqn(self, name: Signature) -> str:
 
         if modname:
             name = f"{modname}.{name}"
+        if namespace:
+            name = f"{namespace}:{name}"
         return name
 
     def add_target_and_index(
@@ -227,13 +232,18 @@ def add_target_and_index(
                 )
 
     @staticmethod
-    def split_fqn(name: str) -> Tuple[str, str]:
+    def split_fqn(name: str) -> Tuple[str, str, str]:
+        if ":" in name:
+            ns, name = name.split(":")
+        else:
+            ns = ""
+
         if "." in name:
             module, name = name.split(".")
         else:
             module = ""
 
-        return (module, name)
+        return (ns, module, name)
 
     def _object_hierarchy_parts(
         self, sig_node: desc_signature
@@ -251,7 +261,7 @@ def _toc_entry_name(self, sig_node: desc_signature) -> str:
             return ""
 
         config = self.env.app.config
-        modname, name = toc_parts
+        namespace, modname, name = toc_parts
 
         if config.toc_object_entries_show_parents == "domain":
             ret = name
@@ -259,6 +269,10 @@ def _toc_entry_name(self, sig_node: desc_signature) -> str:
                 "qapi:module", ""
             ):
                 ret = f"{modname}.{name}"
+            if namespace and namespace != self.env.ref_context.get(
+                "qapi:namespace", ""
+            ):
+                ret = f"{namespace}:{ret}"
             return ret
         if config.toc_object_entries_show_parents == "hide":
             return name
@@ -334,10 +348,15 @@ def handle_signature(self, sig: str, signode: desc_signature) -> Signature:
         As such, the only argument here is "sig", which is just the QAPI
         definition name.
         """
-        modname = self._get_context()
+        # No module or domain info allowed in the signature!
+        assert ":" not in sig
+        assert "." not in sig
 
+        namespace, modname = self._get_context()
         signode["fullname"] = self._get_fqn(sig)
+        signode["namespace"] = namespace
         signode["module"] = modname
+
         sig_prefix = self.get_signature_prefix()
         if sig_prefix:
             signode += addnodes.desc_annotation(
-- 
2.48.1

[PATCH 04/11] docs/qapi-domain: add :namespace: override option

[John Snow]

Akin to the :module: override option, the :namespace: options allows you
to forcibly override the contextual namespace associatied with a
definition.

We don't necessarily actually need this, but I felt compelled to stick
close to how the Python domain works that offers context overrides.

As of this commit, it is possible to add e.g. ":namespace: QMP" to any
QAPI directive to forcibly associate that definition with a given
namespace.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/devel/qapi-domain.rst | 2 ++
 docs/sphinx/qapi_domain.py | 5 +++--
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/docs/devel/qapi-domain.rst b/docs/devel/qapi-domain.rst
index 1475870ca6c..51b283277e1 100644
--- a/docs/devel/qapi-domain.rst
+++ b/docs/devel/qapi-domain.rst
@@ -466,6 +466,8 @@ QAPI standard options
 
 All QAPI directives -- *except* for module -- support these common options.
 
+* ``:namespace: name`` -- This option allows you to override the
+  namespace association of a given definition.
 * ``:module: modname`` -- Borrowed from the Python domain, this option allows
   you to override the module association of a given definition.
 * ``:since: x.y`` -- Allows the documenting of "Since" information, which is
diff --git a/docs/sphinx/qapi_domain.py b/docs/sphinx/qapi_domain.py
index 48a082d489a..6485c432063 100644
--- a/docs/sphinx/qapi_domain.py
+++ b/docs/sphinx/qapi_domain.py
@@ -294,8 +294,9 @@ class QAPIObject(QAPIDescription):
     )
     option_spec.update(
         {
-            # Borrowed from the Python domain:
-            "module": directives.unchanged,  # Override contextual module name
+            # Context overrides:
+            "namespace": directives.unchanged,
+            "module": directives.unchanged,
             # These are QAPI originals:
             "since": directives.unchanged,
             "ifcond": directives.unchanged,
-- 
2.48.1

[PATCH 05/11] docs/qapi-domain: add qapi:namespace directive

[John Snow]

Add a new directive that marks the beginning of a QAPI "namespace", for
example; "QMP", "QGA" or "QSD". This directive will associate all
subsequent QAPI directives in a document with the specified
namespace. This does not change the visual display of any of the
definitions or index entries, but does change the "Fully Qualified Name"
inside the QAPI domain's object table. This allows for two different
"namespaces" to define entities with otherwise identical names -- which
will come in handy for documenting both QEMU QMP and the QEMU Storage
Daemon.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/devel/qapi-domain.rst | 20 +++++++++++++++++++-
 docs/sphinx/qapi_domain.py | 13 +++++++++++++
 2 files changed, 32 insertions(+), 1 deletion(-)

diff --git a/docs/devel/qapi-domain.rst b/docs/devel/qapi-domain.rst
index 51b283277e1..73e13ab45ca 100644
--- a/docs/devel/qapi-domain.rst
+++ b/docs/devel/qapi-domain.rst
@@ -464,7 +464,8 @@ removed in a future version.
 QAPI standard options
 ---------------------
 
-All QAPI directives -- *except* for module -- support these common options.
+All QAPI directives -- *except* for namespace and module -- support
+these common options.
 
 * ``:namespace: name`` -- This option allows you to override the
   namespace association of a given definition.
@@ -482,6 +483,23 @@ All QAPI directives -- *except* for module -- support these common options.
   production code.
 
 
+qapi:namespace
+--------------
+
+The ``qapi:namespace`` directive marks the start of a QAPI namespace. It
+does not take a content body, nor any options. All subsequent QAPI
+directives are associated with the most recent namespace. This affects
+the definition's "fully qualified name", allowing two different
+namespaces to create an otherwise identically named definition.
+
+Example::
+
+   .. qapi:namespace:: QMP
+
+
+This directive has no visible effect.
+
+
 qapi:module
 -----------
 
diff --git a/docs/sphinx/qapi_domain.py b/docs/sphinx/qapi_domain.py
index 6485c432063..a204af9b062 100644
--- a/docs/sphinx/qapi_domain.py
+++ b/docs/sphinx/qapi_domain.py
@@ -38,6 +38,7 @@
 from sphinx.locale import _, __
 from sphinx.roles import XRefRole
 from sphinx.util import logging
+from sphinx.util.docutils import SphinxDirective
 from sphinx.util.nodes import make_id, make_refnode
 
 
@@ -645,6 +646,17 @@ def run(self) -> List[Node]:
         return ret
 
 
+class QAPINamespace(SphinxDirective):
+    has_content = False
+    required_arguments = 1
+
+    def run(self) -> List[Node]:
+        namespace = self.arguments[0].strip()
+        self.env.ref_context["qapi:namespace"] = namespace
+
+        return []
+
+
 class QAPIIndex(Index):
     """
     Index subclass to provide the QAPI definition index.
@@ -726,6 +738,7 @@ class QAPIDomain(Domain):
     # Each of these provides a rST directive,
     # e.g. .. qapi:module:: block-core
     directives = {
+        "namespace": QAPINamespace,
         "module": QAPIModule,
         "command": QAPICommand,
         "event": QAPIEvent,
-- 
2.48.1

[PATCH 06/11] docs/qapidoc: add :namespace: option to qapi-doc directive

[John Snow]

Add a :namespace: option to the qapi-doc directive, which inserts a
qapi:namespace directive into the start of the generated document. This,
in turn, associates all auto-generated definitions by this directive
with the specified namespace.

The source info for these generated lines are credited to the start of
the qapi-doc directive, which isn't precisely correct, but I wasn't sure
how to get it more accurate without some re-parsing shenanigans.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/sphinx/qapidoc.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index 604ab109a19..cc7d9c1df9c 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -451,6 +451,12 @@ def visit_entity(self, ent: QAPISchemaDefinition) -> None:
         finally:
             self._curr_ent = None
 
+    def set_namespace(self, namespace: str, source: str, lineno: int) -> None:
+        self.add_line_raw(
+            f".. qapi:namespace:: {namespace}", source, lineno + 1
+        )
+        self.ensure_blank_line()
+
 
 class QAPISchemaGenDepVisitor(QAPISchemaVisitor):
     """A QAPI schema visitor which adds Sphinx dependencies each module
@@ -496,6 +502,7 @@ class QAPIDocDirective(NestedDirective):
     optional_arguments = 1
     option_spec = {
         "qapifile": directives.unchanged_required,
+        "namespace": directives.unchanged,
         "transmogrify": directives.flag,
     }
     has_content = False
@@ -510,6 +517,11 @@ def transmogrify(self, schema: QAPISchema) -> nodes.Element:
         vis = Transmogrifier()
         modules = set()
 
+        if "namespace" in self.options:
+            vis.set_namespace(
+                self.options["namespace"], *self.get_source_info()
+            )
+
         for doc in schema.docs:
             module_source = doc.info.fname
             if module_source not in modules:
-- 
2.48.1

[PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[John Snow]

This patch does three things:

1. Record the current namespace context in pending_xrefs so it can be
   used for link resolution later,
2. Pass that recorded namespace context to find_obj() when resolving a
   reference, and
3. Wildly and completely rewrite find_obj().

cross-reference support is expanded to tolerate the presence or absence
of either namespace or module, and to cope with the presence or absence
of contextual information for either.

References now work like this:

1. If the explicit reference target is recorded in the domain's object
   registry, we link to that target and stop looking. We do this lookup
   regardless of how fully qualified the target is, which allows direct
   references to modules (which don't have a module component to their
   names) or direct references to definitions that may or may not belong
   to a namespace or module.

2. If contextual information is available from qapi:namespace or
   qapi:module directives, try using those components to find a direct
   match to the implied target name.

3. If both prior lookups fail, generate a series of regular expressions
   looking for wildcard matches in order from most to least
   specific. Any explicitly provided components (namespace, module)
   *must* match exactly, but both contextual and entirely omitted
   components are allowed to differ from the search result. Note that if
   more than one result is found, Sphinx will emit a warning (a build
   error for QEMU) and list all of the candidate references.

The practical upshot is that in the large majority of cases, namespace
and module information is not required when creating simple `references`
to definitions from within the same context -- even when identical
definitions exist in other contexts.

Even when using simple `references` from elsewhere in the QEMU
documentation manual, explicit namespace info is not required if there
is only one definition by that name.

Disambiguation *will* be required from outside of the QAPI documentation
when referencing e.g. block-core definitions, which are shared between
QEMU QMP and the QEMU Storage Daemon. In that case, there are two
options:

A: References can be made partially or fully explicit,
   e.g. `QMP:block-dirty-bitmap-add` will link to the QEMU version of
   the definition, while `QGA:block-dirty-bitmap-add` would link to the
   QGA version.

B: If all of the references in a document are intended to go to the same
   place, you can insert a "qapi:namespace:: QMP" directive to influence
   the fuzzy-searching for later references.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/devel/qapi-domain.rst |  34 ++++++++--
 docs/sphinx/qapi_domain.py | 127 ++++++++++++++++++++++++-------------
 2 files changed, 114 insertions(+), 47 deletions(-)

diff --git a/docs/devel/qapi-domain.rst b/docs/devel/qapi-domain.rst
index 73e13ab45ca..4705ba255e0 100644
--- a/docs/devel/qapi-domain.rst
+++ b/docs/devel/qapi-domain.rst
@@ -400,11 +400,10 @@ Namespaces
 Mimicking the `Python domain target specification syntax
 <https://www.sphinx-doc.org/en/master/usage/domains/python.html#target-specification>`_,
 QAPI allows you to specify the fully qualified path for a data
-type. QAPI enforces globally unique names, so it's unlikely you'll need
-this specific feature, but it may be extended in the near future to
-allow referencing identically named commands and data types from
-different utilities; i.e. QEMU Storage Daemon vs QMP.
+type.
 
+* A namespace can be explicitly provided;
+  e.g. ``:qapi:type:`QMP:BitmapSyncMode``
 * A module can be explicitly provided;
   ``:qapi:type:`block-core.BitmapSyncMode``` will render to:
   :qapi:type:`block-core.BitmapSyncMode`
@@ -413,6 +412,28 @@ different utilities; i.e. QEMU Storage Daemon vs QMP.
   will render to: :qapi:type:`~block-core.BitmapSyncMode`
 
 
+Target resolution
+-----------------
+
+Any cross-reference to a QAPI type, whether using the ```any``` style of
+reference or the more explicit ```:qapi:any:`target``` syntax, allows
+for the presence or absence of either the namespace or module
+information.
+
+When absent, their value will be inferred from context by the presence
+of any ``qapi:namespace`` or ``qapi:module`` directives preceding the
+cross-reference.
+
+If no results are found when using the inferred values, other
+namespaces/modules will be searched as a last resort; but any explicitly
+provided values must always match in order to succeed.
+
+This allows for efficient cross-referencing with a minimum of syntax in
+the large majority of cases, but additional context or namespace markup
+may be required outside of the QAPI reference documents when linking to
+items that share a name across multiple documented QAPI schema.
+
+
 Custom link text
 ----------------
 
@@ -492,6 +513,11 @@ directives are associated with the most recent namespace. This affects
 the definition's "fully qualified name", allowing two different
 namespaces to create an otherwise identically named definition.
 
+This directive also influences how reference resolution works for any
+references that do not explicity specify a namespace, so this directive
+can be used to nudge references into preferring targets from within that
+namespace.
+
 Example::
 
    .. qapi:namespace:: QMP
diff --git a/docs/sphinx/qapi_domain.py b/docs/sphinx/qapi_domain.py
index a204af9b062..a8a85a2de36 100644
--- a/docs/sphinx/qapi_domain.py
+++ b/docs/sphinx/qapi_domain.py
@@ -7,6 +7,7 @@
 
 from __future__ import annotations
 
+import re
 from typing import (
     TYPE_CHECKING,
     List,
@@ -94,6 +95,7 @@ def process_link(
         title: str,
         target: str,
     ) -> tuple[str, str]:
+        refnode["qapi:namespace"] = env.ref_context.get("qapi:namespace")
         refnode["qapi:module"] = env.ref_context.get("qapi:module")
 
         # Cross-references that begin with a tilde adjust the title to
@@ -830,40 +832,44 @@ def merge_domaindata(
                 self.objects[fullname] = obj
 
     def find_obj(
-        self, modname: str, name: str, typ: Optional[str]
-    ) -> list[tuple[str, ObjectEntry]]:
+        self, namespace: str, modname: str, name: str, typ: Optional[str]
+    ) -> List[Tuple[str, ObjectEntry]]:
         """
-        Find a QAPI object for "name", perhaps using the given module.
+        Find a QAPI object for "name", maybe using contextual information.
 
         Returns a list of (name, object entry) tuples.
 
-        :param modname: The current module context (if any!)
-                        under which we are searching.
-        :param name: The name of the x-ref to resolve;
-                     may or may not include a leading module.
-        :param type: The role name of the x-ref we're resolving, if provided.
-                     (This is absent for "any" lookups.)
+        :param namespace: The current namespace context (if any!) under
+            which we are searching.
+        :param modname: The current module context (if any!) under
+            which we are searching.
+        :param name: The name of the x-ref to resolve; may or may not
+            include leading context.
+        :param type: The role name of the x-ref we're resolving, if
+            provided. This is absent for "any" role lookups.
         """
         if not name:
             return []
 
-        names: list[str] = []
-        matches: list[tuple[str, ObjectEntry]] = []
+        # ##
+        # what to search for
+        # ##
 
-        fullname = name
-        if "." in fullname:
-            # We're searching for a fully qualified reference;
-            # ignore the contextual module.
-            pass
-        elif modname:
-            # We're searching for something from somewhere;
-            # try searching the current module first.
-            # e.g. :qapi:cmd:`query-block` or `query-block` is being searched.
-            fullname = f"{modname}.{name}"
+        parts = list(QAPIDescription.split_fqn(name))
+        explicit = tuple(bool(x) for x in parts)
+
+        # Fill in the blanks where possible:
+        if namespace and not parts[0]:
+            parts[0] = namespace
+        if modname and not parts[1]:
+            parts[1] = modname
+
+        implicit_fqn = ""
+        if all(parts):
+            implicit_fqn = f"{parts[0]}:{parts[1]}.{parts[2]}"
 
         if typ is None:
-            # type isn't specified, this is a generic xref.
-            # search *all* qapi-specific object types.
+            # :any: lookup, search everything:
             objtypes: List[str] = list(self.object_types)
         else:
             # type is specified and will be a role (e.g. obj, mod, cmd)
@@ -871,25 +877,57 @@ def find_obj(
             # using the QAPIDomain.object_types table.
             objtypes = self.objtypes_for_role(typ, [])
 
-        if name in self.objects and self.objects[name].objtype in objtypes:
-            names = [name]
-        elif (
-            fullname in self.objects
-            and self.objects[fullname].objtype in objtypes
-        ):
-            names = [fullname]
+        # ##
+        # search!
+        # ##
+
+        def _search(needle: str) -> List[str]:
+            if (
+                needle
+                and needle in self.objects
+                and self.objects[needle].objtype in objtypes
+            ):
+                return [needle]
+            return []
+
+        if found := _search(name):
+            # Exact match!
+            pass
+        elif found := _search(implicit_fqn):
+            # Exact match using contextual information to fill in the gaps.
+            pass
         else:
-            # exact match wasn't found; e.g. we are searching for
-            # `query-block` from a different (or no) module.
-            searchname = "." + name
-            names = [
-                oname
-                for oname in self.objects
-                if oname.endswith(searchname)
-                and self.objects[oname].objtype in objtypes
-            ]
-
-        matches = [(oname, self.objects[oname]) for oname in names]
+            # No exact hits, perform applicable fuzzy searches.
+            searches = []
+
+            esc = tuple(re.escape(s) for s in parts)
+
+            # Try searching for ns:*.name or ns:name
+            if explicit[0] and not explicit[1]:
+                searches.append(f"^{esc[0]}:([^\\.]+\\.)?{esc[2]}$")
+            # Try searching for *:module.name or module.name
+            if explicit[1] and not explicit[0]:
+                searches.append(f"(^|:){esc[1]}\\.{esc[2]}$")
+            # Try searching for context-ns:*.name or context-ns:name
+            if parts[0] and not (explicit[0] or explicit[1]):
+                searches.append(f"^{esc[0]}:([^\\.]+\\.)?{esc[2]}$")
+            # Try searching for *:context-mod.name or context-mod.name
+            if parts[1] and not (explicit[0] or explicit[1]):
+                searches.append(f"(^|:){esc[1]}\\.{esc[2]}$")
+            # Try searching for *:name, *.name, or name
+            if not (explicit[0] or explicit[1]):
+                searches.append(f"(^|:|\\.){esc[2]}$")
+
+            for search in searches:
+                if found := [
+                    oname
+                    for oname in self.objects
+                    if re.search(search, oname)
+                    and self.objects[oname].objtype in objtypes
+                ]:
+                    break
+
+        matches = [(oname, self.objects[oname]) for oname in found]
         if len(matches) > 1:
             matches = [m for m in matches if not m[1].aliased]
         return matches
@@ -904,8 +942,9 @@ def resolve_xref(
         node: pending_xref,
         contnode: Element,
     ) -> nodes.reference | None:
+        namespace = node.get("qapi:namespace")
         modname = node.get("qapi:module")
-        matches = self.find_obj(modname, target, typ)
+        matches = self.find_obj(namespace, modname, target, typ)
 
         if not matches:
             # Normally, we could pass warn_dangling=True to QAPIXRefRole(),
@@ -958,7 +997,9 @@ def resolve_any_xref(
         contnode: Element,
     ) -> List[Tuple[str, nodes.reference]]:
         results: List[Tuple[str, nodes.reference]] = []
-        matches = self.find_obj(node.get("qapi:module"), target, None)
+        matches = self.find_obj(
+            node.get("qapi:namespace"), node.get("qapi:module"), target, None
+        )
         for name, obj in matches:
             rolename = self.role_for_objtype(obj.objtype)
             assert rolename is not None
-- 
2.48.1

[PATCH 08/11] docs/qapi-domain: add namespaced index support

[John Snow]

Generate an index-per-namespace for the QAPI domain. Due to a limitation
with Sphinx's architecture, these indices must be defined during setup
time and cannot be dynamically created on-demand when a namespace
directive is encountered.

Owing to that limitation, add a configuration value to conf.py that
specifies which QAPI namespaces we'll generate indices for.

Indices will be named after their namespace, e.g. the "QMP" namespace
will generate to "qapi-qmp-index.html" and can be referenced using
`qapi-qmp-index`.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/conf.py               |  3 +++
 docs/sphinx/qapi_domain.py | 51 +++++++++++++++++++++++++++-----------
 2 files changed, 40 insertions(+), 14 deletions(-)

diff --git a/docs/conf.py b/docs/conf.py
index a3f9fa63d94..175491148c3 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -161,6 +161,9 @@
     "see also",
 }
 
+# Due to a limitation in Sphinx, we need to know which indices to
+# generate in advance. Adding a namespace here allows that generation.
+qapi_namespaces = set()
 
 # -- Options for HTML output ----------------------------------------------
 
diff --git a/docs/sphinx/qapi_domain.py b/docs/sphinx/qapi_domain.py
index a8a85a2de36..c94af5719ca 100644
--- a/docs/sphinx/qapi_domain.py
+++ b/docs/sphinx/qapi_domain.py
@@ -8,11 +8,13 @@
 from __future__ import annotations
 
 import re
+import types
 from typing import (
     TYPE_CHECKING,
     List,
     NamedTuple,
     Tuple,
+    Type,
     cast,
 )
 
@@ -669,6 +671,7 @@ class QAPIIndex(Index):
     name = "index"
     localname = _("QAPI Index")
     shortname = _("QAPI Index")
+    namespace = ""
 
     def generate(
         self,
@@ -678,25 +681,20 @@ def generate(
         content: Dict[str, List[IndexEntry]] = {}
         collapse = False
 
-        # list of all object (name, ObjectEntry) pairs, sorted by name
-        # (ignoring the module)
-        objects = sorted(
-            self.domain.objects.items(),
-            key=lambda x: x[0].split(".")[-1].lower(),
-        )
-
-        for objname, obj in objects:
+        for objname, obj in self.domain.objects.items():
             if docnames and obj.docname not in docnames:
                 continue
 
-            # Strip the module name out:
-            objname = objname.split(".")[-1]
+            ns, _mod, name = QAPIDescription.split_fqn(objname)
+
+            if self.namespace != ns:
+                continue
 
             # Add an alphabetical entry:
-            entries = content.setdefault(objname[0].upper(), [])
+            entries = content.setdefault(name[0].upper(), [])
             entries.append(
                 IndexEntry(
-                    objname, 0, obj.docname, obj.node_id, obj.objtype, "", ""
+                    name, 0, obj.docname, obj.node_id, obj.objtype, "", ""
                 )
             )
 
@@ -704,10 +702,14 @@ def generate(
             category = obj.objtype.title() + "s"
             entries = content.setdefault(category, [])
             entries.append(
-                IndexEntry(objname, 0, obj.docname, obj.node_id, "", "", "")
+                IndexEntry(name, 0, obj.docname, obj.node_id, "", "", "")
             )
 
-        # alphabetically sort categories; type names first, ABC entries last.
+        # Sort entries within each category alphabetically
+        for category in content:
+            content[category] = sorted(content[category])
+
+        # Sort the categories themselves; type names first, ABC entries last.
         sorted_content = sorted(
             content.items(),
             key=lambda x: (len(x[0]) == 1, x[0]),
@@ -780,6 +782,21 @@ def objects(self) -> Dict[str, ObjectEntry]:
         ret = self.data.setdefault("objects", {})
         return ret  # type: ignore[no-any-return]
 
+    def setup(self) -> None:
+        namespaces = set(self.env.app.config.qapi_namespaces)
+        for namespace in namespaces:
+            new_index: Type[QAPIIndex] = types.new_class(
+                f"{namespace}Index", bases=(QAPIIndex,)
+            )
+            new_index.name = f"{namespace.lower()}-index"
+            new_index.localname = _(f"{namespace} Index")
+            new_index.shortname = _(f"{namespace} Index")
+            new_index.namespace = namespace
+
+            self.indices.append(new_index)
+
+        super().setup()
+
     def note_object(
         self,
         name: str,
@@ -1019,6 +1036,12 @@ def setup(app: Sphinx) -> Dict[str, Any]:
         "env",  # Setting impacts parsing phase
         types=set,
     )
+    app.add_config_value(
+        "qapi_namespaces",
+        set(),
+        "env",
+        types=set,
+    )
     app.add_domain(QAPIDomain)
 
     return {
-- 
2.48.1

[PATCH 09/11] docs: add QAPI namespace "QMP" to qemu-qmp-ref

[John Snow]

This also creates the qapi-qmp-index.html index and cross-reference
target.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/conf.py                  | 4 +++-
 docs/interop/qemu-qmp-ref.rst | 1 +
 qapi/qapi-schema.json         | 2 +-
 3 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/docs/conf.py b/docs/conf.py
index 175491148c3..9a86e84a804 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -163,7 +163,9 @@
 
 # Due to a limitation in Sphinx, we need to know which indices to
 # generate in advance. Adding a namespace here allows that generation.
-qapi_namespaces = set()
+qapi_namespaces = {
+    "QMP",
+}
 
 # -- Options for HTML output ----------------------------------------------
 
diff --git a/docs/interop/qemu-qmp-ref.rst b/docs/interop/qemu-qmp-ref.rst
index e95eeac45e2..ef8792b53f5 100644
--- a/docs/interop/qemu-qmp-ref.rst
+++ b/docs/interop/qemu-qmp-ref.rst
@@ -8,3 +8,4 @@ QEMU QMP Reference Manual
 
 .. qapi-doc:: qapi/qapi-schema.json
    :transmogrify:
+   :namespace: QMP
diff --git a/qapi/qapi-schema.json b/qapi/qapi-schema.json
index 4475e81cc3e..c41c01eb2ab 100644
--- a/qapi/qapi-schema.json
+++ b/qapi/qapi-schema.json
@@ -5,7 +5,7 @@
 #
 # This document describes all commands currently supported by QMP.
 #
-# For locating a particular item, please see the `qapi-index`.
+# For locating a particular item, please see the `qapi-qmp-index`.
 #
 # Most of the time their usage is exactly the same as in the user
 # Monitor, this means that any other document which also describe
-- 
2.48.1

[PATCH 10/11] docs: disambiguate references in qapi-domain.rst

[John Snow]

Before we enable the QGA and QSD namespaces, we need to disambiguate
some of the references that would become ambiguous as a result!

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/devel/qapi-domain.rst | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/docs/devel/qapi-domain.rst b/docs/devel/qapi-domain.rst
index 4705ba255e0..a748529f515 100644
--- a/docs/devel/qapi-domain.rst
+++ b/docs/devel/qapi-domain.rst
@@ -385,13 +385,13 @@ Type names in references can be surrounded by brackets, like
 ``[typename]``, to indicate an array of that type.  The cross-reference
 will apply only to the type name between the brackets. For example;
 ``:qapi:type:`[Qcow2BitmapInfoFlags]``` renders to:
-:qapi:type:`[Qcow2BitmapInfoFlags]`
+:qapi:type:`[QMP:Qcow2BitmapInfoFlags]`
 
 To indicate an optional argument/member in a field list, the type name
 can be suffixed with ``?``. The cross-reference will be transformed to
 "type, Optional" with the link applying only to the type name. For
 example; ``:qapi:type:`BitmapSyncMode?``` renders to:
-:qapi:type:`BitmapSyncMode?`
+:qapi:type:`QMP:BitmapSyncMode?`
 
 
 Namespaces
@@ -405,11 +405,11 @@ type.
 * A namespace can be explicitly provided;
   e.g. ``:qapi:type:`QMP:BitmapSyncMode``
 * A module can be explicitly provided;
-  ``:qapi:type:`block-core.BitmapSyncMode``` will render to:
-  :qapi:type:`block-core.BitmapSyncMode`
+  ``:qapi:type:`QMP:block-core.BitmapSyncMode``` will render to:
+  :qapi:type:`QMP:block-core.BitmapSyncMode`
 * If you don't want to display the "fully qualified" name, it can be
-  prefixed with a tilde; ``:qapi:type:`~block-core.BitmapSyncMode```
-  will render to: :qapi:type:`~block-core.BitmapSyncMode`
+  prefixed with a tilde; ``:qapi:type:`~QMP:block-core.BitmapSyncMode```
+  will render to: :qapi:type:`~QMP:block-core.BitmapSyncMode`
 
 
 Target resolution
@@ -444,7 +444,7 @@ using the ``custom text <target>`` syntax.
 
 For example, ``:qapi:cmd:`Merge dirty bitmaps
 <block-dirty-bitmap-merge>``` will render as: :qapi:cmd:`Merge dirty
-bitmaps <block-dirty-bitmap-merge>`
+bitmaps <QMP:block-dirty-bitmap-merge>`
 
 
 Directives
-- 
2.48.1

[PATCH 11/11] docs: enable transmogrifier for QSD and QGA

[John Snow]

This also creates the `qapi-qsd-index` and `qapi-qga-index` QMP indices.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/conf.py                                 | 2 ++
 docs/interop/qemu-ga-ref.rst                 | 2 ++
 docs/interop/qemu-storage-daemon-qmp-ref.rst | 2 ++
 qga/qapi-schema.json                         | 3 +++
 storage-daemon/qapi/qapi-schema.json         | 8 ++++++++
 5 files changed, 17 insertions(+)

diff --git a/docs/conf.py b/docs/conf.py
index 9a86e84a804..7b5712e122f 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -164,7 +164,9 @@
 # Due to a limitation in Sphinx, we need to know which indices to
 # generate in advance. Adding a namespace here allows that generation.
 qapi_namespaces = {
+    "QGA",
     "QMP",
+    "QSD",
 }
 
 # -- Options for HTML output ----------------------------------------------
diff --git a/docs/interop/qemu-ga-ref.rst b/docs/interop/qemu-ga-ref.rst
index 032d4924552..19b5c7a5498 100644
--- a/docs/interop/qemu-ga-ref.rst
+++ b/docs/interop/qemu-ga-ref.rst
@@ -5,3 +5,5 @@ QEMU Guest Agent Protocol Reference
    :depth: 3
 
 .. qapi-doc:: qga/qapi-schema.json
+   :transmogrify:
+   :namespace: QGA
diff --git a/docs/interop/qemu-storage-daemon-qmp-ref.rst b/docs/interop/qemu-storage-daemon-qmp-ref.rst
index 9fed68152f5..d0228d63b8d 100644
--- a/docs/interop/qemu-storage-daemon-qmp-ref.rst
+++ b/docs/interop/qemu-storage-daemon-qmp-ref.rst
@@ -5,3 +5,5 @@ QEMU Storage Daemon QMP Reference Manual
    :depth: 3
 
 .. qapi-doc:: storage-daemon/qapi/qapi-schema.json
+   :transmogrify:
+   :namespace: QSD
diff --git a/qga/qapi-schema.json b/qga/qapi-schema.json
index 995594aaf43..35ec0e7db31 100644
--- a/qga/qapi-schema.json
+++ b/qga/qapi-schema.json
@@ -3,6 +3,9 @@
 
 ##
 # = QEMU guest agent protocol commands and structs
+#
+# For a concise listing of all commands, events, and types in the QEMU
+# guest agent, please consult the `qapi-qga-index`.
 ##
 
 { 'pragma': { 'doc-required': true } }
diff --git a/storage-daemon/qapi/qapi-schema.json b/storage-daemon/qapi/qapi-schema.json
index f10c9494906..2a562ee32e5 100644
--- a/storage-daemon/qapi/qapi-schema.json
+++ b/storage-daemon/qapi/qapi-schema.json
@@ -13,6 +13,14 @@
 # the array type in the main schema, even if it is unused outside of the
 # storage daemon.
 
+##
+# = QEMU storage daemon protocol commands and structs
+#
+# For a concise listing of all commands, events, and types in the QEMU
+# storage daemon, please consult the `qapi-qsd-index`.
+##
+
+
 { 'include': '../../qapi/pragma.json' }
 
 # Documentation generated with qapi-gen.py is in source order, with
-- 
2.48.1

Re: [PATCH 04/11] docs/qapi-domain: add :namespace: override option

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Akin to the :module: override option, the :namespace: options allows you
> to forcibly override the contextual namespace associatied with a
> definition.
>
> We don't necessarily actually need this, but I felt compelled to stick
> close to how the Python domain works that offers context overrides.
>
> As of this commit, it is possible to add e.g. ":namespace: QMP" to any
> QAPI directive to forcibly associate that definition with a given
> namespace.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  docs/devel/qapi-domain.rst | 2 ++
>  docs/sphinx/qapi_domain.py | 5 +++--
>  2 files changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/docs/devel/qapi-domain.rst b/docs/devel/qapi-domain.rst
> index 1475870ca6c..51b283277e1 100644
> --- a/docs/devel/qapi-domain.rst
> +++ b/docs/devel/qapi-domain.rst
> @@ -466,6 +466,8 @@ QAPI standard options
>  
>  All QAPI directives -- *except* for module -- support these common options.
>  
> +* ``:namespace: name`` -- This option allows you to override the
> +  namespace association of a given definition.
>  * ``:module: modname`` -- Borrowed from the Python domain, this option allows
>    you to override the module association of a given definition.
>  * ``:since: x.y`` -- Allows the documenting of "Since" information, which is
> diff --git a/docs/sphinx/qapi_domain.py b/docs/sphinx/qapi_domain.py
> index 48a082d489a..6485c432063 100644
> --- a/docs/sphinx/qapi_domain.py
> +++ b/docs/sphinx/qapi_domain.py
> @@ -294,8 +294,9 @@ class QAPIObject(QAPIDescription):
>      )
>      option_spec.update(
>          {
> -            # Borrowed from the Python domain:
> -            "module": directives.unchanged,  # Override contextual module name
> +            # Context overrides:
> +            "namespace": directives.unchanged,
> +            "module": directives.unchanged,
>              # These are QAPI originals:
>              "since": directives.unchanged,
>              "ifcond": directives.unchanged,

Did "Borrowed from the Python domain" become wrong(ish) in PATCH 02?

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> This patch does three things:
>
> 1. Record the current namespace context in pending_xrefs so it can be
>    used for link resolution later,
> 2. Pass that recorded namespace context to find_obj() when resolving a
>    reference, and
> 3. Wildly and completely rewrite find_obj().
>
> cross-reference support is expanded to tolerate the presence or absence
> of either namespace or module, and to cope with the presence or absence
> of contextual information for either.
>
> References now work like this:
>
> 1. If the explicit reference target is recorded in the domain's object
>    registry, we link to that target and stop looking. We do this lookup
>    regardless of how fully qualified the target is, which allows direct
>    references to modules (which don't have a module component to their
>    names) or direct references to definitions that may or may not belong
>    to a namespace or module.
>
> 2. If contextual information is available from qapi:namespace or
>    qapi:module directives, try using those components to find a direct
>    match to the implied target name.
>
> 3. If both prior lookups fail, generate a series of regular expressions
>    looking for wildcard matches in order from most to least
>    specific. Any explicitly provided components (namespace, module)
>    *must* match exactly, but both contextual and entirely omitted
>    components are allowed to differ from the search result. Note that if
>    more than one result is found, Sphinx will emit a warning (a build
>    error for QEMU) and list all of the candidate references.
>
> The practical upshot is that in the large majority of cases, namespace
> and module information is not required when creating simple `references`
> to definitions from within the same context -- even when identical
> definitions exist in other contexts.

Can you illustrate this this examples?

> Even when using simple `references` from elsewhere in the QEMU
> documentation manual, explicit namespace info is not required if there
> is only one definition by that name.

Fails safely: if we add a second definition, doc generation errors out.
Correct?

> Disambiguation *will* be required from outside of the QAPI documentation
> when referencing e.g. block-core definitions, which are shared between
> QEMU QMP and the QEMU Storage Daemon. In that case, there are two
> options:
>
> A: References can be made partially or fully explicit,
>    e.g. `QMP:block-dirty-bitmap-add` will link to the QEMU version of
>    the definition, while `QGA:block-dirty-bitmap-add` would link to the
>    QGA version.

Do you mean "QSD:"?

> B: If all of the references in a document are intended to go to the same
>    place, you can insert a "qapi:namespace:: QMP" directive to influence
>    the fuzzy-searching for later references.
>
> Signed-off-by: John Snow <jsnow@redhat.com>

Re: [PATCH 11/11] docs: enable transmogrifier for QSD and QGA

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> This also creates the `qapi-qsd-index` and `qapi-qga-index` QMP indices.
>
> Signed-off-by: John Snow <jsnow@redhat.com>

[...]

> diff --git a/qga/qapi-schema.json b/qga/qapi-schema.json
> index 995594aaf43..35ec0e7db31 100644
> --- a/qga/qapi-schema.json
> +++ b/qga/qapi-schema.json
> @@ -3,6 +3,9 @@
>  
>  ##
>  # = QEMU guest agent protocol commands and structs
> +#
> +# For a concise listing of all commands, events, and types in the QEMU
> +# guest agent, please consult the `qapi-qga-index`.
>  ##
>  
>  { 'pragma': { 'doc-required': true } }
> diff --git a/storage-daemon/qapi/qapi-schema.json b/storage-daemon/qapi/qapi-schema.json
> index f10c9494906..2a562ee32e5 100644
> --- a/storage-daemon/qapi/qapi-schema.json
> +++ b/storage-daemon/qapi/qapi-schema.json
> @@ -13,6 +13,14 @@
>  # the array type in the main schema, even if it is unused outside of the
>  # storage daemon.
>  
> +##
> +# = QEMU storage daemon protocol commands and structs
> +#
> +# For a concise listing of all commands, events, and types in the QEMU
> +# storage daemon, please consult the `qapi-qsd-index`.
> +##
> +
> +
>  { 'include': '../../qapi/pragma.json' }
>  
>  # Documentation generated with qapi-gen.py is in source order, with

Compare qapi/qapi-schema.json:

   # = Introduction
   #
   # This document describes all commands currently supported by QMP.
   #
   # For locating a particular item, please see the `qapi-qmp-index`.
   #

Suggest to pick one phrasing and stick to it, unless there's a reason
for more than one.

Re: [PATCH 04/11] docs/qapi-domain: add :namespace: override option

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2412 bytes --]

On Thu, Mar 13, 2025 at 2:39 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > Akin to the :module: override option, the :namespace: options allows you
> > to forcibly override the contextual namespace associatied with a
> > definition.
> >
> > We don't necessarily actually need this, but I felt compelled to stick
> > close to how the Python domain works that offers context overrides.
> >
> > As of this commit, it is possible to add e.g. ":namespace: QMP" to any
> > QAPI directive to forcibly associate that definition with a given
> > namespace.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  docs/devel/qapi-domain.rst | 2 ++
> >  docs/sphinx/qapi_domain.py | 5 +++--
> >  2 files changed, 5 insertions(+), 2 deletions(-)
> >
> > diff --git a/docs/devel/qapi-domain.rst b/docs/devel/qapi-domain.rst
> > index 1475870ca6c..51b283277e1 100644
> > --- a/docs/devel/qapi-domain.rst
> > +++ b/docs/devel/qapi-domain.rst
> > @@ -466,6 +466,8 @@ QAPI standard options
> >
> >  All QAPI directives -- *except* for module -- support these common
> options.
> >
> > +* ``:namespace: name`` -- This option allows you to override the
> > +  namespace association of a given definition.
> >  * ``:module: modname`` -- Borrowed from the Python domain, this option
> allows
> >    you to override the module association of a given definition.
> >  * ``:since: x.y`` -- Allows the documenting of "Since" information,
> which is
> > diff --git a/docs/sphinx/qapi_domain.py b/docs/sphinx/qapi_domain.py
> > index 48a082d489a..6485c432063 100644
> > --- a/docs/sphinx/qapi_domain.py
> > +++ b/docs/sphinx/qapi_domain.py
> > @@ -294,8 +294,9 @@ class QAPIObject(QAPIDescription):
> >      )
> >      option_spec.update(
> >          {
> > -            # Borrowed from the Python domain:
> > -            "module": directives.unchanged,  # Override contextual
> module name
> > +            # Context overrides:
> > +            "namespace": directives.unchanged,
> > +            "module": directives.unchanged,
> >              # These are QAPI originals:
> >              "since": directives.unchanged,
> >              "ifcond": directives.unchanged,
>
> Did "Borrowed from the Python domain" become wrong(ish) in PATCH 02?
>

:module: is directly borrowed, :namespace: is only spiritually similar.

[-- Attachment #2: Type: text/html, Size: 3268 bytes --]

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 3337 bytes --]

On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > This patch does three things:
> >
> > 1. Record the current namespace context in pending_xrefs so it can be
> >    used for link resolution later,
> > 2. Pass that recorded namespace context to find_obj() when resolving a
> >    reference, and
> > 3. Wildly and completely rewrite find_obj().
> >
> > cross-reference support is expanded to tolerate the presence or absence
> > of either namespace or module, and to cope with the presence or absence
> > of contextual information for either.
> >
> > References now work like this:
> >
> > 1. If the explicit reference target is recorded in the domain's object
> >    registry, we link to that target and stop looking. We do this lookup
> >    regardless of how fully qualified the target is, which allows direct
> >    references to modules (which don't have a module component to their
> >    names) or direct references to definitions that may or may not belong
> >    to a namespace or module.
> >
> > 2. If contextual information is available from qapi:namespace or
> >    qapi:module directives, try using those components to find a direct
> >    match to the implied target name.
> >
> > 3. If both prior lookups fail, generate a series of regular expressions
> >    looking for wildcard matches in order from most to least
> >    specific. Any explicitly provided components (namespace, module)
> >    *must* match exactly, but both contextual and entirely omitted
> >    components are allowed to differ from the search result. Note that if
> >    more than one result is found, Sphinx will emit a warning (a build
> >    error for QEMU) and list all of the candidate references.
> >
> > The practical upshot is that in the large majority of cases, namespace
> > and module information is not required when creating simple `references`
> > to definitions from within the same context -- even when identical
> > definitions exist in other contexts.
>
> Can you illustrate this this examples?
>

do wha?


>
> > Even when using simple `references` from elsewhere in the QEMU
> > documentation manual, explicit namespace info is not required if there
> > is only one definition by that name.
>
> Fails safely: if we add a second definition, doc generation errors out.
> Correct?
>

Yes, see the disambiguation patch for qapi-domain.rst for proof. Roll it
back and see what happens!


>
> > Disambiguation *will* be required from outside of the QAPI documentation
> > when referencing e.g. block-core definitions, which are shared between
> > QEMU QMP and the QEMU Storage Daemon. In that case, there are two
> > options:
> >
> > A: References can be made partially or fully explicit,
> >    e.g. `QMP:block-dirty-bitmap-add` will link to the QEMU version of
> >    the definition, while `QGA:block-dirty-bitmap-add` would link to the
> >    QGA version.
>
> Do you mean "QSD:"?
>

Haha! Yes, I suppose I did.


>
> > B: If all of the references in a document are intended to go to the same
> >    place, you can insert a "qapi:namespace:: QMP" directive to influence
> >    the fuzzy-searching for later references.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
>
>

[-- Attachment #2: Type: text/html, Size: 4635 bytes --]

Re: [PATCH 11/11] docs: enable transmogrifier for QSD and QGA

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 1856 bytes --]

On Thu, Mar 13, 2025 at 2:54 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > This also creates the `qapi-qsd-index` and `qapi-qga-index` QMP indices.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
>
> [...]
>
> > diff --git a/qga/qapi-schema.json b/qga/qapi-schema.json
> > index 995594aaf43..35ec0e7db31 100644
> > --- a/qga/qapi-schema.json
> > +++ b/qga/qapi-schema.json
> > @@ -3,6 +3,9 @@
> >
> >  ##
> >  # = QEMU guest agent protocol commands and structs
> > +#
> > +# For a concise listing of all commands, events, and types in the QEMU
> > +# guest agent, please consult the `qapi-qga-index`.
> >  ##
> >
> >  { 'pragma': { 'doc-required': true } }
> > diff --git a/storage-daemon/qapi/qapi-schema.json
> b/storage-daemon/qapi/qapi-schema.json
> > index f10c9494906..2a562ee32e5 100644
> > --- a/storage-daemon/qapi/qapi-schema.json
> > +++ b/storage-daemon/qapi/qapi-schema.json
> > @@ -13,6 +13,14 @@
> >  # the array type in the main schema, even if it is unused outside of the
> >  # storage daemon.
> >
> > +##
> > +# = QEMU storage daemon protocol commands and structs
> > +#
> > +# For a concise listing of all commands, events, and types in the QEMU
> > +# storage daemon, please consult the `qapi-qsd-index`.
> > +##
> > +
> > +
> >  { 'include': '../../qapi/pragma.json' }
> >
> >  # Documentation generated with qapi-gen.py is in source order, with
>
> Compare qapi/qapi-schema.json:
>
>    # = Introduction
>    #
>    # This document describes all commands currently supported by QMP.
>    #
>    # For locating a particular item, please see the `qapi-qmp-index`.
>    #
>
> Suggest to pick one phrasing and stick to it, unless there's a reason
> for more than one.
>

Nope, just free-handing it. What's your favorite? ;)

[-- Attachment #2: Type: text/html, Size: 2606 bytes --]

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > This patch does three things:
>> >
>> > 1. Record the current namespace context in pending_xrefs so it can be
>> >    used for link resolution later,
>> > 2. Pass that recorded namespace context to find_obj() when resolving a
>> >    reference, and
>> > 3. Wildly and completely rewrite find_obj().
>> >
>> > cross-reference support is expanded to tolerate the presence or absence
>> > of either namespace or module, and to cope with the presence or absence
>> > of contextual information for either.
>> >
>> > References now work like this:
>> >
>> > 1. If the explicit reference target is recorded in the domain's object
>> >    registry, we link to that target and stop looking. We do this lookup
>> >    regardless of how fully qualified the target is, which allows direct
>> >    references to modules (which don't have a module component to their
>> >    names) or direct references to definitions that may or may not belong
>> >    to a namespace or module.
>> >
>> > 2. If contextual information is available from qapi:namespace or
>> >    qapi:module directives, try using those components to find a direct
>> >    match to the implied target name.
>> >
>> > 3. If both prior lookups fail, generate a series of regular expressions
>> >    looking for wildcard matches in order from most to least
>> >    specific. Any explicitly provided components (namespace, module)
>> >    *must* match exactly, but both contextual and entirely omitted
>> >    components are allowed to differ from the search result. Note that if
>> >    more than one result is found, Sphinx will emit a warning (a build
>> >    error for QEMU) and list all of the candidate references.
>> >
>> > The practical upshot is that in the large majority of cases, namespace
>> > and module information is not required when creating simple `references`
>> > to definitions from within the same context -- even when identical
>> > definitions exist in other contexts.
>>
>> Can you illustrate this this examples?
>>
>
> do wha?

Sorry, I went into the curve too fast.

The stuff under "References now work like this" confuses me.  I guess it
describes a series of lookups to try one after the other.

I understand a cross-reference consists of namespace (optional), module
(optional), name, and role.

Let's assume role is "any" for simplicity's sake.

Regarding "1. If the explicit ...":

    What is a reference's "explicit reference target"?  Examples might
    help me understand.

    What is "recorded in the domain's object registry"?

    Can you show me a reference where this lookup succeeds?

Regarding "2. If contextual information ...":

    I guess "contextual information" is the context established by
    qapi:namespace and qapi:module directives, i.e. the current
    namespace and module, if any.

    If the cross reference lacks a namespace, we substitute the current
    namespace.  Same for module.

    We then use that "to find a direct match to the implied target
    name".  Sounds greek to me.  Example(s) might help.

Regarding "3. If both prior lookups fail ...":

    I guess we get here when namespace or module are absent, and
    substituting the current namespace or module doesn't resolve.  We
    then substitute a wildcard, so to speak, i.e. look in all namespaces
    / modules, and succeed if we find exactly one resolution.  Fair?

[...]

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 7335 bytes --]

On Thu, Mar 13, 2025 at 10:41 AM Markus Armbruster <armbru@redhat.com>
wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster <armbru@redhat.com>
> wrote:
> >
> >> John Snow <jsnow@redhat.com> writes:
> >>
> >> > This patch does three things:
> >> >
> >> > 1. Record the current namespace context in pending_xrefs so it can be
> >> >    used for link resolution later,
> >> > 2. Pass that recorded namespace context to find_obj() when resolving a
> >> >    reference, and
> >> > 3. Wildly and completely rewrite find_obj().
> >> >
> >> > cross-reference support is expanded to tolerate the presence or
> absence
> >> > of either namespace or module, and to cope with the presence or
> absence
> >> > of contextual information for either.
> >> >
> >> > References now work like this:
> >> >
> >> > 1. If the explicit reference target is recorded in the domain's object
> >> >    registry, we link to that target and stop looking. We do this
> lookup
> >> >    regardless of how fully qualified the target is, which allows
> direct
> >> >    references to modules (which don't have a module component to their
> >> >    names) or direct references to definitions that may or may not
> belong
> >> >    to a namespace or module.
> >> >
> >> > 2. If contextual information is available from qapi:namespace or
> >> >    qapi:module directives, try using those components to find a direct
> >> >    match to the implied target name.
> >> >
> >> > 3. If both prior lookups fail, generate a series of regular
> expressions
> >> >    looking for wildcard matches in order from most to least
> >> >    specific. Any explicitly provided components (namespace, module)
> >> >    *must* match exactly, but both contextual and entirely omitted
> >> >    components are allowed to differ from the search result. Note that
> if
> >> >    more than one result is found, Sphinx will emit a warning (a build
> >> >    error for QEMU) and list all of the candidate references.
> >> >
> >> > The practical upshot is that in the large majority of cases, namespace
> >> > and module information is not required when creating simple
> `references`
> >> > to definitions from within the same context -- even when identical
> >> > definitions exist in other contexts.
> >>
> >> Can you illustrate this this examples?
> >>
> >
> > do wha?
>
> Sorry, I went into the curve too fast.
>
> The stuff under "References now work like this" confuses me.  I guess it
> describes a series of lookups to try one after the other.
>
> I understand a cross-reference consists of namespace (optional), module
> (optional), name, and role.
>
> Let's assume role is "any" for simplicity's sake.
>
> Regarding "1. If the explicit ...":
>
>     What is a reference's "explicit reference target"?  Examples might
>     help me understand.
>

explicit lookup: `QMP:block-core:block-dirty-bitmap-add`

If that explicit target matches an object in the object database
*directly*, we match immediately and don't consider other potential
targets. This also applies to things like modules, e.g. `QMP:block-core`
even though the "module" is absent (it IS the module)

We always search for the explicit target no matter how un/fully qualified
it is.


>
>     What is "recorded in the domain's object registry"?
>

domain.objects{} - essentially a record of every ObjectDefinition's
"fullname" - the return value from QAPIDefinition._get_fqn().


>
>     Can you show me a reference where this lookup succeeds?
>

`QMP:block-core`
`QMP:block-core.block-dirty-bitmap-add`


>
> Regarding "2. If contextual information ...":
>
>     I guess "contextual information" is the context established by
>     qapi:namespace and qapi:module directives, i.e. the current
>     namespace and module, if any.
>

Yep!


>
>     If the cross reference lacks a namespace, we substitute the current
>     namespace.  Same for module.
>
>     We then use that "to find a direct match to the implied target
>     name".  Sounds greek to me.  Example(s) might help.
>

If namespace or module is missing from the link target, we try to fill in
the blanks with the contextual information if present.

Example, we are in the block-core section of the QEMU QMP reference manual
document and we reference `block-dirty-bitmap-add`. With context, we are
able to assemble a fully qualified name:
"QMP:block-core.block-dirty-bitmap-add`. This matches an item in the
registry directly, so it matches and no further search is performed.


>
> Regarding "3. If both prior lookups fail ...":
>
>     I guess we get here when namespace or module are absent, and
>     substituting the current namespace or module doesn't resolve.  We
>     then substitute a wildcard, so to speak, i.e. look in all namespaces
>     / modules, and succeed if we find exactly one resolution.  Fair?
>

More or less, though the mechanics are quite a bit more complex than your
overview (and what I wrote in qapi-domain.rst.) We can get here for a few
reasons:

(1) We didn't provide a fully qualified target, and we don't have full
context to construct one. For example, we are not "in" a namespace and/or
not "in" a module. This is quite likely to happen when writing simple
references to a definition name from outside of the transmogfrified QAPI
documentation, e.g. from qapi-domain.rst itself, or dirty-bitmaps.rst, etc.

(2) We didn't provide a fully qualified target, and we are referencing
something from outside of the local context. For example, we are "in" a
module but we are trying to link to a different module's definition. e.g.
we are in QMP:transaction and we reference `block-dirty-bitmap-add`. The
implied FQN will be QMP:transaction.block-dirty-bitmap.add, which will not
resolve.

The fuzzy search portion has an order of precedence for how it searches -
and not all searches are tried universally, they are conditional to what
was provided in the reference target and what context is available.

1. match against the explicitly provided namespace (module was not
explicitly provided)
2. match against the explicitly provided module (namespace was not
explicitly provided)
3. match against the implied namespace (neither namespace/module was
explicitly provided)
4. match against the implied module (neither namespace/module was
explicitly provided)
5. match against the definition name only, from anywhere (neither
namespace/module was explicitly provided)

The searches are performed in order: if a search returns zero results, the
next search is tried. If any search returns one or more results, those
results are returned and we stop searching down the list. The priority
order ensures that any explicitly provided information is *always* used to
find a match, but contextually provided information is merely a "hint" and
can be ignored for the sake of a match.

If find_obj() as a whole returns zero results, Sphinx emits a warning for a
dangling reference. if find_obj() as a whole returns multiple results,
Sphinx emits a warning for the ambiguous cross-reference.

QEMU errors out on any such warnings under our normal build settings.

Clear as mud?
--js


> [...]
>
>

[-- Attachment #2: Type: text/html, Size: 9891 bytes --]

Re: [PATCH 11/11] docs: enable transmogrifier for QSD and QGA

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Thu, Mar 13, 2025 at 2:54 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > This also creates the `qapi-qsd-index` and `qapi-qga-index` QMP indices.
>> >
>> > Signed-off-by: John Snow <jsnow@redhat.com>
>>
>> [...]
>>
>> > diff --git a/qga/qapi-schema.json b/qga/qapi-schema.json
>> > index 995594aaf43..35ec0e7db31 100644
>> > --- a/qga/qapi-schema.json
>> > +++ b/qga/qapi-schema.json
>> > @@ -3,6 +3,9 @@
>> >
>> >  ##
>> >  # = QEMU guest agent protocol commands and structs
>> > +#
>> > +# For a concise listing of all commands, events, and types in the QEMU
>> > +# guest agent, please consult the `qapi-qga-index`.
>> >  ##
>> >
>> >  { 'pragma': { 'doc-required': true } }
>> > diff --git a/storage-daemon/qapi/qapi-schema.json b/storage-daemon/qapi/qapi-schema.json
>> > index f10c9494906..2a562ee32e5 100644
>> > --- a/storage-daemon/qapi/qapi-schema.json
>> > +++ b/storage-daemon/qapi/qapi-schema.json
>> > @@ -13,6 +13,14 @@
>> >  # the array type in the main schema, even if it is unused outside of the
>> >  # storage daemon.
>> >
>> > +##
>> > +# = QEMU storage daemon protocol commands and structs
>> > +#
>> > +# For a concise listing of all commands, events, and types in the QEMU
>> > +# storage daemon, please consult the `qapi-qsd-index`.
>> > +##
>> > +
>> > +
>> >  { 'include': '../../qapi/pragma.json' }
>> >
>> >  # Documentation generated with qapi-gen.py is in source order, with
>>
>> Compare qapi/qapi-schema.json:
>>
>>    # = Introduction
>>    #
>>    # This document describes all commands currently supported by QMP.
>>    #
>>    # For locating a particular item, please see the `qapi-qmp-index`.
>>    #
>>
>> Suggest to pick one phrasing and stick to it, unless there's a reason
>> for more than one.
>>
>
> Nope, just free-handing it. What's your favorite? ;)

Let me see...

Alright, I saw, and I hate what I saw.  But addressing that should not
be coupled to this series.  I'm going to take your patch as is, and
improve on top.

"But what exactly do you hate?" you might ask.  Have a look at how the
three rendered manuals look like.

1. QMP (docs/manual/interop/qemu-qmp-ref.html)

    QEMU QMP Reference Manual
    *************************


    Contents
    ^^^^^^^^

    * QEMU QMP Reference Manual

      * Introduction

      * QMP errors

      [More entries at this level, namely all the modules...]


    Introduction
    ============

    This document describes all commands currently supported by QMP.

    For locating a particular item, please see the QMP Index.

    Most of the time their usage is exactly the same as in the user
    Monitor, this means that any other document which also describe
    commands (the manpage, QEMU's manual, etc) can and should be
    consulted.

    QMP has two types of commands: regular and query commands.  Regular
    commands usually change the Virtual Machine's state someway, while
    query commands just return information.  The sections below are
    divided accordingly.

    It's important to observe that all communication examples are
    formatted in a reader-friendly way, so that they're easier to
    understand.  However, in real protocol usage, they're emitted as a
    single line.

    Also, the following notation is used to denote data flow:

    Example:

       -> data issued by the Client
       <- Server data response

    Please refer to the QEMU Machine Protocol Specification for detailed
    information on the Server command and response formats.


    QMP errors
    ==========

    [...]

The manual has a single top-level section, named like the manual.  WTF?

The introduction reads fine, except it should be "all commands and
events".

2. QSD

    QEMU Storage Daemon QMP Reference Manual
    ****************************************


    Contents
    ^^^^^^^^

    * QEMU Storage Daemon QMP Reference Manual

      * QEMU storage daemon protocol commands and structs

      * Common data types

      [More entries at this level, namely all the modules...]


    QEMU storage daemon protocol commands and structs
    =================================================

    For a concise listing of all commands, events, and types in the QEMU
    storage daemon, please consult the QSD Index.


    Common data types
    =================

Same section structure WTF as above.

The introduction section is crap, starting with the title.

3. QGA

    QEMU Guest Agent Protocol Reference
    ***********************************


    Contents
    ^^^^^^^^

    * QEMU Guest Agent Protocol Reference

      * QEMU guest agent protocol commands and structs


    QEMU guest agent protocol commands and structs
    ==============================================

    For a concise listing of all commands, events, and types in the QEMU
    guest agent, please consult the QGA Index.

    Command guest-sync-delimited (Since: 1.1)

       Echo back a unique integer value, and prepend to response a leading
       sentinel byte (0xFF) the client can check scan for.

       [...]

    [More definitions...]

The section structure is completely useless.  Why bury the contents two
levels deep, with meaningless section titles?

Again, the introduction section is crap, starting with the title.

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Thu, Mar 13, 2025 at 10:41 AM Markus Armbruster <armbru@redhat.com>
> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster <armbru@redhat.com> wrote:
>> >
>> >> John Snow <jsnow@redhat.com> writes:
>> >>
>> >> > This patch does three things:
>> >> >
>> >> > 1. Record the current namespace context in pending_xrefs so it can be
>> >> >    used for link resolution later,
>> >> > 2. Pass that recorded namespace context to find_obj() when resolving a
>> >> >    reference, and
>> >> > 3. Wildly and completely rewrite find_obj().
>> >> >
>> >> > cross-reference support is expanded to tolerate the presence or absence
>> >> > of either namespace or module, and to cope with the presence or absence
>> >> > of contextual information for either.
>> >> >
>> >> > References now work like this:
>> >> >
>> >> > 1. If the explicit reference target is recorded in the domain's object
>> >> >    registry, we link to that target and stop looking. We do this lookup
>> >> >    regardless of how fully qualified the target is, which allows direct
>> >> >    references to modules (which don't have a module component to their
>> >> >    names) or direct references to definitions that may or may not belong
>> >> >    to a namespace or module.
>> >> >
>> >> > 2. If contextual information is available from qapi:namespace or
>> >> >    qapi:module directives, try using those components to find a direct
>> >> >    match to the implied target name.
>> >> >
>> >> > 3. If both prior lookups fail, generate a series of regular expressions
>> >> >    looking for wildcard matches in order from most to least
>> >> >    specific. Any explicitly provided components (namespace, module)
>> >> >    *must* match exactly, but both contextual and entirely omitted
>> >> >    components are allowed to differ from the search result. Note that if
>> >> >    more than one result is found, Sphinx will emit a warning (a build
>> >> >    error for QEMU) and list all of the candidate references.
>> >> >
>> >> > The practical upshot is that in the large majority of cases, namespace
>> >> > and module information is not required when creating simple `references`
>> >> > to definitions from within the same context -- even when identical
>> >> > definitions exist in other contexts.
>> >>
>> >> Can you illustrate this this examples?
>> >>
>> >
>> > do wha?
>>
>> Sorry, I went into the curve too fast.
>>
>> The stuff under "References now work like this" confuses me.  I guess it
>> describes a series of lookups to try one after the other.
>>
>> I understand a cross-reference consists of namespace (optional), module
>> (optional), name, and role.
>>
>> Let's assume role is "any" for simplicity's sake.
>>
>> Regarding "1. If the explicit ...":
>>
>>     What is a reference's "explicit reference target"?  Examples might
>>     help me understand.
>>
>
> explicit lookup: `QMP:block-core:block-dirty-bitmap-add`
>
> If that explicit target matches an object in the object database
> *directly*, we match immediately and don't consider other potential
> targets. This also applies to things like modules, e.g. `QMP:block-core`
> even though the "module" is absent (it IS the module)
>
> We always search for the explicit target no matter how un/fully qualified
> it is.
>
>
>>
>>     What is "recorded in the domain's object registry"?
>>
>
> domain.objects{} - essentially a record of every ObjectDefinition's
> "fullname" - the return value from QAPIDefinition._get_fqn().
>
>
>>
>>     Can you show me a reference where this lookup succeeds?
>>
>
> `QMP:block-core`
> `QMP:block-core.block-dirty-bitmap-add`

So, for this lookup to work, the reference must either be of the form
NAMESPACE:MODULE and resolve to that module in that namespace, or of the
form NAMESPACE:MODULE:DEFN and resolve to that definition in that module
in that namespace.  Correct?

These a "fully qualified names (FQN)" in your parlance, right?

Note that the first form is syntactically indistinguishable from
NAMESPACE:DEFN, i.e. a reference to a definition that specifies the
namespace, but not the module.

If the NAMESPACE:MODULE interpretation resolves, we never try the
NAMESPACE:DEFN interpretation, because that happens in later steps.
Correct?

The first form is fully qualified only if it resolves as FQN.  So,
whether such a reference is fully qualified is not syntactically
decidable.  Hmm.

>> Regarding "2. If contextual information ...":
>>
>>     I guess "contextual information" is the context established by
>>     qapi:namespace and qapi:module directives, i.e. the current
>>     namespace and module, if any.
>>
>
> Yep!
>
>
>>
>>     If the cross reference lacks a namespace, we substitute the current
>>     namespace.  Same for module.
>>
>>     We then use that "to find a direct match to the implied target
>>     name".  Sounds greek to me.  Example(s) might help.
>>
>
> If namespace or module is missing from the link target, we try to fill in
> the blanks with the contextual information if present.
>
> Example, we are in the block-core section of the QEMU QMP reference manual
> document and we reference `block-dirty-bitmap-add`. With context, we are
> able to assemble a fully qualified name:
> "QMP:block-core.block-dirty-bitmap-add`. This matches an item in the
> registry directly, so it matches and no further search is performed.

We try this lookup only when the reference lacks a namespace and we are
"in" a namespace, or when it lacks a module and we are "in" a module.
Correct?

We then subsitute current namespace / module for the lacking one(s), and
try the same lookup as in 1.  Correct?

If we have a reference of the form MYSTERY, it could either be a
reference to module MYSTERY in the current namespace, or to definition
MYSTERY in the current namespace and module.  How do we decide?

>> Regarding "3. If both prior lookups fail ...":
>>
>>     I guess we get here when namespace or module are absent, and
>>     substituting the current namespace or module doesn't resolve.  We
>>     then substitute a wildcard, so to speak, i.e. look in all namespaces
>>     / modules, and succeed if we find exactly one resolution.  Fair?
>>
>
> More or less, though the mechanics are quite a bit more complex than your
> overview (and what I wrote in qapi-domain.rst.) We can get here for a few
> reasons:
>
> (1) We didn't provide a fully qualified target, and we don't have full
> context to construct one. For example, we are not "in" a namespace and/or
> not "in" a module. This is quite likely to happen when writing simple
> references to a definition name from outside of the transmogfrified QAPI
> documentation, e.g. from qapi-domain.rst itself, or dirty-bitmaps.rst, etc.
>
> (2) We didn't provide a fully qualified target, and we are referencing
> something from outside of the local context. For example, we are "in" a
> module but we are trying to link to a different module's definition. e.g.
> we are in QMP:transaction and we reference `block-dirty-bitmap-add`. The
> implied FQN will be QMP:transaction.block-dirty-bitmap.add, which will not
> resolve.
>
> The fuzzy search portion has an order of precedence for how it searches -
> and not all searches are tried universally, they are conditional to what
> was provided in the reference target and what context is available.
>
> 1. match against the explicitly provided namespace (module was not
> explicitly provided)

Look for the name in all of the namespace's modules?

> 2. match against the explicitly provided module (namespace was not
> explicitly provided)

Look for the name in all modules so named in all namespaces?

> 3. match against the implied namespace (neither namespace/module was
> explicitly provided)

?

> 4. match against the implied module (neither namespace/module was
> explicitly provided)

?

> 5. match against the definition name only, from anywhere (neither
> namespace/module was explicitly provided)

Look for the name anywhere?

I need examples :)

> The searches are performed in order: if a search returns zero results, the
> next search is tried. If any search returns one or more results, those
> results are returned and we stop searching down the list. The priority
> order ensures that any explicitly provided information is *always* used to
> find a match, but contextually provided information is merely a "hint" and
> can be ignored for the sake of a match.
>
> If find_obj() as a whole returns zero results, Sphinx emits a warning for a
> dangling reference. if find_obj() as a whole returns multiple results,
> Sphinx emits a warning for the ambiguous cross-reference.
>
> QEMU errors out on any such warnings under our normal build settings.
>
> Clear as mud?

Clearer, but not quite mud, yet.

> --js
>
>
>> [...]
>>
>>

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 13207 bytes --]

On Thu, Mar 13, 2025, 11:57 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > On Thu, Mar 13, 2025 at 10:41 AM Markus Armbruster <armbru@redhat.com>
> > wrote:
> >
> >> John Snow <jsnow@redhat.com> writes:
> >>
> >> > On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster <armbru@redhat.com>
> wrote:
> >> >
> >> >> John Snow <jsnow@redhat.com> writes:
> >> >>
> >> >> > This patch does three things:
> >> >> >
> >> >> > 1. Record the current namespace context in pending_xrefs so it can
> be
> >> >> >    used for link resolution later,
> >> >> > 2. Pass that recorded namespace context to find_obj() when
> resolving a
> >> >> >    reference, and
> >> >> > 3. Wildly and completely rewrite find_obj().
> >> >> >
> >> >> > cross-reference support is expanded to tolerate the presence or
> absence
> >> >> > of either namespace or module, and to cope with the presence or
> absence
> >> >> > of contextual information for either.
> >> >> >
> >> >> > References now work like this:
> >> >> >
> >> >> > 1. If the explicit reference target is recorded in the domain's
> object
> >> >> >    registry, we link to that target and stop looking. We do this
> lookup
> >> >> >    regardless of how fully qualified the target is, which allows
> direct
> >> >> >    references to modules (which don't have a module component to
> their
> >> >> >    names) or direct references to definitions that may or may not
> belong
> >> >> >    to a namespace or module.
> >> >> >
> >> >> > 2. If contextual information is available from qapi:namespace or
> >> >> >    qapi:module directives, try using those components to find a
> direct
> >> >> >    match to the implied target name.
> >> >> >
> >> >> > 3. If both prior lookups fail, generate a series of regular
> expressions
> >> >> >    looking for wildcard matches in order from most to least
> >> >> >    specific. Any explicitly provided components (namespace, module)
> >> >> >    *must* match exactly, but both contextual and entirely omitted
> >> >> >    components are allowed to differ from the search result. Note
> that if
> >> >> >    more than one result is found, Sphinx will emit a warning (a
> build
> >> >> >    error for QEMU) and list all of the candidate references.
> >> >> >
> >> >> > The practical upshot is that in the large majority of cases,
> namespace
> >> >> > and module information is not required when creating simple
> `references`
> >> >> > to definitions from within the same context -- even when identical
> >> >> > definitions exist in other contexts.
> >> >>
> >> >> Can you illustrate this this examples?
> >> >>
> >> >
> >> > do wha?
> >>
> >> Sorry, I went into the curve too fast.
> >>
> >> The stuff under "References now work like this" confuses me.  I guess it
> >> describes a series of lookups to try one after the other.
> >>
> >> I understand a cross-reference consists of namespace (optional), module
> >> (optional), name, and role.
> >>
> >> Let's assume role is "any" for simplicity's sake.
> >>
> >> Regarding "1. If the explicit ...":
> >>
> >>     What is a reference's "explicit reference target"?  Examples might
> >>     help me understand.
> >>
> >
> > explicit lookup: `QMP:block-core:block-dirty-bitmap-add`
> >
> > If that explicit target matches an object in the object database
> > *directly*, we match immediately and don't consider other potential
> > targets. This also applies to things like modules, e.g. `QMP:block-core`
> > even though the "module" is absent (it IS the module)
> >
> > We always search for the explicit target no matter how un/fully qualified
> > it is.
> >
> >
> >>
> >>     What is "recorded in the domain's object registry"?
> >>
> >
> > domain.objects{} - essentially a record of every ObjectDefinition's
> > "fullname" - the return value from QAPIDefinition._get_fqn().
> >
> >
> >>
> >>     Can you show me a reference where this lookup succeeds?
> >>
> >
> > `QMP:block-core`
> > `QMP:block-core.block-dirty-bitmap-add`
>
> So, for this lookup to work, the reference must either be of the form
> NAMESPACE:MODULE and resolve to that module in that namespace, or of the
> form NAMESPACE:MODULE:DEFN and resolve to that definition in that module
> in that namespace.  Correct?
>

Yes.


> These a "fully qualified names (FQN)" in your parlance, right?
>

More or less, though as you found below...


> Note that the first form is syntactically indistinguishable from
> NAMESPACE:DEFN, i.e. a reference to a definition that specifies the
> namespace, but not the module.
>
> If the NAMESPACE:MODULE interpretation resolves, we never try the
> NAMESPACE:DEFN interpretation, because that happens in later steps.
> Correct?
>
> The first form is fully qualified only if it resolves as FQN.  So,
> whether such a reference is fully qualified is not syntactically
> decidable.  Hmm.
>

You're right. There is a weirdness here. I might need to do some more
thinking to make sure it isn't theoretically a problem, but in practice,
right now, it isn't.

Stay tuned, I guess.


> >> Regarding "2. If contextual information ...":
> >>
> >>     I guess "contextual information" is the context established by
> >>     qapi:namespace and qapi:module directives, i.e. the current
> >>     namespace and module, if any.
> >>
> >
> > Yep!
> >
> >
> >>
> >>     If the cross reference lacks a namespace, we substitute the current
> >>     namespace.  Same for module.
> >>
> >>     We then use that "to find a direct match to the implied target
> >>     name".  Sounds greek to me.  Example(s) might help.
> >>
> >
> > If namespace or module is missing from the link target, we try to fill in
> > the blanks with the contextual information if present.
> >
> > Example, we are in the block-core section of the QEMU QMP reference
> manual
> > document and we reference `block-dirty-bitmap-add`. With context, we are
> > able to assemble a fully qualified name:
> > "QMP:block-core.block-dirty-bitmap-add`. This matches an item in the
> > registry directly, so it matches and no further search is performed.
>
> We try this lookup only when the reference lacks a namespace and we are
> "in" a namespace, or when it lacks a module and we are "in" a module.
> Correct?
>

or both: if we provided only a name but the context has both a namespace
and module.

essentially the algorithm splits the explicit target into (ns, mod, name)
and for any that are blank, we try to fill in those blanks with context
info where available. Sometimes you have neither explicit nor contextual
info for a component.

Then we do a lookup for an exact match, in order;

1. explicit target name, whatever it was
2. FQN using contextual info

and we stop after the first hit - no chance for multiple results here, just
zero-or-one each step.

i.e. any explicitly provided information is never "overwritten" with
context, context only fills in the blanks where that info was not provided.

If neither of these work, we move on to fuzzy searching.


> We then subsitute current namespace / module for the lacking one(s), and
> try the same lookup as in 1.  Correct?
>

Yes!


> If we have a reference of the form MYSTERY, it could either be a
> reference to module MYSTERY in the current namespace, or to definition
> MYSTERY in the current namespace and module.  How do we decide?
>

fqn a: NS:MYSTERY
fqn b: NS:MOD:MYSTERY

Given we have a current ns/mod context, it's going to pick the second one.

Hm. Maybe it ought to be ambiguous in this case... I'll have to revise
this. (question is: how soon do we need it?)


> >> Regarding "3. If both prior lookups fail ...":
> >>
> >>     I guess we get here when namespace or module are absent, and
> >>     substituting the current namespace or module doesn't resolve.  We
> >>     then substitute a wildcard, so to speak, i.e. look in all namespaces
> >>     / modules, and succeed if we find exactly one resolution.  Fair?
> >>
> >
> > More or less, though the mechanics are quite a bit more complex than your
> > overview (and what I wrote in qapi-domain.rst.) We can get here for a few
> > reasons:
> >
> > (1) We didn't provide a fully qualified target, and we don't have full
> > context to construct one. For example, we are not "in" a namespace and/or
> > not "in" a module. This is quite likely to happen when writing simple
> > references to a definition name from outside of the transmogfrified QAPI
> > documentation, e.g. from qapi-domain.rst itself, or dirty-bitmaps.rst,
> etc.
> >
> > (2) We didn't provide a fully qualified target, and we are referencing
> > something from outside of the local context. For example, we are "in" a
> > module but we are trying to link to a different module's definition. e.g.
> > we are in QMP:transaction and we reference `block-dirty-bitmap-add`. The
> > implied FQN will be QMP:transaction.block-dirty-bitmap.add, which will
> not
> > resolve.
> >
> > The fuzzy search portion has an order of precedence for how it searches -
> > and not all searches are tried universally, they are conditional to what
> > was provided in the reference target and what context is available.
> >
> > 1. match against the explicitly provided namespace (module was not
> > explicitly provided)
>
> Look for the name in all of the namespace's modules?
>

Yeah. search for "ns:*.name" and "ns:name" basically.


> > 2. match against the explicitly provided module (namespace was not
> > explicitly provided)
>
> Look for the name in all modules so named in all namespaces?
>

Yes.


> > 3. match against the implied namespace (neither namespace/module was
> > explicitly provided)
>
> ?
>

User gave `foo`, but we have a namespace from context, so we look for
ns:*.foo or ns:foo.


> > 4. match against the implied module (neither namespace/module was
> > explicitly provided)
>
> ?
>

User gave `foo`, but we have a module from context, so we search for
*:mod.foo and mod.foo


> > 5. match against the definition name only, from anywhere (neither
> > namespace/module was explicitly provided)
>
> Look for the name anywhere?
>
> I need examples :)
>

user gave `foo`. search for any qapi definition in all modules and
namespaces for anything with the name "foo". The broadest possible search.

Doesn't search for stuff outside of the QAPI domain directly, but be aware
when using `references` that all domains are consulted, so it may in fact
match something else from somewhere else, though not by any doing of the
qapi domain.

i.e. :qapi:any:`foo` will definitely only search using the rules laid out
in this patch/thread, but `foo` will consult all domains (and whine if more
than one result between all domains is identified.)


> > The searches are performed in order: if a search returns zero results,
> the
> > next search is tried. If any search returns one or more results, those
> > results are returned and we stop searching down the list. The priority
> > order ensures that any explicitly provided information is *always* used
> to
> > find a match, but contextually provided information is merely a "hint"
> and
> > can be ignored for the sake of a match.
> >
> > If find_obj() as a whole returns zero results, Sphinx emits a warning
> for a
> > dangling reference. if find_obj() as a whole returns multiple results,
> > Sphinx emits a warning for the ambiguous cross-reference.
> >
> > QEMU errors out on any such warnings under our normal build settings.
> >
> > Clear as mud?
>
> Clearer, but not quite mud, yet.
>

Ultimately, Search in this order and stop at any point any of these
searches return at least one result:

1. Explicitly provided name, whatever it is
2. FQN using contextual info
3. Explicitly provided NS; any module
4. Explicitly provided module; any NS
5. Contextual NS; any module
6. Contextual module; any NS
7. any NS/module.

with searches 3-7 being conditional only when the criteria are met:

3. Must have explicit NS (and no explicit module)
4. Must have explicit module (and no explicit NS)
5. Must have contextual NS (must not have explicit NS nor module)
6. Must have contextual module (must not have explicit NS nor module)
7. Must have neither explicit NS nor module.

In summary:

* Anything explicitly provided *must* match. This information is *never*
ignored.
* Anything implicitly provided through context will never overwrite
anything explicitly provided. This is used as a *hint* in the event
multiple candidates are found, but results are allowed to deviate when
stronger matches are not located.

i.e. contextual information is used to *prefer* a result from that context,
but is not used to *limit* you to that context.

by contrast, explicit info is used to directly filter and restrict search.


(With maybe a bug or two for trying to find module names in some
circumstances. Will have to check that over...)


> > --js
> >
> >
> >> [...]
> >>
> >>
>
>

[-- Attachment #2: Type: text/html, Size: 20057 bytes --]

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Thu, Mar 13, 2025, 11:57 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > On Thu, Mar 13, 2025 at 10:41 AM Markus Armbruster <armbru@redhat.com>
>> > wrote:
>> >
>> >> John Snow <jsnow@redhat.com> writes:
>> >>
>> >> > On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster <armbru@redhat.com> wrote:
>> >> >
>> >> >> John Snow <jsnow@redhat.com> writes:
>> >> >>
>> >> >> > This patch does three things:
>> >> >> >
>> >> >> > 1. Record the current namespace context in pending_xrefs so it can be
>> >> >> >    used for link resolution later,
>> >> >> > 2. Pass that recorded namespace context to find_obj() when resolving a
>> >> >> >    reference, and
>> >> >> > 3. Wildly and completely rewrite find_obj().
>> >> >> >
>> >> >> > cross-reference support is expanded to tolerate the presence or absence
>> >> >> > of either namespace or module, and to cope with the presence or absence
>> >> >> > of contextual information for either.
>> >> >> >
>> >> >> > References now work like this:
>> >> >> >
>> >> >> > 1. If the explicit reference target is recorded in the domain's object
>> >> >> >    registry, we link to that target and stop looking. We do this lookup
>> >> >> >    regardless of how fully qualified the target is, which allows direct
>> >> >> >    references to modules (which don't have a module component to their
>> >> >> >    names) or direct references to definitions that may or may not belong
>> >> >> >    to a namespace or module.
>> >> >> >
>> >> >> > 2. If contextual information is available from qapi:namespace or
>> >> >> >    qapi:module directives, try using those components to find a direct
>> >> >> >    match to the implied target name.
>> >> >> >
>> >> >> > 3. If both prior lookups fail, generate a series of regular expressions
>> >> >> >    looking for wildcard matches in order from most to least
>> >> >> >    specific. Any explicitly provided components (namespace, module)
>> >> >> >    *must* match exactly, but both contextual and entirely omitted
>> >> >> >    components are allowed to differ from the search result. Note that if
>> >> >> >    more than one result is found, Sphinx will emit a warning (a build
>> >> >> >    error for QEMU) and list all of the candidate references.
>> >> >> >
>> >> >> > The practical upshot is that in the large majority of cases, namespace
>> >> >> > and module information is not required when creating simple `references`
>> >> >> > to definitions from within the same context -- even when identical
>> >> >> > definitions exist in other contexts.
>> >> >>
>> >> >> Can you illustrate this this examples?
>> >> >>
>> >> >
>> >> > do wha?
>> >>
>> >> Sorry, I went into the curve too fast.
>> >>
>> >> The stuff under "References now work like this" confuses me.  I guess it
>> >> describes a series of lookups to try one after the other.
>> >>
>> >> I understand a cross-reference consists of namespace (optional), module
>> >> (optional), name, and role.
>> >>
>> >> Let's assume role is "any" for simplicity's sake.
>> >>
>> >> Regarding "1. If the explicit ...":
>> >>
>> >>     What is a reference's "explicit reference target"?  Examples might
>> >>     help me understand.
>> >>
>> >
>> > explicit lookup: `QMP:block-core:block-dirty-bitmap-add`
>> >
>> > If that explicit target matches an object in the object database
>> > *directly*, we match immediately and don't consider other potential
>> > targets. This also applies to things like modules, e.g. `QMP:block-core`
>> > even though the "module" is absent (it IS the module)
>> >
>> > We always search for the explicit target no matter how un/fully qualified
>> > it is.
>> >
>> >
>> >>
>> >>     What is "recorded in the domain's object registry"?
>> >>
>> >
>> > domain.objects{} - essentially a record of every ObjectDefinition's
>> > "fullname" - the return value from QAPIDefinition._get_fqn().
>> >
>> >
>> >>
>> >>     Can you show me a reference where this lookup succeeds?
>> >>
>> >
>> > `QMP:block-core`
>> > `QMP:block-core.block-dirty-bitmap-add`
>>
>> So, for this lookup to work, the reference must either be of the form
>> NAMESPACE:MODULE and resolve to that module in that namespace, or of the
>> form NAMESPACE:MODULE:DEFN and resolve to that definition in that module
>> in that namespace.  Correct?
>>
>
> Yes.
>
>
>> These a "fully qualified names (FQN)" in your parlance, right?
>>
>
> More or less, though as you found below...
>
>
>> Note that the first form is syntactically indistinguishable from
>> NAMESPACE:DEFN, i.e. a reference to a definition that specifies the
>> namespace, but not the module.
>>
>> If the NAMESPACE:MODULE interpretation resolves, we never try the
>> NAMESPACE:DEFN interpretation, because that happens in later steps.
>> Correct?
>>
>> The first form is fully qualified only if it resolves as FQN.  So,
>> whether such a reference is fully qualified is not syntactically
>> decidable.  Hmm.
>>
>
> You're right. There is a weirdness here. I might need to do some more
> thinking to make sure it isn't theoretically a problem, but in practice,
> right now, it isn't.

Not a blocker, but please do your thinking :)

> Stay tuned, I guess.
>
>
>> >> Regarding "2. If contextual information ...":
>> >>
>> >>     I guess "contextual information" is the context established by
>> >>     qapi:namespace and qapi:module directives, i.e. the current
>> >>     namespace and module, if any.
>> >>
>> >
>> > Yep!
>> >
>> >
>> >>
>> >>     If the cross reference lacks a namespace, we substitute the current
>> >>     namespace.  Same for module.
>> >>
>> >>     We then use that "to find a direct match to the implied target
>> >>     name".  Sounds greek to me.  Example(s) might help.
>> >>
>> >
>> > If namespace or module is missing from the link target, we try to fill in
>> > the blanks with the contextual information if present.
>> >
>> > Example, we are in the block-core section of the QEMU QMP reference manual
>> > document and we reference `block-dirty-bitmap-add`. With context, we are
>> > able to assemble a fully qualified name:
>> > "QMP:block-core.block-dirty-bitmap-add`. This matches an item in the
>> > registry directly, so it matches and no further search is performed.
>>
>> We try this lookup only when the reference lacks a namespace and we are
>> "in" a namespace, or when it lacks a module and we are "in" a module.
>> Correct?
>>
>
> or both: if we provided only a name but the context has both a namespace
> and module.

So my or is inclusive :)

> essentially the algorithm splits the explicit target into (ns, mod, name)
> and for any that are blank, we try to fill in those blanks with context
> info where available. Sometimes you have neither explicit nor contextual
> info for a component.
>
> Then we do a lookup for an exact match, in order;
>
> 1. explicit target name, whatever it was

Fully qualified name.

If lookup succeeds, we're done.

If lookup fails, we're also done.

*Except* for the ambiguous form NAMESPACE:MYSTERY.  If lookup fails for
that, the name is not fully qualified after all.  Probably.  Maybe.  We
assume it's missing a module, and proceed to 2.

I'm mostly ignoring this exception from now on to keep things simple.

> 2. FQN using contextual info

Partially qualified name, but context can fill the gaps.

If lookup succeeds, we're done.

Else, we proceed to 3.

> and we stop after the first hit - no chance for multiple results here, just
> zero-or-one each step.
>
> i.e. any explicitly provided information is never "overwritten" with
> context, context only fills in the blanks where that info was not provided.
>
> If neither of these work, we move on to fuzzy searching.
>
>
>> We then subsitute current namespace / module for the lacking one(s), and
>> try the same lookup as in 1.  Correct?
>>
>
> Yes!
>
>
>> If we have a reference of the form MYSTERY, it could either be a
>> reference to module MYSTERY in the current namespace, or to definition
>> MYSTERY in the current namespace and module.  How do we decide?
>>
>
> fqn a: NS:MYSTERY
> fqn b: NS:MOD:MYSTERY
>
> Given we have a current ns/mod context, it's going to pick the second one.
>
> Hm. Maybe it ought to be ambiguous in this case... I'll have to revise
> this. (question is: how soon do we need it?)

While we should try to put this on a more solid foundation, it is not a
blocker.

>> >> Regarding "3. If both prior lookups fail ...":
>> >>
>> >>     I guess we get here when namespace or module are absent, and
>> >>     substituting the current namespace or module doesn't resolve.  We
>> >>     then substitute a wildcard, so to speak, i.e. look in all namespaces
>> >>     / modules, and succeed if we find exactly one resolution.  Fair?
>> >>
>> >
>> > More or less, though the mechanics are quite a bit more complex than your
>> > overview (and what I wrote in qapi-domain.rst.) We can get here for a few
>> > reasons:
>> >
>> > (1) We didn't provide a fully qualified target, and we don't have full
>> > context to construct one.

We skipped 1. because not fully qualified, and we skipped 2. because
context can't fill the gaps.

>> >                           For example, we are not "in" a namespace and/or
>> > not "in" a module. This is quite likely to happen when writing simple
>> > references to a definition name from outside of the transmogfrified QAPI
>> > documentation, e.g. from qapi-domain.rst itself, or dirty-bitmaps.rst, etc.

Yes.

>> > (2) We didn't provide a fully qualified target, and we are referencing
>> > something from outside of the local context. For example, we are "in" a
>> > module but we are trying to link to a different module's definition. e.g.
>> > we are in QMP:transaction and we reference `block-dirty-bitmap-add`. The
>> > implied FQN will be QMP:transaction.block-dirty-bitmap.add, which will not
>> > resolve.

We skipped 1. because not fully qualified, and we failed 2. because
context filled the gaps, but lookup failed.

>> >
>> > The fuzzy search portion has an order of precedence for how it searches -
>> > and not all searches are tried universally, they are conditional to what
>> > was provided in the reference target and what context is available.
>> >
>> > 1. match against the explicitly provided namespace (module was not
>> > explicitly provided)
>>
>> Look for the name in all of the namespace's modules?
>>
>
> Yeah. search for "ns:*.name" and "ns:name" basically.

Got it.

>> > 2. match against the explicitly provided module (namespace was not
>> > explicitly provided)
>>
>> Look for the name in all modules so named in all namespaces?
>>
>
> Yes.

Got it.

>> > 3. match against the implied namespace (neither namespace/module was
>> > explicitly provided)
>>
>> ?
>>
>
> User gave `foo`, but we have a namespace from context, so we look for
> ns:*.foo or ns:foo.

Got it.

Detail I had not considered until now: a namespace contains modules that
contain definitions, but it also contains definitions directly.

I can't recall offhand how schema.py represents this.  I'll figure it
out and report back.

>> > 4. match against the implied module (neither namespace/module was
>> > explicitly provided)
>>
>> ?
>>
>
> User gave `foo`, but we have a module from context, so we search for
> *:mod.foo and mod.foo

Got it.

>> > 5. match against the definition name only, from anywhere (neither
>> > namespace/module was explicitly provided)
>>
>> Look for the name anywhere?
>>
>> I need examples :)
>>
>
> user gave `foo`. search for any qapi definition in all modules and
> namespaces for anything with the name "foo". The broadest possible search.
>
> Doesn't search for stuff outside of the QAPI domain directly, but be aware
> when using `references` that all domains are consulted, so it may in fact
> match something else from somewhere else, though not by any doing of the
> qapi domain.
>
> i.e. :qapi:any:`foo` will definitely only search using the rules laid out
> in this patch/thread, but `foo` will consult all domains (and whine if more
> than one result between all domains is identified.)

Got it, I think.

>> > The searches are performed in order: if a search returns zero results, the
>> > next search is tried. If any search returns one or more results, those
>> > results are returned and we stop searching down the list. The priority
>> > order ensures that any explicitly provided information is *always* used to
>> > find a match, but contextually provided information is merely a "hint" and
>> > can be ignored for the sake of a match.
>> >
>> > If find_obj() as a whole returns zero results, Sphinx emits a warning for a
>> > dangling reference. if find_obj() as a whole returns multiple results,
>> > Sphinx emits a warning for the ambiguous cross-reference.
>> >
>> > QEMU errors out on any such warnings under our normal build settings.
>> >
>> > Clear as mud?
>>
>> Clearer, but not quite mud, yet.
>>
>
> Ultimately, Search in this order and stop at any point any of these
> searches return at least one result:
>
> 1. Explicitly provided name, whatever it is
> 2. FQN using contextual info
> 3. Explicitly provided NS; any module
> 4. Explicitly provided module; any NS
> 5. Contextual NS; any module
> 6. Contextual module; any NS
> 7. any NS/module.
>
> with searches 3-7 being conditional only when the criteria are met:
>
> 3. Must have explicit NS (and no explicit module)
> 4. Must have explicit module (and no explicit NS)
> 5. Must have contextual NS (must not have explicit NS nor module)
> 6. Must have contextual module (must not have explicit NS nor module)
> 7. Must have neither explicit NS nor module.
>
> In summary:
>
> * Anything explicitly provided *must* match. This information is *never*
> ignored.
> * Anything implicitly provided through context will never overwrite
> anything explicitly provided. This is used as a *hint* in the event
> multiple candidates are found, but results are allowed to deviate when
> stronger matches are not located.
>
> i.e. contextual information is used to *prefer* a result from that context,
> but is not used to *limit* you to that context.
>
> by contrast, explicit info is used to directly filter and restrict search.

Makes sense.

> (With maybe a bug or two for trying to find module names in some
> circumstances. Will have to check that over...)

Thank you!

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 17322 bytes --]

On Thu, Mar 13, 2025 at 2:30 PM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > On Thu, Mar 13, 2025, 11:57 AM Markus Armbruster <armbru@redhat.com>
> wrote:
> >
> >> John Snow <jsnow@redhat.com> writes:
> >>
> >> > On Thu, Mar 13, 2025 at 10:41 AM Markus Armbruster <armbru@redhat.com
> >
> >> > wrote:
> >> >
> >> >> John Snow <jsnow@redhat.com> writes:
> >> >>
> >> >> > On Thu, Mar 13, 2025 at 2:47 AM Markus Armbruster <
> armbru@redhat.com> wrote:
> >> >> >
> >> >> >> John Snow <jsnow@redhat.com> writes:
> >> >> >>
> >> >> >> > This patch does three things:
> >> >> >> >
> >> >> >> > 1. Record the current namespace context in pending_xrefs so it
> can be
> >> >> >> >    used for link resolution later,
> >> >> >> > 2. Pass that recorded namespace context to find_obj() when
> resolving a
> >> >> >> >    reference, and
> >> >> >> > 3. Wildly and completely rewrite find_obj().
> >> >> >> >
> >> >> >> > cross-reference support is expanded to tolerate the presence or
> absence
> >> >> >> > of either namespace or module, and to cope with the presence or
> absence
> >> >> >> > of contextual information for either.
> >> >> >> >
> >> >> >> > References now work like this:
> >> >> >> >
> >> >> >> > 1. If the explicit reference target is recorded in the domain's
> object
> >> >> >> >    registry, we link to that target and stop looking. We do
> this lookup
> >> >> >> >    regardless of how fully qualified the target is, which
> allows direct
> >> >> >> >    references to modules (which don't have a module component
> to their
> >> >> >> >    names) or direct references to definitions that may or may
> not belong
> >> >> >> >    to a namespace or module.
> >> >> >> >
> >> >> >> > 2. If contextual information is available from qapi:namespace or
> >> >> >> >    qapi:module directives, try using those components to find a
> direct
> >> >> >> >    match to the implied target name.
> >> >> >> >
> >> >> >> > 3. If both prior lookups fail, generate a series of regular
> expressions
> >> >> >> >    looking for wildcard matches in order from most to least
> >> >> >> >    specific. Any explicitly provided components (namespace,
> module)
> >> >> >> >    *must* match exactly, but both contextual and entirely
> omitted
> >> >> >> >    components are allowed to differ from the search result.
> Note that if
> >> >> >> >    more than one result is found, Sphinx will emit a warning (a
> build
> >> >> >> >    error for QEMU) and list all of the candidate references.
> >> >> >> >
> >> >> >> > The practical upshot is that in the large majority of cases,
> namespace
> >> >> >> > and module information is not required when creating simple
> `references`
> >> >> >> > to definitions from within the same context -- even when
> identical
> >> >> >> > definitions exist in other contexts.
> >> >> >>
> >> >> >> Can you illustrate this this examples?
> >> >> >>
> >> >> >
> >> >> > do wha?
> >> >>
> >> >> Sorry, I went into the curve too fast.
> >> >>
> >> >> The stuff under "References now work like this" confuses me.  I
> guess it
> >> >> describes a series of lookups to try one after the other.
> >> >>
> >> >> I understand a cross-reference consists of namespace (optional),
> module
> >> >> (optional), name, and role.
> >> >>
> >> >> Let's assume role is "any" for simplicity's sake.
> >> >>
> >> >> Regarding "1. If the explicit ...":
> >> >>
> >> >>     What is a reference's "explicit reference target"?  Examples
> might
> >> >>     help me understand.
> >> >>
> >> >
> >> > explicit lookup: `QMP:block-core:block-dirty-bitmap-add`
> >> >
> >> > If that explicit target matches an object in the object database
> >> > *directly*, we match immediately and don't consider other potential
> >> > targets. This also applies to things like modules, e.g.
> `QMP:block-core`
> >> > even though the "module" is absent (it IS the module)
> >> >
> >> > We always search for the explicit target no matter how un/fully
> qualified
> >> > it is.
> >> >
> >> >
> >> >>
> >> >>     What is "recorded in the domain's object registry"?
> >> >>
> >> >
> >> > domain.objects{} - essentially a record of every ObjectDefinition's
> >> > "fullname" - the return value from QAPIDefinition._get_fqn().
> >> >
> >> >
> >> >>
> >> >>     Can you show me a reference where this lookup succeeds?
> >> >>
> >> >
> >> > `QMP:block-core`
> >> > `QMP:block-core.block-dirty-bitmap-add`
> >>
> >> So, for this lookup to work, the reference must either be of the form
> >> NAMESPACE:MODULE and resolve to that module in that namespace, or of the
> >> form NAMESPACE:MODULE:DEFN and resolve to that definition in that module
> >> in that namespace.  Correct?
> >>
> >
> > Yes.
> >
> >
> >> These a "fully qualified names (FQN)" in your parlance, right?
> >>
> >
> > More or less, though as you found below...
> >
> >
> >> Note that the first form is syntactically indistinguishable from
> >> NAMESPACE:DEFN, i.e. a reference to a definition that specifies the
> >> namespace, but not the module.
> >>
> >> If the NAMESPACE:MODULE interpretation resolves, we never try the
> >> NAMESPACE:DEFN interpretation, because that happens in later steps.
> >> Correct?
> >>
> >> The first form is fully qualified only if it resolves as FQN.  So,
> >> whether such a reference is fully qualified is not syntactically
> >> decidable.  Hmm.
> >>
> >
> > You're right. There is a weirdness here. I might need to do some more
> > thinking to make sure it isn't theoretically a problem, but in practice,
> > right now, it isn't.
>
> Not a blocker, but please do your thinking :)
>
> > Stay tuned, I guess.
> >
> >
> >> >> Regarding "2. If contextual information ...":
> >> >>
> >> >>     I guess "contextual information" is the context established by
> >> >>     qapi:namespace and qapi:module directives, i.e. the current
> >> >>     namespace and module, if any.
> >> >>
> >> >
> >> > Yep!
> >> >
> >> >
> >> >>
> >> >>     If the cross reference lacks a namespace, we substitute the
> current
> >> >>     namespace.  Same for module.
> >> >>
> >> >>     We then use that "to find a direct match to the implied target
> >> >>     name".  Sounds greek to me.  Example(s) might help.
> >> >>
> >> >
> >> > If namespace or module is missing from the link target, we try to
> fill in
> >> > the blanks with the contextual information if present.
> >> >
> >> > Example, we are in the block-core section of the QEMU QMP reference
> manual
> >> > document and we reference `block-dirty-bitmap-add`. With context, we
> are
> >> > able to assemble a fully qualified name:
> >> > "QMP:block-core.block-dirty-bitmap-add`. This matches an item in the
> >> > registry directly, so it matches and no further search is performed.
> >>
> >> We try this lookup only when the reference lacks a namespace and we are
> >> "in" a namespace, or when it lacks a module and we are "in" a module.
> >> Correct?
> >>
> >
> > or both: if we provided only a name but the context has both a namespace
> > and module.
>
> So my or is inclusive :)
>
> > essentially the algorithm splits the explicit target into (ns, mod, name)
> > and for any that are blank, we try to fill in those blanks with context
> > info where available. Sometimes you have neither explicit nor contextual
> > info for a component.
> >
> > Then we do a lookup for an exact match, in order;
> >
> > 1. explicit target name, whatever it was
>
> Fully qualified name.
>

Yes, for lookup to succeed it should be fully qualified, though if the
target text is "ns:module", that's actually going to succeed here, too.


>
> If lookup succeeds, we're done.
>
> If lookup fails, we're also done.
>

If lookup fails, we actually continue on to #2, but whether or not this
does anything useful depends on whether or not the original target text was
fully qualified or not. If it was, #2 searches with the exact same text and
will fail again and proceed to #3, where because we had a fully qualified
name, none of the search conditions apply and we then just exit.

(It just lacks an early return, but abstractly, if lookup on #1 fails with
a fully qualified name, we are indeed done.)

If lookup fails because it wasn't actually fully qualified, then #2 has
some gaps to try to fill.


>
> *Except* for the ambiguous form NAMESPACE:MYSTERY.  If lookup fails for
> that, the name is not fully qualified after all.  Probably.  Maybe.  We
> assume it's missing a module, and proceed to 2.
>
> I'm mostly ignoring this exception from now on to keep things simple.
>
> > 2. FQN using contextual info
>
> Partially qualified name, but context can fill the gaps.
>
> If lookup succeeds, we're done.
>
> Else, we proceed to 3.
>

That's right.


>
> > and we stop after the first hit - no chance for multiple results here,
> just
> > zero-or-one each step.
> >
> > i.e. any explicitly provided information is never "overwritten" with
> > context, context only fills in the blanks where that info was not
> provided.
> >
> > If neither of these work, we move on to fuzzy searching.
> >
> >
> >> We then subsitute current namespace / module for the lacking one(s), and
> >> try the same lookup as in 1.  Correct?
> >>
> >
> > Yes!
> >
> >
> >> If we have a reference of the form MYSTERY, it could either be a
> >> reference to module MYSTERY in the current namespace, or to definition
> >> MYSTERY in the current namespace and module.  How do we decide?
> >>
> >
> > fqn a: NS:MYSTERY
> > fqn b: NS:MOD:MYSTERY
> >
> > Given we have a current ns/mod context, it's going to pick the second
> one.
> >
> > Hm. Maybe it ought to be ambiguous in this case... I'll have to revise
> > this. (question is: how soon do we need it?)
>
> While we should try to put this on a more solid foundation, it is not a
> blocker.
>
> >> >> Regarding "3. If both prior lookups fail ...":
> >> >>
> >> >>     I guess we get here when namespace or module are absent, and
> >> >>     substituting the current namespace or module doesn't resolve.  We
> >> >>     then substitute a wildcard, so to speak, i.e. look in all
> namespaces
> >> >>     / modules, and succeed if we find exactly one resolution.  Fair?
> >> >>
> >> >
> >> > More or less, though the mechanics are quite a bit more complex than
> your
> >> > overview (and what I wrote in qapi-domain.rst.) We can get here for a
> few
> >> > reasons:
> >> >
> >> > (1) We didn't provide a fully qualified target, and we don't have full
> >> > context to construct one.
>
> We skipped 1. because not fully qualified, and we skipped 2. because
> context can't fill the gaps.
>

we tried #1 and failed, then we tried #2 and failed again.


>
> >> >                           For example, we are not "in" a namespace
> and/or
> >> > not "in" a module. This is quite likely to happen when writing simple
> >> > references to a definition name from outside of the transmogfrified
> QAPI
> >> > documentation, e.g. from qapi-domain.rst itself, or
> dirty-bitmaps.rst, etc.
>
> Yes.
>
> >> > (2) We didn't provide a fully qualified target, and we are referencing
> >> > something from outside of the local context. For example, we are "in"
> a
> >> > module but we are trying to link to a different module's definition.
> e.g.
> >> > we are in QMP:transaction and we reference `block-dirty-bitmap-add`.
> The
> >> > implied FQN will be QMP:transaction.block-dirty-bitmap.add, which
> will not
> >> > resolve.
>
> We skipped 1. because not fully qualified, and we failed 2. because
> context filled the gaps, but lookup failed.
>

Failed #1 and Failed #2.


>
> >> >
> >> > The fuzzy search portion has an order of precedence for how it
> searches -
> >> > and not all searches are tried universally, they are conditional to
> what
> >> > was provided in the reference target and what context is available.
> >> >
> >> > 1. match against the explicitly provided namespace (module was not
> >> > explicitly provided)
> >>
> >> Look for the name in all of the namespace's modules?
> >>
> >
> > Yeah. search for "ns:*.name" and "ns:name" basically.
>
> Got it.
>
> >> > 2. match against the explicitly provided module (namespace was not
> >> > explicitly provided)
> >>
> >> Look for the name in all modules so named in all namespaces?
> >>
> >
> > Yes.
>
> Got it.
>
> >> > 3. match against the implied namespace (neither namespace/module was
> >> > explicitly provided)
> >>
> >> ?
> >>
> >
> > User gave `foo`, but we have a namespace from context, so we look for
> > ns:*.foo or ns:foo.
>
> Got it.
>
> Detail I had not considered until now: a namespace contains modules that
> contain definitions, but it also contains definitions directly.
>
> I can't recall offhand how schema.py represents this.  I'll figure it
> out and report back.
>

I think it gets charged to a module named "qapi-schema". Silly, but it
doesn't really break anything.


> >> > 4. match against the implied module (neither namespace/module was
> >> > explicitly provided)
> >>
> >> ?
> >>
> >
> > User gave `foo`, but we have a module from context, so we search for
> > *:mod.foo and mod.foo
>
> Got it.
>
> >> > 5. match against the definition name only, from anywhere (neither
> >> > namespace/module was explicitly provided)
> >>
> >> Look for the name anywhere?
> >>
> >> I need examples :)
> >>
> >
> > user gave `foo`. search for any qapi definition in all modules and
> > namespaces for anything with the name "foo". The broadest possible
> search.
> >
> > Doesn't search for stuff outside of the QAPI domain directly, but be
> aware
> > when using `references` that all domains are consulted, so it may in fact
> > match something else from somewhere else, though not by any doing of the
> > qapi domain.
> >
> > i.e. :qapi:any:`foo` will definitely only search using the rules laid out
> > in this patch/thread, but `foo` will consult all domains (and whine if
> more
> > than one result between all domains is identified.)
>
> Got it, I think.
>
> >> > The searches are performed in order: if a search returns zero
> results, the
> >> > next search is tried. If any search returns one or more results, those
> >> > results are returned and we stop searching down the list. The priority
> >> > order ensures that any explicitly provided information is *always*
> used to
> >> > find a match, but contextually provided information is merely a
> "hint" and
> >> > can be ignored for the sake of a match.
> >> >
> >> > If find_obj() as a whole returns zero results, Sphinx emits a warning
> for a
> >> > dangling reference. if find_obj() as a whole returns multiple results,
> >> > Sphinx emits a warning for the ambiguous cross-reference.
> >> >
> >> > QEMU errors out on any such warnings under our normal build settings.
> >> >
> >> > Clear as mud?
> >>
> >> Clearer, but not quite mud, yet.
> >>
> >
> > Ultimately, Search in this order and stop at any point any of these
> > searches return at least one result:
> >
> > 1. Explicitly provided name, whatever it is
> > 2. FQN using contextual info
> > 3. Explicitly provided NS; any module
> > 4. Explicitly provided module; any NS
> > 5. Contextual NS; any module
> > 6. Contextual module; any NS
> > 7. any NS/module.
> >
> > with searches 3-7 being conditional only when the criteria are met:
> >
> > 3. Must have explicit NS (and no explicit module)
> > 4. Must have explicit module (and no explicit NS)
> > 5. Must have contextual NS (must not have explicit NS nor module)
> > 6. Must have contextual module (must not have explicit NS nor module)
> > 7. Must have neither explicit NS nor module.
>

I should point out that:

- #3 and #4 are mutually exclusive
- #5 and #6 are mutually exclusive
- #3/#4 and #5/#6 are mutually exclusive
- Only one of #3/#4/#5/#6 will happen for a given reference.
- If we choose #5 or #6, #7 may also happen. If we choose #3 or #4, #7 will
never happen.

I bet I can make the code here cleaner by combining some of these, it's
just a lot of context on the input to keep in mind while designing the
conditionals, so I wrote it out very "explicitly" to help myself make sense
of it ...


> >
> > In summary:
> >
> > * Anything explicitly provided *must* match. This information is *never*
> > ignored.
> > * Anything implicitly provided through context will never overwrite
> > anything explicitly provided. This is used as a *hint* in the event
> > multiple candidates are found, but results are allowed to deviate when
> > stronger matches are not located.
> >
> > i.e. contextual information is used to *prefer* a result from that
> context,
> > but is not used to *limit* you to that context.
> >
> > by contrast, explicit info is used to directly filter and restrict
> search.
>
> Makes sense.
>
> > (With maybe a bug or two for trying to find module names in some
> > circumstances. Will have to check that over...)
>
> Thank you!
>
>
You're welcome!

[-- Attachment #2: Type: text/html, Size: 23362 bytes --]

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Thu, Mar 13, 2025 at 2:30 PM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > On Thu, Mar 13, 2025, 11:57 AM Markus Armbruster <armbru@redhat.com> wrote:
>> >
>> >> John Snow <jsnow@redhat.com> writes:
>> >>
>> >> > On Thu, Mar 13, 2025 at 10:41 AM Markus Armbruster <armbru@redhat.com wrote:

[...]

>> >> >> Regarding "2. If contextual information ...":
>> >> >>
>> >> >>     I guess "contextual information" is the context established by
>> >> >>     qapi:namespace and qapi:module directives, i.e. the current
>> >> >>     namespace and module, if any.
>> >> >>
>> >> >
>> >> > Yep!
>> >> >
>> >> >
>> >> >>
>> >> >>     If the cross reference lacks a namespace, we substitute the current
>> >> >>     namespace.  Same for module.
>> >> >>
>> >> >>     We then use that "to find a direct match to the implied target
>> >> >>     name".  Sounds greek to me.  Example(s) might help.
>> >> >>
>> >> >
>> >> > If namespace or module is missing from the link target, we try to fill in
>> >> > the blanks with the contextual information if present.
>> >> >
>> >> > Example, we are in the block-core section of the QEMU QMP reference manual
>> >> > document and we reference `block-dirty-bitmap-add`. With context, we are
>> >> > able to assemble a fully qualified name:
>> >> > "QMP:block-core.block-dirty-bitmap-add`. This matches an item in the
>> >> > registry directly, so it matches and no further search is performed.
>> >>
>> >> We try this lookup only when the reference lacks a namespace and we are
>> >> "in" a namespace, or when it lacks a module and we are "in" a module.
>> >> Correct?
>> >>
>> >
>> > or both: if we provided only a name but the context has both a namespace
>> > and module.
>>
>> So my or is inclusive :)
>>
>> > essentially the algorithm splits the explicit target into (ns, mod, name)
>> > and for any that are blank, we try to fill in those blanks with context
>> > info where available. Sometimes you have neither explicit nor contextual
>> > info for a component.
>> >
>> > Then we do a lookup for an exact match, in order;
>> >
>> > 1. explicit target name, whatever it was
>>
>> Fully qualified name.
>>
>
> Yes, for lookup to succeed it should be fully qualified, though if the
> target text is "ns:module", that's actually going to succeed here, too.
>
>
>>
>> If lookup succeeds, we're done.
>>
>> If lookup fails, we're also done.
>>
>
> If lookup fails, we actually continue on to #2, but whether or not this
> does anything useful depends on whether or not the original target text was
> fully qualified or not. If it was, #2 searches with the exact same text and
> will fail again and proceed to #3, where because we had a fully qualified
> name, none of the search conditions apply and we then just exit.
>
> (It just lacks an early return, but abstractly, if lookup on #1 fails with
> a fully qualified name, we are indeed done.)
>
> If lookup fails because it wasn't actually fully qualified, then #2 has
> some gaps to try to fill.

So, instead of

    if ref is fully qualified:
        try to look up ref
    else
        proceed to 2.

you do

    look up ref
    if found:
        ref must be fully qualified
    else
        proceed to 2.

Since "proceed to 2. won't do anything useful when ref is fully
qualified", the two should produce the same result.

>> *Except* for the ambiguous form NAMESPACE:MYSTERY.  If lookup fails for
>> that, the name is not fully qualified after all.  Probably.  Maybe.  We
>> assume it's missing a module, and proceed to 2.
>>
>> I'm mostly ignoring this exception from now on to keep things simple.
>>
>> > 2. FQN using contextual info
>>
>> Partially qualified name, but context can fill the gaps.
>>
>> If lookup succeeds, we're done.
>>
>> Else, we proceed to 3.
>>
>
> That's right.
>
>
>>
>> > and we stop after the first hit - no chance for multiple results here, just
>> > zero-or-one each step.
>> >
>> > i.e. any explicitly provided information is never "overwritten" with
>> > context, context only fills in the blanks where that info was not provided.
>> >
>> > If neither of these work, we move on to fuzzy searching.
>> >
>> >
>> >> We then subsitute current namespace / module for the lacking one(s), and
>> >> try the same lookup as in 1.  Correct?
>> >>
>> >
>> > Yes!
>> >
>> >
>> >> If we have a reference of the form MYSTERY, it could either be a
>> >> reference to module MYSTERY in the current namespace, or to definition
>> >> MYSTERY in the current namespace and module.  How do we decide?
>> >>
>> >
>> > fqn a: NS:MYSTERY
>> > fqn b: NS:MOD:MYSTERY
>> >
>> > Given we have a current ns/mod context, it's going to pick the second one.
>> >
>> > Hm. Maybe it ought to be ambiguous in this case... I'll have to revise
>> > this. (question is: how soon do we need it?)
>>
>> While we should try to put this on a more solid foundation, it is not a
>> blocker.
>>
>> >> >> Regarding "3. If both prior lookups fail ...":
>> >> >>
>> >> >>     I guess we get here when namespace or module are absent, and
>> >> >>     substituting the current namespace or module doesn't resolve.  We
>> >> >>     then substitute a wildcard, so to speak, i.e. look in all namespaces
>> >> >>     / modules, and succeed if we find exactly one resolution.  Fair?
>> >> >>
>> >> >
>> >> > More or less, though the mechanics are quite a bit more complex than your
>> >> > overview (and what I wrote in qapi-domain.rst.) We can get here for a few
>> >> > reasons:
>> >> >
>> >> > (1) We didn't provide a fully qualified target, and we don't have full
>> >> > context to construct one.
>>
>> We skipped 1. because not fully qualified, and we skipped 2. because
>> context can't fill the gaps.
>>
>
> we tried #1 and failed, then we tried #2 and failed again.

I think we're describing the same behavior with different words

Behavior:

    try to fill the gaps from context
    if all filled:
        look up
        if found:
            done

When you say "tried #2 and failed", you're referring to the entire
thing.

When I say "skipped 2.", I'm referring to the lookup.

>> >> >                           For example, we are not "in" a namespace and/or
>> >> > not "in" a module. This is quite likely to happen when writing simple
>> >> > references to a definition name from outside of the transmogfrified QAPI
>> >> > documentation, e.g. from qapi-domain.rst itself, or dirty-bitmaps.rst, etc.
>>
>> Yes.
>>
>> >> > (2) We didn't provide a fully qualified target, and we are referencing
>> >> > something from outside of the local context. For example, we are "in" a
>> >> > module but we are trying to link to a different module's definition. e.g.
>> >> > we are in QMP:transaction and we reference `block-dirty-bitmap-add`. The
>> >> > implied FQN will be QMP:transaction.block-dirty-bitmap.add, which will not
>> >> > resolve.
>>
>> We skipped 1. because not fully qualified, and we failed 2. because
>> context filled the gaps, but lookup failed.
>>
>
> Failed #1 and Failed #2.
>
>
>>
>> >> >
>> >> > The fuzzy search portion has an order of precedence for how it searches -
>> >> > and not all searches are tried universally, they are conditional to what
>> >> > was provided in the reference target and what context is available.
>> >> >
>> >> > 1. match against the explicitly provided namespace (module was not
>> >> > explicitly provided)
>> >>
>> >> Look for the name in all of the namespace's modules?
>> >>
>> >
>> > Yeah. search for "ns:*.name" and "ns:name" basically.
>>
>> Got it.
>>
>> >> > 2. match against the explicitly provided module (namespace was not
>> >> > explicitly provided)
>> >>
>> >> Look for the name in all modules so named in all namespaces?
>> >>
>> >
>> > Yes.
>>
>> Got it.
>>
>> >> > 3. match against the implied namespace (neither namespace/module was
>> >> > explicitly provided)
>> >>
>> >> ?
>> >>
>> >
>> > User gave `foo`, but we have a namespace from context, so we look for
>> > ns:*.foo or ns:foo.
>>
>> Got it.
>>
>> Detail I had not considered until now: a namespace contains modules that
>> contain definitions, but it also contains definitions directly.
>>
>> I can't recall offhand how schema.py represents this.  I'll figure it
>> out and report back.
>>
>
> I think it gets charged to a module named "qapi-schema". Silly, but it
> doesn't really break anything.

Alright, here's how it works.

A QAPI schema consists of the main schema file and optionally included
files.

Each file's definitions go into a module named like the file.  The
builtin definitions go into a special module './builtin'.

A definition is *always* in a module.  Even when the schema is
monolithic, i.e. includes nothing.

Why do you need to support definitions directly in the namespace?

[...]

Re: [PATCH 07/11] docs/qapi_domain: add namespace support to cross-references

[Markus Armbruster]

I found the commit message's description of reference resolution hard to
understand.  I think I understand it well enough now to ACK the design.
Would be nice to work some of your explanations into the commit message,
but we don't have time for that.  So I'm merely fixing your QGA vs. QSD
typo.

We might want to put an improved description into the code.  Not today.

Thanks!

Re: [PATCH 09/11] docs: add QAPI namespace "QMP" to qemu-qmp-ref

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> This also creates the qapi-qmp-index.html index and cross-reference
> target.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  docs/conf.py                  | 4 +++-
>  docs/interop/qemu-qmp-ref.rst | 1 +
>  qapi/qapi-schema.json         | 2 +-
>  3 files changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/docs/conf.py b/docs/conf.py
> index 175491148c3..9a86e84a804 100644
> --- a/docs/conf.py
> +++ b/docs/conf.py
> @@ -163,7 +163,9 @@
>  
>  # Due to a limitation in Sphinx, we need to know which indices to
>  # generate in advance. Adding a namespace here allows that generation.
> -qapi_namespaces = set()
> +qapi_namespaces = {
> +    "QMP",
> +}
>  
>  # -- Options for HTML output ----------------------------------------------
>  
> diff --git a/docs/interop/qemu-qmp-ref.rst b/docs/interop/qemu-qmp-ref.rst
> index e95eeac45e2..ef8792b53f5 100644
> --- a/docs/interop/qemu-qmp-ref.rst
> +++ b/docs/interop/qemu-qmp-ref.rst
> @@ -8,3 +8,4 @@ QEMU QMP Reference Manual
>  
>  .. qapi-doc:: qapi/qapi-schema.json
>     :transmogrify:
> +   :namespace: QMP
> diff --git a/qapi/qapi-schema.json b/qapi/qapi-schema.json
> index 4475e81cc3e..c41c01eb2ab 100644
> --- a/qapi/qapi-schema.json
> +++ b/qapi/qapi-schema.json
> @@ -5,7 +5,7 @@
>  #
>  # This document describes all commands currently supported by QMP.
>  #
> -# For locating a particular item, please see the `qapi-index`.
> +# For locating a particular item, please see the `qapi-qmp-index`.

Output changes from "see the QAPI Index" to "see the QMP Index", which
is an improvement.

I think I'd rather spell it "QMP index", though.  "Index" comes from
PATCH 08.  Not worth delaying the series. for that.

>  #
>  # Most of the time their usage is exactly the same as in the user
>  # Monitor, this means that any other document which also describe

Re: [PATCH 08/11] docs/qapi-domain: add namespaced index support

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Generate an index-per-namespace for the QAPI domain. Due to a limitation
> with Sphinx's architecture, these indices must be defined during setup
> time and cannot be dynamically created on-demand when a namespace
> directive is encountered.
>
> Owing to that limitation, add a configuration value to conf.py that
> specifies which QAPI namespaces we'll generate indices for.
>
> Indices will be named after their namespace, e.g. the "QMP" namespace
> will generate to "qapi-qmp-index.html" and can be referenced using
> `qapi-qmp-index`.
>
> Signed-off-by: John Snow <jsnow@redhat.com>

I'm going to state my understanding of a few things.  Please correct
misunderstandings, if any.

Before this patch, the QAPI domain creates a single index, and it's
named "QAPI Index".

It has "everything".

This patch adds one index per namespace.  There are none, yet.  The next
patch will add namespace "QMP", and the last patch adds "QGA" and "QSD".

Each of these indexes will have exactly the stuff in that namespace.

The QAPI Index continues to have everything.

The patch adding namespace "QMP" replaces the QEMU QMP Reference
Manual's link to the QAPI Index by a link to the QMP reference (the QAPI
index is no longer linked from anywhere).  At that time, their contents
is still identical, but that stops when the last patch enables the
transmogrifier more widely, so the replacement is clearly necessary.

Since we put everything QAPI in a namespace, the QAPI index has no
entries that aren't also in a namespace's index.

Since QSD's schema is a subset of QMP's, everything in the QSD index and
also in the QMP index.

The QAPI index has everything...  does it link to QMP's copy or QSD's
copy?  I checked block-commit, and it appears to link to QMP's.

Re: [PATCH 00/11] docs/qapi: enable new guest-agent and storage-daemon docs

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Add namespaces, turn on QGA and QSD.

I'm queuing this as is, with typo fixes in one commit message.

There's some followup work:

* You wanted to think some more on reference resolution.  In particular

    (With maybe a bug or two for trying to find module names in some
    circumstances. Will have to check that over...)

* I had a hard time understanding the reference resolution algorithm
  from the commit message.  Whether the code is sufficiently obvious I
  can't say, because I didn't look closely.  We might want more
  comments.  Not today.  The user documentation on reference resolution
  in qapi-domain.rst looks sufficient.

* I have questions on the QAPI index.  They may or may not lead to
  followup work.

* During review, we realized that the introduction sections of the three
  reference manuals (QMP, QSD, QGA) need work.  Not this series' fault.
  I'll see what I can do.

Thanks!